Utilities
The TileDB Cloud client offers several useful utilities. To use them, you must have the client installed (see Installation).
TileDB Cloud allows you to login (with your username/password or API token) in a way such that the session token can be cached to avoid logging in again for every program execution. This is done as follows:
Python
R
Java
# With username/password
tiledb.cloud.login(username='xxx', password='xxx')
# Or, with token
tiledb.cloud.login(token='xxx')
# With username/password
tiledbcloud::login(username='xxx', password='xxx')
# Or, with token
tiledbcloud::login(api_key='xxx')
// Use the api token for the Java client. You can leave username and password as null.
TileDBClient tileDBClient = new TileDBClient(
new TileDBLogin("username",
"password",
"<TILEDB_API_TOKEN>",
true,
true,
true));
After logging in for the first time, the TileDB Cloud client will store a session token in configuration file
$HOME/.tiledb/cloud.jsoncreated in your home directory.The TileDB Cloud clients have the ability to retry failed HTTP requests automatically. By default this is enabled for retrying when TileDB Cloud indicates there is not enough capacity for the request (HTTP 503 errors). For convenience we also offer the ability to disable retries or to enable more forceful retry settings.
Python
# Set default retry for only retrying "not enough capacity" responses
tiledb.cloud.client.client.retry_mode("default")
# Set do not retry any requests
tiledb.cloud.client.client.retry_mode("disabled")
# Retry for a large number of scenarios
tiledb.cloud.client.client.retry_mode("forceful")
In "forceful" mode it is possible that the client might retry requests which will always fail, such as when there is a syntax error in a SQL query. This mode should be used with care to avoid increased costs from retrying.
All built in modes (besides disabled) will retry a request up to 10 times.
It is also possible to manually set retry conditions to suite your needs.
Python
from urllib3 import Retry
# Set the retries to a urllib3 retry object
tiledb.cloud.config.config.retries=Retry(
total=10,
backoff_factor=0.25,
status_forcelist=[400, 500, 501, 502, 503],
allowed_methods=["HEAD","GET","PUT","DELETE","OPTIONS","TRACE","POST","PATCH",],
raise_on_status=False,
remove_headers_on_redirect=[],
)
# After updating the config make sure to update the package level clients
tiledb.cloud.client.client.update_clients()
There are two helper functions that allow to easily create a
tiledb config or context that has the proper configuration needed for slicing arrays through TileDB Cloud.Python
R
# Create a TileDB Config object with `rest.token` set from the login
cfg = tiledb.cloud.Config()
# Create a TileDB Context which has a config with `rest.token` set from the login
ctx = tiledb.cloud.Ctx()
# Create a TileDB Config object with `rest.token` set from the login
config <- tiledb_config()
# Create a TileDB Context which has a config with `rest.token` set from the login
ctx <- tiledb_ctx(config)
You can see your user profile as follows:
Python
R
Java
prof = tiledb.cloud.user_profile()
print(prof)
prof <- tiledbcloud::user_profile()
print(prof)
UserApi apiInstance = new UserApi(defaultClient);
try {
User result = apiInstance.getUser();
System.out.println(result);
} catch (ApiException e) {
System.err.println("Exception when calling UserApi#getUser");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
You can list arrays from the cloud service, passing a variety of filters:
Python
R
Java
# List all arrays you own
owned_arrays = tiledb.cloud.list_arrays()
print(owned_arrays)
# List all arrays that are shared with you
shared_arrays = tiledb.cloud.list_shared_arrays()
print(shared_arrays)
# List all public arrays
public_arrays = tiledb.cloud.list_public_arrays()
print(public_arrays)
# List arrays in a specific namespace
tiledb_inc_arrays = tiledb.cloud.list_arrays(namespace="TileDB-Inc")
print(tiledb_inc_arrays)
# Filter arrays to only those you have read and write permissions to
rw_arrays = tiledb.cloud.list_arrays(permissions=["read", "write"])
print(rw_arrays)
# You can combine filters
arrays = tiledb.cloud.list_arrays(namespace="TileDB-Inc", permissions=["read"])
print(arrays)
# List all arrays you own
owned_arrays <- tiledbcloud::list_arrays()
str(owned_arrays)
# List all arrays that are shared with you
shared_arrays <- tiledbcloud::list_arrays(shared=TRUE)
str(shared_arrays)
# List all public arrays
public_arrays <- tiledbcloud::list_arrays(public=TRUE)
str(public_arrays)
# List arrays in a specific namespace
tiledb_inc_arrays = tiledbcloud::list_arrays(namespace="TileDB-Inc")
str(tiledb_inc_arrays)
String namespace = "<TILEDB_NAMESPACE>"; // String | namespace array is in (an organization name or user's username)
try {
//get arrays in namespace
List<ArrayInfo> result = apiInstance.getArraysInNamespace(namespace);
} catch (ApiException e) {
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
You can run the following to get basic information about the array, such as its description:
Python
R
Java
info = tiledb.cloud.info("tiledb://TileDB-Inc/quickstart_sparse")
print(info)
info <- tiledbcloud::array_info(namespace="TileDB-Inc", arrayname="quickstart_sparse")
str(info)
String namespace = "TileDB-Inc"; // String | namespace array is in (an organization name or user's username)
String array = "quickstart_sparse"; // String | name/uri of array that is url-encoded
try {
ArrayInfo result = apiInstance.getArrayMetadata(namespace, array);
System.out.println(result);
} catch (ApiException e) {
System.err.println("Exception when calling ArrayApi#getArrayMetadata");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
Array activity can be fetched programmatically as follows:
Python
Java
activity = tiledb.cloud.array_activity("tiledb://TileDB-Inc/quickstart_sparse")
print(activity)
String namespace = "<NAMESAPCE>"; // String | namespace array is in (an organization name or user's username)
String array = "<ARRAY_NAME>"; // String | name/uri of array that is url-encoded
Integer start = null; // Integer | Start time of window of fetch logs, unix epoch in seconds (default: seven days ago)
Integer end = null; // Integer | End time of window of fetch logs, unix epoch in seconds (default: current utc timestamp)
String eventTypes = null; // String | Event values can be one or more of the following read, write, create, delete, register, deregister, comma separated
String taskId = null; // String | Array task ID To filter activity to
Boolean hasTaskId = false; // Boolean | Excludes activity log results that do not contain an array task UUID
try {
List<ArrayActivityLog> result = apiInstance.arrayActivityLog(namespace, array, start, end, eventTypes, taskId, hasTaskId);
System.out.println(result);
} catch (ApiException e) {
System.err.println("Exception when calling ArrayApi#arrayActivityLog");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
You can list tasks from the cloud service, passing a variety of filters:
Python
Java
# List all tasks
all_tasks = tiledb.cloud.tasks()
print(all_tasks)
# List only tasks on a specific array
array_tasks = tiledb.cloud.tasks(array="tiledb://TileDB-Inc/quickstart_sparse")
print(array_tasks)
# Lists tasks within a specific time period
import datetime
ninety_days_ago = datetime.datetime.utcnow() - datetime.timedelta(days=90)
datetime_tasks = tiledb.cloud.tasks(array="tiledb://TileDB-Inc/quickstart_sparse",
start=ninety_days_ago)
print(datetime_tasks)
# Filter tasks by status, valid statuses are RUNNING, FAILED, COMPLETED
running_tasks = tiledb.cloud.tasks(status="RUNNING")
print(running_tasks)
TasksApi apiInstance = new TasksApi(defaultClient);
String namespace = "<NAMESPACE"; // String | namespace to filter
String createdBy = null; // String | username to filter
String array = "<ARRAY_URI>"; // String | name/uri of array that is url-encoded to filter
Integer start = null; // Integer | start time for tasks to filter by
Integer end = null; // Integer | end time for tasks to filter by
Integer page = null; // Integer | pagination offset
Integer perPage = null; // Integer | pagination limit
String type = null; // String | task type, \"QUERY\", \"SQL\", \"UDF\", \"GENERIC_UDF\"
List<String> excludeType = Arrays.asList(); // List<String> | task_type to exclude matching array in results, more than one can be included
List<String> fileType = Arrays.asList(); // List<String> | match file_type of task array, more than one can be included
List<String> excludeFileType = Arrays.asList(); // List<String> | exclude file_type of task arrays, more than one can be included
String status = null; // String | Filter to only return these statuses
String search = null; // String | search string that will look at name, namespace or description fields
String orderby = null; // String | sort by which field valid values include start_time, name
try {
ArrayTaskData result = apiInstance.tasksGet(namespace, createdBy, array, start, end, page, perPage, type, excludeType, fileType, excludeFileType, status, search, orderby);
System.out.println(result);
} catch (ApiException e) {
System.err.println("Exception when calling TasksApi#tasksGet");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
For convenience, you can also see the last SQL or UDF task:
Python
# Get last SQL task
tiledb.cloud.last_sql_task()
# Get last UDF task
tiledb.cloud.last_udf_task()
Or you can get a specific task with a given task ID (which can be found on the UI console):
Python
task = tiledb.cloud.task(id='xxx')
In addition to registering S3-stored TileDB arrays with TileDB cloud via the console, you can also do it programmatically as follows:
Python
Java
tiledb.cloud.register_array(uri="s3://mybucket/myarray",
namespace="user1", # Optional, you may register it under your username, or one of your organizations
array_name="myarray",
description=None, # Optional
access_credentials_name="myCredentials") # You must have already added your AWS credentials on the console
String namespace = "<NAMESPACE>"; // String | namespace array is in (an organization name or user's username)
String array = "s3://<S3_BUCKET>/<ARRAY_NAME>"; // String | name/uri of s3 array that is url-encoded
ArrayInfoUpdate arrayMetadata = new ArrayInfoUpdate(); // ArrayInfoUpdate | metadata associated with array
arrayMetadata.setUri("s3://<S3_BUCKET>/<ARRAY_NAME>");
arrayMetadata.setName("<ARRAY_NAME>");
try {
ArrayInfo result = arrayApi.registerArray(namespace, array, arrayMetadata);
System.out.println(result);
} catch (ApiException e) {
System.err.println("Exception when calling ArrayApi#registerArray");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
You can deregister an array as follows:
Python
Java
tiledb.cloud.deregister_array("tiledb://user1/myarray")
ArrayApi apiInstance = new ArrayApi(defaultClient);
String namespace = "<NAMESPACE>"; // String | namespace array is in (an organization name or user's username)
String array = "<ARRAY_NAME>"; // String | name/uri of array that is url-encoded
try {
apiInstance.deregisterArray(namespace, array);
} catch (ApiException e) {
System.err.println("Exception when calling ArrayApi#deregisterArray");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
Deregistering an array will not physically delete it.
You can programmatically share a registered array, "unshare" a registered array (i.e., revoke access) and list array sharing information as follows:
Python
Java
# Share an array with both read and write permissions with a user
tiledb.cloud.share_array(uri="tiledb://user1/myarray",
namespace="user1", # The user to share the array with
permissions=["read", "write"])
# Revoke access to an array for a particular user
tiledb.cloud.unshare_array(uri="tiledb://user1/myarray", namespace="user1")
# Get sharing information about an array
shared_with = tiledb.cloud.list_shared_with("tiledb://user1/myarray")
print(shared_with)
ArrayApi apiInstance = new ArrayApi(defaultClient);
String namespace = "namespace_example"; // String | namespace array is in (an organization name or user's username) to share the array with.
String array = "array_example"; // String | name/uri of array that is url-encoded
ArraySharing arraySharing = new ArraySharing(); // ArraySharing | Namespace and list of permissions to share with. An empty list of permissions will remove the namespace; if permissions already exist they will be deleted then new ones added. In the event of a failure, the new policies will be rolled back to prevent partial policies, and it's likely the array will not be shared with the namespace at all.
arraySharing.addActionsItem(ArrayActions.READ); //enable read permissions.
try {
//share an array with read persmissions.
apiInstance.shareArray(namespace, array, arraySharing);
//to unshare an array use an empy arraySharing object
arraySharing = null;
apiInstance.shareArray(namespace, array, arraySharing);
//Get sharing information about an array
List<ArraySharing> result = apiInstance.getArraySharingPolicies(namespace, array);
} catch (ApiException e) {
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
While automatic compute region direction is in beta you will need to manually enable it for a query or request. Below is two examples of setting the server address to the redirection domain
https://multi-region.api.tiledb.com .Python - Array Access
Python - Array UDF
Java - Array Access
import tiledb, tiledb.sql
import pandas
# Create the configuration parameters
config = tiledb.Config()
config["rest.username"] = "xxx"
config["rest.password"] = "yyy"
# or, more preferably, config["rest.token"] = "my_token"
# Manually set the server address to the redirection URL
config["rest.server_address"] = "https://multi-region.api.tiledb.com"
# This is the array URI format in TileDB Cloud
array_name = "tiledb://TileDB-Inc/quickstart_sparse-eu-west-2"
# Write code exactly as in TileDB Embedded
with tiledb.open(array_name, 'r', ctx=tiledb.Ctx(config)) as A:
print (A.df[:])
# Using embedded SQL, you need to pass the username/password
# as config parameters as well as the server address in `init_command`
db = tiledb.sql.connect(db="test",
init_command="set mytile_tiledb_config='rest.username=xxx,rest.password=xxx,rest.server_address=https://multi-region.api.tiledb.com'")
pandas.read_sql(sql="select * from `tiledb://TileDB-Inc/quickstart_sparse-eu-west-2`", con=db)
import tiledb, tiledb.cloud, numpy
def median(numpy_ordered_dictionary):
return numpy.median(numpy_ordered_dictionary["a"])
tiledb.cloud.login(username="xxx", password="yyy", host="https://multi-region.api.tiledb.com")
# or tiledb.cloud.login(token="my_token", host="https://multi-region.api.tiledb.com")
with tiledb.open("tiledb://TileDB-Inc/quickstart_sparse-eu-west-2", ctx=tiledb.cloud.Ctx()) as A:
# apply on subarray [1,2]x[1,2]
res = A.apply(median, [(1,2), (1,2)], attrs = ["a"])
print(res)
//set the config
Config config = new Config();
config.set("rest.username", "xxx");
config.set("rest.password", "yyy");
config.set("rest.server_address", "https://multi-region.api.tiledb.com")
Context ctx = new Context(config);
//open the array as you would normally do in the Java API.
Array array = new Array(ctx, "tiledb://TileDB-Inc/quickstart_sparse-eu-west-2");
//print the array schema
System.out.println(array.getSchema());
A compute region can be access directly bypassing automatic redirection. This is helpful if you want to avoid the slight increase in latency that the redirection adds.
To access a region directly the domain is of the scheme:
<region>.aws.api.tiledb.comThe four domains we currently support are:
us-east-1.aws.api.tiledb.comus-west-2.aws.api.tiledb.comeu-west-2.aws.api.tiledb.comap-southeast-1.aws.api.tiledb.com
You can manually set the domain to send a request directly to a region as follows:
Python - Array Access
Python - Array UDF
Java - Array Access
import tiledb, tiledb.sql
import pandas
# Create the configuration parameters
config = tiledb.Config()
config["rest.username"] = "xxx"
config["rest.password"] = "yyy"
# or, more preferably, config["rest.token"] = "my_token"
# Manually set the server address to the redirection URL
config["rest.server_address"] = "https://eu-west-2.aws.api.tiledb.com"
# This is the array URI format in TileDB Cloud
array_name = "tiledb://TileDB-Inc/quickstart_sparse-eu-west-2"
# Write code exactly as in TileDB Developer
with tiledb.open(array_name, 'r', ctx=tiledb.Ctx(config)) as A:
print (A.df[:])
# Using embedded SQL, you need to pass the username/password
# as config parameters as well as the server address in `init_command`
db = tiledb.sql.connect(db="test",
init_command="set mytile_tiledb_config='rest.username=xxx,rest.password=xxx,rest.server_address=https://eu-west-2.aws.api.tiledb.com'")
pandas.read_sql(sql="select * from `tiledb://TileDB-Inc/quickstart_sparse-eu-west-2`", con=db)
import tiledb, tiledb.cloud, numpy
def median(numpy_ordered_dictionary):
return numpy.median(numpy_ordered_dictionary["a"])
tiledb.cloud.login(username="xxx", password="yyy", host="https://eu-west-2.aws.api.tiledb.com")
# or tiledb.cloud.login(token="my_token", host="https://eu-west-2.aws.api.tiledb.com")
with tiledb.open("tiledb://TileDB-Inc/quickstart_sparse-eu-west-2", ctx=tiledb.cloud.Ctx()) as A:
# apply on subarray [1,2]x[1,2]
res = A.apply(median, [(1,2), (1,2)], attrs = ["a"])
print(res)
//set the config
Config config = new Config();
config.set("rest.username", "xxx");
config.set("rest.password", "yyy");
config.set("rest.server_address", "https://eu-west-2.aws.api.tiledb.com")
Context ctx = new Context(config);
//open the array as you would normally do in the Java API.
Array array = new Array(ctx, "tiledb://TileDB-Inc/quickstart_sparse-eu-west-2");
//print the array schema
System.out.println(array.getSchema());
TileDB Cloud has the ability to convert files to and from the TileDB File representation. This allows you to store any arbitrary file as a 1 dimensions dense array. Importing and exporting to and from the original file format is supported directly through TileDB Cloud. The file-arrays can be stored on an object store, such as S3, directly.
Python
Java
# Import from s3 to a TileDB array,
# automatically registering it with TileDB Cloud
tiledb.cloud.file.create_file(
namespace="my_organization",
name="my_file", # optional name to set for registered file
input_uri="s3://my_bucket/files/my_file.pdf",
output_uri="s3://my_bucket/files/arrays/my_file"
)
# Export back to S3 in the original format
# The export happens completely in TileDB cloud
tiledb.cloud.file.export_file(
uri="tiledb://my_organization/my_file",
output_uri="s3://my_bucket/files/arrays/my_file"
)
# Export back to local filesystem in the original format
tiledb.cloud.file.export_file(
uri="tiledb://my_organization/my_file",
output_uri="my_file_exported.pdf",
)
//Use the TileDB-Java package
// Set up the config with your TileDB-Cloud credentials
Config config = new Config();
// For s3 access
config.set("vfs.s3.aws_access_key_id", "<ID>");
config.set("vfs.s3.aws_secret_access_key", "<KEY>");
// For TileDB-Cloud access.
// You can either use rest.username and rest.password
config.set("rest.username", "<USERNAME>");
config.set("rest.password", "<PASSWORD>");
// Or rest.token
config.set("rest.token", "<TOKEN>");
Context ctx = new Context(config);
// Create the array schema of an array based on the file to be saved
ArraySchema arraySchema = FileStore.schemaCreate(ctx, "<FILENAME>");
// Create a TileDB array with the schema
Array.create("tiledb://<NAMESPACE_NAME>/s3://<BUCKET_NAME>/<ARRAY_NAME>", arraySchema);
// Import the file to be saved to the TileDB array
FileStore.uriImport(
ctx,
"tiledb://<NAMESPACE_NAME>/<ARRAY_NAME>",
"<FILENAME>",
MimeType.TILEDB_MIME_AUTODETECT);
// Export/download the file from TileDB and save it with a given name.
FileStore.uriExport(ctx, "tiledb://<NAMESPACE_NAME>/<ARRAY_NAME>", "<OUTPUT_FILENAME>");
In addition to registering S3-stored TileDB groups with TileDB cloud via the console, you can also do it programmatically as follows:
Python
Java
tiledb.cloud.groups.register("s3://mybucket/mygroup",
namespace="user1", # Optional, you may register it under your username, or one of your organizations
name="mygroup",
description=None, # Optional
credentials_name="myCredentials") # You must have already added your AWS credentials on the console
String namespace = "<NAMESPACE>"; // String | namespace array is in (an organization name or user's username)
String array = "s3://<S3_BUCKET>/<ARRAY_NAME>"; // String | name/uri of s3 array that is url-encoded
GroupInfoUpdate groupInfo = new GroupInfoUpdate(); // ArrayInfoUpdate | metadata associated with array
groupInfo.setUri("s3://<S3_BUCKET>/<ARRAY_NAME>");
groupInfo.setName("<ARRAY_NAME>");
try {
GroupInfo result = arrayApi.registerGroup(namespace, array, groupInfo);
System.out.println(result);
} catch (ApiException e) {
System.err.println("Exception when calling ArrayApi#registerArray");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
You can deregister an group as follows:
Python
Java
tiledb.cloud.groups.deregister("tiledb://user1/mygroup", recursive=False)
GroupApi apiInstance = new GroupApi(defaultClient);
String namespace = "<NAMESPACE>"; // String | namespace array is in (an organization name or user's username)
String array = "<ARRAY_NAME>"; // String | name/uri of array that is url-encoded
try {
apiInstance.deregisterGroup(namespace, array);
} catch (ApiException e) {
System.err.println("Exception when calling GroupApi#deregisterGroup");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
Deregistering an array will not physically delete it.
Last modified 11d ago