Utilities

The TileDB Cloud client offers several useful utilities. To use them, you must have the client installed (see Installation).

Login Sessions

TileDB Cloud allows you to login (with your username/password or API token) in a way such that the session token can be cached to avoid logging in again for every program execution. This is done as follows:

# With username/password
tiledb.cloud.login(username='xxx', password='xxx')

# Or, with token
tiledb.cloud.login(token='xxx')

After logging in for the first time, the TileDB Cloud client will store a session token in configuration file $HOME/.tiledb/cloud.jsoncreated in your home directory.

Retry Settings

The TileDB Cloud clients have the ability to retry failed HTTP requests automatically. By default this is enabled for retrying when TileDB Cloud indicates there is not enough capacity for the request (HTTP 503 errors). For convenience we also offer the ability to disable retries or to enable more forceful retry settings.

Built in modes

# Set default retry for only retrying "not enough capacity" responses
tiledb.cloud.client.client.retry_mode("default")

# Set do not retry any requests
tiledb.cloud.client.client.retry_mode("disabled")

# Retry for a large number of scenarios
tiledb.cloud.client.client.retry_mode("forceful")

In "forceful" mode it is possible that the client might retry requests which will always fail, such as when there is a syntax error in a SQL query. This mode should be used with care to avoid increased costs from retrying.

All built in modes (besides disabled) will retry a request up to 10 times.

Custom Retry Logic

It is also possible to manually set retry conditions to suite your needs.

from urllib3 import Retry

# Set the retries to a urllib3 retry object
tiledb.cloud.config.config.retries=Retry(
        total=10,
        backoff_factor=0.25,
        status_forcelist=[400, 500, 501, 502, 503],
        allowed_methods=["HEAD","GET","PUT","DELETE","OPTIONS","TRACE","POST","PATCH",],
        raise_on_status=False,
        remove_headers_on_redirect=[],
    )
    
# After updating the config make sure to update the package level clients
tiledb.cloud.client.client.update_clients()

Context and Config

There are two helper functions that allow to easily create a tiledb config or context that has the proper configuration needed for slicing arrays through TileDB Cloud.

# Create a TileDB Config object with `rest.token` set from the login
cfg = tiledb.cloud.Config()

# Create a TileDB Context which has a config with `rest.token` set from the login
ctx = tiledb.cloud.Ctx()

Viewing the User Profile

You can see your user profile as follows:

prof = tiledb.cloud.user_profile()
print(prof)

Listing Arrays

You can list arrays from the cloud service, passing a variety of filters:

# List all arrays you own
owned_arrays = tiledb.cloud.list_arrays()
print(owned_arrays)

# List all arrays that are shared with you
shared_arrays = tiledb.cloud.list_shared_arrays()
print(shared_arrays)

# List all public arrays
public_arrays = tiledb.cloud.list_public_arrays()
print(public_arrays)

# List arrays in a specific namespace
tiledb_inc_arrays = tiledb.cloud.list_arrays(namespace="TileDB-Inc")
print(tiledb_inc_arrays)

# Filter arrays to only those you have read and write permissions to
rw_arrays = tiledb.cloud.list_arrays(permissions=["read", "write"])
print(rw_arrays)

# You can combine filters
arrays = tiledb.cloud.list_arrays(namespace="TileDB-Inc", permissions=["read"])
print(arrays)

# Search keywords
arrays = tiledb.cloud.list_public_arrays(namespace="TileDB-Inc", search="nyc")
print(arrays)

# List specific asset types that are based upon Arrays. See also Listing Groups
# Available asset types:
# FileType.FILE
# FileType.USER_DEFINED_FUNCTION
# FileType.REGISTERED_TASK_GRAPH
# FileType.ML_MODEL
# FileType.NOTEBOOK
assets = tiledb.cloud.list_public_arrays(namespace="TileDB-Inc", file_type=[tiledb.cloud.rest_api.FileType.NOTEBOOK, tiledb.cloud.rest_api.FileType.ML_MODEL])
print(assets)

Getting Array Information

You can run the following to get basic information about the array, such as its description:

info = tiledb.cloud.info("tiledb://TileDB-Inc/quickstart_sparse")
print(info)

Array Activity

Array activity can be fetched programmatically as follows:

activity = tiledb.cloud.array_activity("tiledb://TileDB-Inc/quickstart_sparse")
print(activity)

Listing Tasks

You can list tasks from the cloud service, passing a variety of filters:

# List all tasks
all_tasks = tiledb.cloud.fetch_tasks()
print(all_tasks)

# List only tasks on a specific array
array_tasks = tiledb.cloud.fetch_tasks(array="tiledb://TileDB-Inc/quickstart_sparse")
print(array_tasks)

# Lists tasks within a specific time period
import datetime
ninety_days_ago = datetime.datetime.utcnow() - datetime.timedelta(days=90)
datetime_tasks = tiledb.cloud.fetch_tasks(
    array="tiledb://TileDB-Inc/quickstart_sparse",
    start=ninety_days_ago
)
print(datetime_tasks)

# Filter tasks by status, valid statuses are RUNNING, FAILED, COMPLETED
running_tasks = tiledb.cloud.fetch_tasks(status="RUNNING")
print(running_tasks)

For convenience, you can also see the last SQL or UDF task:

# Get last SQL task
tiledb.cloud.last_sql_task()

# Get last UDF task
tiledb.cloud.last_udf_task()

Or you can get a specific task with a given task ID (which can be found on the UI console):

task = tiledb.cloud.task(id='xxx')

Registering an Array

In addition to registering S3-stored TileDB arrays with TileDB cloud via the console, you can also do it programmatically as follows:

tiledb.cloud.register_array(uri="s3://mybucket/myarray",
                namespace="user1", # Optional, you may register it under your username, or one of your organizations
                array_name="myarray",
                description=None,  # Optional 
                access_credentials_name="myCredentials") # You must have already added your AWS credentials on the console

Deregistering an Array

You can deregister an array as follows:

tiledb.cloud.deregister_array("tiledb://user1/myarray")

Deregistering an array will not physically delete it.

Sharing Arrays

You can programmatically share a registered array, "unshare" a registered array (i.e., revoke access) and list array sharing information as follows:

# Share an array with both read and write permissions with a user
tiledb.cloud.share_array(uri="tiledb://user1/myarray",
                         namespace="user1", # The user to share the array with
                         permissions=["read", "write"])

# Revoke access to an array for a particular user                         
tiledb.cloud.unshare_array(uri="tiledb://user1/myarray", namespace="user1")                      

# Get sharing information about an array
shared_with = tiledb.cloud.list_shared_with("tiledb://user1/myarray")
print(shared_with)

Invite to Array

Similar to Sharing Arrays, you can invite users to an array as follows:

tiledb.cloud.invites.invite_to_array(
    "tiledb://user1/myarray",
    recipients=[<tiledb-username1>, <tiledb-username2>, <email>, ...],
    actions=["READ", "WRITE", ...]
)
  • recipients can include any combination of TileDB usernames and email addresses.

  • actions allowed values are: READ, WRITE, EDIT, READ_ARRAY_LOGS, READ_ARRAY_INFO, READ_ARRAY_SCHEMA

You can cancel an invitation to an array as follows:

tiledb.cloud.invites.cancel_invite_to_array(
    "tiledb://user1/myarray", 
    invitation_id=<invitation_id>
)

Region Redirection

When accessing an array or group via the API, your request will be automatically routed to the instance closest to the data. If you already know the region, a compute region can be accessed directly with a configured parameter to manually bypass automatic redirection. Manually specifying the region can be helpful if you want to avoid the slight increase in latency that the redirection adds.

To access a region directly the domain is of the scheme: <region>.aws.api.tiledb.com

The five domains we currently support are:

  • us-east-1.aws.api.tiledb.com

  • us-west-2.aws.api.tiledb.com

  • eu-west-1.aws.api.tiledb.com

  • eu-west-2.aws.api.tiledb.com

  • ap-southeast-1.aws.api.tiledb.com

You can manually set the domain to send a request directly to a region as follows:

import tiledb, tiledb.sql
import pandas

# Create the configuration parameters
config = tiledb.Config()
config["rest.username"] = "xxx"
config["rest.password"] = "yyy"
# or, more preferably, config["rest.token"] = "my_token"

# Manually set the server address to the redirection URL
config["rest.server_address"] = "https://eu-west-2.aws.api.tiledb.com"

# This is the array URI format in TileDB Cloud
array_name = "tiledb://TileDB-Inc/quickstart_sparse-eu-west-2"

# Write code exactly as in TileDB Developer
with tiledb.open(array_name, 'r', ctx=tiledb.Ctx(config)) as A:
    print (A.df[:])
    
# Using embedded SQL, you need to pass the username/password 
# as config parameters as well as the server address in `init_command`
db = tiledb.sql.connect(db="test",
        init_command="set mytile_tiledb_config='rest.username=xxx,rest.password=xxx,rest.server_address=https://eu-west-2.aws.api.tiledb.com'")
pandas.read_sql(sql="select * from `tiledb://TileDB-Inc/quickstart_sparse-eu-west-2`", con=db)

Files

TileDB Cloud has the ability to convert files to and from the TileDB File representation. This allows you to store any arbitrary file as a 1 dimensions dense array. Importing and exporting to and from the original file format is supported directly through TileDB Cloud. The file-arrays can be stored on an object store, such as S3, directly.

# Import from s3 to a TileDB array,
# automatically registering it with TileDB Cloud
tiledb.cloud.files.utils.create_file(
    namespace="my_organization",
    name="my_file", # optional name to set for registered file
    input_uri="s3://my_bucket/files/my_file.pdf",
    output_uri="s3://my_bucket/files/arrays/my_file"
)

# Export back to S3 in the original format
# The export happens completely in TileDB cloud
tiledb.cloud.files.utils.export_file(
    uri="tiledb://my_organization/my_file",
    output_uri="s3://my_bucket/files/arrays/my_file"
)

# Export back to local filesystem in the original format
tiledb.cloud.files.utils.export_file(
    uri="tiledb://my_organization/my_file",
    output_uri="my_file_exported.pdf",
)

Registering a Group

In addition to registering S3-stored TileDB groups with TileDB cloud via the console, you can also do it programmatically as follows:

tiledb.cloud.groups.register("s3://mybucket/mygroup",
                namespace="user1", # Optional, you may register it under your username, or one of your organizations
                name="mygroup",
                description=None,  # Optional 
                credentials_name="myCredentials") # You must have already added your AWS credentials on the console

Deregistering a Group

You can deregister an group as follows:

tiledb.cloud.groups.deregister("tiledb://user1/mygroup", recursive=False)

Deregistering a group will not physically delete it.

Listing Group

You can list arrays from the cloud service, passing a variety of filters:

# List all groups you own
owned_groups = tiledb.cloud.list_groups()
print(owned_groups)

# List all arrays that are shared with you
shared_arrays = tiledb.cloud.list_shared_groups()
print(shared_groups)

# List all public groups
public_groups = tiledb.cloud.list_public_groups()
print(public_groups)

# List arrays in a specific namespace
tiledb_inc_groups = tiledb.cloud.list_groups(namespace="TileDB-Inc")
print(tiledb_inc_groups)

# Search keywords
groups = tiledb.cloud.list_public_groups(namespace="TileDB-Inc", search="dragen")
print(groups)

# You can combine filters
groups = tiledb.cloud.list_public_groups(namespace="TileDB-Inc", tag="genomics", search="dragen")
print(groups)

# List specific asset types that are based upon Groups. See also Listing Arrays
# Available asset types:
# GroupType.BIOIMG
# GroupType.SOMA
# GroupType.VCF
# GroupType.POINTCLOUD
# GroupType.RASTER
assets = tiledb.cloud.list_public_groups(namespace="TileDB-Inc", group_type=tiledb.cloud.rest_api.GroupType.SOMA)
print(assets)

Getting Group Information

You can run the following to get basic information about the array, such as its description:

info = tiledb.cloud.groups.info("tiledb://TileDB-Inc/vcf-1kghicov-dragen-v376")
print(info)

Invite to a Group

You can invite users to a group as follows:

tiledb.cloud.invites.invite_to_group(
    "tiledb://user1/mygroup", 
    recipients=[<tiledb-username1>, <tiledb-username2>, <email>, ...],
    array_actions=["READ", ...],
    group_actions=["WRITE", ...]
)
  • recipients can include any combination of TileDB usernames and email addresses.

  • array_actions allowed values are: READ, WRITE, EDIT, READ_ARRAY_LOGS, READ_ARRAY_INFO, READ_ARRAY_SCHEMA

  • group_actions allowed values are: READ, WRITE, EDIT

You can cancel an invitation to a group as follows:

tiledb.cloud.invites.cancel_invite_to_group(
    "tiledb://user1/mygroup", 
    invitation_id=<invitation_id>
)

Invite to Organization

You can invite users to an organization as follows:

tiledb.cloud.invites.invite_to_organization(
    "Organization_Name", 
    recipients=[<tiledb-username1>, <tiledb-username2>, <email>, ...],
    role="READ_WRITE"
)
  • recipients any combination of TileDB usernames and email addresses.

  • role can be one of the following values: OWNER, ADMIN, READ_WRITE, READ_ONLY

Accept an Invite

You can accept an invite by its ID as follows:

tiledb.cloud.invites.accept_invitation(<invitation_id>)

List invitations

You can fetch a paginated list of invitations as follows:

tiledb.cloud.invites.fetch_invitations(**filters)

Available Filters

  • organization: name or ID of organization to filter

  • array: name/uri of array that is url-encoded to filter

  • group: name or ID of group to filter

  • start: start time for tasks to filter by

  • end: end time for tasks to filter by

  • page: pagination offset

  • per_page: pagination limit

  • type: invitation type, "ARRAY_SHARE", "JOIN_ORGANIZATION"

  • status: Filter to only return "PENDING", "ACCEPTED"

  • orderby: sort by which field valid values include

Last updated