Concepts
API Reference
More Help
Utilities
The TileDB Cloud client offers several useful utilities. To use them, you must have the client installed (see Installation).
Login Sessions
TileDB Cloud allows you to login (with your username/password or API token) in a way such that the session token can be cached to avoid logging in again for every program execution. This is done as follows:
Python
R
1
# With username/password
2
tiledb.cloud.login(username='xxx', password='xxx')
3
​
4
# Or, with token
5
tiledb.cloud.login(token='xxx')
Copied!
1
# With username/password
2
tiledbcloud::login(username='xxx', password='xxx')
3
​
4
# Or, with token
5
tiledbcloud::login(api_key='xxx')
Copied!
After logging in for the first time, the TileDB Cloud client will store a session token in configuration file
$HOME/.tiledb/cloud.jsoncreated in your home directory.Retry Settings
The TileDB Cloud clients have the ability to retry failed HTTP requests automatically. By default this is enabled for retrying when TileDB Cloud indicates there is not enough capacity for the request (HTTP 503 errors). For convenience we also offer the ability to disable retries or to enable more forceful retry settings.
Built in modes
Python
1
# Set default retry for only retrying "not enough capacity" responses
2
tiledb.cloud.client.client.retry_mode("default")
3
​
4
# Set do not retry any requests
5
tiledb.cloud.client.client.retry_mode("disabled")
6
​
7
# Retry for a large number of scenarios
8
tiledb.cloud.client.client.retry_mode("forceful")
9
​
Copied!
In "forceful" mode it is possible that the client might retry requests which will always fail, such as when there is a syntax error in a SQL query. This mode should be used with care to avoid increased costs from retrying.
All built in modes (besides disabled) will retry a request up to 10 times.
Custom Retry Logic
It is also possible to manually set retry conditions to suite your needs.
Python
1
from urllib3 import Retry
2
​
3
# Set the retries to a urllib3 retry object
4
tiledb.cloud.config.config.retries=Retry(
5
total=10,
6
backoff_factor=0.25,
7
status_forcelist=[400, 500, 501, 502, 503],
8
allowed_methods=["HEAD","GET","PUT","DELETE","OPTIONS","TRACE","POST","PATCH",],
9
raise_on_status=False,
10
remove_headers_on_redirect=[],
11
)
12
13
# After updating the config make sure to update the package level clients
14
tiledb.cloud.client.client.update_clients()
Copied!
Context and Config
There are two helper functions that allow to easily create a
tiledb config or context that has the proper configuration needed for slicing arrays through TileDB Cloud.Python
R
1
# Create a TileDB Config object with `rest.token` set from the login
2
cfg = tiledb.cloud.Config()
3
​
4
# Create a TileDB Context which has a config with `rest.token` set from the login
5
ctx = tiledb.cloud.Ctx()
Copied!
1
# Create a TileDB Config object with `rest.token` set from the login
2
config <- tiledb_config()
3
​
4
# Create a TileDB Context which has a config with `rest.token` set from the login
5
ctx <- tiledb_ctx(config)
Copied!
Viewing the User Profile
You can see your user profile as follows:
Python
R
1
prof = tiledb.cloud.user_profile()
2
print(prof)
Copied!
1
prof <- tiledbcloud::user_profile()
2
print(prof)
Copied!
Listing Arrays
You can list arrays from the cloud service, passing a variety of filters:
Python
R
1
# List all arrays you own
2
owned_arrays = tiledb.cloud.list_arrays()
3
print(owned_arrays)
4
​
5
# List all arrays that are shared with you
6
shared_arrays = tiledb.cloud.list_shared_arrays()
7
print(shared_arrays)
8
​
9
# List all public arrays
10
public_arrays = tiledb.cloud.list_public_arrays()
11
print(public_arrays)
12
​
13
# List arrays in a specific namespace
14
tiledb_inc_arrays = tiledb.cloud.list_arrays(namespace="TileDB-Inc")
15
print(tiledb_inc_arrays)
16
​
17
# Filter arrays to only those you have read and write permissions to
18
rw_arrays = tiledb.cloud.list_arrays(permissions=["read", "write"])
19
print(rw_arrays)
20
​
21
# You can combine filters
22
arrays = tiledb.cloud.list_arrays(namespace="TileDB-Inc", permissions=["read"])
23
print(arrays)
Copied!
1
# List all arrays you own
2
owned_arrays <- tiledbcloud::list_arrays()
3
str(owned_arrays)
4
​
5
# List all arrays that are shared with you
6
shared_arrays <- tiledbcloud::list_arrays(shared=TRUE)
7
str(shared_arrays)
8
​
9
# List all public arrays
10
public_arrays <- tiledbcloud::list_arrays(public=TRUE)
11
str(public_arrays)
12
​
13
# List arrays in a specific namespace
14
tiledb_inc_arrays = tiledbcloud::list_arrays(namespace="TileDB-Inc")
15
str(tiledb_inc_arrays)
Copied!
Getting Array Information
You can run the following to get basic information about the array, such as its description:
Python
R
1
info = tiledb.cloud.info("tiledb://TileDB-Inc/quickstart_sparse")
2
print(info)
Copied!
1
info <- tiledbcloud::array_info(namespace="TileDB-Inc", arrayname="quickstart_sparse")
2
str(info)
Copied!
Array Activity
Array activity can be fetched programmatically as follows:
Python
1
activity = tiledb.cloud.array_activity("tiledb://TileDB-Inc/quickstart_sparse")
2
print(activity)
Copied!
Listing Tasks
You can list tasks from the cloud service, passing a variety of filters:
Python
1
# List all tasks
2
all_tasks = tiledb.cloud.tasks()
3
print(all_tasks)
4
​
5
# List only tasks on a specific array
6
array_tasks = tiledb.cloud.tasks(array="tiledb://TileDB-Inc/quickstart_sparse")
7
print(array_tasks)
8
​
9
# Lists tasks within a specific time period
10
import datetime
11
ninety_days_ago = datetime.datetime.utcnow() - datetime.timedelta(days=90)
12
datetime_tasks = tiledb.cloud.tasks(array="tiledb://TileDB-Inc/quickstart_sparse",
13
start=ninety_days_ago)
14
print(datetime_tasks)
15
​
16
# Filter tasks by status, valid statuses are RUNNING, FAILED, COMPLETED
17
running_tasks = tiledb.cloud.tasks(status="RUNNING")
18
print(running_tasks)
Copied!
For convenience, you can also see the last SQL or UDF task:
Python
1
# Get last SQL task
2
tiledb.cloud.last_sql_task()
3
​
4
# Get last UDF task
5
tiledb.cloud.last_udf_task()
Copied!
Or you can get a specific task with a given task ID (which can be found on the UI console):
Python
1
task = tiledb.cloud.task(id='xxx')
Copied!
Registering an Array
In addition to registering S3-stored TileDB arrays with TileDB cloud via the console, you can also do it programmatically as follows:
Python
1
tiledb.cloud.register_array(uri="s3://mybucket/myarray",
2
namespace="user1", # Optional, you may register it under your username, or one of your organizations
3
array_name="myarray",
4
description=None, # Optional
5
access_credentials_name="myCredentials") # You must have already added your AWS credentials on the console
Copied!
Deregistering an Array
You can deregister an array as follows:
Python
1
tiledb.cloud.deregister_array("tiledb://user1/myarray")
Copied!
Deregistering an array will not physically delete it.
Sharing Arrays
You can programmatically share a registered array, "unshare" a registered array (i.e., revoke access) and list array sharing information as follows:
Python
1
# Share an array with both read and write permissions with a user
2
tiledb.cloud.share_array(uri="tiledb://user1/myarray",
3
namespace="user1", # The user to share the array with
4
permissions=["read", "write"])
5
​
6
# Revoke access to an array for a particular user
7
tiledb.cloud.unshare_array(uri="tiledb://user1/myarray", namespace="user1")
8
​
9
# Get sharing information about an array
10
shared_with = tiledb.cloud.list_shared_with("tiledb://user1/myarray")
11
print(shared_with)
Copied!
Automatic Region Redirection
While automatic compute region direction is in beta you will need to manually enable it for a query or request. Below is two examples of setting the server address to the redirection domain
https://multi-region.api.tiledb.com .Python - Array Access
Python - Array UDF
1
import tiledb, tiledb.sql
2
import pandas
3
​
4
# Create the configuration parameters
5
config = tiledb.Config()
6
config["rest.username"] = "xxx"
7
config["rest.password"] = "yyy"
8
# or, more preferably, config["rest.token"] = "my_token"
9
​
10
# Manually set the server address to the redirection URL
11
config["rest.server_address"] = "https://multi-region.api.tiledb.com"
12
​
13
# This is the array URI format in TileDB Cloud
14
array_name = "tiledb://TileDB-Inc/quickstart_sparse-eu-west-2"
15
​
16
# Write code exactly as in TileDB Embedded
17
with tiledb.open(array_name, 'r', ctx=tiledb.Ctx(config)) as A:
18
print (A.df[:])
19
20
# Using embedded SQL, you need to pass the username/password
21
# as config parameters as well as the server address in `init_command`
22
db = tiledb.sql.connect(db="test",
23
init_command="set mytile_tiledb_config='rest.username=xxx,rest.password=xxx,rest.server_address=https://multi-region.api.tiledb.com'")
24
pandas.read_sql(sql="select * from `tiledb://TileDB-Inc/quickstart_sparse-eu-west-2`", con=db)
Copied!
1
import tiledb, tiledb.cloud, numpy
2
​
3
def median(numpy_ordered_dictionary):
4
return numpy.median(numpy_ordered_dictionary["a"])
5
​
6
tiledb.cloud.login(username="xxx", password="yyy", host="https://multi-region.api.tiledb.com")
7
# or tiledb.cloud.login(token="my_token", host="https://multi-region.api.tiledb.com")
8
​
9
with tiledb.open("tiledb://TileDB-Inc/quickstart_sparse-eu-west-2", ctx=tiledb.cloud.Ctx()) as A:
10
# apply on subarray [1,2]x[1,2]
11
res = A.apply(median, [(1,2), (1,2)], attrs = ["a"])
12
print(res)
Copied!
Direct Region Access
A compute region can be access directly bypassing automatic redirection. This is helpful if you want to avoid the slight increase in latency that the redirection adds.
To access a region directly the domain is of the scheme:
<region>.aws.api.tiledb.comThe four domains we currently support are:
us-east-1.aws.api.tiledb.comus-west-2.aws.api.tiledb.comeu-west-2.aws.api.tiledb.comap-southeast-1.aws.api.tiledb.com
You can manually set the domain to send a request directly to a region as follows:
Python - Array Access
Python - Array UDF
1
import tiledb, tiledb.sql
2
import pandas
3
​
4
# Create the configuration parameters
5
config = tiledb.Config()
6
config["rest.username"] = "xxx"
7
config["rest.password"] = "yyy"
8
# or, more preferably, config["rest.token"] = "my_token"
9
​
10
# Manually set the server address to the redirection URL
11
config["rest.server_address"] = "https://eu-west-2.aws.api.tiledb.com"
12
​
13
# This is the array URI format in TileDB Cloud
14
array_name = "tiledb://TileDB-Inc/quickstart_sparse-eu-west-2"
15
​
16
# Write code exactly as in TileDB Developer
17
with tiledb.open(array_name, 'r', ctx=tiledb.Ctx(config)) as A:
18
print (A.df[:])
19
20
# Using embedded SQL, you need to pass the username/password
21
# as config parameters as well as the server address in `init_command`
22
db = tiledb.sql.connect(db="test",
23
init_command="set mytile_tiledb_config='rest.username=xxx,rest.password=xxx,rest.server_address=https://eu-west-2.aws.api.tiledb.com'")
24
pandas.read_sql(sql="select * from `tiledb://TileDB-Inc/quickstart_sparse-eu-west-2`", con=db)
Copied!
1
import tiledb, tiledb.cloud, numpy
2
​
3
def median(numpy_ordered_dictionary):
4
return numpy.median(numpy_ordered_dictionary["a"])
5
​
6
tiledb.cloud.login(username="xxx", password="yyy", host="https://eu-west-2.aws.api.tiledb.com")
7
# or tiledb.cloud.login(token="my_token", host="https://eu-west-2.aws.api.tiledb.com")
8
​
9
with tiledb.open("tiledb://TileDB-Inc/quickstart_sparse-eu-west-2", ctx=tiledb.cloud.Ctx()) as A:
10
# apply on subarray [1,2]x[1,2]
11
res = A.apply(median, [(1,2), (1,2)], attrs = ["a"])
12
print(res)
Copied!
Files
TileDB Cloud has the ability to convert files to and from the TileDB File representation. This allows you to store any arbitrary file as a 1 dimensions dense array. Importing and exporting to and from the original file format is supported directly through TileDB Cloud. The file-arrays can be stored on an object store, such as S3, directly.
Python
1
# Import from s3 to a TileDB array,
2
# automatically registering it with TileDB Cloud
3
tiledb.cloud.file.create_file(
4
namespace="my_organization",
5
name="my_file", # optional name to set for registered file
6
input_uri="s3://my_bucket/files/my_file.pdf",
7
output_uri="s3://my_bucket/files/arrays/my_file"
8
)
9
​
10
# Export back to S3 in the original format
11
# The export happens completely in TileDB cloud
12
tiledb.cloud.file.export_file(
13
uri="tiledb://my_organization/my_file",
14
output_uri="s3://my_bucket/files/arrays/my_file"
15
)
16
​
17
# Export back to local filesystem in the original format
18
tiledb.cloud.file.export_file(
19
uri="tiledb://my_organization/my_file",
20
output_uri="my_file_exported.pdf",
21
)
Copied!
Last modified 3mo ago