Serverless SQL

How It Works

TileDB Cloud allows you to perform any SQL query on your TileDB arrays in a serverless manner. No need to spin up or tear down any computational resources. You just need to have the TileDB Cloud client installed (see Installation). You get charged only for the time it took to run the SQL operation (see Pricing for more details).

TileDB Cloud currently supports serverless SQL only through Python, but support for more languages will be added soon.

TileDB Cloud receives your SQL query and executes it on a stateless worker that runs a warm MariaDB instance using the MyTile storage engine. The results of the SQL query can be returned directly to you (when using TileDB-Cloud-Py version 0.4.0 or newer) or they can be written back to an S3 array of your choice (either existing or new).

When results are returned directly, they are sent to the client as JSON and in Python they are converted into a Pandas dataframe. This is most suitable for small results, such as aggregations or limit queries.

Writing results to an S3 array is necessary to allow processing of SQL queries with large results, without overwhelming the user (who may be on their laptop). The user can always open the created TileDB array and fetch any desired data afterwards. If you need to write the SQL results to an array you are permitted to access, you need to set up your AWS keys to use the serverless SQL functionality.

Each TileDB Cloud worker running a SQL query uses 16 CPUs and has a limit of 2GB RAM. Therefore, you must consider "sharding" a SQL query so that each result fits in 2GB of memory (see also Parallel Computing). In the future, TileDB Cloud will offer flexibility in choosing the types of resources to run SLQ on.

Usage

Small results such as an average can easily be run and returned as follows:

Python
Python
import tiledb.cloud
tiledb.cloud.login(username="my_username", password="my_password")
# or tiledb.cloud.login(token="my_token")
# res is a Pandas dataframe
res = tiledb.cloud.sql.exec(
query = "select `rows`, AVG(a) as avg_a from `tiledb://TileDB-Inc/quickstart_dense` GROUP BY `rows`")

Supposing that there is an array tiledb://user/array_name that you have write permissions for, you can run a SQL query that writes the results to that array as follows:

Python
Python
import tiledb.cloud
tiledb.cloud.login(username="my_username", password="my_password")
# or tiledb.cloud.login(token="my_token")
tiledb.cloud.sql.exec(
query = "select `rows`, AVG(a) as avg_a from `tiledb://TileDB-Inc/quickstart_dense` GROUP BY `rows`",
output_uri = "tiledb://user/array_name")

You can run any SQL statement, including CTEs.

Python
Python
import tiledb.cloud
tiledb.cloud.login(username="my_username", password="my_password")
# or tiledb.cloud.login(token="my_token")
# INSERT INTO ...
tiledb.cloud.sql.exec(query = "insert into `tiledb://user1/output1` select AVG(a) from `tiledb://user1/myarray1`")
# CTE
tiledb.cloud.sql.exec(query =
"""
WITH t1 AS (select `dim`, `dim2`, AVG(a), AVG(a) OVER (partition by dim2) FROM `tiledb://user1/array1`)
insert into `tiledb://user/array_name`
select * from t1
""")

If the array does not exist, you will just need to create and pass an array schema that complies with the result schema.

Python
Python
import tiledb.cloud
import numpy as np
import pandas
tiledb.cloud.login(username="my_username", password="my_password")
# or tiledb.cloud.login(token="my_token")
# The output array has a single dimension
dom = tiledb.Domain(tiledb.Dim(name="rows", domain=(1, 4), tile=4, dtype=np.int32))
# The output array has a single attribute
attr = tiledb.Attr(name="avg_a", dtype=np.float32)
# The output array is sparse
schema = tiledb.ArraySchema(domain=dom, sparse=True, attrs=[attr])
# The following will create the output array and write the results to it
res = tiledb.cloud.sql.exec_and_fetch(
query = "select `rows`, AVG(a) as avg_a from `tiledb://TileDB-Inc/quickstart_dense` GROUP BY `rows`",
output_schema = schema,
output_uri = "tiledb://user/s3://my_bucket/my_array") # Notice the full path here
print(pandas.DataFrame(res[:]))

We also provide an auxiliary function exec_and_fetch, which executes a SQL query as above, but then opens the resulting output array so that it is ready for use.

Python
Python
import tiledb.cloud
import pandas
tiledb.cloud.login(username="my_username", password="my_password")
# or tiledb.cloud.login(token="my_token")
res = tiledb.cloud.sql.exec_and_fetch(
query = "select `rows`, AVG(a) as avg_a from `tiledb://TileDB-Inc/quickstart_dense` GROUP BY `rows`",
output_uri="tiledb://user/array_name")
# `res` is an opened array for reading, ready for use
print(pandas.DataFrame(res[:]))

Selecting Who to Charge

If you you are a member of an organization, then by default the organization is changed for your SQL query. If you would like to charge the SQL query to yourself, you just need to add one extra argument namespace.

Python
Python
import tiledb.cloud
tiledb.cloud.login(username="my_username", password="my_password")
# or tiledb.cloud.login(token="my_token")
res = tiledb.cloud.sql.exec(
namespace ="my_username", # who to charge the query to
query = "select `rows`, AVG(a) as avg_a from `tiledb://TileDB-Inc/quickstart_dense` GROUP BY `rows`",
output_uri = "tiledb://user/array_name")

Async Execution

Similar to array UDFs and generic UDFs, an asynchronous version of severless SQL is available. The _async version returns a future.

Python
Python
import tiledb.cloud
tiledb.cloud.login(username="my_username", password="my_password")
# or tiledb.cloud.login(token="my_token")
# res is a future
res = tiledb.cloud.sql.exec_async(
query = "select `rows`, AVG(a) as avg_a from `tiledb://TileDB-Inc/quickstart_dense` GROUP BY `rows`")
# call res.get() to block on the results
print(res.get())

Creating New Arrays

It is also possible to use SQL to create a new array. Full examples for the SQL syntax are available in the MariaDB Usage Docs.

Python
Python
import tiledb.cloud
tiledb.cloud.login(username="my_username", password="my_password")
# or tiledb.cloud.login(token="my_token")
# Create an array that will be registered in TileDB Cloud and stored in the
# s3 bucket and location specied in the `uri` parameter. The `uri` parameter
# overrides the normal use of the table name `example`.
# The array is automatically registered in TileDB cloud
# under `my_username`/`create_example_1`
tiledb.cloud.sql.exec("""
CREATE TABLE `example` (
dim0 integer DIMENSION=1 lower_bound="0" upper_bound="100" tile_extent="10",
dim1 integer DIMENSION=1 lower_bound="0" upper_bound="100" tile_extent="10",
attr1 varchar(255) filters="GZIP=-1",
attr2 FLOAT filters="GZIP=-1",
attr3 DATETIME filters="GZIP=-1"
) engine=MyTile
array_type='SPARSE'
coordinate_filters="NONE"
offset_filters="POSITIVE_DELTA=128"
uri='tiledb://my_username/s3://my_bucket/create_example_1';
""")
# You can also use `Primary Key` to specify which fields are the dimensions
# This replaces the `DIMENSION=1` parameter
# The array is automatically registered in TileDB cloud
# under `my_username`/`create_example_2`
tiledb.cloud.sql.exec("""
CREATE TABLE `example` (
dim0 integer lower_bound="0" upper_bound="100" tile_extent="10" NOT NULL,
dim1 bigint lower_bound="0" upper_bound="100" tile_extent="10" NOT NULL,
attr1 varchar(255) filters="GZIP=-1",
PRIMARY KEY(dim0, dim1)
) engine=MyTile
array_type='SPARSE'
coordinate_filters="NONE"
offset_filters="POSITIVE_DELTA=128"
uri='tiledb://my_username/s3://my_bucket/create_example_2';
""")
# Lastly you can also use `INDEX()` to set dimensions and allow duplicates
# By default duplicate coordinates for dimensions are not allowed,
# uniqueness is enforced just like a primary key. Using `Index()`
# instead triggers the array to be created allowing duplicates coordinate values.
# The array is automatically registered in TileDB cloud
# under `my_username`/`create_example_3`
tiledb.cloud.sql.exec("""
CREATE TABLE `example` (
dim0 integer lower_bound="0" upper_bound="100" tile_extent="10" NOT NULL,
dim1 varchar(255) NOT NULL,
attr1 varchar(255) filters="GZIP=-1",
INDEX(dim0, dim1)
) engine=MyTile
array_type='SPARSE'
coordinate_filters="NONE"
offset_filters="POSITIVE_DELTA=128"
uri='tiledb://my_username/s3://my_bucket/create_example_3';
""")