In this tutorial, you will learn:

1. How to access (slice) a public array

2. How to perform a SQL query on a public array.

3. How to perform serverless UDFs on a public array.

We will use public TileDB Cloud array TileDB-Inc/MBTA_Average_Monthly_Ridership_by_Mode, which stores the data from the Boston MBTA's Open Data Portal.

Running on TileDB Cloud

You can preview this tutorial as TileDB Cloud notebook (no login is needed). You can also easily launch it within the TileDB Cloud UI console, but you will need to sign up / login to do so.

Running on your own client

You can run all the commands of this notebook in your own client. The only changes required are:

1. Install the TileDB Cloud client

2. Log in using the TileDB Cloud client before running any notebook command

What is TileDB Cloud?

TileDB Cloud is the commercial platform built by the TileDB team that allows you and your organization to unify all types of data, automate distributed analysis and pipelines, and securely explore and share data and code, while enjoying extreme interoperability with programming languages and data science tools.

TileDB Cloud takes a radically different approach than the current data management landscape. Instead of dealing with a large number of different data formats and special-purpose databases, TileDB Cloud builds a unified data management stack, storing all types of data in a single unified format, pushing access control, logging and a growing set of computational primitives to storage, and emphasizing on integrations with every popular programming language and computational tool. And it does all that providing a 100% serverless experience to the user.

TileDB Cloud is based on the universal storage engine, which models and efficiently stores all data as (dense or sparse) multi-dimensional arrays, providing a common API and a large number of APIs and tool integrations.

Is TileDB Cloud for you?

TileDB Cloud is ideal for you if you struggle with:

• Data storage and access

  • inefficient files and domain-specific formats

  • wrangling data across tools and languages

  • controlling and monitoring access to data and code

  • sharing data and code at extreme scale

• Mixed data and workloads

  • different data types (e.g., dataframes, images, genomics, etc)

  • combination of SQL, data science and Machine Learning

• Scalability and deployment

  • scaling analysis easily and inexpensively

  • setting up and monitoring machine clusters

  • managing numerous disparate data solutions and silos

• Finding and contributing public data and code

  • sharing files in cloud buckets and code in repos

  • easily and reproducibly running code on different data

  • monitoring usage stats

• Monetizing data and code

  • having to build an entire infrastructure to sell data and code

Use Cases

There is a variety of application domains our team has expertise on and you can use TileDB Cloud for:

• Geospatial

  • point cloud (LiDAR, SONAR, AIS)

  • SAR

  • optical imaging

• Genomics

  • population genomics

  • single-cell multi-omics

• Dataframes

  • any tabular data, accessible with various APIs as well as SQL

• Time series

  • any data that could benefit from indexing on date/time fields

• Biomedical imaging

  • any imaging data requiring pyramid structures

• Many others

  • video, automotive, telecommunications, etc.

Capabilities

• Data and code management Manage all you data (modeled as multi-dimensional arrays) and code (UDFs and notebooks) in a single platform.

• Access control and sharing Securely share your data and code with access policies.

• Logging and auditing See all the activity on your data and code from detailed audit logs.

• Organizations Create organizations and define different access policies for data and code.

• Jupyter notebooks Create, share and spin up Jupyter notebooks directly in the platform with a few clicks.

• Serverless SQL and UDFs You can run any SQL query on any array, without having to provision any clusters. You can also define and register any user-defined function, as well as share it with other, which can be run in a serverless manner (similar to lambdas).

• Serverless task graphs Create any pipeline or any sophisticated distributed algorithm with TileDB's task graphs. TileDB executes the various tasks in the graph in parallel respecting the dependencies, without forcing the user to create or define any clusters.

Getting Started

If you'd like to take a deep dive into the TileDB Cloud internals, you can navigate to CONCEPTS in the left navigation menu. You can also always consult the HOW TO guides and API REFERENCE.

How to Use the Docs

To make it easy to understand where to find what you are looking for, the documentation is structured in the following sections:

• Tutorials A series of steps to address key problems and use cases

• Concepts Background information and explanation of key topics and concepts

• How To Short how-to guides based on FAQ

• API Reference

  Technical reference to the client APIs

TileDB Cloud Enterprise

Looking for an on-prem solution?

More Help

In case you do not find the information you need in these docs, there is a variety of channels you can get more help from:

In this tutorial you will learn how to:

1. Sign up / sign in to TileDB Cloud

2. Access a public array using the TileDB-Py library

3. Access basic array information using the TileDB Cloud client

4. View the public array on the TileDB Cloud console

5. View a task on the TileDB Cloud console

6. View and edit your profile on the TileDB Cloud console

Sign Up & Sign In

First, sign up and you will get $10 in free credit. Feel free to contact us if you run out of credits and you are still evaluating TileDB Cloud, we'll be happy to help out with more credits. Then simply sign in with the username and password you created.

Access a Public Array

It is extremely easy to access arrays registered with TileDB Cloud via the core TileDB Open Source APIs, with literally a single change: adding your username/password as configuration parameters. In this tutorial we will use TileDB-Py, TileDB-R and TileDB-Java. Let's first install it with:

# Pip:
$ pip install tiledb

# Or Conda:
$ conda install -c conda-forge tiledb-py

# Via CRAN
install.packages("tiledb")

// Maven
//Include this in your Maven project:
<dependency>
  <groupId>io.tiledb</groupId>
  <artifactId>tiledb-java</artifactId>
  <version>X.X.X</version>
</dependency>

// Or build from source
$ git clone https://github.com/TileDB-Inc/TileDB-Java.git
$ cd TileDB-Java
$ ./gradlew assemble

To validate the installation, run:

$ python
>>> import tiledb
>>> tiledb.__version__
'x.y.z' # The version will appear here

$ R
> library(tiledb)
> packageVersion("tiledb")
'x.y.z' # The version will appear here

// Compile and run:
Version version = new Version();
System.out.println(version);

For Python, also make sure you have already installed pandas and pyarrow, or alternatively run:

$ conda install pandas
$ conda install -c conda-forge pyarrow

We have added a couple of public arrays that every user on TileDB Cloud can access, under our TileDB-Inc organization. Below we show an example for getting the schema of array tiledb://TileDB-Inc/quickstart_sparse, and slicing its contents:

import tiledb

# Set your username and password
config = tiledb.Config()
config["rest.username"] = "xxx"
config["rest.password"] = "yyy"

# This is the array URI format in TileDB Cloud
array_name = "tiledb://TileDB-Inc/quickstart_sparse"

# Open the array
A = tiledb.open(array_name, 'r', ctx=tiledb.Ctx(config))

# Print the array schema
print(A.schema)
    
# This will print:
#
# ArraySchema(
#  domain=Domain(*[
#    Dim(name='rows', domain=(1, 4), tile=4, dtype='int32'),
#    Dim(name='cols', domain=(1, 4), tile=4, dtype='int32'),
#  ]),
#  attrs=[
#    Attr(name='a', dtype='uint32', var=False, nullable=False),
#  ],
#  cell_order='row-major',
#  tile_order='row-major',
#  capacity=10000,
#  sparse=True,
#  allows_duplicates=False,
#  coords_filters=FilterList([ZstdFilter(level=-1)]),
#)

# Print all the contents of the array
print(A.df[:])

# This will print
#
#    rows  cols  a
# 0     1     1  1
# 1     2     3  3
# 2     2     4  2

# Close the array
A.close()

library(tiledb)

# Set your username and password
config <- tiledb_config()
config["rest.username"] <- 'xxx'
config["rest.password"] <- 'yyy'
ctx <- tiledb_ctx(config)

# This is the array URI format in TileDB Cloud
array_name = "tiledb://TileDB-Inc/quickstart_sparse"

# Open the array
arr <- tiledb_array(array_name, return_as='data.frame')

# Print the array schema
schema(arr)

# This will print:
# tiledb_array_schema(
#     domain=tiledb_domain(c(tiledb_dim(name="rows", domain=c(1L,4L), tile=4L, type="INT32"), tiledb_dim(name="cols", domain=c(1L,4L), tile=4L, type="INT32"))),
#     attrs=c(tiledb_attr(name="a", type="UINT32", ncells=1, nullable=FALSE)),
#     cell_order="ROW_MAJOR", tile_order="ROW_MAJOR", capacity=10000, sparse=TRUE, allows_dups=FALSE,
#     coords_filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))),
#     offsets_filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))))
    
# Print all the contents of the array
df <- arr[]
df

# This will print
#   rows cols a
# 1    1    1 1
# 2    2    3 3
# 3    2    4 2

# Close the array
tiledb_array_close(arr)

//set the config
Config config = new Config();
config.set("rest.username", "xxx");
config.set("rest.password", "yyy");
Context ctx = new Context(config);

//open the array
Array array = new Array(ctx, "tiledb://TileDB-Inc/quickstart_sparse");

//print the array schema
System.out.println(array.getSchema());

//this will print:
//ArraySchema<TILEDB_SPARSE io.tiledb.java.api.Domain@51462713 Attr<a,TILEDB_UINT32,1>>

Congratulations! You just performed your first array query to a public array in TileDB Cloud!

Access via the TileDB Cloud Client

There are several TileDB Clients that allow you to perform pretty much any kind of task you would otherwise do via the TileDB Cloud online console (also described later in this tutorial). Let's first install it with:

# At your shell prompt
pip install tiledb-cloud

# At your shell prompt
# See https://github.com/TileDB-Inc/TileDB-Cloud-R/releases for latest:
remotes::install_github('TileDB-Inc/TileDB-Cloud-R@v0.0.8')

// Maven
//Include this in your project:
<dependency>
  <groupId>io.tiledb</groupId>
  <artifactId>tiledb-cloud-java</artifactId>
  <version>X.X.X</version>
</dependency>

// Or build from source
$ git clone https://github.com/TileDB-Inc/TileDB-Cloud-Java.git
$ cd TileDB-Cloud-Java
$ ./gradlew assemble

Check that it installed properly as follows:

$ python
>>> import tiledb.cloud

$ R
> library(tiledbcloud)

Let's get the description of the array we used above:

import tiledb.cloud

# Set your username and password
config = tiledb.Config()
config["rest.username"] = "xxx"
config["rest.password"] = "yyy"

info = tiledb.cloud.info("tiledb://TileDB-Inc/quickstart_sparse")
print(info)

# This prints:
#
# {'access_credentials_name': None,
# 'allowed_actions': ['read', 'read_array_info', 'read_array_schema'],
# 'description': '# Quickstart Sparse Array\n'
#                '\n'
#                'This is a very simple 2D sparse array, used to quickly '
#                'demonstrate basic TileDB Cloud functionality.',
# 'file_properties': None,
# 'file_type': None,
# 'id': '69ca9424-2578-44d0-8662-87897d9e2941',
# 'last_accessed': datetime.datetime(2021, 5, 3, 19, 1, 15, tzinfo=tzutc()),
# 'license_id': 'MIT',
# 'license_text': 'MIT License Copyright (c) <year> <copyright holders>\n'
#                 '\n'
#                 'Permission is hereby granted, free of charge, to any person '
#                 'obtaining a copy of this software and associated '
#                 'documentation files (the "Software"), to deal in the '
#                 'Software without restriction, including without limitation '
#                 'the rights to use, copy, modify, merge, publish, distribute, '
#                 'sublicense, and/or sell copies of the Software, and to '
#                 'permit persons to whom the Software is furnished to do so, '
#                 'subject to the following conditions:\n'
#                 '\n'
#                 'The above copyright notice and this permission notice '
#                 '(including the next paragraph) shall be included in all '
#                 'copies or substantial portions of the Software.\n'
#                 '\n'
#                 'THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY '
#                 'KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE '
#                 'WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR '
#                 'PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS '
#                 'OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR '
#                 'OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR '
#                 'OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE '
#                 'SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.',
# 'logo': None,
# 'name': 'quickstart_sparse',
# 'namespace': 'TileDB-Inc',
# 'namespace_subscribed': False,
# 'pricing': None,
# 'public_share': True,
# 'share_count': 1.0,
# 'size': 903.0,
# 'subscriptions': None,
# 'tags': None,
# 'tiledb_uri': 'tiledb://TileDB-Inc/quickstart_sparse',
# 'type': 'sparse',
# 'uri': None

library(tiledbcloud)

# Set your username and password
config <- tiledb_config()
config["rest.username"] <- 'xxx'
config["rest.password"] <- 'yyy'
ctx <- tiledb_ctx(config)

info <- tiledbcloud::array_info(
  namespace="TileDB-Inc",
  arrayname="quickstart_dense"))

str(info)

# This prints:
# List of 14
#  $ id                  : chr "2d6e7def-851e-4832-ae92-202af1f6940d"
#  $ namespace           : chr "TileDB-Inc"
#  $ size                : int 835
#  $ last_accessed       : chr "2022-02-28T17:39:48Z"
#  $ description         : chr "# Quickstart Dense Array\n\nThis array is the results of running the quickstart dense example program. This arr"| __truncated__
#  $ name                : chr "quickstart_dense"
#  $ type                : chr "dense"
#  $ share_count         : int 1
#  $ public_share        : logi TRUE
#  $ namespace_subscribed: logi FALSE
#  $ tiledb_uri          : chr "tiledb://TileDB-Inc/quickstart_dense"
#  $ tags                : list()
#  $ license_id          : chr "MIT"
#  $ license_text        : chr "MIT License Copyright (c) <year> <copyright holders>\n\nPermission is hereby granted, free of charge, to any pe"| __truncated__

ApiClient defaultClient = Configuration.getDefaultApiClient();
defaultClient.setBasePath("https://api.tiledb.com/v1");

// Configure HTTP basic authorization: BasicAuth
HttpBasicAuth BasicAuth = (HttpBasicAuth) defaultClient.getAuthentication("BasicAuth");
BasicAuth.setUsername("xxx");
BasicAuth.setPassword("yyy");

ArrayApi apiInstance = new ArrayApi(defaultClient);

String namespace = "TileDB-Inc"; // String | namespace array is in (an organization name or user's username)
String array = "quickstart_sparse"; // String | name/uri of array that is url-encoded
try {
    ArrayInfo result = apiInstance.getArrayMetadata(namespace, array);
    System.out.println(result);
} catch (ApiException e) {
    System.err.println("Exception when calling ArrayApi#getArrayMetadata");
    System.err.println("Status code: " + e.getCode());
    System.err.println("Reason: " + e.getResponseBody());
    System.err.println("Response headers: " + e.getResponseHeaders());
    e.printStackTrace();
}

//this prints:
//class ArrayInfo {
//  id: 69ca9424-2578-44d0-8662-87897d9e2941
//  fileType: null
//  fileProperties: null
//  uri: null
//  namespace: TileDB-Inc
//  size: 903
//  lastAccessed: 2022-06-01T15:01:16Z
//  description: # Quickstart Sparse Array  
//  This is a very simple 2D sparse array, used to quickly demonstrate basic TileDB Cloud functionality.
//  name: quickstart_sparse
//  allowedActions: [read, read_array_info, read_array_schema]
//  pricing: null
//  subscriptions: null
//  logo: null
//  accessCredentialsName: null
//  type: sparse
//  shareCount: 1
//  publicShare: true
//  namespaceSubscribed: false
//  tiledbUri: tiledb://TileDB-Inc/quickstart_sparse
//  tags: [one, three, two]
//  licenseId: MIT
//  licenseText: MIT License Copyright (c) <year> <copyright holders>  
//  Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
//  The above copyright notice and this permission notice (including the next paragraph) shall be included in all copies or substantial portions of the Software.  
//  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
//  readOnly: false
//  isFavorite: null
//}

Great work! In the following we will see how to view useful information directly through the TileDB Cloud console.

Using the TileDB Cloud Console

You can view the information of public arrays by directly navigating to the array's URL on TileDB Cloud, even if you are not signed in (TileDB Cloud provides "static view" of arrays). For example, you can click directly on https://cloud.tiledb.com/arrays/details/TileDB-Inc/quickstart_sparse/overview, which is the URL of the array we used in the previous sections. Once you do so, you will see:

Click on the Schema tab to see the array schema:

Next, let's sign in and see some activity logs. Click on Activity on the left-hand side menu, under Assets:

Here you can see the various tasks that you have performed (like slicing an array), along with other information, such as the time, cost, duration, etc. Clicking on a task provides further information (to be covered in other tutorials).

Finally, you can view your profile information by clicking Profileon the left-hand side menu

Here you can edit your personal information, change your password, etc.

What's Next?

Are interested in diving deeper into TileDB Cloud? Here is what we recommend you to do next.

Quick recipes:

• Create API tokens for faster programmatic login.

• Set up AWS credentials so that you can access and share your arrays through TileDB Cloud.

• Add billing info, so that you can use the service once you run out of credits.

Next tutorials:

• Get a console walkthrough

• Start learning about serverless compute

• Familiarize yourself with the power of task graphs

• Learn to use TileDB Cloud in specific use cases

The need to manage data has existed for decades and there are hundreds of different data management solutions available today. What is TileDB and how does it innovate in such a crowded space?

In this section we explain what we define as "universal data management" and the practical problem it solves. We start with the main motivation behind TileDB.

Motivation

TileDB started as a research project at Intel Labs and MIT in 2014, where we made some observations.

Observations

• Data

  • Most database systems deal with one type of data, mostly tables.

  • Most data out there is not really tabular. Look at satellite imaging, biomedical imaging, video, weather, point cloud, and many others.

  • A lot of organizations maintain a lot of diverse data. For example, hospitals have clinical records, but also genomics, MRI scans, etc. Insurance companies may have weather data coupled with satellite imaging. Telecommunications companies may have customer records, in addition to LiDAR and location data.

• Tools

  • Teams within an organization may be using a variety of programming languages and tools. Some may use SQL, but others may prefer Python or R. Some may wish to perform data analytics, but others may want to run Machine Learning tasks or simply visualize the data.

• Collaboration

  • Collaboration and governance within and across organizations is very challenging when the data (and code) does not live within a centralized database that would ordinarily manage access policies and maintain logs for auditing.

Since there is no database that can tackle the above challenges, organizations often resort to building specialized data solutions in-house. This takes a lot of time, costs a lot of money, and often involves combining a lot of disparate data software which make data management even more challenging.

Having those observations in mind, we asked ourselves a few questions.

Questions

• Data storage

  • Is there a single data structure that can capture all data? In our mind, tables are constrained, whereas key-values, documents and graphs do not seem to be efficient models for data like images, video, weather, point clouds, genomics, and more.

  • Is there a way to abstract all storage in a way that the system can work on any backend (memory, cloud object store, or other)?

• Common layers

  • What are the common components of a "data management" solution, regardless of the application domain (e.g., storage layer, an authentication layer, APIs, access control, logging, etc.)? Could these be common in Genomics, Earth Observation, Time Series, etc?

• Data access

  • Can we abstract all access in a way that the system can efficiently work with any API or tool?

• Collaboration

  • Can we scale access control to any number of users, anywhere in the world and beyond the limits of a data center?

  • Can we share code in a similar way to sharing data? In other words, can we treat code as data?

• Monetization

  • Can we enable users to sell data and code they share with others?

• Scalability

  • Can we take multi-tenancy to the extreme, without being constrained by clusters. Can we scale easily and elastically?

  • Does a scalable "data infrastructure" also imply a scalable "compute infrastructure"? In other words, if we build the former, can we also gain the foundation for the latter?

• Future proofness

  • Is there a way to make the system future proof? That is, can we build it in a way that in the future we can easily extend it with any storage backend, any language API, any data analytics and visualization tool, any new hardware, and practically any technological advancement?

And while contemplating about the answers to these questions, we developed TileDB and introduced the concept of universal data management.

Going Universal

There is a need for rearchitecting data management from the ground up, in a universal way that is flexible and extendible to future user needs and technological advancements. Below we describe the most important aspects of TileDB Cloud that make it a universal data management system.

Universal Storage

We needed a single data format and storage engine that could handle all types of data. We observed that multi-dimensional arrays constitute a great candidate for that. We can prove that any data, from biomedical imaging to genomics to SAR to tables to anything, can be modeled very efficiently with (dense or sparse) multi-dimensional arrays. Also multi-dimensional arrays are the currency of data analytics and machine learning, because a lot of advanced mathematical computations (e.g., using Linear Algebra) are applied to vectors, matrices and tensors -- in other words, multi-dimensional arrays!

• Dense and sparse multi-dimensional array support

• "Columnar" format and compression

• Multi-threading and parallel IO

• Cloud-optimized implementation

• Rapid updates and array slicing

• Data versioning and time traveling

• Arbitrary metadata stored along with the array data

TileDB Cloud relies on TileDB Open Source for data (and code) storage. All code written with TileDB Open Source can be used with TileDB Cloud by changing only a few configuration parameters. This allows the users to test their code locally, and transition to the cloud offering by changing 1-2 lines of code.

Governance

TileDB authentication and access control works as follows:

• The user stores their data in the open-source TileDB Open Source array format on some scalable shared storage backend (e.g., AWS S3).

• The user owns the data, TileDB Cloud does not do any hosting. The user only registers the array with TileDB Cloud, granting authentication keys to TileDB Cloud for accessing the data.

• The user can create organizations, and share data and code with other organizations and users with various access policies. There is no bound on the number of users and organizations one can share data and code with. Users can collaborate with anyone within or beyond their organization.

• When data and code is accessed, TileDB Cloud is responsible for securely checking and enforcing all the appropriate access policies.

There is no need to manage IAM roles, or Apache Sentry/Range setups anymore. TileDB Cloud handles everything transparently.

In addition, TileDB Cloud allows users to make data and code public, attaching descriptions, metadata and arbitrary tags. The data and code can then be discovered and used by any other TileDB Cloud user on the planet.

Logging

All access to arrays and code is logged and can be viewed for auditing purposes. TileDB Cloud allows users to keep track of how their shared or public arrays are being used and gain valuable insights.

Interoperability

All the APIs and integrations (existing and future ones) inherit the authentication, access control and logging functionality we built directly on top of array storage. In other words, modeling all data universally as arrays allowed us to build a single layer for authentication, access control and logging, instead of building custom support for all the data types and APIs/tools used across different applications.

Side Benefits of Universality

Building a universal data management system that can provide extreme multi-tenancy and scale requires building an entire distributed system infrastructure from scratch. Our implementation revealed additional capabilities that proved to be very valuable for scalable data analytics, ease of use, extracting monetary value from data and code and remaining relevant in the rapidly paced data technology space. Therefore, we gradually exposed these capabilities within TileDB Cloud, described below.

Serverless Compute

We outlined the following requirements around accessing arrays registered with TileDB Cloud:

1. Any user on the planet with appropriate access policies should be able to access data at any time.

2. There should be no limit on how many users can simultaneously access an array.

3. The user who accesses the array should not be responsible for spinning up dedicated machines.

4. The user who shares the array should not be responsible for spinning up dedicated machines.

The architecture we built to meet these requirements resulted in the following:

1. Totally "serverless" compute from the user's stand point. Any request "just works" without reserving resources in advance.

2. TileDB Cloud uses an elastic compute infrastructure, which automatically expands and shrinks based on user demand.

3. The user is charged in a pay-as-you-go fashion, and only for compute and data egress they consume.

4. The compute is sent to the data, respecting geographical cloud storage regions to eliminate egress cloud provider costs and maximize performance.

Users wanted to do more than just slicing arrays. For example, they wished to run advanced SQL queries and user-defined functions (UDFs), i.e., arbitrary code in Python, R or other language, potentially using external libraries and integrations, and manipulating the data efficiently, securely and inexpensively. In the most general scenario, users wanted to create task graphs, i.e., task workflows that can implement sophisticated distributed algorithms to take advantage of the computational power and ease of use of TileDB Cloud. But this functionality could readily be provided by the infrastructure we built. Therefore, we optimized it and exposed it.

We also took this one step further. SQL queries and UDFs are runnable code. But it is also shareable data. We stored UDFs as TileDB arrays (recall that arrays can model any data) and we unlocked all the TileDB Cloud capabilities even for UDFs, such as sharing, logging and exploration of public code.

Jupyter Notebooks

A lot of data scientists find it convenient to use Jupyter notebooks for writing code and performing exploratory analysis. TileDB Cloud allows launching JupyterLab instances within the online console. The instances come with prepackaged images that include useful libraries, but the user can also install any library inside the Jupyter environment. TileDB Cloud handles this within its distributed infrastructure, without requiring the user to manually deploy servers.

Jupyter notebooks have become a standard tool for scientific analysis and reproducibility. Therefore, TileDB Cloud allows users to share notebooks in the same manner as arrays. Users can also make notebooks public, or explore notebooks that others have shared with the world.

Marketplace

This is convenient for several reasons:

1. Sellers do not need to ship potentially huge quantities of data to buyers.

2. Sellers do not need to build their own infrastructure to serve data and code, as well as perform all billing and accounting.

3. Buyers can perform exploration and analysis on data from multiple vendors in a single platform.

4. Buyers do not need to download and host the potentially massive quantities they are purchasing.

5. A pay-as-you-go model is an alternative, more flexible model to the standard annual license model, which may be more economical for both buyers (due to scale) and sellers (due to paying only for what they use).

Future Proofness

So far we have described that TileDB Cloud is universal in the following respects:

1. Data: TileDB can manage any data type, and hence can support any future data type.

2. Storage backends: TileDB abstracts the backend layer and thus can easily add support for new backends (on the cloud, in memory, or other).

3. APIs and tools: TileDB is all about extreme interoperability and it is designed to easily add support for any new popular language and tool.

4. Deployment: TileDB is cloud and data center agnostic and therefore can be deployed anywhere.

5. Hardware: TileDB is being implemented in a way that can benefit from hardware accelerators, and boost performance in clusters with heterogeneous instances.

6. Algorithms: TileDB allows the development of any arbitrary distributed algorithm (from SQL to Linear Algebra to genomics pipelines), which can easily be shared and improved through collaboration.

To sum up, TileDB Cloud is flexible and can adapt to change throughout its lifetime in an organization's software stack. User requirements and creativity around data processing continually increase. TileDB Cloud remains valuable and relevant by evolving based on user feedback, rather than becoming obsolete.

In this tutorial, you will learn:

1. How to use task graphs and specifically the Delayed API

2. How to scale your computation, significantly boosting performance, all serverless

3. How to eliminate egress costs

We will use public TileDB Cloud array TileDB-Inc/nyctlcyellowtripdata_2019, which stores the data from the NYC yellow taxi dataset for the year of 2019. The original data is in CSV format with collective size of about 7GB, which is converted into a TileDB 1D sparse array with the size being compressed down to ~1GB. The selected sparse dimension is tpep_pickup_datetime, which means that the array supports very fast range slicing (and, therefore, also partitioning) on that column of the dataset.

Running on TileDB Cloud

You can preview this tutorial as TileDB Cloud notebook (no login is needed). You can also easily launch it within the TileDB Cloud UI console, but you will need to sign up / login to do so.

Running on your own client

You can run all the commands of this notebook in your own client. The only changes required are:

1. Install the TileDB Cloud client

2. Log in using the TileDB Cloud client before running any notebook command

This page group provides details on the TileDB Cloud internal mechanics. You can navigate from the menu on the left, or through the following links:

In this tutorial you will learn how to navigate and use the main components of the , namely:

Namespace selector

TileDB Cloud has a versatile namespace selector designed to enhance your experience in managing data and collaborations.

Upon signing up, each user is allocated a dedicated, private primary namespace. This namespace serves as your personal workspace, ensuring your data remains isolated and organized (until explicitly shared with other users or orgs).

In addition to your private namespace, TileDB offers the capability to create or join multiple organization namespaces. These spaces fosters seamless teamwork by allowing users to collaborate on projects, share resources, and collectively manage data.

The namespace selector enables effortless movement between these private and organizational spaces, facilitating a smooth transition as you navigate between different projects and contexts.

Notifications

You can receive notifications for various actions happening in TileDB Cloud. For instance, you can be notified when you're invited to join an organization or when someone shares an asset with you.

Overview

When you log in, the first page you see is Overview. Here you can see a summary of your assets, your current bill and your recent activity.

Launch server

You can easily launch Jupyter notebook server instances within TileDB Cloud.

Assets sidebar

You can catalogue and access a wide-range of asset types in TileDB Cloud. From generic, fundamental assets like arrays, files, notebooks and UDFs to more sophisticated assets used across broader applications verticals, including geospatial analysis, genomics research and machine learning.

This holistic approach ensures that whether you're working with traditional data types or delving into specialized domains, TileDB Cloud is your one-stop solution for streamlined asset management across diverse fields.

Asset browsers

Each asset category has it's dedicated browser where you can also filter and search for specific assets. In the asset browser you can navigate between:

• My tab : Your registered assets

• Shared tab : Assets that are shared with you

• Public tab : Publicly available assets

• Favorites tab : Assets that are marked as favorite

Assets

Assets constitutes data, code and data products that belong to you or an organization you are a member of, as well as data and code shared with you by other users and organizations.

Asset types

The asset categories currently supported by TileDB Cloud are listed in the table below. These assets can be registered and accessed in TileDB Cloud with various methods, described later in an another section.

Overview

You can preview various information from the overview tab of an asset. Rich descriptions, tags, URIs, permissions, versioning information along with some asset specific information.

Preview

The preview tab displays important information relative to the asset contents.

Schema

Explicitly on array assets you can view detailed information regarding the schema of the array.

Metadata

Most of the asset types come with metadata, either inherited by the asset type itself, or defined by the user.

Sharing

Any asset can be shared with explicit permissions via username or email. If the email invited is not a TileDB account already, it will prompt the user for a signup first.

Settings

From the settings tab you can update your asset description and license, assign tags, rename or remove your asset, change the cloud credentials and make your asset publicly accessible.

Actions

Some assets have specific actions associated with them (highlighted by the blue buttons) such as the ability to download the asset, copy it to another namespace, launch a notebook or quickly add a description to the asset.

Adding assets

Adding assets to TileDB Cloud usually consists of two actions:

1. The creation or transformation of an existing or new asset to a multi-dimensional array. This can be done programmatically, via ingestion or straight from TileDB Cloud.

2. The registration of the asset from it's original storage location (usually S3, Azure or other cloud storage provider) to TileDB Cloud. Again this can happen either programmatically, via ingestion or from TileDB Cloud.

Activity

You can view all the logged activity of assets you have access to.

Profile

You can view and edit your primary and organization profile, add cloud credentials, default storage paths, API tokens, and manage your billing.

On organizations you can also manage your team members.

TileDB Cloud allows you to build arbitrary (directed acyclic) task graphs to combine any number of different tasks into one workflow. You can combine serverless UDFs, SQL and array access along with even local execution of any function.

The task graph is currently driven by the client. The client can be in a hosted notebook, your local laptop, or even a serverless UDF itself. The client manages the graph, and dispatches the execution of severless jobs or local functions.

The local driver uses the Python ThreadPoolExecutor by default to drive the tasks. The default number of workers is 4 * #cores on the client machine. Python allows multiple serverless tasks to run as they use asynchronous HTTP requests. Serverless tasks will scale elastically. As you request more tasks to be run, TileDB Cloud launches more resources to accommodate the tasks.

Array access refers to any read or write operation to an array registered with TileDB Cloud and referenced via its tiledb:// URI. Each array access is directed to a particular Kubernetes cluster in a specific cloud region as explained in Automatic Redirection. Then this request is assigned to a REST worker pod in an elastic and load balanced manner. That worker uses 16 CPU cores and sets the total result buffer size for TileDB Open Source to 2GB RAM.

The REST worker performs authentication (looking up the system state), logs all activity, manages billing and monetization, and enforces the access policies. Most importantly, each REST worker is totally stateless, and requires no synchronization or locking, allowing TileDB Cloud to scale very gracefully and quickly recover from failure via retry policies.

One of the most powerful feature of TileDB Cloud is that it allows users to share arrays, UDFs and notebooks at extreme scale, with anyone on the planet, and with diverse polices (e.g., read, write, read/write). There are no restrictions on the number of users data and code can be shared with.

TileDB Cloud also enables users to create organizations, in order to better manage access to their assets and manage billing. You can create any number of organizations.

TileDB Cloud maintains a global system state using MariaDB, recording all information required to know which assets belong to which users and who has access to the various assets.

TileDB Cloud logs everything: the task types, the users that initiated them, duration, cost, etc. All this information gets logged by the REST workers into the persistent and encrypted MariaDB instance. The activity can then be browsed on the TileDB Cloud UI console or retrieved programmatically using the TileDB Cloud client. Six months of logs are made available for instant retrieval. Contact us if you need longer retention or ways to perform offline audits of historical logs for your organization.

By default, sessions on TileDB Cloud will timeout after 8 hours. SSO session timeout is controlled by organizational policies.

TileDB Cloud allows you to perform any SQL query on your TileDB arrays in a serverless manner. No need to spin up or tear down any computational resources. You just need to have the TileDB Cloud client installed (see Installation). You get charged only for the time it took to run the SQL operation.

TileDB Cloud receives your SQL query and executes it on a stateless REST worker that runs a warm MariaDB instance using the MyTile storage engine. The results of the SQL query can be returned directly to you (when using TileDB-Cloud-Py version 0.4.0 or newer) or they can be written back to an S3 array of your choice (either existing or new). Any array access happens on the same REST instance running the SQL query to optimize performance.

When results are returned directly, they are sent to the client in either JSON or Apache Arrow format, and in Python they are converted into a Pandas dataframe. This is most suitable for small results, such as aggregations or limit queries.

Writing results to an S3 array is necessary to allow processing of SQL queries with large results, without overwhelming the user (who may be on their laptop). The user can always open the created TileDB array and fetch any desired data afterwards.

Each TileDB Cloud REST worker running a SQL query uses 16 CPUs and has a limit of 2GB RAM. Therefore, you must consider "sharding" a SQL query so that each result fits in 2GB of memory (see Task Graphs). In the future, TileDB Cloud will offer flexibility in choosing the types of resources to run SQL on.

TileDB Cloud allows you to run any lambda-like user-defined function (UDF). More specifically, you write the code on your laptop using the TileDB Cloud client (see Installation), your function gets shipped and executed on stateless TileDB Cloud REST workers. You get charged only for the time it took to run the function and the amount of data that got returned to your laptop. You do not need to worry about launching or managing any computational resources.

There are two types of supported UDFs:

• Generic: These can include any code.

• Array UDFs: These are UDFs that are applied to slices of one or more arrays.

Running UDFs is particularly useful if you want to perform reductions (such as a sum or an average), since the amount of data returned is very small regardless of how much data the UDF processes.

TileDB Cloud runs your UDF in a separate dedicated container for security. Any array access is executed in parallel on the same REST worker but separate containers, and the results are sent to the UDF container using zero-copy techniques for performance.

We offer Python and R UDF images based on the following versions:

In the default environment that the UDF runs, we include the following Python packages:

In the default environment that the UDF runs, we include the following R packages:

Geospacial image (geo) is based on Python images and include the following packages:

Genomics image (genomics) is based on Python images and include the following packages:

Imaging image (imaging-dev) is based on Python images and include the following packages:

Vector search image (vectorsearch) is based on Python images and includes the following packages:

If you would like additional packages added to the UDF environment, please leave your suggestion on our feedback request board.

Each UDF allows for the following configurations to be used:

In the future, TileDB Cloud will offer more flexibility in choosing the types of resources to run the UDF on.

This page describes the architecture of our TileDB Cloud SaaS offering.

The following figure outlines the TileDB Cloud architecture, which is comprised of the following components:

• Automatic Redirection

• Orchestration

• UI Console

• System State

• REST Workers

• Jupyter Notebooks

We explain each of those components below.

Automatic Redirection

TileDB Cloud maintains compute clusters in multiple cloud regions, geographically distributed across the globe. The reason is that users may store their data in cloud buckets located in different regions, and it is always faster and more economical to send the compute to the data; that eliminates egress costs, reduces latency and increases network speeds. However, users may not know which region the array they are accessing is located.

To facilitate sending the compute to the appropriate region, TileDB Cloud supports automatic redirection using the Cloudflare Workers service. This provides a scalable and serverless way to lookup the region of the array being accessed (maintaining a fast key-value store that is always in sync with the System State) and issue a 302 temporary redirect to the HTTP request. TileDB Open Source and the TileDB Cloud client will honor the redirection and send the the request to the TileDB Cloud service in the proper region (see Orchestration).

Orchestration

In every cloud region, TileDB Cloud maintains a Kubernetes cluster that carries out all tasks, properly autoscaling and load balancing to match capacity with demand based upon several factors. We use the Kubernetes built in metrics and monitoring toolchain to ensure pod memory usage is monitored and we have an accurate picture of the real world workloads at all times.

Currently supported regions:

• us-east-1

• us-west-2

• eu-west-2

• ap-southeast-1

In each region we use a variety of compute EC2 instance types, predominantly from m5, c5 and r5classes.

UI Console

The TileDB Cloud user interface console (https://cloud.tiledb.com) is a web app written in React that uses the REST Workers API across the same procedures and protocols as the clients. Many of the same routes are also used directly from one of the many clients, such a TileDB-Cloud-Py or TileDB-Cloud-R. The console web app autoscales based on the load, but currently it runs only inside the us-east-1cluster.

System State

TileDB Cloud maintains persistent state about user records, arrays, UDFs, billing, activity and more by using an always encrypted MariaDB instance. This instance is maintained in the us-east-1 region. In addition, this state is replicated and synced at all times with a read-only MariaDB instance maintained in every other supported region, in order to reduce latency for the queries executed in those regions.

REST Workers

TileDB Cloud's architecture is centered around a REST API Service. The service is a Go based application which provides all of the base functionality such as user management, authentication and access control, billing and monetization (via integration with Stripe), UDF execution, and serverless SQL orchestration used in TileDB Cloud. The REST Service is deployed in Kubernetes with a stateless design that allows for distributed orchestration and execution without the need for centralized coordination or locking.

The REST Service monitors resource usage and does its own book keeping in order to determine if it can service a request or if it should inform the client to retry later. By allowing the client to manage retries and with the high availability of the REST service architecture. TileDB Cloud is able to gracefully load balance and distribute the work across multiple instances.

The REST service handles the following types of serverless tasks, building upon the TileDB Open Source library:

• Access (read/write)

• SQL

• UDFs

Jupyter Notebooks

TileDB Cloud offers hosted Jupyter notebooks by using Jupyter Docker Stacks for the base conda environments, and Jupyterhub / Zero to Jupyterhub K8S for the runtime environment. The notebooks are spawned inside Kubernetes using kubespawner to offer an isolated environment for each user with their own dedicated and persisted storage.

Connectivity (Firewall) Requirements

TileDB Cloud runs over standard http connectivity, using tcp ports 80 and443. Connection made on port 80 are automatically redirected to https over port 443.

Open ID Connect

TileDB Cloud provides Open ID Connect support that can be used with any Open ID Connect compatible service. TileDB Cloud provide a fixed set of IP address used for the outbound request as part of the Open ID Connect sequence.

eu-west-2

• 13.41.67.254

• 18.134.194.194

• 18.135.61.196

us-west-2

• 35.81.95.218

• 54.185.206.57

• 54.189.31.204

us-east-1

• 52.21.38.106

• 54.87.160.2

• 52.70.6.129

ap-southeast-1

• 13.213.235.67

• 54.255.255.186

• 52.76.199.70

See Corporate SSO with TileDB Cloud SaaS if you are interested in enabling OIDC support for TileDB Cloud SaaS in your own environment.

TileDB Cloud charges in three ways:

• $0.11 / CPU / hour for a task time: This is the time it takes for a task from beginning to completion. Depending on the task and user parameters (see , and ), the number of CPUs used will be different. This price of $0.11 per CPU-hour is based on 1 CPU usage. The time is multiplied by the number of CPUs used in the task. At the end of the month, usage is summed up and rounded to the nearest second.

• $0.14 / GB of data egress: This is the amount of data retrieved from the service to your client, not the data processed by a query. Egress is summed up and rounded to the GB at the end of the month.

• $0.06 / CPU / hour for the notebook server duration: Notebooks are charged for how long the server is running and based on the size of the server. See for the instance types along with the number of CPUs for each type.

In TileDB Cloud you need to register your own data, which you must currently store in your own cloud buckets. TileDB Cloud does not offer storage hosting. You just your already existing cloud-stored arrays (created with ), and TileDB Cloud manages access and computation on those arrays without unnecessary copying and movement of the data.

All access via the web console, excluding notebook usage, is free and no charges are applied. TileDB Cloud bills you monthly.

Example Pricing Scenarios

Billing with Organizations

If you do not belong to any organization, then all charges (array access, UDF, SQL, notebooks) are directed to your personal account. If you belong to one or more organizations, the following rules apply:

1. For SQL, UDFs and notebooks , TileDB Cloud charges the user's first organization (the one the user joined first).

2. For array access, TileDB Cloud determines how the user got access to the array. Suppose the user is called user and belongs to one or more organizations, but the "first" is called org. Here are the possible scenarios:

   1. Array owned by user -> user is charged.

   2. Array owned by org -> org is charged

   3. Array owned by another organization org2, shared with org -> org is charged

   4. Array owned by org2, shared with org AND shared with user directly -> Error unless Default namespace to charge is configured (see below)

   5. Array owned by another user user2 (not an organization) -> user is charged

TileDB Cloud also allows you to specify explicitly who to charge:

2. In your profile Settings, there is a field called Default namespace to charge.

You can create an API token by navigating to API tokensin Settings , as shown below. You can also provide an expiration date for your token upon creation. You can create multiple tokens and revoke them any time.

This page group contains simple recipes for managing your account. You can find the contents below:

TileDB Cloud allows users to monetize their data and code, with its full-fledged marketplace that integrates with .

You can select any of the arrays you own and specify the following:

• $ per CPU per hour: This is a cost that TileDB Cloud will apply on top of the CPU cost that it would otherwise charge the users when accessing that array (see ).

• $ per GB of egress: This is a cost that TileDB Cloud will apply on top of the egress cost that it would otherwise charge the users when accessing that array (see ).

Users accessing an array that comes with extra pricing from its owner get charged for two separate costs, one from TileDB Cloud (see ) and one from the array owner. Sellers can see all their revenue details directly on the Stripe platform. Buyers can see a complete breakdown of their costs in their TileDB Cloud invoices.

This page group contains simple recipes for managing your arrays. You can find the contents below:

You can add your payment method and see your billing information, including your current balance breakdown and all invoices, by navigating to the Billing tab of your Settings :

When you create UDFs, notebooks, dashboards, groups, and ML models, TileDB Cloud stores each asset as a TileDB object (array or group). In order to do so, it requires you to provide a default storage path for storing those assets. You can do so from your profile settings as follows:

In order to be able to create, register, and access arrays through the TileDB Cloud service, you need to set up access credentials. For S3 compatible object stores, TileDB Cloud supports both IAM Roles and Access Credential key pairs. TileDB Cloud securely stores all keys in an encrypted database and never grants your keys to any other user. TileDB Cloud uses your keys in containarized stateless workers, which are under TileDB's full control and inaccessible by any other user's code (e.g., SQL or UDFs).

Note: You can add multiple AWS keys to TileDB Cloud, register different arrays with different keys, select a key to be your default key, and revoke any key at any time.

AWS Access Keys

You can add your AWS keys from the AWS credentials tab of Settings as follows:

AWS Assume Role

With an AWS AssumeRole policy we are solving the very same issue we used keys before: Enable AWS cross-account access, so that a role in one account can access a bucket in a separate account.

When using AWS AssumeRole, temporary keys are created through the Service Token Service (STS), and used from the deployment party (TileDB Cloud Console). This means that for organisation purposes there is no need to create an AWS IAM User for every user logging into TileDB Cloud Console and generate key pairs. Instead, after a User is authenticated, the AssumeRole functionality enables TileDB Cloud Console to access the bucket on behalf of a User and the credentials used in that case can be reused by multiple Users in the same organisation that need to access the same S3 buckets.

As an example, let 's consider the account (Account A) we are signing up with TileDB Cloud to access bucket(s) in User's AWS account (Account B). For that purpose, Account B has a bucket created. The most common setup is to create an IAM role for TileDB Cloud to use and then allow it to access a specific bucket with an AWS S3 bucket policy. Requests for access to the bucket will only be granted coming from our AWS account with our external ID.

Steps:

In TileDB Cloud Console navigate to Settings then select the tab Cloud Credentials

Click Add credentials, then select ARN Role and click Next and Next in the following step which is just a short description

Select tab Existing Role that presents the Account A ID as well as the External ID

Select tab New Role that proposes the JSON Account B can use to create the role. Please note Account A Account ID as well as the External ID

In Account B, User (or Admin) can create the bucket policy

In Account B User (or Admin) can create the role, using Account A ID and External ID

In Account B User (or Admin) has to attach the policy to the role

Obtain ARN for the new role

Then press Next in TileDB Console Add Credentials modal dialog and enter a name for the new AssumeRole Credentials and the ARN obtained in previous step

Test the connection

Example configurations have been detailed below:

AWS IAM Role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "<PROVIDED IN UI>"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "<PROVIDED IN UI>"
                }
            }
        }
    ]
}

Note: Both the AWS Principal and External ID will be provided when attempting to register the ARN role

AWS S3 Bucket Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Action": ["s3:ListBucket", "s3:ListBucketMultipartUploads"],
            "Principal": {
                "AWS": "<IAM ROLE ARN>"
            },
            "Resource": ["arn:aws:s3:::<BUCKET NAME>"]
        },
        {
            "Sid": "AllObjectActions",
            "Effect": "Allow",
            "Action": ["s3:GetObject",
                       "s3:PutObject",
                       "s3:AbortMultipartUpload",
                       "s3:ListMultipartUploadParts",
                       "s3:DeleteObject"],
            "Principal": {
                "AWS": "<IAM ROLE ARN>"
            },
            "Resource": ["arn:aws:s3:::<BUCKET NAME>/*"]
        }
    ]
}

Setup KMS for the target bucket

It is possible that encryption is needed for the target bucket

To enable KMS usage for the target bucket, it is needed to edit the policy for the KMS key and add a statement that gives access to the role used previously

Example configuration is provided below

    "Version": "2012-10-17",
    "Statement": [
        ...
        {
            "Sid": "Example KMS Role",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<IAM ROLE ARN>"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        ...
    ]
}

TileDB Cloud enables the user to launch Jupyter notebooks within the UI console. It spins up Jupyter notebook instances in the Kubernetes cluster in us-east-1. The user can install any extra packages in the notebook. The notebook server environment is destroyed on shutdown. Any extra packages installed will not persist across server instances.

Every user gets a 2GB persistent storage in an EBS volume (also in us-east-1). This is mounted as the home directory in the notebook server. All contents in the home directory will persist across server restarts. The user does not get charged for storage!

Currently, TileDB offers two notebook server sizes:

As explained in the Pricing and Billing section, notebooks are charged based on the size of the notebook server and duration it is run for.

TileDB Cloud offers three notebook images, with the following installed packages:

• Basic Data Science:tiledb, libtiledb-sql-py, plotly, ipywidgets, graphviz, pandas, pydot, trimesh, numpy, chardet, numba, tiledb-r, voila, opencv, tiledb-cloud, pybabylonjs, envbash, tiledb-ml

• Genomics: Everything in the Basic Data Science notebook plus:snakemake, tiledb-vcf, htslib, bcftools, pybedtools

• Geospatial: Everything in the Basic Data Science notebook plus:cartopy, datashader, descartes, folium, geos, geotiff, holoviews, imagemagick, laszip, libnetcdf, proj, shapely, scikit-build, proj, gdal, rasterio, mb-system, pdal, fiona, geopandas, scikit-mobility, xarray, tiledb-segy, capella-tools

You can change any information in your profile from the Profile tab in your Settings:

You can register a TileDB array that is already created in a cloud bucket under your control, either via the , or in the TileDB Cloud console as follows. Make sure that you have already , which are needed to access the array you are registering, as those will have to be selected in the drop down list in the pop up, after you click on the "add array" button in Assets -> Arrays. You can also choose who will be the owner of the array (you or an organization you belong to). Your array will be accessible by tiledb://<namespace>/<array-name> after it is registered, where namespace is the owner of the array and <array-name> is the name of the array that you can select during this registration process in the pop up.

To create an array you will need to use . After you create the array, if you would like to make it "visible" to TileDB Cloud, you need to in a subsequent step. However, TileDB Open Source allows you to create and register the array in a single step, with a very simple change in the way you would otherwise create the array:

• Instead of using <array-uri> as you would typically in TileDB Open Source, you must use tiledb://<username>/<array-uri>. For example, if you wish to create an array at s3://my_bucket/my_array, you need to set the array URI to tiledb://my_username/s3://my_bucket/my_array and TileDB Open Source will instruct TileDB Cloud to automatically register the array as tiledb://my_username/my_array.

You can also make an array public, effectively sharing it with every user on TileDB Cloud. This can make your array discoverable by other users that wish to , or you can discover useful datasets that others wish to share with the world.

To make an array public, you just need to navigate to the Settings of that array and click on Make public as shown below. Similarly, you can always switch the array back to private mode at any time.

You can access the array details programmatically via the TileDB Cloud API (e.g., see ), or on the console as follows. You can see the array overview, which includes the physical path of the array, permissions, a description, etc. Moreover, you can see the array schema, activity logs, metadata, sharing properties, monetization and settings.

You can explore public notebooks, adding a variety of filters, from the Explore page.

This page group contains simple recipes for managing your notebooks. You can find the contents below:

TileDB Cloud support uploading notebooks from git repositories. Its common that you might have notebooks stored in source control such as github or gitlab and wish to automatically upload them to TileDB Cloud for usage.

Github Actions

TileDB provides public github actions that let you upload a notebook.

Example usage includes:

   - uses: TileDB-Inc/github-actions/upload-notebooks@main
      with:
        notebooks_local: >-
          path/to/notebook-1.ipynb
          path/to/notebook-2.ipynb
          path/to/notebook-3.ipynb
        notebooks_remote: >-
          tiledb://namespace/s3://cloud-path/notebook-1
          tiledb://namespace/s3://cloud-path/notebook-2
          tiledb://namespace/s3://cloud-path/notebook-3
        TILEDB_CLOUD_TOKEN: ${{ secrets.TILEDB_CLOUD_TOKEN }}
        TILEDB_CLOUD_STORAGE_CREDENTIAL_NAME: <name-of-credential-saved-on-tiledb-cloud>

Gitlab CI

Gitlab CI doesn't offer a marketplace or public template support currently. Instead find the bellow example for setting up a gitlab ci setup.

Secrets

An API Token is required to access TileDB Cloud. Its recommended to set this as a secret in gitlab. This should be set as an environmental variable for TILEDB_REST_TOKEN.

upload_notebook:
  stage: upload
  image: python:3.9
  variables:
    TILEDB_STORAGE_CREDENTIAL_NAME: "example-credential-name"
    TILEDB_NAMESPACE: "namespace-to-upload-to"
  script:
    # Install tiledb-cloud
    - pip install tiledb-cloud
    # Define notebook to upload
    - export NOTEBOOK_FILE="example.ipynb"
    # Invoke tiledb-cloud's upload functionality
    - python -c "import tiledb, os; tiledb.cloud.upload_notebook_from_file(
        ipynb_file_name=${NOTEBOOK_FILE},
        namespace=${TILEDB_NAMESPACE},
        array_name=os.path.basename(${NOTEBOOK_FILE}),
        on_exists=OnExists.OVERWRITE,
    )"

You can also make a notebook public, effectively sharing it with every user on TileDB Cloud. This can make your notebook discoverable by other users that wish to explore public notebooks, or you can discover useful code that others wish to share with the world.

To make a notebook public, you just need to navigate to the Settings of that notebook and click on Make public as shown below. Similarly, you can always switch the notebook back to private mode at any time.

You can share a registered notebook with any other user on TileDB Cloud. To share a notebook, find it on Assets -> Notebooks and either click on the sharing button located on the right end of the notebook card in the list, or click on the notebook card and navigate to the Sharing tab. The added member will appear in the notebook members list, where you will be able to change the access policy or revoke access from the user. Users get notified by email when someone shares a notebook with them.

To create a Jupyter notebook, you first need to navigate to the notebooks tab from the sidebar under your assets and then click the "plus" icon. That will prompt a dialog asking you to specify the notebook name, the path of the cloud storage space where the physical notebook will live and your cloud credentials that will allow TileDB Cloud to access that storage space. Once you create the notebook, you can launch it, edit and save it as you would otherwise do for any Jupyter notebook.

Alternatively you can create a notebook from the launch a JupyterLab notebook instance. Once you are in the Jupyter notebook environment, navigate to File -> New -> TileDB notebook.

You can deregister a notebook by navigating to its Settings and clicking on Deregister Notebook.

You can explore public arrays, adding a variety of filters, from the Explore page.

TileDB Cloud notebook servers include a persistent home directory for you to install custom packages.

TileDB Cloud supports pip, conda , cran, and other methods of installing packages. When installing a package, you should install it in your home directory if you want it persisted after reboots.

Pip

For Python, you can use pip to install packages into your home directory with the --user option.

pip install --user <package>

Conda

Both conda and mamba are available in the notebook environment to install any available packages. Creating a custom environment in your home directory will allow you to install and persist any packages.

mamba create -n my_env 
mamba activate my_env
mamba install -c conda-forge <package>

CRAN

R packages can be installed from CRAN by setting the `lib location to the home directory for persistence.

install.packages("package_name", lib="~/R")

You can create, register and run UDFs only programmatically. See Serverless UDFs and Serverless Array UDFs for more information.

Once you create and register a UDF using the TileDB Cloud client, you will be able to see the UDF on the UDFs page in the left menu. After selecting a UDF from the list, you can see its description and basic information, preview its code, share the UDF with others and change its settings.

You can share a registered UDF with any other user on TileDB Cloud. To share an array, find it on Assets -> UDFs and either click on the sharing button located on the right end of the UDF card in the list, or click on the UDF card and navigate to the Sharing tab. The added member will appear in the UDF members list, where you will be able to change the access policy or revoke access from the user. Users get notified by email when someone shares a UDF with them.

To make a UDF public, you just need to navigate to the Settings of that UDF and click on Make public as shown below. Similarly, you can always switch the UDF back to private mode at any time.

This page group contains simple recipes for managing your UDFs. You can find the contents below:

You can explore public UDFs, adding a variety of filters, from the Explore page.

Jupyter notebooks are a common way to run Python code for data science or data exploration. You can find more information about how TileDB Cloud manages JupyterLab instances here.

You can launch a JupyterLab instance by clicking on Launch Notebook from the left menu, select and an image and the server size, and click on the Start button. After about 30 seconds, you are all set to start writing some code. To shut down the notebook server, simply click on the Shut down button (it also takes a few seconds).

TileDB Cloud allows you to extract monetary value from your arrays, by setting a dollar amount for egress (per GB of data read by another user) and CPU (per hour of compute spent when another user reads your array).

Before you start monetizing your arrays, you need to create an account with Stripe and connect it with your TileDB Cloud account:

Now you are ready to add pricing to your array from the array's Pricing tag:

When you under one of your organizations, TileDB Cloud stores the UDF code as a 1D dense array. In order to do so, it requires you to provide a default storage path for storing that UDF array. You can do so from your organization's profile settings as follows:

You can create a new organization by clicking on Organizations in the left menu and then on the add organization button. Your new organization will appear on the screen. You can click on it to view its details, add a description, see its settings, etc.

Similarly to setting up AWS credentials for your account, you need to set up AWS keys to grant access to TileDB Cloud for the arrays that belong to your organizations. Recall that, when you register an array, you need to specify the owner of the array, which can be one of your organizations. If that is the case, you need to provide the corresponding AWS keys that you set up for that organization.

To add AWS keys to an organization, click on Organizations in the left menu, then select the organization you are interested in, then click on AWS credentials in the left side menu, and then on the add button. After entering the key information, the new credentials will appear in the list, where you will be able to edit or revoke them, or select the default key.

To enjoy access to TileDB arrays registered with TileDB Cloud, you do not need an extra client. You can continue to use your favorite TileDB Open Source API or integration, tweak a couple of parameters and you are all set. See Array Access for details.

If you wish to enjoy the TileDB Cloud serverless capabilities and perform TileDB Cloud console actions (such as viewing tasks and array descriptions, or sharing data and code) programmatically, you need to add one of our cloud clients: TileDB-Cloud-Py, TileDB-Cloud-R, TileDB-Cloud-Java. You can install the latest releases as follows:

pip install tiledb-cloud

# For latest released version please see
# https://github.com/TileDB-Inc/TileDB-Cloud-R/releases
remotes::install_github('TileDB-Inc/TileDB-Cloud-R@v0.0.9')

// Maven
//Include this in your Maven project:
<dependency>
  <groupId>io.tiledb</groupId>
  <artifactId>tiledb-cloud-java</artifactId>
  <version>X.X.X</version>
</dependency>

// Or build from source
$ git clone https://github.com/TileDB-Inc/TileDB-Cloud-Java.git
$ cd TileDB-Cloud-Java
$ ./gradlew assemble

The latest development version of TileDB-Cloud-Py can be installed directly from Github:

pip install -e git+"ssh://git@github.com/TileDB-Inc/TileDB-Cloud-Py#egg=tiledb-cloud"

remotes::install_github('TileDB-Inc/TileDB-Cloud-R')

Running code samples and Jupyter notebooks locally

While the preferred method of running code samples and notebooks in this section is directly within TileDB Cloud (as all dependencies are installed for you), you can run most of the code samples and notebooks in this section locally. To run these code samples and notebooks locally, install the following dependencies:

conda install -c conda-forge -c anaconda ipykernel jupyterlab graphviz
pip install tiledb-cloud[all]

conda install -c conda-forge -c anaconda ipykernel jupyterlab graphviz
pip install tiledb-cloud\[all\]

When it comes to scalable analytics, we observed the following challenges:

• Spinning up and monitoring clusters on the cloud is cumbersome and can get expensive.

• Users frequently do not know how many machines to provision in a cluster for a given workload. This results in either under provisioning that impacts performance, or over provisioning that leads to wasted cost due to idle compute.

• When users slice array data from TileDB Cloud only to further process it in their own compute environment, (1) they get charged for egress, and (2) the performance is impacted by the extra network transmission cost that occurs between the TileDB Cloud machines and their own machines.

TileDB Cloud allows users to access and compute on arrays in a serverless manner from the user's standpoint, i.e., without thinking about provisioning for machines, and paying for idle compute or unnecessary egress. TileDB Cloud automatically parallelizes all tasks across thousands of machines and monitors their progress.

TileDB Cloud supports the following tasks:

• , e.g., basic ingestion and slicing.

• , from simple selections and filters, to aggregate queries and joins.

• , i.e., arbitrary code in Python, R or other languages.

Users can submit numerous such tasks concurrently and TileDB Cloud will process all of them in parallel, elastically expanding and shrinking its computational resources on demand without supervision by the user. That is, TileDB Cloud provides extreme multi-tenancy by default.

Serverless SQL and array UDFs (i.e., UDFs that are specifically applied to one or more TileDB arrays) have the additional benefit that they can minimize egress by reducing the returned results size, which is true especially in aggregation queries.

Any distributed algorithm can be modeled as a directed graph, where the nodes represent atomic tasks and the edges represent tasks dependencies (i.e., a task cannot begin its execution before all the tasks from the incoming edges have completed their execution). TileDB Cloud supports such task graphs, which can be programmatically created by the user and submitted to the platform. TileDB Cloud is responsible for parallelizing all tasks while respecting the dependencies, and for monitoring all progress. Task graphs are a powerful tool for creating any sophisticated algorithm and scale it on TileDB Cloud.

TileDB Cloud also provides automation for spinning up JupyterLab instances, so that users can run Jupyter notebooks without having to manually set up servers and deploy JupyterLab. This makes it very easy for user to kickstart their data analysis on TileDB Cloud.

Finally, TileDB Cloud treats UDFs and notebooks as data and thus allows users to share runnable code just as easily as they share data. This makes TileDB Cloud a powerful platform for collaboration and reproducibility of scientific results.

The TileDB Cloud Python API provides tools to access, manage, and share notebooks stored in the TileDB Cloud service.

Before accessing notebooks, you’ll need to log in to TileDB Cloud:

Accessing notebook contents

Uploading notebooks

The TileDB Cloud Python API provides two ways to upload notebooks: you can provide a filename to upload, or you can provide the notebook’s contents as a string.

Downloading notebooks

Likewise, notebooks can be downloaded and either saved as a file or kept as a string in memory:

Sharing notebooks

Notebooks are stored as TileDB arrays, so sharing can be managed just like any other array.

Listing users with access to a notebook

Use array.list_shared_with to see who has what kind of access to a notebook:

Sharing with another user

To share a notebook with another user, share the array. In most cases, you will want to provide a user or organization with read or write access to the array:

Revoking access to a notebook

Use unshare_array to revoke access (“unshare”) a notebook from a user or namespace:

Making a notebook public

To make a notebook public, share it with the special public namespace:

To make a public notebook private again, revoke access from the public namespace:

TileDB Cloud comes with two user-facing components:

• Console: This is the TileDB Cloud UI you can use by signing up and logging into https://cloud.tiledb.com. On the console you can manage your arrays, see billing information, spin up a Jupyter notebook, etc. The Console Walkthrough tutorial is a good starting point for getting familiar with the TileDB Cloud console.

• TileDB Cloud client: This is a library for programmatic API access of the various features of TileDB Cloud. You can use the client API to perform pretty much every action you can perform on the console (except for signing up and running Jupyter notebooks). You need to have a TileDB Cloud account to use the client, since it requires you to log in with your username/password or an API token.

For reads, writes, embedded SQL, any integration, and any API, you can use TileDB Open Source with only two changes:

• Set the TileDB configuration parameters rest.username and rest.passwordwith your TileDB Cloud username and password, or alternatively rest.tokenwith the API token you created.

• Every array registered with TileDB Cloud must be accessed using a URI of the form tiledb://<namespace>/<array-name>, where<namespace> is the user or organization who owns the array and <array-name> is the array name set by the owner upon array registration. This URI is displayed on the console when viewing the array details.

Here are some Python/R/Java examples, although the above changes will work with any TileDB API or integration:

import tiledb, tiledb.sql
import pandas

# Create the configuration parameters
config = tiledb.Config()
config["rest.username"] = "xxx"
config["rest.password"] = "yyy"
# or, more preferably, config["rest.token"] = "ttt"

# This is the array URI format in TileDB Cloud
array_name = "tiledb://TileDB-Inc/quickstart_sparse"

# Write code exactly as in TileDB Developer
with tiledb.open(array_name, 'r', ctx=tiledb.Ctx(config)) as A:
    print (A.df[:])

# A helper function tiledb.cloud.Ctx() exists to create a context
# automatically based on a previous call to tiledb.cloud.login()
with tiledb.open(array_name, 'r', ctx=tiledb.cloud.Ctx()) as A:
    print (A.df[:])
    
# Using embedded SQL, you need to pass the username/password or token
# as config parameters in `init_command`
db = tiledb.sql.connect(db="test",
        init_command="set mytile_tiledb_config='rest.username=xxx,rest.password=yyy'")
pandas.read_sql(sql="select * from `tiledb://TileDB-Inc/quickstart_sparse`", con=db)

library(tiledb)

# Create the configuration parameters
config <- tiledb_config()
config["rest.username"] <- "xxx"
config["rest.password"] <- "yyy"
# or, more preferably, config["rest.token"] <- "ttt"
ctx <- tiledb_ctx(config)

# This is the array URI format in TileDB Cloud
array_name <- "tiledb://TileDB-Inc/quickstart_sparse"

# Write code exactly as in TileDB Developer
arr <- tiledb_array(array_name, query_type="READ", ctx=ctx, return_as='data.frame')
df <- arr[]
str(df)

//set the config
Config config = new Config();
config.set("rest.username", "xxx");
config.set("rest.password", "yyy");
Context ctx = new Context(config);

//open the array
Array array = new Array(ctx, "tiledb://TileDB-Inc/quickstart_sparse");

//print the array schema
System.out.println(array.getSchema());

//this will print:
//ArraySchema<TILEDB_SPARSE io.tiledb.java.api.Domain@51462713 Attr<a,TILEDB_UINT32,1>>

You can create an array inside or outside TileDB Cloud. The benefit of creating an array with TileDB Cloud is that it will be logged for auditing purposes. Moreover, it will be registered automatically with your account upon creation.

Create New Arrays

To instruct TileDB Open Source that you are creating an array through the TileDB Cloud service, you just need a single change:

• Instead of using <array-uri> as you would typically in TileDB Open Source, you must use tiledb://<username>/<array-uri>. For example, if you wish to create an array at s3://my_bucket/my_array, you need to set the array URI to tiledb://my_username/s3://my_bucket/my_array.

import tiledb
import numpy as np

#####################
# Define the Schema #
#####################
# The array will be 4x4 with dimensions "rows" and "cols", with domain [1,4].
dom = tiledb.Domain(
    tiledb.Dim(name="rows", domain=(1, 4), tile=4, dtype=np.int32),
    tiledb.Dim(name="cols", domain=(1, 4), tile=4, dtype=np.int32),
)

# The array will be sparse with a single attribute "a" so each (i,j) cell can store an integer.
schema = tiledb.ArraySchema(
    domain=dom, sparse=True, attrs=[tiledb.Attr(name="a", dtype=np.int32)]
)

####################
# Create the Array #
####################
array_uri = "tiledb://my_username/s3://my_bucket/my_array"
# Create the (empty) array on disk.
tiledb.SparseArray.create(array_uri, schema)

library(tiledb)

#####################
# Define the Schema #
#####################
dom <- tiledb_domain(dims = c(
  tiledb_dim("rows", c(1L, 4L), type = "INT32"),
  tiledb_dim("cols", c(1L, 4L), type = "INT32")
))
attrs <- c(
  tiledb_attr("a",  type = "INT32")
)
schema <- tiledb_array_schema(domain=dom, sparse=TRUE, attrs=attrs)

####################
# Create the Array #
####################
array_uri <- "tiledb://my_username/s3://my_bucket/my_array"
# Create the (empty) array on disk.
tiledb_array_create(array_uri, schema)