Task Graphs 101

In this tutorial, you will learn:

1. How to use task graphs and specifically the Delayed API

2. How to scale your computation, significantly boosting performance, all serverless

3. How to eliminate egress costs

We will use public TileDB Cloud array TileDB-Inc/nyctlcyellowtripdata_2019, which stores the data from the NYC yellow taxi dataset for the year of 2019. The original data is in CSV format with collective size of about 7GB, which is converted into a TileDB 1D sparse array with the size being compressed down to ~1GB. The selected sparse dimension is tpep_pickup_datetime, which means that the array supports very fast range slicing (and, therefore, also partitioning) on that column of the dataset.

Running on TileDB Cloud

You can preview this tutorial as TileDB Cloud notebook (no login is needed). You can also easily launch it within the TileDB Cloud UI console, but you will need to sign up / login to do so.

Running on your own client

You can run all the commands of this notebook in your own client. The only changes required are:

1. Install the TileDB Cloud client

2. Log in using the TileDB Cloud client before running any notebook command