Task Graphs 101
In this tutorial, you will learn:
- 1.How to use task graphs and specifically the
Delayed
API - 2.How to scale your computation, significantly boosting performance, all serverless
- 3.How to eliminate egress costs
We will use public TileDB Cloud array TileDB-Inc/nyctlcyellowtripdata_2019, which stores the data from the NYC yellow taxi dataset for the year of 2019. The original data is in CSV format with collective size of about 7GB, which is converted into a TileDB 1D sparse array with the size being compressed down to ~1GB. The selected sparse dimension is
tpep_pickup_datetime
, which means that the array supports very fast range slicing (and, therefore, also partitioning) on that column of the dataset.
You can run all the commands of this notebook in your own client. The only changes required are:
Last modified 1yr ago