The TileDB GDAL driver enables indexed and efficient access to integral and partial tiles of geospatial data. The Python community typically use the Rasterio library to access GDAL drivers. Here we show how to ingest large raster data into TileDB in parallel using GDAL, Rasterio, xarray and Dask.
To highlight how TileDB works with large dense arrays we will use a dataset from the Sentinel-2 mission. You can either register and download a sample yourself from the Copernicushub or use the requestor pays bucket from the AWS open data program, the latter being preferable if you wish to run your code on AWS using public data. The directory size of the Sentinel-2 image we are using is 788 MB.
You can run the Rasterio code as follows:
docker run -it --rm -u 0 -v /local/path:/data tiledb/tiledb-geospatial /bin/bash
To verify that the Sentinel-2 dataset can be read by our installation of Rasterio,run the following (changing the filename):
rio info --subdatasets S2A_MSIL1C_20190829T163901_N0208_R126_T16TDM_20190829T201831.SAFE/MTD_MSIL1C.xml
Rasterio should successfully print out the metadata about the subdatasets within the the Sentinel-2 dataset as follows: