AWS S3
Quickstart
If you have already:
saved your AWS credentials using the AWS CLI
aws login
commandinstalled TileDB-Py with
pip install tiledb
ormamba install -c conda-forge tiledb-py
Then the following should work for any bucket in the us-east-1
region:
If the bucket is in another region, then use:
Configuration Overview
This is a simple guide that demonstrates how to use TileDB on AWS S3. After configuring TileDB to work with AWS S3, your TileDB programs will function properly without any API change! Instead of using local file system paths for referencing files (e.g. arrays, groups, VFS files) use must format your URIs to start with s3://
. For instance, if you wish to create (and subsequently write/read) an array on AWS S3, you use URI s3://<your-bucket>/<your-array-name>
for the array name.
AWS Setup
First, we need to set up an AWS account and generate access keys.
Create a new AWS account.
Visit the AWS console and sign in.
On the AWS console, click on the
Services
drop-down menu and selectStorage->S3
. You can create S3 buckets there.On the AWS console, click on the
Services
drop-down menu and selectSecurity, Identity & Compliance->IAM
.Click on
Users
from the left-hand side menu, and then click on theAdd User
button. Provide the email or username of the user you wish to add, select theProgrammatic Access
checkbox and click onNext: Permissions
.Click on the
Attach existing policies directly
button, search for the S3-related policies and add the policy of your choice (e.g., full-access, read-only, etc.). Click onNext
and thenCreate User
.Using TileDB with an existing bucket requires at least the following S3 permissions:
Upon successful creation, the next page will show the user along with two keys:
Access key ID
andSecret access key
. Write down both these keys.
AWS Security Credentials
There are multiple supported ways to access resources on AWS. You can either request access with long-term access credentials (e.g. Access Keys) or temporary ones by utilizing AWS Security Token Service.
Access Keys
Access keys are long-term credentials for an IAM user or the AWS account root user. To be able to access AWS resource this way you need to follow the next steps:
1. Export these keys to your environment from a console:
2. Or, set the following keys in a configuration object (see Configuration):
Session Tokens
TileDB (version 1.8+
) supports authentication with temporary credentials from the AWS Session Token. This method of acquiring temporary credentials is preferred in case you want to maintain permissions solely within your organization
Assume Role
TileDB (version 2.1+
) supports authentication with temporary credentials from the AWS Assume Role. If you prefer to maintain permissions within AWS, this method's base permissions for the temporary credentials will be derived from the policy on a role. You can use them to access AWS resources, that you might not normally have access to. In this case you will need to configure the following:
Using this method, the IAM
user credentials used by your proxy server only requires the ability to call sts:AssumeRole
. There is one extra step of creating a new role and attaching trust policy to it, which is not the case with Session Tokens approach.
Now you are ready to start writing TileDB programs! When creating a TileDB context or a VFS object, you need to set up a configuration object with the following parameters for AWS S3 (supposing that your S3 buckets are on region us-east-
- you can set an arbitrary region).
The above configuration parameters are currently set as shown in TileDB by default. However, we suggest to always check whether the default values are the desired ones for your application.
Physical Organization
So far we explained that TileDB arrays and groups are stored as directories. There is no directory concept on S3 and other similar object stores. However, S3 uses character /
in the object URIs which allows the same conceptual organization as a directory hierarchy in local storage. At a physical level, TileDB stores on S3 all the files it would create locally as objects. For instance, for array s3://bucket/path/to/array
, TileDB creates array schema object s3://bucket/path/to/array/__array_schema.tdb
, fragment metadata object s3://bucket/path/to/array/<fragment>/__fragment_metadata.tdb
, and similarly all the other files/objects. Since there is no notion of a “directory” on S3, nothing special is persisted on S3 for directories, e.g., s3://bucket/path/to/array/<fragment>/
does not exist as an object.
The AWS S3 CLI allows you to sync (i.e., download) the S3 objects having a common URI prefix to local storage, organizing them into a directory hierarchy based on the use of /
in the object URIs. This makes it very easy to clone TileDB arrays or entire groups locally from S3. For instance, given an array my_array
you created and wrote on an S3 bucket my_bucket
, you can clone it locally to an array my_local_array
with the following command from your console:
After downloading an array locally, your TileDB program will function properly by changing the array name from s3://my_bucket/my_array
to my_local_array
, without any other modification.
Performance
TileDB writes the various fragment files as append-only objects using the multi-part upload API of the AWS C++ SDK. In addition to enabling appends, this API renders the TileDB writes to S3 particularly amenable to optimizations via parallelization. Since TileDB updates arrays only by writing (appending to) new files (i.e., it never updates a file in-place), TileDB does not need to download entire objects, update them, and re-upload them to S3. This leads to excellent write performance.
TileDB reads utilize the range GET request API of the AWS SDK, which retrieves only the requested (contiguous) bytes from a file/object, rather than downloading the entire file from the cloud. This results in extremely fast subarray reads, especially because of the array tiling. Recall that a tile (which groups cell values that are stored contiguously in the file) is the atomic unit of IO. The range GET API enables reading each tile from S3 in a single request. Finally, TileDB performs all reads in parallel using multiple threads, which is a tunable configuration parameter.
Advanced
Proxy Server Settings
The AWS backend supports several settings for proxy servers:
It is necessary to override "vfs.s3.proxy_scheme" to `http`
for most proxy setups. TileDB currently 2.0.8 uses the default setting, which will be updated in the next release.
Logging
TileDB uses the AWS C++ SDK and cURL for access to S3. The AWS SDK logging level is a process-global setting. The configuration of the most recently constructed context will set the process state. Log files are written to the process working directory.
TileDB uses the AWS C++ SDK and cURL for access to S3. The AWS SDK logging level is a process-global setting. The configuration of the most recently constructed context will set the process state. Log files are written to the process working directory.
Certificate Paths
There is no universal location for SSL/TLS certificates on Linux. While TileDB searches for the default CA store on several major distributions, other systems or custom certificates may require path specification. to SSL/TLS certificate file to be used by cURL for for S3 HTTPS encryption. These parameters follow cURL conventions: https://curl.haxx.se/docs/manpage.html
For debugging purposes only it is possible to disable SSL/TLS certificate verification:
Last updated