Data Model
TileDB-VCF uses 3D sparse arrays to store genomic variant data. This section describes the technical implementation details about the underlying model, including array schemas, dimensions, tiling order, attributes, and metadata. It is recommended that you read the following section to fully understand the mapping between the TileDB-VCF data and TileDB arrays and groups:
A TileDB-VCF dataset is composed of a group of two separate TileDB arrays:
- a 3D sparse array for the actual genomic variants and associated fields/attributes
- a 1D sparse array for the metadata stored in each single-sample VCF header
Parameter | Value |
Array type | Sparse |
Rank | 3D |
Cell order | Row-major |
Tile order | Row-major |
The dimensions in the schema are:
Dimension Name | TileDB Datatype | Corresponding VCF Field |
contig | TILEDB_STRING_ASCII | CHR |
start_pos | uint32_t | VCF POS plus TileDB anchors |
sample | TILEDB_STRING_ASCII | Sample name |
The coordinates of the 3D array are
contig
along the first dimension, chromosomal location of the variants start position along the second dimension, and sample
names along the third dimension.For each field in a single-sample VCF record there is a corresponding attribute in the schema.
Attribute Name | TileDB Datatype | Description |
end_pos | uint32_t | VCF END position of VCF records |
qual | float | VCF QUAL field |
alleles | var<char> | CSV list of REF and ALT VCF fields |
id | var<char> | VCF ID field |
filter_ids | var<int32_t> | Vector of integer IDs of entries in the FILTER VCF field |
real_start_pos | uint32_t | VCF POS (no anchors) |
info | var<uint8_t> | Byte blob containing any INFO fields that are not stored as explicit attributes |
fmt | var<uint8_t> | Byte blob containing any FMT fields that are not stored as explicit attributes |
info_* | var<uint8_t> | One or more attributes storing specific VCF INFO fields, e.g. info_DP , info_MQ , etc. |
fmt_* | var<uint8_t> | One or more attributes storing specific VCF FORMAT fields, e.g. fmt_GT , fmt_MIN_DP , etc. |
info_TILEDB_IAF | var<float> | Computed allele frequency |
The
info_*
and fmt_*
attributes allow individual INFO
or FMT
VCF fields to be extracted into explicit array attributes. This can be beneficial if your queries frequently access only a subset of the INFO
or FMT
fields, as no unrelated data then needs to be fetched from storage.The choice of which fields to extract as explicit array attributes is user-configurable during array creation.
Any extra info or format fields not extracted as explicit array attributes are stored in the byte blob attributes,
info
and fmt
. anchor_gap
Anchor gap valueextra_attributes
List ofINFO
orFMT
field names that are stored as explicit array attributesversion
Array schema version
These metadata values are updated during array creation, and are used during the export phase. The metadata is stored as "array metadata" in the sparse
data
array.When ingesting samples, the sample header must be identical for all samples with respect to the contig mappings. That means all samples must have the exact same set of contigs listed in the VCF header. This requirement will be relaxed in future versions.
The
vcf_headers
array stores the original text of every ingested VCF header in order to:- 1.ensure the original VCF file can be fully recovered for any given sample
- 2.reconstruct an
htslib
header instance when reading from the dataset, which is used for operations such as mapping a filter ID back to the filter string, etc.
Parameter | Value |
Array type | Sparse |
Rank | 1D |
Cell order | Row-major |
Tile order | Row-major |
Dimension Name | TileDB Datatype | Description |
sample | TILEDB_STRING_ASCII | Sample name |
Attribute Name | TileDB Datatype | Description |
header | var<char> | Original text of the VCF header |
To summarize, we've described three main entities:
- The variant data array (3D sparse)
- The general metadata, stored in the variant data array as metadata
- The VCF header array (1D sparse)
All together we term this a "TileDB-VCF dataset." Expressed as a directory hierarchy, a TileDB-VCF dataset has the following structure:
<dataset_uri>/
|_ __tiledb_group.tdb
|_ data/
|_ __array_schema.tdb
|_ __meta/
|_ <general-metadata-here>
... <other array directories/fragments and files>
|_ vcf_headers/
|_ __array_schema.tdb
... <other array directories/fragments and files>
The root of the dataset,
<dataset_uri>
is a TileDB group. The data
member is the TileDB 3D sparse array storing the variant data. This array stores the general TileDB-VCF metadata as its array metadata in folder data/__meta
. The vcf_headers
member is the TileDB 1D sparse array containing the VCF header data.During array creation, there are several array-related parameters that the user can control. These are:
- Array data tile capacity (default 10,000)
- The "anchor gap" size (default 1,000)
- The list of
INFO
andFMT
fields to store as explicit array attributes (default is none).
Once chosen, these parameters cannot be changed.
During sample ingestion, the user can specify the:
- Sample batch size (default 10)
The above parameters may impact read and write performance, as well as the size of the persisted array. Therefore, some care should be taken to determine good values for these parameters before ingesting a large amount of data into an array.
Last modified 4mo ago