Setting Filters

Array attributes and dimensions accept compressors and other filters.

Creating a Filter List

Compression

The following example shows how to create a filter list with a GZIP compressor and compression level 10.

#include <tiledb/tiledb.h>

// Create context
tiledb_ctx_t* ctx;
tiledb_ctx_alloc(nullptr, &ctx);

// Create compressor as a filter
tiledb_filter_t* filter;
tiledb_filter_alloc(ctx, TILEDB_FILTER_GZIP, &filter);
int level = 10;
tiledb_filter_set_option(ctx, filter, TILEDB_COMPRESSION_LEVEL, &level);

// Create filter list
tiledb_filter_list_t* filter_list;
tiledb_filter_list_alloc(ctx, &filter_list);

// Add compressor to filter list
tiledb_filter_list_add_filter(ctx, filter_list, filter);

// Clean up
tiledb_filter_free(&filter);
tiledb_filter_list_free(&filter_list);
tiledb_ctx_free(&ctx);

Supported compressors:

CompressorDescriptionOption (type)

TILEDB_FILTER_GZIP

GZIP

TILEDB_COMPRESSION_LEVEL (int)

TILEDB_FILTER_ZSTD

Zstandard

TILEDB_COMPRESSION_LEVEL (int)

TILEDB_FILTER_LZ4

LZ4

TILEDB_COMPRESSION_LEVEL (int)

TILEDB_FILTER_RLE

RLE

TILEDB_COMPRESSION_LEVEL (int)

TILEDB_FILTER_BZIP2

BZIP2

TILEDB_COMPRESSION_LEVEL (int)

TILEDB_FILTER_DOUBLE_DELTA

Double Delta

None

Other Filters

TileDB supports some more filters:

#include <tiledb/tiledb.h>

// Create context
tiledb_ctx_t* ctx;
tiledb_ctx_alloc(nullptr, &ctx);

// Create filters
tiledb_filter_t *f1, *f2;
tiledb_filter_alloc(ctx, TILEDB_FILTER_BITWIDTH, &f1);
tiledb_filter_alloc(ctx, TILEDB_FILTER_ZSTD, &f2);

// Create filter list
tiledb_filter_list_t* filter_list;
tiledb_filter_list_alloc(ctx, &filter_list);

// Add filters to filter list
tiledb_filter_list_add_filter(ctx, filter_list, f1);
tiledb_filter_list_add_filter(ctx, filter_list, f2);

// Clean up
tiledb_filter_free(&f1);
tiledb_filter_free(&f2);
tiledb_filter_list_free(&filter_list);
tiledb_ctx_free(&ctx);

Supported filters (beyond compressors):

FilterDescriptionOption

TILEDB_BIT_WIDTH_REDUCTION

Bit width reduction

TILEDB_BIT_WIDTH_MAX_WINDOW (uint32)

TILEDB_BITSHUFFLE

Bit shuffle

None

TILEDB_BYTESHUFFLE

Byte shuffle

None

TILEDB_POSITIVE_DELTA

Positive Delta

TILEDB_POSITIVE_DELTA_MAX_WINDOW (uint32)

TILEDB_FILTER_CHECKSUM_MD5

MD5 checksum filter

None

TILEDB_FILTER_CHECKSUM_SHA256

SHA256 checksum filter

None

Setting the Tile Chunk Size

The default tile chunk size used by TileDB is 64KB, which is the size of many common processor L1 caches. You can control the chunk size by changing the option on a filter list:

// ... create context and filter list
tiledb_filter_list_set_max_chunk_size(ctx, filter_list, 10000);

Setting a Filter List for an Attribute

This is done as follows:

// ... create context, filter list and attribute

// Set filter list to attribute
tiledb_attribute_set_filter_list(ctx, attr, filter_list);

Setting a Filter List for a Dimension

This is done as follows:

// ... create context, filter list and dimension

// Set filter list to dimension
tiledb_dimension_set_filter_list(ctx, dim, filter_list);

Setting a Filter List for All Dimensions

If you wish all the dimensions to have the same filter list, you can set it once as follows:

// ... create context ctx 
// ... create array schema
// ... create filter list fl

// Set coordinates filter list
tiledb_array_schema_set_coords_filter_list(ctx, schema, fl);

If you do not specify a filter for a dimension separately, the dimension will inherit the filters set to all dimensions collectively as shown above.

Setting a Filter List for Variable-sized Value Offsets

You can set filter lists for the offsets of variable-sized attributes or dimensions as follows:

#import <tiledb/tiledb.h>

// ... create context ctx 
// ... create array schema
// ... create filter list fl

// Set offsets filter list
tiledb_array_schema_set_offsets_filter_list(ctx, schema, fl);

The offset filters are applied to all variable-sized attributes and dimensions.

Last updated