Using Performance Statistics
A lot of performance optimization for TileDB programs involves minimizing wasted work. TileDB comes with an internal statistics reporting system that can help identify potential areas of performance improvement for your TileDB programs, including reducing wasted work.
The TileDB statistics can be enabled and disabled at runtime, and a report can be dumped at any point. A typical situation is to enable the statistics immediately before submitting a query, submit the query, and then immediately dump the report. This can be done like so:
C
C++
Python
R
Java
Go
1
tiledb_stats_enable();
2
3
// ... create some query here
4
5
tiledb_query_submit(ctx, query);
6
7
// Dump the statistics
8
tiledb_stats_dump(FILE* out);
9
tiledb_stats_disable();
10
11
// You can also reset the stats as follows
12
tiledb_stats_reset();
Copied!
1
tiledb::Stats::enable();
2
3
// ... create some query here
4
5
// Submit the query
6
query.submit();
7
8
// Dump the statistics
9
tiledb::Stats::dump(stdout);
10
tiledb::Stats::disable();
11
12
// You can also reset the stats as follows
13
tiledb::Stats::reset();
Copied!
1
tiledb.stats_enable()
2
3
# Do some work
4
data = A[:]
5
6
# Dump the statistics
7
tiledb.stats_dump()
8
tiledb.stats_disable()
9
10
# You can also reset the stats as follows
11
tiledb.stats_reset()
12
13
# You can get the report for invidual context and query runs
14
with tiledb.open(array_uri, mode="r", ctx=ctx) as A:
15
qry = A.query()[:]
16
17
# Dump the stats for a given query call
18
qry.get_stats()
19
20
# Dump the stats for a given context
21
ctx.get_stats()
Copied!
1
# Start collecting statistics
2
tiledb_stats_enable()
3
4
# ... create some query here
5
6
A[1:4]
7
8
# Stop collecting statistics
9
tiledb_stats_disable()
10
11
# Show the statistics on the console
12
tiledb_stats_print()
13
14
# Save the statistics to a file
15
tiledb_stats_dump(my_file_name)
16
17
# You can reset the stats as follows
18
tiledb_stats_reset()
19
20
# Statistics can also be collected on a per-query basis
21
arr <- tiledb_array(uri, query_statistics = TRUE)
22
res <- arr[]
23
24
# we can access the 'query_statistics' attribute of the result
25
qstats <- attr(res, "query_statistics")
26
## then use jsonlite::fromJSON(qstats) to parse
27
28
# Statistics can also be retrieved for the current context
29
cstats <- tiledb_ctx_stats()
30
## then jsonlite::fromJSON(cstats)
31
Copied!
1
try (Array array = new Array(ctx, "<array-uri>", TILEDB_READ);
2
ArraySchema schema = array.getSchema();
3
Query query = new Query(array, TILEDB_READ)) {
4
query.addRange(0, 1, 2);
5
query.addRange(1, 2, 4);
6
7
query.setLayout(TILEDB_ROW_MAJOR);
8
9
NativeArray dim1Array = new NativeArray(ctx, 6, Integer.class);
10
NativeArray dim2Array = new NativeArray(ctx, 6, Integer.class);
11
NativeArray a1Array = new NativeArray(ctx, 12, Character.class);
12
NativeArray a2Array = new NativeArray(ctx, 6, Float.class);
13
14
query.setBuffer("rows", dim1Array);
15
query.setBuffer("cols", dim2Array);
16
query.setBuffer("a1", a1Array);
17
query.setBuffer("a2", a2Array);
18
19
// Submit query
20
query.submit();
21
String stats = query.getStats();
22
System.out.println(query.getStats());
Copied!
1
tiledb.StatsEnable()
2
3
// ... create some query here
4
5
// Submit the query
6
query.Submit()
7
8
// Dump the statistics
9
tiledb.StatsDumpSTDOUT()
10
tiledb.StatsDisable()
11
12
// You can also reset the stats as follows
13
tiledb.StatsReset()
Copied!
With the dump call, a report containing the gathered statistics will be printed. The report prints values of many individual counters. Typically the summary contains the necessary information to make high-level performance tuning decisions. An example summary is shown below:
1
{
2
"timers": {
3
"Context.StorageManager.Query.Subarray.read_load_relevant_rtrees.sum": 0.000225025,
4
"Context.StorageManager.Query.Subarray.read_load_relevant_rtrees.avg": 0.000112513,
5
"Context.StorageManager.Query.Subarray.read_compute_tile_overlap.sum": 0.000541446,
6
"Context.StorageManager.Query.Subarray.read_compute_tile_overlap.avg": 0.000270723,
7
"Context.StorageManager.Query.Subarray.read_compute_tile_coords.sum": 3.001e-06,
8
"Context.StorageManager.Query.Subarray.read_compute_tile_coords.avg": 3.001e-06,
9
"Context.StorageManager.Query.Subarray.read_compute_relevant_tile_overlap.sum": 0.000177502,
10
"Context.StorageManager.Query.Subarray.read_compute_relevant_tile_overlap.avg": 8.8751e-05,
11
"Context.StorageManager.Query.Subarray.read_compute_relevant_frags.sum": 0.000110015,
12
"Context.StorageManager.Query.Subarray.read_compute_relevant_frags.avg": 5.50075e-05,
13
"Context.StorageManager.Query.Subarray.read_compute_est_result_size.sum": 0.000230768,
14
"Context.StorageManager.Query.Subarray.read_compute_est_result_size.avg": 2.8846e-05,
15
"Context.StorageManager.Query.Reader.unfilter_attr_tiles.sum": 0.000114806,
16
"Context.StorageManager.Query.Reader.unfilter_attr_tiles.avg": 5.7403e-05,
17
"Context.StorageManager.Query.Reader.read.sum": 0.00153529,
18
"Context.StorageManager.Query.Reader.read.avg": 0.00153529,
19
"Context.StorageManager.Query.Reader.load_tile_offsets.sum": 0.000285125,
20
"Context.StorageManager.Query.Reader.load_tile_offsets.avg": 0.000285125,
21
"Context.StorageManager.Query.Reader.init_state.sum": 2.1814e-05,
22
"Context.StorageManager.Query.Reader.init_state.avg": 2.1814e-05,
23
"Context.StorageManager.Query.Reader.fill_dense_coords.sum": 4.537e-06,
24
"Context.StorageManager.Query.Reader.fill_dense_coords.avg": 4.537e-06,
25
"Context.StorageManager.Query.Reader.copy_fixed_attr_values.sum": 2.6773e-05,
26
"Context.StorageManager.Query.Reader.copy_fixed_attr_values.avg": 1.33865e-05,
27
"Context.StorageManager.Query.Reader.copy_attr_values.sum": 0.000641413,
28
"Context.StorageManager.Query.Reader.copy_attr_values.avg": 0.000641413,
29
"Context.StorageManager.Query.Reader.compute_sparse_result_tiles.sum": 2.39e-06,
30
"Context.StorageManager.Query.Reader.compute_sparse_result_tiles.avg": 2.39e-06,
31
"Context.StorageManager.Query.Reader.compute_sparse_result_cell_slabs_dense.sum": 3.941e-05,
32
"Context.StorageManager.Query.Reader.compute_sparse_result_cell_slabs_dense.avg": 3.941e-05,
33
"Context.StorageManager.Query.Reader.compute_result_coords.sum": 7.273e-06,
34
"Context.StorageManager.Query.Reader.compute_result_coords.avg": 7.273e-06,
35
"Context.StorageManager.Query.Reader.attr_tiles.sum": 0.000193725,
36
"Context.StorageManager.Query.Reader.attr_tiles.avg": 0.000193725,
37
"Context.StorageManager.Query.Reader.SubarrayPartitioner.read_next_partition.sum": 0.000815125,
38
"Context.StorageManager.Query.Reader.SubarrayPartitioner.read_next_partition.avg": 0.000815125
39
},
40
"counters": {
41
"Context.StorageManager.Query.Subarray.precompute_tile_overlap.tile_overlap_byte_size": 160,
42
"Context.StorageManager.Query.Subarray.precompute_tile_overlap.relevant_fragment_num": 2,
43
"Context.StorageManager.Query.Subarray.precompute_tile_overlap.ranges_requested": 2,
44
"Context.StorageManager.Query.Subarray.precompute_tile_overlap.ranges_computed": 2,
45
"Context.StorageManager.Query.Subarray.precompute_tile_overlap.fragment_num": 2,
46
"Context.StorageManager.Query.Reader.result_num": 3,
47
"Context.StorageManager.Query.Reader.read_unfiltered_byte_num": 72,
48
"Context.StorageManager.Query.Reader.overlap_tile_num": 2,
49
"Context.StorageManager.Query.Reader.loop_num": 1,
50
"Context.StorageManager.Query.Reader.dim_num": 2,
51
"Context.StorageManager.Query.Reader.dim_fixed_num": 2,
52
"Context.StorageManager.Query.Reader.cell_num": 8,
53
"Context.StorageManager.Query.Reader.attr_num": 2,
54
"Context.StorageManager.Query.Reader.attr_fixed_num": 2,
55
"Context.StorageManager.Query.Reader.SubarrayPartitioner.compute_current_start_end.not_found": 1,
56
"Context.StorageManager.Query.Reader.SubarrayPartitioner.compute_current_start_end.fixed_result_size_overflow": 1
57
}
58
}
Copied!
The TileDB library is built by default with statistics enabled. You can disable statistics gathering with the -DTILEDB_STATS=OFF CMake variable.
Last modified 2mo ago
Copy link