Consolidation and Vacuuming
Consolidation
Fragments
You can consolidate an array as follows. See Consolidation for how to fine-tune the consolidation algorithm with configuration parameters, and Configuration for more details on creating configuration objects.
C
C++
Python
R
Java
Go
1
// Create TileDB context
2
tiledb_ctx_t* ctx;
3
tiledb_ctx_alloc(NULL, &ctx);
4
​
5
// Consolidate, using ctx's configuration
6
tiledb_array_consolidate(ctx, array_name, NULL);
7
​
8
// Alteratively, you can create and pass a configuration object
9
tiledb_config_t* config;
10
tiledb_config_alloc(&config, NULL);
11
tiledb_config_set(config, "sm.consolidation.steps", "3", &error);
12
tiledb_config_set(config, "sm.consolidation.mode", "fragments", &error);
13
tiledb_array_consolidate(ctx, array_name, config);
14
​
15
// Clean-up
16
tiledb_ctx_free(&ctx);
17
tiledb_config_free(&config);
Copied!
1
Context ctx;
2
​
3
// Consolidate, using ctx's configuration
4
Array::consolidate(ctx, "<array-uri>");
5
​
6
// Alteratively, you can create and pass a configuration object
7
Config config;
8
config["sm.consolidation.steps"] = "3";
9
config["sm.consolidation.mode"] = "fragments";
10
Array::consolidate(ctx, "<array-uri>", config);
Copied!
1
# Consolidate array at array_uri
2
tiledb.consolidate(array_uri)
3
​
4
# Alternatively, you can create and pass a configuration object
5
config = tiledb.Config(
6
{"sm.consolidation.steps": 3, "sm.consolidation.mode": "fragments"}
7
)
8
ctx = tiledb.Ctx(config)
9
tiledb.consolidate(array_uri, ctx=ctx)
10
​
11
# You may also consolidate a subset of fragments across a timestamp range
12
​
13
# Set tuple of timestamp ranges as an argument to the timestamp parameter
14
tiledb.consolidate(array_uri, timestamp=(5, 9))
15
​
16
# Or set it in the configuration object
17
config = tiledb.Config(
18
{
19
"sm.consolidation.timestamp_start": 5,
20
"sm.consolidation.timestamp_end": 9,
21
"sm.consolidation.mode": "fragments",
22
}
23
)
24
ctx = tiledb.Ctx(config)
25
tiledb.consolidate(array_uri, ctx=ctx)
Copied!
1
# An array URI
2
uri <- "<array_uri>"
3
​
4
# Consolidate with default configuration
5
array_consolidate(uri)
6
7
# Alteratively, create and pass a configuration object
8
cfg <- tiledb_config()
9
cfg["sm.consolidation.steps"] <- "3"
10
cfg["sm.consolidation.mode"] <- "fragments"
11
array_consolidate(uri, cfg)
12
​
13
# We can also consolidate over a timestamp range
14
windowStart <- as.POSIXct("2021-01-01 00:00:00")
15
windowEnd <- as.POSIXct("2021-06-30 23:59:59.999")
16
array_consolidate(uri, start_time=windowStart, end_time=windowEnd)
17
​
Copied!
1
try(Context ctx = new Context() {
2
// Consolidate, using ctx's configuration
3
consolidate(ctx, "<array-uri>");
4
5
// Alteratively, you can create and pass a configuration object
6
Config config = new Config();
7
config.set("sm.consolidation.steps") = "3";
8
config.set("sm.consolidation.mode") = "fragments";
9
consolidate(ctx, "<array-uri>", config);
10
}
Copied!
1
// Create configuration
2
config, _ := NewConfig()
3
​
4
// Test context with config
5
ctx, _ := NewContext(config)
6
​
7
// Consolidate, using ctx's configuration
8
config.Set("sm.consolidation.steps", "3")
9
config.Set("sm.consolidation.mode", "fragments")
10
array.Consolidate(ctx, config)
Copied!
If you do not set a configuration object to the consolidation function, the consolidation algorithm will inherit the (default or set) configuration of the context. Otherwise, the set options in the passed configuration object will override those of the context's, but the rest of the options will still be inherited from the context's configuration.
Fragment Metadata
The same consolidation API is used for consolidating fragment metadata. The only change is configuration parameter
"sm.consolidation.mode" which should now be set to "fragment_meta". Array Metadata
The same consolidation API is used for consolidating array metadata. The only change is configuration parameter
"sm.consolidation.mode" which should now be set to "array_meta". Vacuuming
Fragments
After consolidation, you may optionally remove the consolidated fragments with the "vacuum" step. Note that vacuuming must not be run if you intend to use the time-traveling feature (opening array at a specific prior timestamp).
C
C++
Python
R
Java
Go
1
// Create TileDB context
2
tiledb_ctx_t* ctx;
3
tiledb_ctx_alloc(NULL, &ctx);
4
​
5
// Vacuum, using ctx's configuration
6
tiledb_array_vacuum(ctx, array_uri, nullptr);
7
​
8
// Alteratively, you can create and pass a configuration object
9
tiledb_config_t* config;
10
tiledb_config_alloc(&config, NULL);
11
tiledb_config_set(config, "sm.vacuum.mode", "fragments", &error);
12
tiledb_array_vacuum(ctx, array_name, config);
13
​
14
// Clean-up
15
tiledb_ctx_free(&ctx);
16
tiledb_config_free(&config);
Copied!
1
Context ctx;
2
​
3
// Consolidate, using ctx's configuration
4
Array::vacuum(ctx, "<array-uri>");
5
​
6
// Alteratively, you can create and pass a configuration object
7
Config config;
8
config["sm.vacuum.mode"] = "fragments";
9
Array::vacuum(ctx, "<array-uri>", config);
Copied!
1
# Consolidate array at array_uri
2
tiledb.vacuum(array_uri)
3
​
4
# Alteratively, you can create and pass a configuration object
5
config = tiledb.Config({"sm.vacuum.mode": "fragments"})
6
​
7
tiledb.vacuum(array_uri, config=config)
Copied!
1
# An array URI
2
uri <- "<array_uri>"
3
​
4
# Vacuum with default configuration
5
array_vacuum(uri)
6
7
# Alternatively, create and pass a configuration object
8
cfg <- tiledb_config()
9
cfg["sm.vacuum.mode"] <- "fragments"
10
array_vacuum(uri, cfg)
11
​
12
# We can also vacuum over a timestamp range
13
windowStart <- as.POSIXct("2021-01-01 00:00:00")
14
windowEnd <- as.POSIXct("2021-06-30 23:59:59.999")
15
array_vacuum(uri, start_time=windowStart, end_time=windowEnd)
16
​
Copied!
1
try(Context ctx = new Context()){
2
// Vacuum, using ctx's configuration
3
Config conf = new Config();
4
conf.set("sm.vacuum.mode", "fragments");
5
6
// Alteratively, you can create and pass a configuration object
7
Array.vacuum(ctx,"<array-uri>", conf);
8
}
Copied!
1
// Create configuration
2
config, _ := NewConfig()
3
​
4
// Test context with config
5
ctx, _ := NewContext(config)
6
​
7
// Consolidate, using ctx's configuration
8
config.Set("sm.vacuum.mode", "fragments")
9
array.Vacuum(ctx, config)I
Copied!
As with consolidation above, vacuuming inherits the context configuration if no configuration override is specified.
Fragment Metadata
The same vacuuming API is used for vacuuming fragment metadata. The only change is configuration parameter
"sm.vacuum.mode" which should now be set to "fragment_meta". Array Metadata
The same vacuuming API is used for vacuuming array metadata. The only change is configuration parameter
"sm.vacuum.mode" which should now be set to "array_meta". Last modified 6mo ago