Consolidation and Vacuuming

Consolidation

Fragments

You can consolidate an array as follows. See Consolidation for how to fine-tune the consolidation algorithm with configuration parameters, and Configuration for more details on creating configuration objects.

C
C++
Python
R
Java
Go
C
// Create TileDB context
tiledb_ctx_t* ctx;
tiledb_ctx_alloc(NULL, &ctx);
// Consolidate, using ctx's configuration
tiledb_array_consolidate(ctx, array_name, NULL);
// Alteratively, you can create and pass a configuration object
tiledb_config_t* config;
tiledb_config_alloc(&config, NULL);
tiledb_config_set(config, "sm.consolidation.steps", "3", &error);
tiledb_config_set(config, "sm.consolidation.mode", "fragments", &error);
tiledb_array_consolidate(ctx, array_name, config);
// Clean-up
tiledb_ctx_free(&ctx);
tiledb_config_free(&config);
C++
Context ctx;
// Consolidate, using ctx's configuration
Array::consolidate(ctx, "<array-uri>");
// Alteratively, you can create and pass a configuration object
Config config;
config["sm.consolidation.steps"] = "3";
config["sm.consolidation.mode"] = "fragments";
Array::consolidate(ctx, "<array-uri>", config);
Python
# Consolidate array at array_uri
tiledb.consolidate(array_uri)
# Alteratively, you can create and pass a configuration object
config = tiledb.Config({"sm.consolidation.steps": 3, "sm.consolidation.mode": "fragments"})
ctx = tiledb.Ctx(config)
tiledb.consolidate(array_uri, ctx=ctx)
R
# TODO: Consolidation is currently not supported in R
Java
try(Context ctx = new Context() {
// Consolidate, using ctx's configuration
consolidate(ctx, "<array-uri>");
// Alteratively, you can create and pass a configuration object
Config config = new Config();
config.set("sm.consolidation.steps") = "3";
config.set("sm.consolidation.mode") = "fragments";
consolidate(ctx, "<array-uri>", config);
}
Go
// Create configuration
config, _ := NewConfig()
// Test context with config
ctx, _ := NewContext(config)
// Consolidate, using ctx's configuration
config.Set("sm.consolidation.steps", "3")
config.Set("sm.consolidation.mode", "fragments")
array.Consolidate(ctx, config)

If you do not set a configuration object to the consolidation function, the consolidation algorithm will inherit the (default or set) configuration of the context. Otherwise, the set options in the passed configuration object will override those of the context's, but the rest of the options will still be inherited from the context's configuration.

Fragment Metadata

The same consolidation API is used for consolidating fragment metadata. The only change is configuration parameter "sm.consolidation.mode" which should now be set to "fragment_meta".

Array Metadata

The same consolidation API is used for consolidating array metadata. The only change is configuration parameter "sm.consolidation.mode" which should now be set to "array_meta".

Vacuuming

Fragments

After consolidation, you may optionally remove the consolidated fragments with the "vacuum" step. Note that vacuuming must not be run if you intend to use the time-traveling feature (opening array at a specific prior timestamp).

C
C++
Python
R
Java
Go
C
// Create TileDB context
tiledb_ctx_t* ctx;
tiledb_ctx_alloc(NULL, &ctx);
// Vacuum, using ctx's configuration
tiledb_array_vacuum(ctx, array_uri, nullptr);
// Alteratively, you can create and pass a configuration object
tiledb_config_t* config;
tiledb_config_alloc(&config, NULL);
tiledb_config_set(config, "sm.vacuum.mode", "fragments", &error);
tiledb_array_vacuum(ctx, array_name, config);
// Clean-up
tiledb_ctx_free(&ctx);
tiledb_config_free(&config);
C++
Context ctx;
// Consolidate, using ctx's configuration
Array::vacuum(ctx, "<array-uri>");
// Alteratively, you can create and pass a configuration object
Config config;
config["sm.vacuum.mode"] = "fragments";
Array::vacuum(ctx, "<array-uri>", config);
Python
# Consolidate array at array_uri
tiledb.vacuum(array_uri)
# Alteratively, you can create and pass a configuration object
config = tiledb.Config({"sm.vacuum.mode": "fragments"})
tiledb.vacuum(array_uri, config=config)
R
# TODO: Consolidation is currently not supported in R
Java
// Vacuuming is not currently supported in the Java API
Go
// Create configuration
config, _ := NewConfig()
// Test context with config
ctx, _ := NewContext(config)
// Consolidate, using ctx's configuration
config.Set("sm.vacuum.mode", "fragments")
array.Vacuum(ctx, config)I

As with consolidation above, vacuuming inherits the context configuration if no configuration override is specified.

Fragment Metadata

The same vacuuming API is used for vacuuming fragment metadata. The only change is configuration parameter "sm.vacuum.mode" which should now be set to "fragment_meta".

Array Metadata

The same vacuuming API is used for vacuuming array metadata. The only change is configuration parameter "sm.vacuum.mode" which should now be set to "array_meta".