Consolidation and Vacuuming

Consolidation

Fragments

You can consolidate an array as follows. See Consolidation for how to fine-tune the consolidation algorithm with configuration parameters, and Configuration for more details on creating configuration objects.

C

// Create TileDB context
tiledb_ctx_t* ctx;
tiledb_ctx_alloc(NULL, &ctx);

// Consolidate, using ctx's configuration
tiledb_array_consolidate(ctx, array_name, NULL);

// Alteratively, you can create and pass a configuration object
tiledb_config_t* config;
tiledb_config_alloc(&config, NULL);
tiledb_config_set(config, "sm.consolidation.steps", "3", &error);
tiledb_config_set(config, "sm.consolidation.mode", "fragments", &error);
tiledb_array_consolidate(ctx, array_name, config);

// Clean-up
tiledb_ctx_free(&ctx);
tiledb_config_free(&config);

C++

Context ctx;

// Consolidate, using ctx's configuration
Array::consolidate(ctx, "<array-uri>");

// Alteratively, you can create and pass a configuration object
Config config;
config["sm.consolidation.steps"] = "3";
config["sm.consolidation.mode"] = "fragments";
Array::consolidate(ctx, "<array-uri>", config);

Python

# Consolidate array at array_uri
tiledb.consolidate(array_uri)

# Alternatively, you can create and pass a configuration object
config = tiledb.Config(
    {"sm.consolidation.steps": 3, "sm.consolidation.mode": "fragments"}
)
ctx = tiledb.Ctx(config)
tiledb.consolidate(array_uri, ctx=ctx)

# You may also consolidate a subset of fragments across a timestamp range

# Set tuple of timestamp ranges as an argument to the timestamp parameter
tiledb.consolidate(array_uri, timestamp=(5, 9))

# Or set it in the configuration object
config = tiledb.Config(
    {
        "sm.consolidation.timestamp_start": 5,
        "sm.consolidation.timestamp_end": 9,
        "sm.consolidation.mode": "fragments",
    }
)
ctx = tiledb.Ctx(config)
tiledb.consolidate(array_uri, ctx=ctx)

R

# An array URI
uri <- "<array_uri>"

# Consolidate with default configuration
array_consolidate(uri)
 
# Alteratively, create and pass a configuration object
cfg <- tiledb_config()
cfg["sm.consolidation.steps"] <- "3"
cfg["sm.consolidation.mode"] <- "fragments"
array_consolidate(uri, cfg)

# We can also consolidate over a timestamp range
windowStart <- as.POSIXct("2021-01-01 00:00:00")
windowEnd <- as.POSIXct("2021-06-30 23:59:59.999")
array_consolidate(uri, start_time=windowStart, end_time=windowEnd)

Java

try(Context ctx = new Context() {
    // Consolidate, using ctx's configuration
    consolidate(ctx, "<array-uri>");
    
    // Alteratively, you can create and pass a configuration object
    Config config = new Config();
    config.set("sm.consolidation.steps") = "3";
    config.set("sm.consolidation.mode") = "fragments";
    consolidate(ctx, "<array-uri>", config);
}

Go

// Create configuration
config, _ := NewConfig()

// Test context with config
ctx, _ := NewContext(config)

// Consolidate, using ctx's configuration
config.Set("sm.consolidation.steps", "3")
config.Set("sm.consolidation.mode", "fragments")
array.Consolidate(ctx, config)

C#

using TileDB.CSharp;

// Create TileDB context
using Context ctx = new Context();

// Consolidate, using ctx's configuration
Array.Consolidate(ctx, array_name);

// Alternatively, you can create and pass a configuration object
using Config config = new Config();
config.Set("sm.consolidation.steps", "3");
config.Set("sm.consolidation.mode", "fragments");
Array.Consolidate(ctx, array_name, config);

If you do not set a configuration object to the consolidation function, the consolidation algorithm will inherit the (default or set) configuration of the context. Otherwise, the set options in the passed configuration object will override those of the context's, but the rest of the options will still be inherited from the context's configuration.

Fragment Metadata

The same consolidation API is used for consolidating fragment metadata. The only change is configuration parameter "sm.consolidation.mode" which should now be set to "fragment_meta".

Array Metadata

The same consolidation API is used for consolidating array metadata. The only change is configuration parameter "sm.consolidation.mode" which should now be set to "array_meta".

Commits

The same consolidation API is used for consolidating commits. The only change is configuration parameter "sm.consolidation.mode" which should now be set to "commits".

Vacuuming

Fragments

After consolidation, you may optionally remove the consolidated fragments with the "vacuum" step. Note that vacuuming must not be run if you intend to use the time-traveling feature (opening array at a specific prior timestamp).

C

// Create TileDB context
tiledb_ctx_t* ctx;
tiledb_ctx_alloc(NULL, &ctx);

// Vacuum, using ctx's configuration
tiledb_array_vacuum(ctx, array_name, nullptr);

// Alteratively, you can create and pass a configuration object
tiledb_config_t* config;
tiledb_config_alloc(&config, NULL);
tiledb_config_set(config, "sm.vacuum.mode", "fragments", &error);
tiledb_array_vacuum(ctx, array_name, config);

// Clean-up
tiledb_ctx_free(&ctx);
tiledb_config_free(&config);

C++

Context ctx;

// Consolidate, using ctx's configuration
Array::vacuum(ctx, "<array-uri>");

// Alteratively, you can create and pass a configuration object
Config config;
config["sm.vacuum.mode"] = "fragments";
Array::vacuum(ctx, "<array-uri>", config);

Python

# Consolidate array at array_uri
tiledb.vacuum(array_uri)

# Alteratively, you can create and pass a configuration object
config = tiledb.Config({"sm.vacuum.mode": "fragments"})

tiledb.vacuum(array_uri, config=config)

R

# An array URI
uri <- "<array_uri>"

# Vacuum with default configuration
array_vacuum(uri)
 
# Alternatively, create and pass a configuration object
cfg <- tiledb_config()
cfg["sm.vacuum.mode"] <- "fragments"
array_vacuum(uri, cfg)

# We can also vacuum over a timestamp range
windowStart <- as.POSIXct("2021-01-01 00:00:00")
windowEnd <- as.POSIXct("2021-06-30 23:59:59.999")
array_vacuum(uri, start_time=windowStart, end_time=windowEnd)

Java

try(Context ctx = new Context()){
      // Vacuum, using ctx's configuration
      Config conf = new Config();
      conf.set("sm.vacuum.mode", "fragments");
      
      // Alteratively, you can create and pass a configuration object 
      Array.vacuum(ctx,"<array-uri>", conf);
}

Go

// Create configuration
config, _ := NewConfig()

// Test context with config
ctx, _ := NewContext(config)

// Consolidate, using ctx's configuration
config.Set("sm.vacuum.mode", "fragments")
array.Vacuum(ctx, config)I

C#

using TileDB.CSharp;

// Create TileDB context
using Context ctx = new Context();

// Vacuum, using ctx's configuration
Array.Vacuum(ctx, array_name);

// Alternatively, you can create and pass a configuration object
using Config config = new Config();
config.Set("sm.vacuum.mode", "fragments");
Array.Vacuum(ctx, array_name, config);

As with consolidation above, vacuuming inherits the context configuration if no configuration override is specified.

Fragment Metadata

The same vacuuming API is used for vacuuming fragment metadata. The only change is configuration parameter "sm.vacuum.mode" which should now be set to "fragment_meta".

Array Metadata

The same vacuuming API is used for vacuuming array metadata. The only change is configuration parameter "sm.vacuum.mode" which should now be set to "array_meta".

Commits

The same vacuuming API is used for vacumming commits. The only change is configuration parameter "sm.vacuum.mode" which should now be set to "commits".