Consolidation Plan

This will create a plan for an array that can be later used during consolidation. The desired fragment size gives the maximum size for a fragment. The maximum fragment size will be used to find fragments that are already too large and should be split as well as fragments that are too small and should be combined. The value should be used as is in the consolidation parameters ("sm.consolidation.max_fragment_size") when running the different nodes. The plan divides the fragments of the array into different nodes that need to be consolidated together. The plan is created as follows:

C

#include <tiledb/tiledb.h>
#include <tiledb/tiledb_experimental.h>

// Create TileDB context
tiledb_ctx_t* ctx;
tiledb_ctx_alloc(NULL, &ctx);

// Open array for reading
tiledb_array_t* array;
tiledb_array_alloc(ctx, array_name, &array);
tiledb_array_open(ctx, array, TILEDB_READ);

// Create the plan with a desired fragment size of 1000 bytes.
tiledb_consolidation_plan_t* consolidation_plan;
tiledb_consolidation_plan_create_with_mbr(
    ctx, array, 1000, &consolidation_plan);

uint64_t num_nodes;
tiledb_consolidation_plan_get_num_nodes(ctx, consolidation_plan, &num_nodes);
printf("Consolidation plan for %llu node(s):\n", num_nodes);

for (uint64_t n = 0; n < num_nodes; n++) {
  uint64_t num_fragments;
  tiledb_consolidation_plan_get_num_fragments(
      ctx, consolidation_plan, n, &num_fragments);
  printf("  Node %llu with %llu fragment(s):\n", n, num_fragments);

  for (uint64_t f = 0; f < num_fragments; f++) {
    const char* frag_uri;
    tiledb_consolidation_plan_get_fragment_uri(
        ctx, consolidation_plan, n, f, &frag_uri);
    printf("    %s\n", frag_uri);
  }
}

// Close array
tiledb_array_close(ctx, array);

// Clean up
tiledb_consolidation_plan_free(&consolidation_plan);
tiledb_array_free(&array);
tiledb_ctx_free(&ctx);

C++

#include <tiledb/tiledb>
#include <tiledb/tiledb_experimental>
using namespace tiledb;

// Create context
Context ctx;

// Open the array for reading
Array array(ctx, array_name, TILEDB_READ);

// Create the plan with a desired fragment size of 1000 bytes.
ConsolidationPlan plan(ctx, array, 1000);

auto num_nodes = plan.num_nodes();
std::cout << "Consolidation plan for " << num_nodes << " node(s):" << std::endl;
for (uint64_t n = 0; n < num_nodes; n++) {
  auto num_fragments = plan.num_fragments(n);
  std::cout << "  Node " << n << " with " << num_fragments
            << " fragment(s):" << std::endl;
  for (uint64_t f = 0; f < num_fragments; f++) {
    std::cout << "    " << plan.fragment_uri(n, f) << std::endl;
  }
}
  
// Close the array  
array.close();

Java

// Create array and context 

ConsolidationPlan consolidationPlan =
    new ConsolidationPlan(ctx, BigInteger.valueOf(1024 * 1024), arrayURI);

// Get number of nodes and fragments
long numNodes = consolidationPlan.getNumNodes();
long numFragments = consolidationPlan.getNumFragments(BigInteger.ZERO);

// Get the number of fragments for a specific node of a consolidation plan object
String uri = consolidationPlan.getFragmentURI(BigInteger.ZERO, BigInteger.ZERO);

The code snippets above also shows how to print the different nodes with the fragment URIs they contain. The list of fragments can be passed into the tiledb_array_consolidate_fragments API. Note that at the moment there is no CPP API that allows to consolidate a list of fragments.