Atomicity
In TileDB, reads, writes, consolidation and vacuuming are all atomic and will never lead to array corruption.
A read operation is the process of (i) creating a read query object and (ii) submitting the query (potentially multiple times in the case of incomplete queries) until the query is completed. Each such read operation is atomic and can never corrupt the state of an array.
A write operation is the process of (i) creating a write query object, (ii) submitting the query (potentially multiple times in the case of global writes) and (iii) finalizing the query object (important only in global writes). Each such write operation is atomic, i.e., a set of functions (which depends on the API) that must be treated atomically by each thread. For example, multiple threads should not submit the query for the same query object. Instead, you can have multiple threads create separate query objects for the same array (even sharing the same context or array object), and prepare and submit them in parallel with each thread.
A write operation either succeeds and creates a fragment that is visible to future reads, or it fails and any folder and file relevant to the failed fragment is entirely ignored by future reads. A fragment creation is successful if a file
<fragment_name>.ok appears in the array folder for the created fragment <fragment_name>. There will never be the case that a fragment will be partially written and still accessible by the reader. The user just needs to eventually delete the partially written folder to save space (i.e., a fragment folder without an associated .ok file). Furthermore, each fragment is immutable, so there is no way for a write operation to corrupt another fragment created by another operation.Consolidation entails a read and a write and, therefore, it is atomic in the same sense as for writing. There is no way for consolidation to lead to a corrupted array state.
Vacuuming simply deletes fragment folders and array/fragment metadata files. Vacuuming always deletes the
.ok files before proceeding to erasing the corresponding folders. It is atomic in the sense that it cannot lead to array corruption and if the vacuuming process is interrupted, it can be restarted without issues. Last modified 2yr ago