Uploading from Git

TileDB Cloud support uploading notebooks from git repositories. Its common that you might have notebooks stored in source control such as github or gitlab and wish to automatically upload them to TileDB Cloud for usage.

Github Actions

TileDB provides public github actions that let you upload a notebook.

Example usage includes:

   - uses: TileDB-Inc/github-actions/upload-notebooks@main
      with:
        notebooks_local: >-
          path/to/notebook-1.ipynb
          path/to/notebook-2.ipynb
          path/to/notebook-3.ipynb
        notebooks_remote: >-
          tiledb://namespace/s3://cloud-path/notebook-1
          tiledb://namespace/s3://cloud-path/notebook-2
          tiledb://namespace/s3://cloud-path/notebook-3
        TILEDB_CLOUD_TOKEN: ${{ secrets.TILEDB_CLOUD_TOKEN }}
        TILEDB_CLOUD_STORAGE_CREDENTIAL_NAME: <name-of-credential-saved-on-tiledb-cloud>

Gitlab CI

Gitlab CI doesn't offer a marketplace or public template support currently. Instead find the bellow example for setting up a gitlab ci setup.

Secrets

An API Token is required to access TileDB Cloud. Its recommended to set this as a secret in gitlab. This should be set as an environmental variable for TILEDB_REST_TOKEN.

upload_notebook:
  stage: upload
  image: python:3.9
  variables:
    TILEDB_STORAGE_CREDENTIAL_NAME: "example-credential-name"
    TILEDB_NAMESPACE: "namespace-to-upload-to"
  script:
    # Install tiledb-cloud
    - pip install tiledb-cloud
    # Define notebook to upload
    - export NOTEBOOK_FILE="example.ipynb"
    # Invoke tiledb-cloud's upload functionality
    - python -c "import tiledb, os; tiledb.cloud.upload_notebook_from_file(
        ipynb_file_name=${NOTEBOOK_FILE},
        namespace=${TILEDB_NAMESPACE},
        array_name=os.path.basename(${NOTEBOOK_FILE}),
        on_exists=OnExists.OVERWRITE,
    )"