Serverless Array UDFs

Search…
Basic Usage
Below we show how to use Python UDFs in TileDB Cloud, with an example that computes the median on the values of attribute a on a slice of a 2D dense array. You just need to write your function (median in this example) that takes as input an ordered numpy dictionary, i.e., in the form {"a" : <numpy-array>, "b" : <numpy-array>, ...}, where the keys are attribute or dimension names of the array you are querying. The reason is that this function will be applied on an array slice; recall that the Python API of TileDB returns an ordered dictionary of numpy arrays on each attribute and dimension upon a read. Then you just use the apply function of the TileDB Cloud client, which takes as input your function, a slice, and optionally a list of attributes (default is all attributes). Note that only the selected attributes must appear in the ordered dictionary that you provide as input to your function.
Python
1
import tiledb, tiledb.cloud, numpy
2
​
3
def median(numpy_ordered_dictionary):
4
  return numpy.median(numpy_ordered_dictionary["a"])
5
​
6
tiledb.cloud.login(username="my_username", password="my_password")
7
# or tiledb.cloud.login(token="my_token")
8
​
9
with tiledb.open("tiledb://TileDB-Inc/quickstart_dense", ctx=tiledb.cloud.Ctx()) as A:
10
    # apply on subarray [1,2]x[1,2]
11
    res = A.apply(median, [(1,2), (1,2)], attrs = ["a"])
12
    print(res)
13
​
Copied!
All slices provided as input to theapply function are inclusive.
Multi-Index Usage
Multi-index queries are supported when applying an array UDF. You can pass any number of tuples or slices using a list of lists syntax (one per dimension).
Python
1
import tiledb, tiledb.cloud, numpy
2
​
3
def median(numpy_ordered_dictionary):
4
  return numpy.median(numpy_ordered_dictionary["a"])
5
​
6
tiledb.cloud.login(username="my_username", password="my_password")
7
# or tiledb.cloud.login(token="my_token")
8
​
9
with tiledb.open("tiledb://TileDB-Inc/quickstart_dense", ctx=tiledb.cloud.Ctx()) as A:
10
    # apply on subarrays [1,2]x[1,4] and [4,4]x[1,4]
11
    res = A.apply(median, [[(1,2), 4], [slice(1,4)]], attrs = ["a"])
12
    print(res)
Copied!
All slices provided as input to theapply function are inclusive.
Apply Without Opening The Array
To execute an array UDF, it is not always necessary to have the array opened locally. An alternative function to apply the UDF on an array URI is provided.
Python
1
import tiledb, tiledb.cloud, numpy
2
​
3
def median(numpy_ordered_dictionary):
4
  return numpy.median(numpy_ordered_dictionary["a"])
5
​
6
tiledb.cloud.login(username="my_username", password="my_password")
7
# or tiledb.cloud.login(token="my_token")
8
​
9
# apply on subarrays [1,2]x[1,4] and [4,4]x[1,4]
10
res = tiledb.cloud.array.apply("tiledb://TileDB-Inc/quickstart_dense", median, [[(1,2), 4], [slice(1,4)]], attrs = ["a"])
11
print(res)
Copied!
Asynchronous Execution
An asynchronous version of the array UDFs is available. 
Python
1
import tiledb, tiledb.cloud, numpy, random
2
​
3
def median():
4
  vals = []
5
  for i in range(1, random.randrange(2,50)):
6
    vals.append(random.randrange(0, i))
7
  return numpy.median(vals)
8
​
9
tiledb.cloud.login(username="my_username", password="my_password")
10
# or tiledb.cloud.login(token="my_token")
11
​
12
​
13
with tiledb.open("tiledb://TileDB-Inc/quickstart_dense", ctx=tiledb.cloud.Ctx()) as A:
14
    # apply on subarrays [1,2]x[1,4] and [4,4]x[1,4]
15
    # res will be a future
16
    res = A.apply_async(median, [[(1,2), 4], [slice(1,4)]], attrs = ["a"])
17
​
18
    # call res.get() to block on the results
19
    print(res.get())
Copied!
Selecting Who to Charge
If you you are a member of an organization, then by default the organization is charged for your array UDF. If you would like to charge the array UDF to yourself, you just need to add an extra argument namespace. 
Python
1
import tiledb, tiledb.cloud, numpy
2
​
3
def median(numpy_ordered_dictionary):
4
  return numpy.median(numpy_ordered_dictionary["a"])
5
​
6
tiledb.cloud.login(username="my_username", password="my_password")
7
# or tiledb.cloud.login(token="my_token")
8
​
9
with tiledb.open("tiledb://TileDB-Inc/quickstart_dense", ctx=tiledb.cloud.Ctx()) as A:
10
    # apply on subarray [1,2]x[1,2]
11
    res = A.apply(median, [(1,2), (1,2)], attrs = ["a"], namespace="my_username")
12
    print(res)
Copied!
Registering Array UDFs
You can register an array UDF (similar to arrays) as follows:
Python
1
import tiledb, tiledb.cloud, numpy
2
​
3
def median(numpy_ordered_dictionary):
4
  return numpy.median(numpy_ordered_dictionary["a"])
5
​
6
tiledb.cloud.login(username="my_username", password="my_password")
7
# or tiledb.cloud.login(token="my_token")
8
​
9
tiledb.cloud.udf.register_single_array_udf(median, name="median_test", namespace="my_username")
Copied!
In order to be able to register a UDF you need to set up the default storage path for you and/or your organization.
Multi-Array UDFs
TileDB Cloud provides also multi-array UDFs, i.e., UDFs that are applied to more than one arrays. 
1
import numpy as np
2
import tiledb
3
import tiledb.cloud
4
​
5
array_1 = "tiledb://TileDB-Inc/array_1"
6
array_2 = "tiledb://TileDB-Inc/array_2"
7
​
8
def median(numpy_ordered_dictionary_list):
9
  # When you have multiple arrays, the parameter 
10
  # we pass in is actually a list of ordered dictionaries. 
11
  # The list is in the order of the arrays you asked for.
12
    return (
13
        np.median(numpy_ordered_dictionary[0]["a"] + numpy_ordered_dictionary[1]["a"])
14
    )
15
​
16
# The following will create the list of array to take part
17
# in the multi-array UDF. Each has as input the array name,
18
# a multi-index for slicing and a list of attributes to subselect on.
19
array_list = tiledb.cloud.array.ArrayList()    
20
array_list.add(array_1, [(1, 4), (1, 4)], ["a"])
21
array_list.add(array_2, [(1, 2), (1, 4)], ["a"])
22
​
23
# This will execute `median` using as input the result of the
24
# slicing and subselection for each of the arrays in `array_list` 
25
res = tiledb.cloud.array.exec_multi_array_udf(median, array_list)
26
​
27
print("Median Multi-array UDF:\n{}\n".format(res))
Copied!
You can register a multi-array UDF simply as follows:
Python
1
import tiledb, tiledb.cloud, numpy
2
​
3
def median(numpy_ordered_dictionary_list):
4
  # When you have multiple arrays, the parameter 
5
  # we pass in is actually a list of ordered dictionaries. 
6
  # The list is in the order of the arrays you asked for.
7
    return (
8
        np.median(numpy_ordered_dictionary[0]["a"] + numpy_ordered_dictionary[1]["a"])
9
    )
10
​
11
tiledb.cloud.login(username="my_username", password="my_password")
12
# or tiledb.cloud.login(token="my_token")
13
​
14
tiledb.cloud.udf.register_multi_array_udf(median, name="median_multi_array", namespace="my_username")
Copied!
Retry Settings
See Retry Settings.
API Reference - Previous
Serverless UDFs
Next - API Reference
Task Graphs
Last modified 8mo ago
Export as PDF
Copy link
Contents
Basic Usage
Multi-Index Usage
Apply Without Opening The Array
Asynchronous Execution
Selecting Who to Charge
Registering Array UDFs
Multi-Array UDFs
Retry Settings