Task Graphs

Generic Functions
Any Python function can be wrapped in a Delayed object making the function executable as a future.
Python
Python
from tiledb.cloud.compute import Delayed
import numpy
​
# Wrap numpy median in a delayed object
x = Delayed(numpy.median)
​
# It can be called like a normal function to set the parameters
# Note at this point the function does not get executed since it
# is of "delayed" type
x([1,2,3,4,5])
​
# To force execution and get the result call `compute()`
print(x.compute())
SQL and Arrays
Besides arbitrary Python functions, serverless SQL queries and array UDFs can also be called with the delayed API.
Python
Python
from tiledb.cloud.compute import DelayedSQL, DelayedArrayUDF
import numpy
​
# SQL
y = DelayedSQL("select AVG(`a`) FROM `tiledb://TileDB-Inc/quickstart_sparse`")
​
# Run query
print(y.compute())
​
# Array
z = DelayedArrayUDF("tiledb://TileDB-Inc/quickstart_sparse",
        lambda x: numpy.average(x["a"]))([(1, 4), (1, 4)])
​
# Run the UDF on the array
z.compute()
Local Functions
It is also possible to include a generic Python function as delayed, but have it run locally instead of serverless on TileDB Cloud. This is useful for testing or for saving finalized results to your local machine, e.g., saving an image.
Python
Python
from tiledb.cloud.compute import Delayed
import numpy
​
# Set the `local` argument to `True`
local = Delayed(numpy.median, local=True)([1,2,3])
​
# This will compute locally
local.compute()
Task Graphs
Delayed objects can be combined into a task graph, which is typically a directed acyclic graph (DAG). The output from one function or query can be passed into another, and dependencies are automatically determined.
Python
Python
from tiledb.cloud.compute import DelayedArrayUDF, Delayed, DelayedSQL
import numpy
​
# Build several delayed objects to define a graph
# Note that package numpy is aliased as np in the UDFs
local = Delayed(lambda x: x * 2, local=True)(100)
array_apply = DelayedArrayUDF("tiledb://TileDB-Inc/quickstart_sparse", 
        lambda x: np.sum(x["a"]), name="array_apply")([(1, 4), (1, 4)])
sql = DelayedSQL("select SUM(`a`) as a from `tiledb://TileDB-Inc/quickstart_dense`"), name="sql")
​
# Custom function for averaging all the results we are passing in
def mean(local, array_apply, sql):
    return np.mean([local, array_apply, sql.iloc(0)[0]])
​
# This is essentially a task graph that looks like
#                 mean
#          /       |      \
#         /        |       \    
#      local  array_apply  sql 
#
# The `local`, `array_apply` and `sql` tasks will computed first,
# and once all three are finished, `mean` will computed on their results
res = Delayed(func_exec=mean, name="node_exec")(local, array_apply, sql)
print(res.compute())
Visualization
Any task graph created using the delayed API can be visualized with visualize(). The graph will be auto-updated by default as the computation progresses. If you wish to disable auto-updating, then simply set auto_update=False as a parameter to visualize(). If you are inside a Jupyter notebook, the graph will render as a widget. If you are not on the notebook, you can set notebook=False as a parameter to render in a normal Python window.
Python
Python
res.visualize()
Manually Setting Task Dependencies
There are cases where you might have one function to depend on another without using its results directly. A common case is when one function manipulates data stored somewhere else (on S3 or a database). To facilitate this, we provide function depends_on.
Python
Python
# A few base functions:
import random
from tiledb.cloud.compute import Delayed
​
# Set three initial nodes
node_1 = Delayed(numpy.median, local=True, name="node_1")([1, 2, 3])
node_2 = Delayed(lambda x: x * 2, local=True, name="node_2")(node_1)
node_3 = Delayed(lambda x: x * 2, local=True, name="node_3")(node_2)
​
# Create a dictionary to hold the nodes so we can ranodmly pick dependencies
nodes_by_name= {'node_1': node_1, 'node_2': node_2, 'node_3': node_3}
​
#Function which sleeps for some time so we can see the graph in different states
def f():
    import time
    import random
    time.sleep(random.randrange(0, 30))
    return x
​
# Randomly add 96 other nodes to the graph. All of these will use the sleep function
for i in range(4, 100):
    name = "node_{}".format(i)
    node = Delayed(f, local=True, name=name)()
    
    dep = random.randrange(1, i-1)
    # Randomly set dependency on one other node
    node_dep = nodes_by_name["node_{}".format(dep)]
    # Force the dependency to be set
    node.depends_on(node_dep)
    
    nodes_by_name[name] = node
​
# You can call visualize on any member node and see the whole graph
node_1.visualize()
​
# Get the last function's results
node_99 = nodes_by_name["node_99"]
node_99.compute()
The above code, after the call to node_1.visualize(), produces a task graph similar to that shown below:
Selecting Who to Charge
If you you are a member of an organization, then by default the organization is changed for your Delayed tasks. If you would like to charge the task to yourself, you just need to add one extra argument namespace. 
Python
Python
import tiledb.cloud
​
tiledb.cloud.login(username="my_username", password="my_password")
# or tiledb.cloud.login(token="my_token")
​
res = DelayedSQL("select `rows`, AVG(a) as avg_a from `tiledb://TileDB-Inc/quickstart_dense` GROUP BY `rows`"
      namespace ="my_username", # who to charge the query to
      )
You can also set who to charge for the entire task graph instead of individual Delayed objects. This is often useful when building a large task graph to avoid having to set the extra parameter on every object. Taking the example above, you just pass namespace="my_username" to the compute call.
Python
Python
import tiledb.cloud.compute
import numpy
​
# Build several delayed objects to define a graph
local = Delayed(lambda x: x * 2, local=True)(100)
array_apply = DelayedArrayUDF("tiledb://TileDB-Inc/quickstart_sparse", 
        lambda x: numpy.sum(x["a"]), name="array_apply")([(1, 4), (1, 4)])
sql = DelayedSQL("select SUM(`a`) as a from `tiledb://TileDB-Inc/quickstart_dense`"), name="sql")
​
# Custom function to use to average all the results we are passing in
def mean(local, array_apply, sql):
    import numpy
    return numpy.mean([local, array_apply, sql.iloc(0)[0]])
​
res = Delayed(func_exec=mean, name="node_exec")(local, array_apply, sql)
​
# Set all tasks to run under your username
print(res.compute(namespace="my_username"))
API Reference - Previous
Serverless Array UDFs
Next - API Reference
Utilities
Last updated 4 months ago