DaskBackend

The class is specific implementation of Backend interface using Dask.

API

class unidist.core.backends.dask.backend.DaskBackend

The class that implements the interface in Backend using Dask.

static cluster_resources()

Get resources of Dask cluster.

Returns:

Dictionary with cluster nodes info in the form {“node_ip0”: {“CPU”: x0}, “node_ip1”: {“CPU”: x1}, …}.

Return type:

dict

static get(futures)

Get a remote object or a list of remote objects from distributed memory.

Parameters:

futures (distributed.client.Future or a list of distributed.client.Future-s) – Dask Future or a list of Dask Future objects to get data from.

Returns:

A Python object or a list of Python objects.

Return type:

object

static get_ip()

Get node IP address.

Returns:

Node IP address.

Return type:

str

static is_initialized()

Check if Dask backend has already been initialized.

Returns:

True or False.

Return type:

bool

static make_actor(cls, num_cpus, resources)

Define an actor class.

clsobject

Class to be an actor class.

num_cpusint

The number of CPUs to reserve for the lifetime of the actor.

resourcesdict

Custom resources to reserve for the lifetime of the actor.

Returns:

• DaskActor – The actor class type to create.

• list – The list of arguments for DaskActor constructor.

static make_remote_function(function, num_cpus, num_returns, resources)

Define a remote function.

functioncallable

Function to be a remote function.

num_cpusint

The number of CPUs to reserve for the remote function.

num_returnsint

The number of ObjectRef-s returned by the remote function invocation.

resourcesdict

Custom resources to reserve for the remote function.

Return type:

DaskRemoteFunction

static num_cpus()

Get the number of CPUs used by the execution backend.

Return type:

int

static put(data)

Put data into distributed memory.

Parameters:

data (object) – Data to be put.

Returns:

Dask Future matching to data.

Return type:

distributed.client.Future

static shutdown()

Shutdown Dask execution backend.

static wait(futures, num_returns=1)

Wait until futures are finished.

This method returns two lists. The first list consists of futures that correspond to objects that completed computations. The second list corresponds to the rest of the futures (which may or may not be ready).

Parameters:

• futures (distributed.client.Future or list) – Dask Future or list of Dask Futures to be waited.

• num_returns (int, default: 1) – The number of Dask Futures that should be returned as ready.

Returns:

Two lists of futures that are ready and list of the remaining futures.

Return type:

tuple