Shared Object Storage

Python Multiprocessing ObjectStore stores shared between processes data in a dict implemented using Python multiprocessing.Manager.

class unidist.core.backends.pymp.core.object_store.ObjectStore

Class that stores objects and provides access to these from different processes.

Notes

Shared storage is organized using multiprocessing.Manager.dict. This is separate process which starts work in the class constructor.

get(data_ids)

Get a object(s) associated with data_ids from the shared internal dictionary.

Parameters:: data_ids (unidist.core.backends.common.data_id.DataID or list) – An ID(s) of object(s) to get data from.
Returns:: A Python object.
Return type:: object

classmethod get_instance()

Get instance of ObjectStore.

Return type:: unidist.core.backends.pymp.core.object_store.ObjectStore

put(data, data_id=None)

Put data to internal shared dictionary.

Parameters:

data (object) – Data to be put.
data_id (unidist.core.backends.common.data_id.DataID, optional) – An ID to data. If it isn’t provided, will be created automatically.

Returns:

An ID of object in internal shared dictionary.

Return type:

unidist.core.backends.common.data_id.DataID

wait(data_ids, num_returns=1)

Wait until data_ids are finished.

This method returns two lists. The first list consists of DataID-s that correspond to objects that completed computations. The second list corresponds to the rest of the DataID-s (which may or may not be ready).

Parameters:

data_ids (unidist.core.backends.common.data_id.DataID or list) – DataID or list of DataID-s to be waited.
num_returns (int, default: 1) – The number of DataID-s that should be returned as ready.

Returns:

List of data IDs that are ready and list of the remaining data IDs.

Return type:

tuple