Python Multiprocessing High-level API

Python Multiprocessing API module provides high-level functions for initialization of the backend, for working with object storage and submitting tasks.

API

Function is_initialized() allows to check if the execution backend has been initialized.

unidist.core.backends.pymp.core.api.is_initialized()

Check if Python Multiprocessing backend has already been initialized.

Returns:: True or False.
Return type:: bool

Function init() creates instances of singleton classes ObjectStore and ProcessManager.

unidist.core.backends.pymp.core.api.init(num_workers=2)

Initialize shared object storage and workers pool.

Parameters:: num_workers (int, default: number of CPUs) – Number of worker-processes to start.

Notes

Run initialization of singleton objects unidist.core.backends.pymp.core.object_store.ObjectStore and unidist.core.backends.pymp.core.process_manager.ProcessManager.

Functions get() and put() are responsible for read/write, respectively, objects from/to ObjectStore. Both of the functions block execution until read/write finishes.

unidist.core.backends.pymp.core.api.get(data_ids)

Get a object(s) associated with data_ids from the shared object storage.

Parameters:: data_ids (unidist.core.backends.common.data_id.DataID or list) – An ID(s) to object(s) to get data from.
Returns:: A Python object.
Return type:: object

unidist.core.backends.pymp.core.api.put(data)

Put data into shared object storage.

Parameters:: data (object) – Data to be put.
Returns:: An ID of object in shared object storage.
Return type:: unidist.core.backends.common.data_id.DataID

wait() carries out blocking of execution until a requested number of DataID isn’t ready.

unidist.core.backends.pymp.core.api.wait(data_ids, num_returns=1)

Wait until data_ids are finished.

This method returns two lists. The first list consists of DataID-s that correspond to objects that completed computations. The second list corresponds to the rest of the DataID-s (which may or may not be ready).

Parameters:

data_ids (unidist.core.backends.common.data_id.DataID or list) – DataID or list of DataID-s to be waited.
num_returns (int, default: 1) – The number of DataID-s that should be returned as ready.

Returns:

List of data IDs that are ready and list of the remaining data IDs.

Return type:

tuple

submit() wraps an operation to Task and adds it to task queue of one of the workers. Specific worker will be chosen by ProcessManager.

unidist.core.backends.pymp.core.api.submit(func, *args, num_returns=1, **kwargs)

Execute function in a worker process.

Parameters:

func (callable) – Function to be executed in the worker.
*args (iterable) – Positional arguments to be passed in the func.
num_returns (int, default: 1) – Number of results to be returned from func.
**kwargs (dict) – Keyword arguments to be passed in the func.

Returns:

Type of returns depends on num_returns value:

if num_returns == 1, DataID will be returned.
if num_returns > 1, list of DataID-s will be returned.
if num_returns == 0, None will be returned.

Return type:

unidist.core.backends.common.data_id.DataID, list or None