Local Object Store

MPI LocalObjectStore stores data in-process memory in a local dict. In depend on MpiSharedObjectStoreThreshold`, data can be stored in SharedObjectStore.

API

class unidist.core.backends.mpi.core.local_object_store.LocalObjectStore

Class that stores local objects and provides access to them.

Notes

The storage is local to the current worker process only.

cache_send_info(data_id, rank)

Save communication event for this data_id and rank.

Parameters:
cache_serialized_data(data_id, data)

Save serialized object for this data_id.

Parameters:
clear(cleanup_list)

Clear “strong” references to data IDs from cleanup_list.

Parameters:

cleanup_list (list) – List of data IDs.

Notes

The actual data will be collected later when there is no weak or strong reference to data in the current worker.

contains(data_id)

Check if the data associated with data_id exists in a local dictionary.

Parameters:

data_id (unidist.core.backends.mpi.core.common.MpiDataID) – An ID to data.

Returns:

Return the status if an object exist in local dictionary.

Return type:

bool

contains_data_owner(data_id)

Check if the data location info associated with data_id exists in a local dictionary.

Parameters:

data_id (unidist.core.backends.mpi.core.common.MpiDataID) – An ID to data.

Returns:

Return the True status if an object location is known.

Return type:

bool

generate_data_id(gc)

Generate unique MpiDataID instance.

Parameters:

gc (unidist.core.backends.mpi.core.executor.GarbageCollector) – Local garbage collector reference.

Returns:

Unique data ID instance.

Return type:

unidist.core.backends.mpi.core.common.MpiDataID

generate_output_data_id(dest_rank, gc, num_returns=1)

Generate unique list of unidist.core.backends.mpi.core.common.MpiDataID instance.

Parameters:

  • dest_rank (int) – Ranks number where generated list will be located.

  • gc (unidist.core.backends.mpi.core.executor.GarbageCollector) – Local garbage collector reference.

  • num_returns (int, default: 1) – Generated list size.

Returns:

A list of unique MpiDataID instances.

Return type:

list

get(data_id)

Get the data from a local dictionary.

Parameters:

data_id (unidist.core.backends.mpi.core.common.MpiDataID) – An ID to data.

Returns:

Return local data associated with data_id.

Return type:

object

get_data_owner(data_id)

Get the data owner rank.

Parameters:

data_id (unidist.core.backends.mpi.core.common.MpiDataID) – An ID to data.

Returns:

Rank number where the data resides.

Return type:

int

classmethod get_instance()

Get instance of LocalObjectStore.

Return type:

LocalObjectStore

get_serialized_data(data_id)

Get serialized data on this data_id.

Parameters:

data_id (unidist.core.backends.mpi.core.common.MpiDataID) – An ID to data.

Returns:

Cached serialized data associated with data_id.

Return type:

object

is_already_sent(data_id, rank)

Check if communication event on this data_id and rank happened.

Parameters:
Returns:

True if communication event already happened.

Return type:

bool

is_already_serialized(data_id)

Check if the data on this data_id is already serialized.

Parameters:

data_id (unidist.core.backends.mpi.core.common.MpiDataID) – An ID to data.

Returns:

True if the data is already serialized.

Return type:

bool

maybe_update_data_id_map(data_id)

Add a strong reference to the data_id if necessary.

Parameters:

data_id (unidist.core.backends.mpi.core.common.MpiDataID) – An ID to data.

Notes

The worker must have a strong reference to the external data_id until the owner process send the unidist.core.backends.common.Operation.CLEANUP operation with this data_id.

put(data_id, data)

Put data to internal dictionary.

Parameters:
put_data_owner(data_id, rank)

Put data location (owner rank) to internal dictionary.

Parameters: