Workers related functionality API
Worker
Python Multiprocessing Worker
represents a Python process that has a task queue. Tasks from the queue are run sequentially.
API
- class unidist.core.backends.pymp.core.process_manager.Worker(task_queue, obj_store)
Class-process that executes tasks from self.task_queue.
- Parameters:
task_queue (multiprocessing.JoinableQueue) – A queue of task to execute.
obj_store (unidist.core.backends.pymp.core.object_store.ObjectStore) – Shared object storage to read/write data.
- add_task(task)
Add task to self.task_queue.
- Parameters:
task (unidist.core.backends.pymp.core.process_manager.Task) – Task to be added in the queue.
- run()
Run main infinite loop of process to execute tasks from self.task_queue.
Task
Python Multiprocessing Task
is an object-wrapper for a free functions and actor methods.
API
- class unidist.core.backends.pymp.core.process_manager.Task(func, data_ids, obj_store, *args, **kwargs)
Class poses as unified callable object to execute in Multiprocessing Worker.
- Parameters:
func (callable) – A function to be called in object invocation.
data_ids (unidist.core.backends.common.data_id.DataID or list) –
DataID
-(s) associated with result(s) of func invocation.obj_store (unidist.core.backends.pymp.core.object_store.ObjectStore) – Object storage to share data between workers.
*args (iterable) – Positional arguments to be passed in the func.
**kwargs (dict) – Keyword arguments to be passed in the func.
ProcessManager
Python Multiprocessing ProcessManager
schedules operations (Task
objects)
to free Worker
-s by round-robin algorithm.
Free Worker
is a worker, that isn’t used
by Python Multiprocessing Actor
.
API
- class unidist.core.backends.pymp.core.process_manager.ProcessManager(num_workers=None)
Class that controls worker pool and assings task to workers.
- Parameters:
num_workers (int, optional) – Number of worker-processes to start. If isn’t provided, will be equal to number of CPUs.
Notes
Constructor starts num_workers Multiprocessing Workers.
- free_worker(idx)
Free worker by index idx.
- Parameters:
idx (int) – Index of worker to be freed.
- classmethod get_instance(num_workers=None)
Get instance of
ProcessManager
.
- grab_worker()
Grab a worker from worker pool.
Grabbed worker is marked as blocked and doesn’t participate in the tasks submission.
- Returns:
unidist.core.backends.pymp.core.process_manager.Worker – Grabbed worker.
int – Index of grabbed worker.
- submit(task)
Add task to task queue of one of workers using round-robin.
- Parameters:
task (unidist.core.backends.pymp.core.process_manager.Task) – Task to be added in task queue.