
What is unidist?
unidist (Unified Distributed Execution) is a framework that is intended to provide the unified API for distributed execution by supporting various performant execution backends. At the moment the following backends are supported under the hood:
Also, the framework provides a Python Sequential backend (pyseq), that can be used for debugging.
unidist is designed to work in a task-based parallel model. The framework mimics Ray API and expands the existing frameworks (Ray and Dask Distributed) with additional features.
Quick Start Guide
Installation
To install the most recent stable release for unidist run the following:
pip install unidist[all] # Install unidist with dependencies for all the backends
For further instructions on how to install unidist with concrete execution backends or
using conda
see our Installation section.
Usage
The example below describes squaring the numbers from a list using unidist:
# script.py
if __name__ == "__main__":
import unidist
unidist.init() # Initialize unidist's backend. MPI is used by default.
@unidist.remote # Apply a decorator to make `foo` a remote function.
def foo(x):
return x * x
# This will run `foo` on a pool of workers in parallel;
# `refs` will contain object references to actual data.
refs = [foo.remote(i) for i in range(4)]
# Get materialized data.
print(unidist.get(refs)) # [0, 1, 4, 9]
Run the script.py with:
$ mpiexec -n 1 python script.py # for MPI backend
# python script.py # for any other supported backend
To get started with unidist refer to the getting started page.
To deep dive into unidist internals refer to the framework architecture.