What is unidist?

unidist (Unified Distributed Execution) is a framework that is intended to provide the unified API for distributed execution by supporting various performant execution backends. At the moment the following backends are supported under the hood:

• MPI

• Dask Distributed

• Ray

• Python Multiprocessing

Also, the framework provides a Python Sequential backend (pyseq), that can be used for debugging.

unidist is designed to work in a task-based parallel model. The framework mimics Ray API and expands the existing frameworks (Ray and Dask Distributed) with additional features.

Quick Start Guide

Installation

To install the most recent stable release for unidist run the following:

pip install unidist[all] # Install unidist with dependencies for all the backends

For further instructions on how to install unidist with concrete execution backends or using conda see our Installation section.

Usage

The example below describes squaring the numbers from a list using unidist:

# script.py
if __name__ == "__main__":
   import unidist

   unidist.init() # Initialize unidist's backend. MPI is used by default.

   @unidist.remote # Apply a decorator to make `foo` a remote function.
   def foo(x):
      return x * x

   # This will run `foo` on a pool of workers in parallel;
   # `refs` will contain object references to actual data.
   refs = [foo.remote(i) for i in range(4)]

   # Get materialized data.
   print(unidist.get(refs)) # [0, 1, 4, 9]

Run the script.py with:

$ mpiexec -n 1 python script.py  # for MPI backend
# python script.py  # for any other supported backend

To get started with unidist refer to the getting started page.

To deep dive into unidist internals refer to the framework architecture.