MPI backend

We highly encourage to use external (either custom-built or system-provided) MPI installations in production scenarious to get ultimate performance.

Open Source MPI implementaions

Building MPI from source

In Building MPI from source section of mpi4py documentation you can find executive instructions for building some of the open-source MPI implementations out there with support for shared/dynamic libraries on POSIX environments.

Once you have a working MPI implementation, you will have to adapt your PATH environment variable to use the installed MPI version.

export PATH=path/to/mpi/bin:$PATH

Then, you can install unidist and the required dependencies for the MPI backend.

pip install unidist[mpi]

Now you can use unidist on MPI backend using the installed MPI implementaion.

Proprietary MPI implementations

Intel MPI From Intel oneAPI HPC Toolkit

The following instructions will help you install Intel MPI from Intel oneAPI HPC Toolkit to use it as the unidist’s backend. We will use an offline installer an an example but you are free to use other installation options.

  1. Create a directory for installing Intel MPI and go into it. You can do this in a terminal by typing

mkdir local
cd local

  1. Download a toolkit installer from Intel oneAPI HPC Toolkit, e.g., using wget command

wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/1ff1b38a-8218-4c53-9956-f0b264de35a4/l_HPCKit_p_2023.1.0.46346_offline.sh

Note that we use the specific version of the toolkit as an example. You can install any version you want.

  1. Launch the installer

sh ./l_HPCKit_p_2023.1.0.46346_offline.sh

During installation process you can choose a directory in which the toolkit should be installed (e.g., path/to/local/<toolkit>).

  1. Source the setvars.sh (global to the toolkit) or the vars.sh (local to the Intel MPI)

# source path/to/local/<toolkit>/oneapi/setvars.sh
source path/to/local/<toolkit>/oneapi/mpi/latest/env/vars.sh

  1. Install unidist and the required dependencies for the MPI backend

pip install unidist[mpi]

Now you can use unidist on MPI backend using Intel MPI implementaion.

  1. Remove the installer (optional):

rm l_HPCKit_p_2023.1.0.46346_offline.sh