Unidist Configuration Settings

Using these configuration settings, the user can tune unidist’s behavior. Below you can find configuration settings currently provided by unidist.

Public API

Potentially, the source of configuration settings can be any, but for now only environment variables are implemented. Any environment variable originates from EnvironmentVariable, which contains most of the config API implementation.

class unidist.config.parameter.EnvironmentVariable

Base class for environment variables-based configuration.

classmethod get()

Get config value.

Returns:

Decoded and verified config value.

Return type:

Any

classmethod put(value)

Set config value.

Parameters:

value (Any) – Config value to set.

Unidist Configuration Settings List

Config Name	Env. Variable Name	Description
Backend	UNIDIST_BACKEND	Distribution backend to run queries by
CpuCount	UNIDIST_CPUS	How many CPU cores to use during initialization of the unidist backend
RayGpuCount	UNIDIST_RAY_GPUS	How many GPU devices to use during initialization of the Ray backend
IsRayCluster	UNIDIST_RAY_CLUSTER	Whether Ray is running on pre-initialized Ray cluster
RayRedisAddress	UNIDIST_RAY_REDIS_ADDRESS	Redis address to connect to when running in Ray cluster
RayRedisPassword	UNIDIST_RAY_REDIS_PASSWORD	What password to use for connecting to Redis
RayObjectStoreMemory	UNIDIST_RAY_OBJECT_STORE_MEMORY	How many bytes of memory to start the Ray object store with
DaskMemoryLimit	UNIDIST_DASK_MEMORY_LIMIT	How many bytes of memory that Dask worker should use
IsDaskCluster	UNIDIST_DASK_CLUSTER	Whether Dask is running on pre-initialized Dask cluster
DaskSchedulerAddress	UNIDIST_DASK_SCHEDULER_ADDRESS	Dask Scheduler address to connect to when running in Dask cluster
MpiSpawn	UNIDIST_MPI_SPAWN	Whether to enable MPI spawn or not
MpiHosts	UNIDIST_MPI_HOSTS	MPI hosts to run unidist on
MpiPickleThreshold	UNIDIST_MPI_PICKLE_THRESHOLD	Minimum buffer size for serialization with pickle 5 protocol
MpiBackoff	UNIDIST_MPI_BACKOFF	Backoff time for preventing the “busy wait” in loops exchanging messages
MpiLog	UNIDIST_MPI_LOG	Whether to enable logging for MPI backend or not
MpiSharedObjectStore	UNIDIST_MPI_SHARED_OBJECT_STORE	Whether to enable shared object store or not
MpiSharedObjectStoreMemory	UNIDIST_MPI_SHARED_OBJECT_STORE_MEMORY	How many bytes of memory to start the shared object store with
MpiSharedServiceMemory	UNIDIST_MPI_SHARED_SERVICE_MEMORY	How many bytes of memory to start the shared service memory with
MpiSharedObjectStoreThreshold	UNIDIST_MPI_SHARED_OBJECT_STORE_THRESHOLD	Minimum size of data to put into the shared object store
MpiRuntimeEnv	Only the config API is available	Runtime environment for MPI worker processes

Usage Guide

As it can be seen below a config value can be set either by setting the environment variable or by using config API.

import os

# Setting `UNIDIST_BACKEND` environment variable.
# Also can be set outside the script.
os.environ["UNIDIST_BACKEND"] = "mpi"

import unidist.config as cfg

# Checking initially set `Backend` config,
# which corresponds to `UNIDIST_BACKEND` environment variable
print(cfg.Backend.get()) # prints 'mpi'

# Checking default value of `CpuCount`
print(cfg.CpuCount.get()) # prints the number of CPUs on your machine

# Changing value of `CpuCount`
cfg.CpuCount.put(16)
print(cfg.CpuCount.get()) # prints '16'

Note

Make sure that setting configuration values happens before unidist initialization (init() call)! Otherwise, unidist will opt for the default settings.