Platform Specs

libEnsemble detects platform specifications including MPI runners and resources. Usually this will result in the correct settings. However, users can configure platform specifications via the platform_specs option or indicate a known platform via the platform option.

platform_specs

A Platform object or dictionary specifying settings for a platform.

To define a platform (in calling script):

Platform Object

from libensemble.resources.platforms import Platform

libE_specs["platform_specs"] = Platform(
    mpi_runner="srun",
    cores_per_node=64,
    logical_cores_per_node=128,
    gpus_per_node=8,
    gpu_setting_type="runner_default",
    gpu_env_fallback="ROCR_VISIBLE_DEVICES",
    scheduler_match_slots=False,
)

Dictionary

libE_specs["platform_specs"] = {
    "mpi_runner": "srun",
    "cores_per_node": 64,
    "logical_cores_per_node": 128,
    "gpus_per_node": 8,
    "gpu_setting_type": "runner_default",
    "gpu_env_fallback": "ROCR_VISIBLE_DEVICES",
    "scheduler_match_slots": False,
}

The list of platform fields is given below. Any fields not given will be auto-detected by libEnsemble.

Platform Fields

pydantic model libensemble.resources.platforms.Platform

Class to define attributes of a target platform.

All are optional, and any not defined will be determined by libEnsemble’s auto-detection.

Config:

arbitrary_types_allowed: bool = True
populate_by_name: bool = True
extra: str = forbid
validate_assignment: bool = True

field mpi_runner: str | None: MPI runner: One of "mpich", "openmpi", "aprun", "srun", "jsrun", "msmpi", "custom"

field runner_name: str | None

Literal string of MPI runner command. Only needed if different to the default

Note that "mpich" and "openmpi" runners have the default command "mpirun"

field cores_per_node: int | None: Number of physical CPU cores on a compute node of the platform

field logical_cores_per_node: int | None: Number of logical CPU cores on a compute node of the platform

field gpus_per_node: int | None: Number of GPU devices on a compute node of the platform

field tiles_per_gpu: int | None: Number of tiles on a GPU

field gpu_setting_type: str | None

How GPUs will be assigned.

Must take one of the following string options.

"runner_default": Use default setting for MPI runner (same as if not set).
"env": Use an environment variable (comma-separated list of slots)
"option_gpus_per_node": Expresses GPUs per node on MPI runner command line.
"option_gpus_per_task": Expresses GPUs per task on MPI runner command line.

With the exception of “runner_default”, the gpu_setting_name attribute is also required when this attribute is set.

If “gpu_setting_type” is not provided (same as runner_default) and the MPI runner does not have a default GPU setting in libEnsemble, and no other information is present, then the environment variable CUDA_VISIBLE_DEVICES is used.

Examples:

Use environment variable ROCR_VISIBLE_DEVICES to assign GPUs.

"gpu_setting_type" = "env"
"gpu_setting_name" = "ROCR_VISIBLE_DEVICES"

Use command line option --gpus-per-node

"gpu_setting_type" = "option_gpus_per_node"
"gpu_setting_name" = "--gpus-per-node"

field gpu_setting_name: str | None

Name of GPU setting

See gpu_setting_type for more details.

field gpu_env_fallback: str | None

GPU fallback environment setting if not using an MPI runner.

For example:

"gpu_setting_type" = "runner_default"
"gpu_env_fallback" = "ROCR_VISIBLE_DEVICES"

This example will use the MPI runner default settings when using an MPI runner, but will otherwise use ROCR_VISIBLE_DEVICES (e.g., if setting via function set_env_to_gpus).

If this is not set, the default is “CUDA_VISIBLE_DEVICES”.

field scheduler_match_slots: bool | None

Whether the libEnsemble resource scheduler should only assign matching slots when there are multiple (partial) nodes assigned to a sim function.

Defaults to True, within libEnsemble.

Useful if setting an environment variable such as CUDA_VISIBLE_DEVICES, where the value should match on each node of an MPI run (choose True).

When using command-line options just as --gpus-per-node, which allow the system’s application-level scheduler to manage GPUs, then match_slots can be False (allowing for more efficient scheduling when MPI runs cross nodes).

To use an existing platform:

from libensemble.resources.platforms import PerlmutterGPU

libE_specs["platform_specs"] = PerlmutterGPU()

See known platforms.

platform

A string giving the name of a known platform defined in the platforms module.

libE_specs["platform"] = "perlmutter_g"

Note: the environment variable LIBE_PLATFORM is an alternative way of setting.

E.g., in the command line or batch submission script:

export LIBE_PLATFORM="perlmutter_g"

Platform Specs

platform_specs

platform

Known Platforms List