General Specs

libEnsemble is primarily customized by setting options within a libE_specs dictionary or using the LibeSpecs class. When provided as a Python class, options are validated immediately on instantiation.

libE_specs = {
    "comm": MPI.COMM_WORLD,
    "comms": "mpi",
    "save_every_k_gens": 1000,
    "sim_dirs_make": True,
    "ensemble_dir_path": "/scratch/ensemble",
    "profile_worker": False,
}

Complete Class API

pydantic model libensemble.specs.LibeSpecs

Specifications for configuring libEnsemble’s runtime behavior. Equivalent to a libE_specs dictionary.

field abort_on_exception: bool | None = True: In MPI mode, whether to call MPI_ABORT on an exception. If False, an exception will be raised by the manager

field authkey: str | None = 'libE_auth_1015': TCP Only: Authkey for Manager’s system

field comms: str | None = 'mpi': Manager/Worker communications mode. 'mpi', 'local', or 'tcp'

field dedicated_mode: bool | None = False: Instructs libEnsemble to not run applications on resources where libEnsemble processes (manager and workers) are running

field disable_log_files: bool | None = False: Disable the creation of ensemble.log and libE_stats.txt log files

field disable_resource_manager: bool | None = False: Disable the built-in resource manager. If True, automatic resource detection and/or assignment of resources to workers is disabled. resource_info will also be ignored

field dry_run: bool | None = False: Whether libEnsemble should immediately exit after validating all inputs

field enforce_worker_core_bounds: bool | None = False: If False, the Executor will permit submission of tasks with a higher processor count than the CPUs available to the worker as detected by the resource manager. Larger node counts are not allowed. When "disable_resource_manager" is True, this argument is ignored

field ensemble_copy_back: bool | None = False: Whether to copy back directories within ensemble_dir_path back to launch location. Useful if ensemble directory placed on node-local storage

field ensemble_dir_path: str | Path | None = PosixPath('ensemble'): Path to main ensemble directory containing calculation directories. Can serve as single working directory for workers, or contain calculation directories

field final_fields: List[str] | None = []: List of fields in H that the manager will return to persistent workers along with the PERSIS_STOP tag at the end of a run

field gen_dir_copy_files: List[str | Path] | None = []: Paths to files or directories to copy into each generator or ensemble directory. List of strings or pathlib.Path objects

field gen_dir_symlink_files: List[str | Path] | None = []: Paths to files or directories to symlink into each generator directory. List of strings or pathlib.Path objects

field gen_dirs_make: bool | None = False: Whether to make generator-specific calculation directories for each generator function call. By default all workers operate within the top-level ensemble directory

field gen_input_dir: str | Path | None = None: Copy this directory and its contents for each generator-instance-specific directory. If not using calculation directories, contents are copied to the ensemble directory

field ip: str | None = None: TCP Only: IP address for Manager’s system

field kill_canceled_sims: bool | None = True: Instructs libEnsemble to send kill signals to sims with their cancel_requested field set. If False, the manager avoids this moderate overhead

field mpi_comm: MPI_Communicator | None = None: libEnsemble communicator. Default: MPI.COMM_WORLD

field num_resource_sets: int | None = None: Total number of resource sets. Resources will be divided into this number. If not set, resources will be divided evenly (excluding zero_resource_workers).

field nworkers: int | None = None: Number of worker processes to spawn (only in local/tcp modes)

field platform: str | None = ''

Name of a known platform defined in the platforms module.

See Known Platforms List

Example:

libE_specs["platform"] = "perlmutter_g"

Note: the environment variable LIBE_PLATFORM is an alternative way of setting.

E.g., on command line or batch submission script:

export LIBE_PLATFORM="perlmutter_g"

Known Platforms List

Platform Fields

Platform Fields

pydantic model libensemble.resources.platforms.Platform

Class to define attributes of a target platform.

All are optional, and any not defined will be determined by libEnsemble’s auto-detection.

field mpi_runner: str | None: MPI runner: One of "mpich", "openmpi", "aprun", "srun", "jsrun", "msmpi", "custom"

field runner_name: str | None

Literal string of MPI runner command. Only needed if different to the default

Note that "mpich" and "openmpi" runners have the default command "mpirun"

field cores_per_node: int | None: Number of physical CPU cores on a compute node of the platform

field logical_cores_per_node: int | None: Number of logical CPU cores on a compute node of the platform

field gpus_per_node: int | None: Number of GPU devices on a compute node of the platform

field gpu_setting_type: str | None

How GPUs will be assigned.

Must take one of the following string options.

"runner_default": Use default setting for MPI runner (same as if not set).
"env": Use an environment variable (comma separated list of slots)
"option_gpus_per_node": Expresses GPUs per node on MPI runner command line.
"option_gpus_per_task": Expresses GPUs per task on MPI runner command line.

With the exception of “runner_default”, the gpu_setting_name attribute is also required when this attribute is set.

If “gpu_setting_type” is not provided (same as runner_default) and the MPI runner does not have a default GPU setting in libEnsemble, and no other information is present, then the environment variable CUDA_VISIBLE_DEVICES is used.

Examples:

Use environment variable ROCR_VISIBLE_DEVICES to assign GPUs.

"gpu_setting_type" = "env"
"gpu_setting_name" = "ROCR_VISIBLE_DEVICES"

Use command line option --gpus-per-node

"gpu_setting_type" = "option_gpus_per_node"
"gpu_setting_name" = "--gpus-per-node"

field gpu_setting_name: str | None

Name of GPU setting

See gpu_setting_type for more details.

field scheduler_match_slots: bool | None

Whether the libEnsemble resource scheduler should only assign matching slots when there are multiple (partial) nodes assigned to a sim function.

Defaults to True, within libEnsemble.

Useful if setting an environment variable such as CUDA_VISIBLE_DEVICES, where the value should match on each node of an MPI run (choose True).

When using command-line options just as --gpus-per-node, which allow the systems application level scheduler to manager GPUs, then match_slots can be False (allowing for more efficient scheduling when MPI runs cross nodes).

Scheduler Options

See options for built-in scheduler.

Overriding Resource Auto-Detection

Note that "cores_on_node" and "gpus_on_node" are supported for backward compatibility, but use of platform_specs is recommended for these settings.

Formatting libE_stats.txt

The allowable libE_specs["stats_fmt"] fields are:

"task_timing" [bool] = ``False``:
    Outputs elapsed time for each task launched by the executor.

"task_datetime" [bool] = ``False``:
    Outputs the elapsed time and start and end time for each task launched by the executor.
    Can be used with the ``"plot_libe_tasks_util_v_time.py"`` to give task utilization plots.

"show_resource_sets" [bool] = ``False``:
    Shows the resource set IDs assigned to each worker for each call of the user function.