Summit is an IBM AC922 system located at the Oak Ridge Leadership Computing Facility (OLCF). Each of the approximately 4,600 compute nodes on Summit contains two IBM POWER9 processors and six NVIDIA Volta V100 accelerators.

Summit features three tiers of nodes: login, launch, and compute nodes.

Users on login nodes submit batch runs to the launch nodes. Batch scripts and interactive sessions run on the launch nodes. Only the launch nodes can submit MPI runs to the compute nodes via jsrun.

Configuring Python

Begin by loading the Python 3 Anaconda module:

$ module load python

You can now create and activate your own custom conda environment:

conda create --name myenv python=3.9
export PYTHONNOUSERSITE=1 # Make sure get python from conda env
. activate myenv

If you are installing any packages with extensions, ensure that the correct compiler module is loaded. If using mpi4py, this must be installed from source, referencing the compiler. Currently, mpi4py must be built with gcc:

module load gcc

With your environment activated, run

CC=mpicc MPICC=mpicc pip install mpi4py --no-binary mpi4py

Installing libEnsemble

Obtaining libEnsemble is now as simple as pip install libensemble. Your prompt should be similar to the following line:

(my_env) user@login5:~$ pip install libensemble


If you encounter pip errors, run python -m pip install --upgrade pip first

Or, you can install via conda:

(my_env) user@login5:~$ conda config --add channels conda-forge
(my_env) user@login5:~$ conda install -c conda-forge libensemble

See here for more information on advanced options for installing libEnsemble.

Special note on resource sets and Executor submit options

When using the portable MPI run configuration options (e.g., num_nodes) to the MPIExecutor submit function, it is important to note that, due to the resource sets used on Summit, the options refer to resource sets as follows:

  • num_procs (int, optional) – The total number resource sets for this run.

  • num_nodes (int, optional) – The number of nodes on which to submit the run.

  • procs_per_node (int, optional) – The number of resource sets per node.

It is recommended that the user defines a resource set as the minimal configuration of CPU cores/processes and GPUs. These can be added to the extra_args option of the submit function. Alternatively, the portable options can be ignored and everything expressed in extra_args.

For example, the following jsrun line would run three resource sets, each having one core (with one process), and one GPU, along with some extra options:

jsrun -n 3 -a 1 -g 1 -c 1 --bind=packed:1 --smpiargs="-gpu"

To express this line in the submit function may look something like the following:

exctr = Executor.executor
task = exctr.submit(app_name="mycode",
                    extra_args="-a 1 -g 1 -c 1 --bind=packed:1 --smpiargs="-gpu""
                    app_args="-i input")

This would be equivalent to:

exctr = Executor.executor
task = exctr.submit(app_name="mycode",
                    extra_args="-n 3 -a 1 -g 1 -c 1 --bind=packed:1 --smpiargs="-gpu""
                    app_args="-i input")

The libEnsemble resource manager works out the resources available to each worker, but unlike some other systems, jsrun on Summit dynamically schedules runs to available slots across and within nodes. It can also queue tasks. This allows variable size runs to easily be handled on Summit. If oversubscription to the jsrun system is desired, then libEnsemble’s resource manager can be disabled in the calling script via:

libE_specs["disable_resource_manager"] = True

In the above example, the task being submitted used three GPUs, which is half those available on a Summit node, and thus two such tasks may be allocated to each node (from different workers), if they were running at the same time.

Job Submission

Summit uses LSF for job management and submission. For libEnsemble, the most important command is bsub for submitting batch scripts from the login nodes to execute on the launch nodes.

It is recommended to run libEnsemble on the launch nodes (assuming workers are submitting MPI applications) using the local communications mode (multiprocessing).

Interactive Runs

You can run interactively with bsub by specifying the -Is flag, similarly to the following:

$ bsub -W 30 -P [project] -nnodes 8 -Is

This will place you on a launch node.


You will need to reactivate your conda virtual environment.

Batch Runs

Batch scripts specify run settings using #BSUB statements. The following simple example depicts configuring and launching libEnsemble to a launch node with multiprocessing. This script also assumes the user is using the parse_args() convenience function from libEnsemble’s tools module.

#!/bin/bash -x
#BSUB -P <project code>
#BSUB -J libe_mproc
#BSUB -W 60
#BSUB -nnodes 128
#BSUB -alloc_flags "smt1"

# --- Prepare Python ---

# Load conda module and gcc.
module load python
module load gcc

# Name of conda environment
export CONDA_ENV_NAME=my_env

# Activate conda environment
source activate $CONDA_ENV_NAME

# --- Prepare libEnsemble ---

# Name of calling script

# Communication Method
export COMMS="--comms local"

# Number of workers.
export NWORKERS="--nworkers 128"

hash -r # Check no commands hashed (pip/python...)

# Launch libE
python $EXE $COMMS $NWORKERS > out.txt 2>&1

With this saved as, allocating, configuring, and queueing libEnsemble on Summit is achieved by running

$ bsub

Example submission scripts are also given in the examples.

Launching User Applications from libEnsemble Workers

Only the launch nodes can submit MPI runs to the compute nodes via jsrun. This can be accomplished in user sim_f functions directly. However, it is highly recommended that the Executor interface be used inside the sim_f or gen_f, because this provides a portable interface with many advantages including automatic resource detection, portability, launch failure resilience, and ease of use.

Additional Information

See the OLCF guides for more information about Summit.