Improv

Improv is an LCRC production cluster at Argonne National Laboratory with 825 dual-socket compute nodes with AMD 7713 64-core processors.

Installing libEnsemble and Dependencies

To create a conda environment and install libEnsemble:

module load anaconda3
conda create --name improv_libe_env python=3
conda activate improv_libe_env
pip install libensemble

See here for more information on advanced options for installing libEnsemble, including using Spack.

Job Submission

Improv uses the PBS scheduler to submit jobs from login nodes to run on compute nodes.

Example

To run the forces_simple tutorial on Improv.

To obtain the example you can git clone libEnsemble - although only the forces sub-directory is needed:

git clone https://github.com/Libensemble/libensemble
cd libensemble/libensemble/tests/scaling_tests/forces/forces_app

To compile forces:

module load gcc openmpi aocl
mpicc -O3 -o forces.x forces.c -lm

Now go to forces_simple directory:

cd ../forces_simple/

Now grab an interactive session on one node:

qsub -A <project_id> -l select=1:mpiprocs=64 -l walltime=20:00 -qdebug -I

Once in the interactive session, you may need to reload the modules:

cd $PBS_O_WORKDIR
module load anaconda3 gcc openmpi aocl
conda activate improv_libe_env

Now run forces with five workers (one for generator and four for simulations):

python run_libe_forces.py --comms local --nworkers 5

mpi4py comms

You can install mpi4py as usual having installed the Open-MPI module:

pip install mpi4py

Note if using mpi4py comms with Open-MPI, you may need to set export OMPI_MCA_coll_hcoll_enable=0 to prevent HCOLL warnings.