MPI Executor - MPI apps
This module launches and controls the running of MPI applications.
In order to create an MPI executor, the calling script should contain:
exctr = MPIExecutor()
The MPIExecutor will use system resource information supplied by the libEnsemble resource manager when submitting tasks.
See this example for usage.
- class libensemble.executors.mpi_executor.MPIExecutor(custom_info={})
Bases:
Executor
The MPI executor can create, poll and kill runnable MPI tasks
- Parameters:
custom_info (dict, Optional) – Provide custom overrides to selected variables that are usually auto-detected. See below.
custom_info usage
The MPIExecutor automatically detects MPI runners and launch mechanisms. However it is possible to override the detected information using the
custom_info
argument. This takes a dictionary of values.The allowable fields are:
'mpi_runner' [string]: Select runner: 'mpich', 'openmpi', 'aprun', 'srun', 'jsrun', 'custom' All except 'custom' relate to runner classes in libEnsemble. Custom allows user to define their own run-lines but without parsing arguments or making use of auto-resources. 'runner_name' [string]: Runner name: Replaces run command if present. All runners have a default except for 'custom'. 'subgroup_launch' [bool]: Whether MPI runs should be initiated in a new process group. This needs to be correct for kills to work correctly. Use the standalone test at libensemble/tests/standalone_tests/kill_test to determine correct value for a system.
For example:
customizer = {'mpi_runner': 'mpich', 'runner_name': 'wrapper -x mpich'} from libensemble.executors.mpi_executor import MPIExecutor exctr = MPIExecutor(custom_info=customizer)
- submit(calc_type=None, app_name=None, num_procs=None, num_nodes=None, procs_per_node=None, num_gpus=None, machinefile=None, app_args=None, stdout=None, stderr=None, stage_inout=None, hyperthreads=False, dry_run=False, wait_on_start=False, extra_args=None, auto_assign_gpus=False, match_procs_to_gpus=False, env_script=None, mpi_runner_type=None)
Creates a new task, and either executes or schedules execution.
The created
task
object is returned.The user must supply either the app_name or calc_type arguments (app_name is recommended). All other arguments are optional.
- Parameters:
calc_type (str, Optional) – The calculation type: ‘sim’ or ‘gen’ Only used if app_name is not supplied. Uses default sim or gen application.
app_name (str, Optional) – The application name. Must be supplied if calc_type is not.
num_procs (int, Optional) – The total number of processes (MPI ranks)
num_nodes (int, Optional) – The number of nodes
procs_per_node (int, Optional) – The processes per node
num_gpus (int, Optional) – The total number of GPUs
machinefile (str, Optional) – Name of a machinefile
app_args (str, Optional) – A string of the application arguments to be added to task submit command line
stdout (str, Optional) – A standard output filename
stderr (str, Optional) – A standard error filename
stage_inout (str, Optional) – A directory to copy files from; default will take from current directory
hyperthreads (bool, Optional) – Whether to submit MPI tasks to hyperthreads
dry_run (bool, Optional) – Whether this is a dry_run - no task will be launched; instead runline is printed to logger (at INFO level)
wait_on_start (bool, Optional) – Whether to wait for task to be polled as RUNNING (or other active/end state) before continuing.
extra_args (str, Optional) – Additional command line arguments to supply to MPI runner. If arguments are recognized as MPI resource configuration (num_procs, num_nodes, procs_per_node) they will be used in resources determination unless also supplied in the direct options.
auto_assign_gpus (bool, Optional) – Auto-assign GPUs available to this worker using either the method supplied in configuration or determined by detected environment. Default: False
match_procs_to_gpus (bool, Optional) – For use with auto_assign_gpus. Auto-assigns MPI processors to match the assigned GPUs. Default: False unless auto_assign_gpus is True and no other CPU configuration is supplied.
env_script (str, Optional) – The full path of a shell script to set up the environment for the launched task. This will be run in the subprocess, and not affect the worker environment. The script should start with a shebang.
mpi_runner_type ((str|dict), Optional) – An MPI runner to be used for this submit only. Supply either a string for the MPI runner type or a dictionary for detailed configuration (see custom_info on MPIExecutor constructor). This will not change the default MPI runner for the executor. Example string inputs are “mpich”, “openmpi”, “srun”, “jsrun”, “aprun”.
- Returns:
task – The launched task object
- Return type:
Note that if some combination of num_procs, num_nodes, and procs_per_node is provided, these will be honored if possible. If resource detection is on and these are omitted, then the available resources will be divided among workers.
- manager_kill_received()
Return True if received kill signal from the manager
- Return type:
bool
- manager_poll()
Polls for a manager signal
The executor manager_signal attribute will be updated.
- Return type:
int
- polling_loop(task, timeout=None, delay=0.1, poll_manager=False)
Optional, blocking, generic task status polling loop. Operates until the task finishes, times out, or is optionally killed via a manager signal. On completion, returns a presumptive calc_status integer. Useful for running an application via the Executor until it stops without monitoring its intermediate output.
- Parameters:
task (object) – a Task object returned by the executor on submission
timeout (int, Optional) – Maximum number of seconds for the polling loop to run. Tasks that run longer than this limit are killed. Default: No timeout
delay (int, Optional) – Sleep duration between polling loop iterations. Default: 0.1 seconds
poll_manager (bool, Optional) – Whether to also poll the manager for ‘finish’ or ‘kill’ signals. If detected, the task is killed. Default: False.
- Returns:
calc_status – presumptive integer attribute describing the final status of a launched task
- Return type:
int
- register_app(full_path, app_name=None, calc_type=None, desc=None, precedent='')
Registers a user application to libEnsemble.
The
full_path
of the application must be supplied. Eitherapp_name
orcalc_type
can be used to identify the application in user scripts (in the submit function).app_name
is recommended.- Parameters:
full_path (str) – The full path of the user application to be registered
app_name (str, Optional) – Name to identify this application.
calc_type (str, Optional) – Calculation type: Set this application as the default ‘sim’ or ‘gen’ function.
desc (str, Optional) – Description of this application
precedent (str, Optional) – Any str that should directly precede the application full path.
- Return type:
None
Class-specific Attributes
Class-specific attributes can be set directly to alter the behavior of the MPI Executor. However, they should be used with caution, because they may not be implemented in other executors.
- max_submit_attempts:
(int) Maximum number of launch attempts for a given task. Default: 5.
- fail_time:
(int or float) Only if wait_on_start is set. Maximum run time to failure in seconds that results in relaunch. Default: 2.
- retry_delay_incr:
(int or float) Delay increment between launch attempts in seconds. Default: 5. (i.e., First retry after 5 seconds, then 10 seconds, then 15, etc…)
Example. To increase resilience against submission failures:
taskctrl = MPIExecutor()
taskctrl.max_launch_attempts = 8
taskctrl.fail_time = 5
taskctrl.retry_delay_incr = 10