Base Executor - Local apps

This module contains the classes Executor and Task. The class Executor can be used to subprocess an application locally. For MPI programs, or any program using non-local compute resources, one of the inherited classes should be used. Inherited classes include MPI and Balsam variants. An executor can create and manage multiple tasks. The user function can issue and manage tasks using the submit, poll, wait, and kill functions. Task attributes are queried to determine status. Functions are also provided to access and interrogate files in the task’s working directory. A manager_poll function can be used to poll for STOP signals from the manager.

See this example for usage.

See the Executor APIs for optional arguments.

Executor Class

Only create an object of this class for running local serial-launched applications. To run MPI applications and use detected resources, use the MPIExecutor

class libensemble.executors.executor.Executor

The executor can create, poll and kill runnable tasks

Class Attributes:

Variables:

Executor – executor: The executor object is stored here and can be retrieved in user functions.

__init__()

Instantiate a new Executor instance.

Returns:

A new Executor object is created. This is typically created in the user calling script.

Return type:

Executor

register_app(full_path, app_name=None, calc_type=None, desc=None, precedent='')

Registers a user application to libEnsemble.

The full_path of the application must be supplied. Either app_name or calc_type can be used to identify the application in user scripts (in the submit function). app_name is recommended.

Parameters:

  • full_path (str) – The full path of the user application to be registered

  • app_name (str, Optional) – Name to identify this application.

  • calc_type (str, Optional) – Calculation type: Set this application as the default ‘sim’ or ‘gen’ function.

  • desc (str, Optional) – Description of this application

  • precedent (str, Optional) – Any str that should directly precede the application full path.

Return type:

None

manager_poll()

Polls for a manager signal

The executor manager_signal attribute will be updated.

Return type:

int

manager_kill_received()

Return True if received kill signal from the manager

Return type:

bool

polling_loop(task, timeout=None, delay=0.1, poll_manager=False)

Optional, blocking, generic task status polling loop. Operates until the task finishes, times out, or is Optionally killed via a manager signal. On completion, returns a presumptive calc_status integer. Potentially useful for running an application via the Executor until it stops without monitoring its intermediate output.

Parameters:

  • task (object) – a Task object returned by the executor on submission

  • timeout (int, Optional) – Maximum number of seconds for the polling loop to run. Tasks that run longer than this limit are killed. Default: No timeout

  • delay (int, Optional) – Sleep duration between polling loop iterations. Default: 0.1 seconds

  • poll_manager (bool, Optional) – Whether to also poll the manager for ‘finish’ or ‘kill’ signals. If detected, the task is killed. Default: False.

Returns:

calc_status – presumptive integer attribute describing the final status of a launched task

Return type:

int

submit(calc_type=None, app_name=None, app_args=None, stdout=None, stderr=None, dry_run=False, wait_on_start=False, env_script=None)

Create a new task and run as a local serial subprocess.

The created task object is returned.

Parameters:

  • calc_type (str, Optional) – The calculation type: ‘sim’ or ‘gen’ Only used if app_name is not supplied. Uses default sim or gen application.

  • app_name (str, Optional) – The application name. Must be supplied if calc_type is not.

  • app_args (str, Optional) – A str of the application arguments to be added to task submit command line

  • stdout (str, Optional) – A standard output filename

  • stderr (str, Optional) – A standard error filename

  • dry_run (bool, Optional) – Whether this is a dry_run - no task will be launched; instead runline is printed to logger (at INFO level)

  • wait_on_start (bool, Optional) – Whether to wait for task to be polled as RUNNING (or other active/end state) before continuing

  • env_script (str, Optional) – The full path of a shell script to set up the environment for the launched task. This will be run in the subprocess, and not affect the worker environment. The script should start with a shebang.

Returns:

task – The launched task object

Return type:

Task

Task Class

Tasks are created and returned through the Executor submit() function. Tasks can be polled, killed, waited on with the respective poll, kill, and wait functions. Task information can be queried through the task attributes below and the query functions.

class libensemble.executors.executor.Task(app=None, app_args=None, workdir=None, stdout=None, stderr=None, workerid=None, dry_run=False)

Manages the creation, configuration and status of a launchable task

workdir_exists()

Returns true if the task’s workdir exists

Return type:

bool | None

file_exists_in_workdir(filename)

Returns true if the named file exists in the task’s workdir

Parameters:

filename (str) –

Return type:

bool

read_file_in_workdir(filename)

Opens and reads the named file in the task’s workdir

Parameters:

filename (str) –

Return type:

str

stdout_exists()

Returns true if the task’s stdout file exists in the workdir

Return type:

bool

read_stdout()

Opens and reads the task’s stdout file in the task’s workdir

Return type:

str

stderr_exists()

Returns true if the task’s stderr file exists in the workdir

Return type:

bool

read_stderr()

Opens and reads the task’s stderr file in the task’s workdir

Return type:

str

poll()

Polls and updates the status attributes of the task

Return type:

None

wait(timeout=None)

Waits on completion of the task or raises TimeoutExpired exception

Status attributes of task are updated on completion.

Parameters:

timeout (int or float, Optional) – Time in seconds after which a TimeoutExpired exception is raised. If not set, then simply waits until completion. Note that the task is not automatically killed on timeout.

Return type:

None

result(timeout=None)

Wrapper for task.wait() that also returns the task’s status on completion.

Parameters:

timeout (int or float, Optional) – Time in seconds after which a TimeoutExpired exception is raised. If not set, then simply waits until completion. Note that the task is not automatically killed on timeout.

Return type:

str

exception(timeout=None)

Wrapper for task.wait() that instead returns the task’s error code on completion.

Parameters:

timeout (int or float, Optional) – Time in seconds after which a TimeoutExpired exception is raised. If not set, then simply waits until completion. Note that the task is not automatically killed on timeout.

running()

Return True if task is currently running.

Return type:

bool

done()

Return True if task is finished.

Return type:

bool

kill(wait_time=60)

Kills or cancels the supplied task

Parameters:

wait_time (int, Optional) – Time in seconds to wait for termination between sending SIGTERM and a SIGKILL signals.

Return type:

None

Sends SIGTERM, waits for a period of <wait_time> for graceful termination, then sends a hard kill with SIGKILL. If <wait_time> is 0, we go immediately to SIGKILL; if <wait_time> is none, we never do a SIGKILL.

cancel()

Wrapper for task.kill() without waiting

Return type:

None

cancelled()

Return `True if task successfully cancelled.

Return type:

bool

Task Attributes

Following is a list of task status and configuration attributes that can be retrieved from a task.

Note

These should not be set directly. Tasks are launched by the Executor, and task information can be queried through the task attributes below and the query functions.

Task status attributes include the following:

task.state:

(string) The task status. One of (“UNKNOWN”|”CREATED”|”WAITING”|”RUNNING”|”FINISHED”|”USER_KILLED”|”FAILED”)

task.process:

(process obj) The process object used by the underlying process manager (e.g., return value of subprocess.Popen).

task.errcode:

(int) The errorcode/return code used by the underlying process manager.

task.finished:

(boolean) True means task has finished running - not whether it was successful.

task.success:

(boolean) Did task complete successfully (e.g., the returncode is zero)?

task.runtime:

(int) Time in seconds that task has been running.

task.submit_time:

(int) Time since epoch that task was submitted

task.total_time:

(int) Total time from task submission to completion (only available when task is finished).

Run configuration attributes - some will be autogenerated:

task.workdir:

(string) Work directory for the task

task.name:

(string) Name of task - autogenerated

task.app:

(app obj) Use application/executable, registered using exctr.register_app

task.app_args:

(string) Application arguments as a string

task.stdout:

(string) Name of file where the standard output of the task is written (in task.workdir)

task.stderr:

(string) Name of file where the standard error of the task is written (in task.workdir)

task.dry_run:

(boolean) True if task corresponds to dry run (no actual submission)

task.runline:

(string) Complete, parameterized command to be subprocessed to launch app