Convenience Tools and Functions

Calling Script Function Support

tools.add_unique_random_streams(persis_info, nstreams, seed='')

Creates nstreams random number streams for the libE manager and workers when nstreams is num_workers + 1. Stream i is initialized with seed i by default. Otherwise the streams can be initialized with a provided seed.

The entries are appended to the provided persis_info dictionary.

persis_info = add_unique_random_streams(old_persis_info, nworkers + 1)
Parameters:

  • persis_info (dict) – Persistent information dictionary (example)

  • nstreams (int) – Number of independent random number streams to produce

  • seed (int) – (Optional) Seed for identical random number streams for each worker. If explicitly set to None, random number streams are unique and seed via other pseudorandom mechanisms.

tools.eprint(*args, **kwargs)

Prints a user message to standard error

class tools.ForkablePdb(completekey='tab', stdin=None, stdout=None, skip=None, nosigint=False, readrc=True)

A Pdb subclass that may be used from a forked multiprocessing child

Usage:

from libensemble.tools import ForkablePdb

ForkablePdb().set_trace()
tools.parse_args()

Parses command-line arguments. Use in calling script.

from libensemble.tools import parse_args

nworkers, is_manager, libE_specs, misc_args = parse_args()

From the shell:

$ python calling_script --comms local --nworkers 4

Usage:

usage: test_... [-h] [--comms [{local, tcp, ssh, client, mpi}]]
                [--nworkers [NWORKERS]] [--workers WORKERS [WORKERS ...]]
                [--nsim_workers [NSIM_WORKERS]]
                [--nresource_sets [NRESOURCE_SETS]]
                [--workerID [WORKERID]] [--server SERVER SERVER SERVER]
                [--pwd [PWD]] [--worker_pwd [WORKER_PWD]]
                [--worker_python [WORKER_PYTHON]]
                [--tester_args [TESTER_ARGS [TESTER_ARGS ...]]]

Note that running via an MPI runner uses the default 'mpi' comms, and '--nworkers'
will be ignored. The number of processes are supplied via the MPI run line. One being
the manager, and the rest are workers.

--comms,          Communications medium for manager and workers. Default is 'mpi'.
--nworkers,       (For 'local' or 'tcp' comms) Set number of workers.
--nresource_sets, Explicitly set the number of resource sets. This sets
                  libE_specs["num_resource_sets"]. By default, resources will be
                  divided by workers (excluding zero_resource_workers).
--nsim_workers,   (For 'local" or 'mpi' comms) A convenience option for cases with
                  persistent generators - sets the number of simulation workers.
                  If used with no other criteria, one additional worker for running a
                  generator will be added, and the number of resource sets will be assigned
                  the given value. If '--nworkers' has also been specified, will generate
                  enough additional workers to match the other criteria. If '--nresource_sets'
                  is also specified, will not override resource sets.

Example command lines:

Run with 'local' comms and 4 workers
$ python calling_script --comms local --nworkers 4

Run with 'local' comms and 5 workers - one gen worker (no resources), and 4 sim workers.
$ python calling_script --comms local --nsim_workers 4

Run with 'local' comms with 4 workers and 8 resource sets. The extra resource sets will
be used for larger simulations (using variable resource assignment).
$ python calling_script --comms local --nresource_sets 8

Previous example with 'mpi' comms.
$ mpirun -np 5 python calling_script --nresource_sets 8
Returns:

  • nworkers (int) – Number of workers libEnsemble will initiate

  • is_manager (boolean) – Indicates whether the current process is the manager process

  • libE_specs (dict) – Settings and specifications for libEnsemble (example)

tools.save_libE_output(H, persis_info, calling_file, nworkers, dest_path='/home/docs/checkouts/readthedocs.org/user_builds/libensemble/checkouts/main/docs', mess='Run completed')

Writes out history array and persis_info to files.

Format: <calling_script>_results_History_length=<length>_evals=<Completed evals>_ranks=<nworkers>

save_libE_output(H, persis_info, __file__, nworkers)
Parameters:

  • H (NumPy structured array) – History array storing rows for each point. (example)

  • persis_info (dict) – Persistent information dictionary (example)

  • calling_file (string) – Name of user-calling script (or user chosen name) to prefix output files. The convention is to send __file__ from user calling script.

  • nworkers (int) – The number of workers in this ensemble. Added to output file names.

  • mess (String) – A message to print/log when saving the file.

Persistent Function Support

These routines are commonly used within persistent generator functions like persistent_aposmm in libensemble/gen_funcs/ for intermediate communication with the manager. Persistent simulator functions are also supported.

class persistent_support.PersistentSupport(libE_info, calc_type)

A helper class to assist with writing persistent user functions.

Parameters:

  • libE_info (Dict[str, Dict[Any, Any]]) –

  • calc_type (int) –

send(output, calc_status=0, keep_state=False)

Send message from worker to manager.

Parameters:

  • output (ndarray[Any, dtype[ScalarType]]) – Output array to be sent to manager

  • calc_status (int) – Optional, Provides a task status

  • keep_state – Optional, If True the manager will not modify its record of the workers state (usually the manager changes the worker’s state to inactive, indicating the worker is ready to receive more work, unless using active receive mode).

Return type:

None

recv(blocking=True)

Receive message to worker from manager.

Parameters:

blocking (bool) – Optional, If True (default), will block until a message is received.

Returns:

message tag, Work dictionary, calc_in array

Return type:

(<class ‘int’>, <class ‘dict’>, numpy.ndarray[Any, numpy.dtype[+ScalarType]])

send_recv(output, calc_status=0)

Send message from worker to manager and receive response.

Parameters:

  • output (ndarray[Any, dtype[ScalarType]]) – Output array to be sent to manager

  • calc_status (int) – Optional, Provides a task status

Returns:

message tag, Work dictionary, calc_in array

Return type:

(<class ‘int’>, <class ‘dict’>, numpy.ndarray[Any, numpy.dtype[+ScalarType]])

request_cancel_sim_ids(sim_ids)

Request cancellation of sim_ids

Parameters:

sim_ids (List[int]) – A list of sim_ids to cancel

A message is sent to the manager to mark requested sim_ids as cancel_requested

Allocation Function Support

These routines are used within custom allocation functions to help prepare Work structures for workers. See the routines within libensemble/alloc_funcs/ for examples.

exception alloc_support.AllocException

Raised for any exception in the alloc support

class alloc_support.AllocSupport(W, manage_resources=False, persis_info={}, libE_info={}, user_resources=None, user_scheduler=None)

A helper class to assist with writing allocation functions.

This class contains methods for common operations like populating work units, determining which workers are available, evaluating what values need to be distributed to workers, and others.

Note that since the alloc_f is called periodically by the Manager, this class instance (if used) will be recreated/destroyed on each loop.

assign_resources(rsets_req, use_gpus=None, user_params=[])

Schedule resource sets to a work record if possible.

For default scheduler, if more than one group (node) is required, will try to find even split, otherwise allocates whole nodes.

Raises InsufficientFreeResources if the required resources are not currently available, or InsufficientResourcesError if the required resources do not exist.

Parameters:

  • rsets_req – Int. Number of resource sets to request.

  • use_gpus – Bool. Whether to use GPU resource sets.

  • user_params – list of Integers. User parameters num_procs, num_gpus

Returns:

List of Integers. Resource set indices assigned.

avail_worker_ids(persistent=None, active_recv=False, zero_resource_workers=None)

Returns available workers as a list of IDs, filtered by the given options.

Parameters:

  • persistent – Optional int. Only return workers with given persis_state (1=sim, 2=gen).

  • active_recv – Optional boolean. Only return workers with given active_recv state.

  • zero_resource_workers – Optional boolean. Only return workers that require no resources

Returns:

List of worker IDs

If there are no zero resource workers defined, then the zero_resource_workers argument will be ignored.

count_gens()

Returns the number of active generators.

test_any_gen()

Returns True if a generator worker is active.

count_persis_gens()

Return the number of active persistent generators.

sim_work(wid, H, H_fields, H_rows, persis_info, **libE_info)

Add sim work record to given Work dictionary.

Includes evaluation of required resources if the worker is not in a persistent state.

Parameters:

  • wid – Int. Worker ID.

  • H – History array. For parsing out requested resource sets.

  • H_fields – Which fields from H to send

  • H_rows – Which rows of H to send.

  • persis_info – Worker specific persis_info dictionary

Returns:

a Work entry

Additional passed parameters are inserted into libE_info in the resulting work record.

If rset_team is passed as an additional parameter, it will be honored, assuming that any resource checking has already been done.

gen_work(wid, H_fields, H_rows, persis_info, **libE_info)

Add gen work record to given Work dictionary.

Includes evaluation of required resources if the worker is not in a persistent state.

Parameters:

  • Work – Work dictionary

  • wid – Worker ID.

  • H_fields – Which fields from H to send

  • H_rows – Which rows of H to send.

  • persis_info – Worker specific persis_info dictionary

Returns:

A Work entry

Additional passed parameters are inserted into libE_info in the resulting work record.

If rset_team is passed as an additional parameter, it will be honored, and assume that any resource checking has already been done. For example, passing rset_team=[], would ensure that no resources are assigned.

all_sim_started(H, pt_filter=None, low_bound=None)

Returns True if all expected points have started their sim

Excludes cancelled points.

Parameters:

  • pt_filter – Optional boolean array filtering expected returned points in H.

  • low_bound – Optional lower bound for testing all returned.

Returns:

True if all expected points have started their sim

all_sim_ended(H, pt_filter=None, low_bound=None)

Returns True if all expected points have had their sim_end

Excludes cancelled points that were not already sim_started.

Parameters:

  • pt_filter – Optional boolean array filtering expected returned points in H.

  • low_bound – Optional lower bound for testing all returned.

Returns:

True if all expected points have had their sim_end

all_gen_informed(H, pt_filter=None, low_bound=None)

Returns True if gen has been informed of all expected points

Excludes cancelled points that were not already given out.

Parameters:

  • pt_filter – Optional boolean array filtering expected sim_end points in H.

  • low_bound – Optional lower bound for testing all returned.

Returns:

True if gen have been informed of all expected points

points_by_priority(H, points_avail, batch=False)

Returns indices of points to give by priority

Parameters:

  • points_avail – Indices of points that are available to give

  • batch – Optional boolean. Should batches of points with the same priority be given simultaneously.

Returns:

An array of point indices to give.