Convenience Tools and Functions

Setup Helpers

tools.add_unique_random_streams(persis_info, nstreams, seed='')

Creates nstreams random number streams for the libE manager and workers when nstreams is num_workers + 1. Stream i is initialized with seed i by default. Otherwise the streams can be initialized with a provided seed.

The entries are appended to the provided persis_info dictionary.

persis_info = add_unique_random_streams(old_persis_info, nworkers + 1)

Parameters:

persis_info (dict) – Persistent information dictionary. (example)
nstreams (int) – Number of independent random number streams to produce.
seed (int) – (Optional) Seed for identical random number streams for each worker. If explicitly set to None, random number streams are unique and seed via other pseudorandom mechanisms.

tools.check_npy_file_exists(filename, basename=False, max_wait=3)

Checks a file is created in a parallel environment

Parameters:

filename (str) – Name of file.
basename (bool) – Whether name provided is a basename for save_libE_output.
max_wait (int) – The maximum number of seconds to wait for the file to exist.

tools.eprint(*args, **kwargs): Prints a user message to standard error

class tools.ForkablePdb(completekey='tab', stdin=None, stdout=None, skip=None, nosigint=False, readrc=True)

A Pdb subclass that may be used from a forked multiprocessing child

Usage:

from libensemble.tools import ForkablePdb

ForkablePdb().set_trace()

tools.parse_args()

Parses command-line arguments. Use in calling script.

from libensemble.tools import parse_args

nworkers, is_manager, libE_specs, misc_args = parse_args()

Or for object interface, when creating the ensemble object.

from libensemble import Ensemble

ensemble = Ensemble(parse_args=True)

From the shell:

$ python calling_script --comms local --nworkers 4

Usage:

usage: test_... [-h] [--comms [{local, tcp, ssh, client, mpi}]]
                [--nworkers [NWORKERS]] [--workers WORKERS [WORKERS ...]]
                [--nsim_workers [NSIM_WORKERS]]
                [--nresource_sets [NRESOURCE_SETS]]
                [--workerID [WORKERID]] [--server SERVER SERVER SERVER]
                [--pwd [PWD]] [--worker_pwd [WORKER_PWD]]
                [--worker_python [WORKER_PYTHON]]
                [--tester_args [TESTER_ARGS [TESTER_ARGS ...]]]

Note that running via an MPI runner uses the default 'mpi' comms, and '--nworkers'
will be ignored. The number of processes are supplied via the MPI run line. One being
the manager, and the rest are workers.

--comms,          Communications medium for manager and workers.
                  Default is 'local' if --nworkers is provided, otherwise  'mpi'.
--nworkers/-n,    (For 'local' or 'tcp' comms) Set number of workers.
--nresource_sets, Explicitly set the number of resource sets. This sets
                  libE_specs["num_resource_sets"]. By default, resources will be
                  divided by workers (excluding zero_resource_workers).
--nsim_workers,   (For 'local" or 'mpi' comms) A convenience option for cases with
                  persistent generators - sets the number of simulation workers.
                  If used with no other criteria, one additional worker for running a
                  generator will be added, and the number of resource sets will be assigned
                  the given value. If '--nworkers' has also been specified, will generate
                  enough additional workers to match the other criteria. If '--nresource_sets'
                  is also specified, will not override resource sets.

Example command lines:

Run with 'local' comms and 4 workers (the following are equivalent).
$ python calling_script --comms local --nworkers 4
$ python calling_script --nworkers 4
$ python calling_script -n 4

Run with 'local' comms and 5 workers - one gen worker (no resources), and 4 sim workers.
$ python calling_script --comms local --nsim_workers 4

Run with 'local' comms with 4 workers and 8 resource sets. The extra resource sets will
be used for larger simulations (using variable resource assignment).
$ python calling_script --comms local --nresource_sets 8

Previous example with 'mpi' comms.
$ mpirun -np 5 python calling_script --nresource_sets 8

Returns:

nworkers (int) – Number of workers libEnsemble will initiate
is_manager (bool) – Indicates whether the current process is the manager process
libE_specs (dict) – Settings and specifications for libEnsemble (example)

tools.save_libE_output(H, persis_info, calling_file, nworkers, dest_path='/home/docs/checkouts/readthedocs.org/user_builds/libensemble/checkouts/main/docs', mess='Run completed')

Writes out history array and persis_info to files.

Format: <calling_script>_results_History_length=<length>_evals=<Completed evals>_ranks=<nworkers>

save_libE_output(H, persis_info, __file__, nworkers)

Parameters:

H (NumPy structured array) – History array storing rows for each point. (example)
persis_info (dict) – Persistent information dictionary. (example)
calling_file (str) – Name of user-calling script (or user chosen name) to prefix output files. The convention is to send __file__ from user calling script.
nworkers (int) – The number of workers in this ensemble. Added to output file names.
mess (str) – A message to print/log when saving the file.

Persistent Helpers

These routines are commonly used within persistent generator functions such as persistent_aposmm in libensemble/gen_funcs/ for intermediate communication with the manager. Persistent simulator functions are also supported.

class persistent_support.PersistentSupport(libE_info, calc_type)

A helper class to assist with writing persistent user functions.

Parameters:

libE_info (Dict[str, Dict[Any, Any]])
calc_type (int)

send(output, calc_status=0, keep_state=False)

Send message from worker to manager.

Parameters:

output (ndarray[Any, dtype[_ScalarType_co]]) – Output array to be sent to manager.
calc_status (int) – (Optional) Provides a task status.
keep_state – (Optional) If True the manager will not modify its record of the workers state (usually the manager changes the worker’s state to inactive, indicating the worker is ready to receive more work, unless using active receive mode).

Return type:

None

recv(blocking=True)

Receive message to worker from manager.

Parameters:: blocking (bool) – (Optional) If True (default), will block until a message is received.
Returns:: message tag, Work dictionary, calc_in array
Return type:: (<class ‘int’>, <class ‘dict’>, numpy.ndarray[Any, numpy.dtype[+_ScalarType_co]])

send_recv(output, calc_status=0)

Send message from worker to manager and receive response.

Parameters:

output (ndarray[Any, dtype[_ScalarType_co]]) – Output array to be sent to manager.
calc_status (int) – (Optional) Provides a task status.

Returns:

message tag, Work dictionary, calc_in array

Return type:

(<class ‘int’>, <class ‘dict’>, numpy.ndarray[Any, numpy.dtype[+_ScalarType_co]])

request_cancel_sim_ids(sim_ids)

Request cancellation of sim_ids.

Parameters:: sim_ids (List[int]) – A list of sim_ids to cancel.

A message is sent to the manager to mark requested sim_ids as cancel_requested.

Allocation Helpers

These routines are used within custom allocation functions to help prepare Work structures for workers. See the routines within libensemble/alloc_funcs/ for examples.

exception alloc_support.AllocException: Raised for any exception in the alloc support

class alloc_support.AllocSupport(W, manage_resources=False, persis_info={}, libE_info={}, user_resources=None, user_scheduler=None)

A helper class to assist with writing allocation functions.

This class contains methods for common operations like populating work units, determining which workers are available, evaluating what values need to be distributed to workers, and others.

Note that since the alloc_f is called periodically by the Manager, this class instance (if used) will be recreated/destroyed on each loop.

assign_resources(rsets_req, use_gpus=None, user_params=[])

Schedule resource sets to a work record if possible.

For default scheduler, if more than one group (node) is required, will try to find even split, otherwise allocates whole nodes.

Raises InsufficientFreeResources if the required resources are not currently available, or InsufficientResourcesError if the required resources do not exist.

Parameters:

rsets_req – Int. Number of resource sets to request.
use_gpus – Bool. Whether to use GPU resource sets.
user_params – List of Integers. User parameters num_procs, num_gpus.

Returns:

List of Integers. Resource set indices assigned.

avail_worker_ids(persistent=None, active_recv=False, zero_resource_workers=None, gen_workers=None)

Returns available workers as a list of IDs, filtered by the given options.

Parameters:

persistent – (Optional) Int. Only return workers with given persis_state (1=sim, 2=gen).
active_recv – (Optional) Boolean. Only return workers with given active_recv state.
zero_resource_workers – (Optional) Boolean. Only return workers that require no resources.
gen_workers – (Optional) Boolean. If True, return gen-only workers. If False, return all other workers.

Returns:

List of worker IDs.

If there are no zero resource workers defined, then the zero_resource_workers argument will be ignored.

count_gens(): Returns the number of active generators.

test_any_gen(): Returns True if a generator worker is active.

count_persis_gens(): Return the number of active persistent generators.

sim_work(wid, H, H_fields, H_rows, persis_info, **libE_info)

Add sim work record to given Work dictionary.

Includes evaluation of required resources if the worker is not in a persistent state.

Parameters:

wid – Int. Worker ID.
H – History array. For parsing out requested resource sets.
H_fields – Which fields from H to send.
H_rows – Which rows of H to send.
persis_info – Worker specific persis_info dictionary.

Returns:

a Work entry.

Additional passed parameters are inserted into libE_info in the resulting work record.

If rset_team is passed as an additional parameter, it will be honored, assuming that any resource checking has already been done.

gen_work(wid, H_fields, H_rows, persis_info, **libE_info)

Add gen work record to given Work dictionary.

Includes evaluation of required resources if the worker is not in a persistent state.

Parameters:

Work – Work dictionary.
wid – Worker ID.
H_fields – Which fields from H to send.
H_rows – Which rows of H to send.
persis_info – Worker specific persis_info dictionary.

Returns:

A Work entry.

Additional passed parameters are inserted into libE_info in the resulting work record.

If rset_team is passed as an additional parameter, it will be honored, and assume that any resource checking has already been done. For example, passing rset_team=[], would ensure that no resources are assigned.

all_sim_started(H, pt_filter=None, low_bound=None)

Returns True if all expected points have started their sim.

Excludes cancelled points.

Parameters:

pt_filter – (Optional) Boolean array filtering expected returned points in H.
low_bound – (Optional) Lower bound for testing all returned.

Returns:

True if all expected points have started their sim.

all_sim_ended(H, pt_filter=None, low_bound=None)

Returns True if all expected points have had their sim_end.

Excludes cancelled points that were not already sim_started.

Parameters:

pt_filter – (Optional) Boolean array filtering expected returned points in H.
low_bound – (Optional) Lower bound for testing all returned.

Returns:

True if all expected points have had their sim_end.

all_gen_informed(H, pt_filter=None, low_bound=None)

Returns True if gen has been informed of all expected points.

Excludes cancelled points that were not already given out.

Parameters:

pt_filter – (Optional) Boolean array filtering expected sim_end points in H.
low_bound – (Optional) Lower bound for testing all returned.

Returns:

True if gen have been informed of all expected points.

points_by_priority(H, points_avail, batch=False)

Returns indices of points to give by priority.

Parameters:

points_avail – Indices of points that are available to give.
batch – (Optional) Boolean. Should batches of points with the same priority be given simultaneously.

Returns:

An array of point indices to give.

skip_canceled_points(H, persis_info): Increments the “next_to_give” field in persis_info to skip any cancelled points

Live Data

These classes provide a means to capture and display data during a workflow run. Users may provide an initialized object via libE_specs["live_data"]. For example:

from libensemble.tools.live_data.plot2n import Plot2N
libE_specs["live_data"] = Plot2N(plot_type='2d')

class libensemble.tools.live_data.live_data.LiveData

A base class for capturing and processing data during an ensemble

abstract live_update(hist)

Process history data after simulation results have been added to history

Parameters:: hist (libensemble.history.History) – A libEnsemble history type object.
Return type:: None

finalize(hist)

Finzalize live data processing

Parameters:: hist (libensemble.history.History) – A libEnsemble history type object.
Return type:: None

class plot2n.Plot2N(plot_type='2d', func=<function six_hump_camel_func>, bounds=((-2, 2), (-1, 1.1)))

Bases: LiveData

Plot2N class for generating plots of a 2D domain during a workflow run.

Parameters:

plot_type (str) – Type of plot (“2d” or “3d”)
func (Callable) – Function to plot. Default is six hump camel.
bounds (tuple) – Bounds for the plot.

live_update(hist): Function called after every f update in the manager

finalize(hist)

Finzalize live data processing

Parameters:: hist (libensemble.history.History) – A libEnsemble history type object.