Allocation Functions

Although the included allocation functions are sufficient for most users, those who want to fine-tune how data or resources may be allocated to their generator or simulator can write their own.

We encourage experimenting with:

  1. Prioritization of simulations

  2. Sending results immediately or in batch

  3. Assigning varying resources to evaluations

Example
libensemble.alloc_funcs.fast_alloc.give_sim_work_first
from libensemble.tools.alloc_support import AllocSupport, InsufficientFreeResources


def give_sim_work_first(W, H, sim_specs, gen_specs, alloc_specs, persis_info, libE_info):
    """
    This allocation function gives (in order) entries in ``H`` to idle workers
    to evaluate in the simulation function. The fields in ``sim_specs["in"]``
    are given. If all entries in `H` have been given a be evaluated, a worker
    is told to call the generator function, provided this wouldn't result in
    more than ``gen_specs["num_active_gens"]`` or ``alloc_specs["user"]["num_active_gens"]`` active generators.

    This fast_alloc variation of give_sim_work_first is useful for cases that
    simply iterate through H, issuing evaluations in order and, in particular,
    is likely to be faster if there will be many short simulation evaluations,
    given that this function contains fewer column length operations.

    tags: alloc, simple, fast

    .. seealso::
        `test_fast_alloc.py <https://github.com/Libensemble/libensemble/blob/develop/libensemble/tests/functionality_tests/test_fast_alloc.py>`_ # noqa
    """

    if libE_info["sim_max_given"] or not libE_info["any_idle_workers"]:
        return {}, persis_info

    user = {**gen_specs, **alloc_specs.get("user", {})}

    manage_resources = libE_info["use_resource_sets"]

    support = AllocSupport(W, manage_resources, persis_info, libE_info)

    gen_count = support.count_gens()
    Work = {}
    gen_in = gen_specs.get("in", [])

    # Give sim work if possible
    for wid in support.avail_worker_ids(gen_workers=False):
        persis_info = support.skip_canceled_points(H, persis_info)
        if persis_info["next_to_give"] < len(H):
            try:
                Work[wid] = support.sim_work(wid, H, sim_specs["in"], [persis_info["next_to_give"]], [])
            except InsufficientFreeResources:
                break
            persis_info["next_to_give"] += 1

    # Give gen work if possible
    if persis_info["next_to_give"] >= len(H):
        for wid in support.avail_worker_ids(gen_workers=True):
            if wid not in Work and gen_count < user.get("num_active_gens", gen_count + 1):
                return_rows = range(len(H)) if gen_in else []
                try:
                    Work[wid] = support.gen_work(wid, gen_in, return_rows, persis_info.get(wid))
                except InsufficientFreeResources:
                    break
                gen_count += 1
                persis_info["total_gen_calls"] += 1

    return Work, persis_info

The alloc_f function definition resembles:

def my_allocator(W, H, sim_specs, gen_specs, alloc_specs, persis_info, libE_info):

where:

  • W is an array containing worker state info

  • H is the trimmed History array, containing rows from the generator

  • libE_info is a set of statistics to determine the progress of work or exit conditions

Most users first check that it is appropriate to allocate work:

if libE_info["sim_max_given"] or not libE_info["any_idle_workers"]:
    return {}, persis_info

If the allocation is to continue, instantiate a support class to assist with the Work dictionary construction:

manage_resources = "resource_sets" in H.dtype.names or libE_info["use_resource_sets"]
support = AllocSupport(W, manage_resources, persis_info, libE_info)
Work = {}

The Work dictionary is populated with integer keys wid for each worker and dictionary values to give to those workers:

Example Work
{
    1: {
        "H_fields": ["x"],
        "persis_info": {"rand_stream": RandomState(...) at ..., "worker_num": 1},
        "tag": 1,
        "libE_info": {"H_rows": array([368])}
    },

    2: {
        "H_fields": ["x"],
        "persis_info": {"rand_stream": RandomState(...) at ..., "worker_num": 2},
        "tag": 1,
        "libE_info": {"H_rows": array([369])}
    },

    3: {
        "H_fields": ["x"],
        "persis_info": {"rand_stream": RandomState(...) at ..., "worker_num": 3},
        "tag": 1,
        "libE_info": {"H_rows": array([370])}
    },
    ...

}

This Work dictionary instructs each worker to call the sim_f (tag: 1) with data from "x" and a given "H_row" from the History array. A worker-specific persis_info is also given.

Constructing these arrays and determining which workers are available for receiving data is simplified by the AllocSupport class available within the libensemble.tools.alloc_support module:

AllocSupport
class libensemble.tools.alloc_support.AllocSupport(W, manage_resources=False, persis_info={}, libE_info={}, user_resources=None, user_scheduler=None)

A helper class to assist with writing allocation functions.

This class contains methods for common operations like populating work units, determining which workers are available, evaluating what values need to be distributed to workers, and others.

Note that since the alloc_f is called periodically by the Manager, this class instance (if used) will be recreated/destroyed on each loop.

__init__(W, manage_resources=False, persis_info={}, libE_info={}, user_resources=None, user_scheduler=None)

Instantiate a new AllocSupport instance

W is passed in for convenience on init; it is referenced by the various methods, but never modified.

By default, an AllocSupport instance uses any initiated libEnsemble resource module and the built-in libEnsemble scheduler.

Parameters:

  • W – A Worker array

  • manage_resources – (Optional) Boolean for if to assign resource sets when creating work units.

  • persis_info – (Optional) A dictionary of persistent information..

  • scheduler_opts – (Optional) A dictionary of options to pass to the resource scheduler.

  • user_resources – (Optional) A user supplied resources object.

  • user_scheduler – (Optional) A user supplied user_scheduler object.

assign_resources(rsets_req, use_gpus=None, user_params=[])

Schedule resource sets to a work record if possible.

For default scheduler, if more than one group (node) is required, will try to find even split, otherwise allocates whole nodes.

Raises InsufficientFreeResources if the required resources are not currently available, or InsufficientResourcesError if the required resources do not exist.

Parameters:

  • rsets_req – Int. Number of resource sets to request.

  • use_gpus – Bool. Whether to use GPU resource sets.

  • user_params – List of Integers. User parameters num_procs, num_gpus.

Returns:

List of Integers. Resource set indices assigned.

avail_worker_ids(persistent=None, active_recv=False, gen_workers=None)

Returns available workers as a list of IDs, filtered by the given options.

Parameters:

  • persistent – (Optional) Int. Only return workers with given persis_state (1=sim, 2=gen).

  • active_recv – (Optional) Boolean. Only return workers with given active_recv state.

  • gen_workers – (Optional) Boolean. If True, return gen-only workers. If False, return all other workers.

Returns:

List of worker IDs.

count_gens()

Returns the number of active generators.

test_any_gen()

Returns True if a generator worker is active.

count_persis_gens()

Return the number of active persistent generators.

sim_work(wid, H, H_fields, H_rows, persis_info, **libE_info)

Add sim work record to given Work dictionary.

Includes evaluation of required resources if the worker is not in a persistent state.

Parameters:

  • wid – Int. Worker ID.

  • H – History array. For parsing out requested resource sets.

  • H_fields – Which fields from H to send.

  • H_rows – Which rows of H to send.

  • persis_info – Worker specific persis_info dictionary.

Returns:

a Work entry.

Additional passed parameters are inserted into libE_info in the resulting work record.

If rset_team is passed as an additional parameter, it will be honored, assuming that any resource checking has already been done.

gen_work(wid, H_fields, H_rows, persis_info, **libE_info)

Add gen work record to given Work dictionary.

Includes evaluation of required resources if the worker is not in a persistent state.

Parameters:

  • Work – Work dictionary.

  • wid – Worker ID.

  • H_fields – Which fields from H to send.

  • H_rows – Which rows of H to send.

  • persis_info – Worker specific persis_info dictionary.

Returns:

A Work entry.

Additional passed parameters are inserted into libE_info in the resulting work record.

If rset_team is passed as an additional parameter, it will be honored, and assume that any resource checking has already been done. For example, passing rset_team=[], would ensure that no resources are assigned.

all_sim_started(H, pt_filter=None, low_bound=None)

Returns True if all expected points have started their sim.

Excludes cancelled points.

Parameters:

  • pt_filter – (Optional) Boolean array filtering expected returned points in H.

  • low_bound – (Optional) Lower bound for testing all returned.

Returns:

True if all expected points have started their sim.

all_sim_ended(H, pt_filter=None, low_bound=None)

Returns True if all expected points have had their sim_end.

Excludes cancelled points that were not already sim_started.

Parameters:

  • pt_filter – (Optional) Boolean array filtering expected returned points in H.

  • low_bound – (Optional) Lower bound for testing all returned.

Returns:

True if all expected points have had their sim_end.

all_gen_informed(H, pt_filter=None, low_bound=None)

Returns True if gen has been informed of all expected points.

Excludes cancelled points that were not already given out.

Parameters:

  • pt_filter – (Optional) Boolean array filtering expected sim_end points in H.

  • low_bound – (Optional) Lower bound for testing all returned.

Returns:

True if gen have been informed of all expected points.

points_by_priority(H, points_avail, batch=False)

Returns indices of points to give by priority.

Parameters:

  • points_avail – Indices of points that are available to give.

  • batch – (Optional) Boolean. Should batches of points with the same priority be given simultaneously.

Returns:

An array of point indices to give.

skip_canceled_points(H, persis_info)

Increments the “next_to_give” field in persis_info to skip any cancelled points

The Work dictionary is returned to the manager alongside persis_info. If 1 is returned as the third value, this instructs the ensemble to stop.

Note

An error occurs when the alloc_f returns nothing while all workers are idle

Information from the manager describing the progress of the current libEnsemble routine can be found in libE_info:

libE_info =  {
        "any_idle_workers": bool,            # True if there are any idle workers
        "exit_criteria": {...},              # Criteria for ending routine
        "elapsed_time": float,               # Time elapsed since start of routine
        "gen_informed_count": int,           # Total number of evaluated points given back to a generator function
        "manager_kill_canceled_sims": bool,  # True if manager is to send kills to cancelled simulations
        "scheduler_opts": {...},             # Options passed to the scheduler. "split2fit" and "match_slots"
        "sim_started_count": int,            # Total number of points given for simulation function evaluation
        "sim_ended_count": int,              # Total number of points returned from simulation function evaluations
        "sim_max_given": bool,               # True if `sim_max` simulations have been given out to workers
        "use_resource_sets": bool,           # True if num_resource_sets has been explicitly set.
        "gen_num_procs": int,                # Number of processes used for generator function evaluations
        "gen_num_gpus": int,                 # Number of GPUs used for generator function evaluations
        "gen_on_worker": bool}               # True if generator function is running on a worker

Most often, the allocation function will just return once sim_max_given is True, but the user could choose to do something different, such as cancel points or keep returning completed points to the generator.

Generators that construct models based on all evaluated points, for example, may need simulation work units at the end of an ensemble to be returned to the generator anyway.

Alternatively, users can use elapsed_time to track runtime inside their allocation function and detect impending timeouts, then pack up cleanup work requests, or mark points for cancellation.

The remaining values above are useful for efficient filtering of H values (e.g., sim_ended_count saves filtering by an entire column of H.)

The default allocation function is start_only_persistent. During its worker ID loop, it checks if there’s unallocated work and assigns simulations for that work. Otherwise, it initializes generators for up to "num_active_gens" instances. Other settings like batch_mode are also supported. See here for more information.

Examples

Below are example allocation functions available in libEnsemble.

Many users use these unmodified.

Important

The default allocation function changed in libEnsemble v2.0 from give_sim_work_first to start_only_persistent.

Note

The most commonly used allocation function for non-persistent generators is give_sim_work_first.

start_only_persistent

start_only_persistent.only_persistent_gens(W, H, sim_specs, gen_specs, alloc_specs, persis_info, libE_info)

This allocation function will give simulation work if possible, but otherwise start up to gen_specs["num_active_gens"] or alloc_specs["user"]["num_active_gens"] persistent generators (defaulting to one).

By default, evaluation results are given back to the generator once all generated points have been returned from the simulation evaluation. If gen_specs["async_return"] or alloc_specs["user"]["async_return"] is set to True, then any returned points are given back to the generator.

“” or alloc_specs["user"]["num_active_gens"] persistent generators (defaulting to one).

If any of the persistent generators has exited, then ensemble shutdown is triggered.

User options:

To be provided in calling script: E.g., alloc_specs["user"]["async_return"] = True

init_sample_size: int, optional

Initial sample size - always return in batch. Default: 0

num_active_gens: int, optional

Maximum number of persistent generators to start. Default: 1

async_return: Boolean, optional

Return results to gen as they come in (after sample). Default: False (batch return).

batch_evaluate_same_priority: Boolean, optional

If True, then all points with the same priority value are given as a batch to the sim. Default is False

active_recv_gen: Boolean, optional

Create gen in active receive mode. If True, the manager does not need to wait for a return from the generator before sending further returned points. Default: False

tags: alloc, batch, async, persistent, priority

See also

test_persistent_uniform_sampling.py test_persistent_uniform_sampling_async.py test_persistent_surmise_calib.py test_persistent_uniform_gen_decides_stop.py

start_only_persistent.py
  1import numpy as np
  2
  3from libensemble.message_numbers import EVAL_GEN_TAG, EVAL_SIM_TAG
  4from libensemble.tools.alloc_support import AllocSupport, InsufficientFreeResources
  5
  6
  7def only_persistent_gens(W, H, sim_specs, gen_specs, alloc_specs, persis_info, libE_info):
  8    """
  9    This allocation function will give simulation work if possible, but
 10    otherwise start up to ``gen_specs["num_active_gens"]`` or ``alloc_specs["user"]["num_active_gens"]``
 11    persistent generators (defaulting to one).
 12
 13    By default, evaluation results are given back to the generator once
 14    all generated points have been returned from the simulation evaluation.
 15    If ``gen_specs["async_return"]`` or ``alloc_specs["user"]["async_return"]`` is set to True, then any
 16    returned points are given back to the generator.
 17
 18    "" or ``alloc_specs["user"]["num_active_gens"]``
 19    persistent generators (defaulting to one).
 20
 21    If any of the persistent generators has exited, then ensemble shutdown
 22    is triggered.
 23
 24    **User options**:
 25
 26    To be provided in calling script: E.g., ``alloc_specs["user"]["async_return"] = True``
 27
 28    init_sample_size: int, optional
 29        Initial sample size - always return in batch. Default: 0
 30
 31    num_active_gens: int, optional
 32        Maximum number of persistent generators to start. Default: 1
 33
 34    async_return: Boolean, optional
 35        Return results to gen as they come in (after sample). Default: False (batch return).
 36
 37    batch_evaluate_same_priority: Boolean, optional
 38        If True, then all points with the same priority value are given as a batch to the sim.
 39        Default is False
 40
 41    active_recv_gen: Boolean, optional
 42        Create gen in active receive mode. If True, the manager does not need to wait
 43        for a return from the generator before sending further returned points.
 44        Default: False
 45
 46    tags: alloc, batch, async, persistent, priority
 47
 48    .. seealso::
 49        `test_persistent_uniform_sampling.py <https://github.com/Libensemble/libensemble/blob/develop/libensemble/tests/functionality_tests/test_persistent_uniform_sampling.py>`_ # noqa
 50        `test_persistent_uniform_sampling_async.py <https://github.com/Libensemble/libensemble/blob/develop/libensemble/tests/functionality_tests/test_persistent_uniform_sampling_async.py>`_ # noqa
 51        `test_persistent_surmise_calib.py <https://github.com/Libensemble/libensemble/blob/develop/libensemble/tests/regression_tests/test_persistent_surmise_calib.py>`_ # noqa
 52        `test_persistent_uniform_gen_decides_stop.py <https://github.com/Libensemble/libensemble/blob/develop/libensemble/tests/functionality_tests/test_persistent_uniform_gen_decides_stop.py>`_ # noqa
 53    """
 54
 55    if libE_info["sim_max_given"] or not libE_info["any_idle_workers"]:
 56        return {}, persis_info
 57
 58    # Initialize alloc_specs["user"] as user.
 59    user = {**gen_specs, **alloc_specs.get("user", {})}
 60
 61    manage_resources = libE_info["use_resource_sets"]
 62
 63    active_recv_gen = user.get("active_recv_gen", False)  # Persistent gen can handle irregular communications
 64    initial_batch_size = user.get("initial_batch_size", 0)  # Always batch return until this many evals complete
 65    batch_give = user.get("batch_evaluate_same_priority", False)
 66
 67    support = AllocSupport(W, manage_resources, persis_info, libE_info)
 68    gen_count = support.count_persis_gens()
 69    Work = {}
 70
 71    # Asynchronous return to generator
 72    async_return = user.get("async_return", False) and sum(H["sim_ended"]) >= initial_batch_size
 73
 74    if gen_count < persis_info.get("num_gens_started", 0):
 75        # When a persistent worker is done, trigger a shutdown (returning exit condition of 1)
 76        return Work, persis_info, 1
 77
 78    # Give evaluated results back to a running persistent gen
 79    for wid in support.avail_worker_ids(persistent=EVAL_GEN_TAG, active_recv=active_recv_gen):
 80        gen_inds = H["gen_worker"] == wid
 81        returned_but_not_given = np.logical_and.reduce((H["sim_ended"], ~H["gen_informed"], gen_inds))
 82        if np.any(returned_but_not_given):
 83            if async_return or support.all_sim_ended(H, gen_inds):
 84                point_ids = np.where(returned_but_not_given)[0]
 85                Work[wid] = support.gen_work(
 86                    wid,
 87                    gen_specs["persis_in"],
 88                    point_ids,
 89                    persis_info.get(wid),
 90                    persistent=True,
 91                    active_recv=active_recv_gen,
 92                )
 93                returned_but_not_given[point_ids] = False
 94
 95    # Now the give_sim_work_first part
 96    points_to_evaluate = ~H["sim_started"] & ~H["cancel_requested"]
 97    avail_workers = support.avail_worker_ids(persistent=False, gen_workers=False)
 98    if user.get("alt_type"):
 99        avail_workers = list(
100            set(support.avail_worker_ids(persistent=False)) | set(support.avail_worker_ids(persistent=EVAL_SIM_TAG))
101        )
102    for wid in avail_workers:
103        if not np.any(points_to_evaluate):
104            break
105
106        sim_ids_to_send = support.points_by_priority(H, points_avail=points_to_evaluate, batch=batch_give)
107
108        try:
109            if user.get("alt_type"):
110                Work[wid] = support.sim_work(
111                    wid, H, sim_specs["in"], sim_ids_to_send, persis_info.get(wid), persistent=True
112                )
113            else:
114                Work[wid] = support.sim_work(wid, H, sim_specs["in"], sim_ids_to_send, persis_info.get(wid))
115        except InsufficientFreeResources:
116            break
117
118        points_to_evaluate[sim_ids_to_send] = False
119
120    # Start persistent gens if no worker to give out.
121    if not np.any(points_to_evaluate):
122        avail_workers = support.avail_worker_ids(persistent=False, gen_workers=True)
123
124        for wid in avail_workers:
125            if gen_count < user.get("num_active_gens", 1):
126                # Finally, start a persistent generator as there is nothing else to do.
127                try:
128                    Work[wid] = support.gen_work(
129                        wid,
130                        gen_specs.get("in", []),
131                        range(len(H)),
132                        persis_info.get(wid),
133                        persistent=True,
134                        active_recv=active_recv_gen,
135                    )
136                except InsufficientFreeResources:
137                    break
138
139                persis_info["num_gens_started"] = persis_info.get("num_gens_started", 0) + 1
140                gen_count += 1
141
142    return Work, persis_info, 0

give_sim_work_first

give_sim_work_first.give_sim_work_first(W, H, sim_specs, gen_specs, alloc_specs, persis_info, libE_info)

Decide what should be given to workers. This allocation function gives any available simulation work first, and only when all simulations are completed or running does it start (at most gen_specs["num_active_gens"] or alloc_specs["user"]["num_active_gens"]) generator instances.

Allows for a gen_specs["batch_mode"] or alloc_specs["user"]["batch_mode"] where no generation work is given out unless all entries in H are returned.

Can give points in highest priority, if "priority" is a field in H. If gen_specs["batch_evaluate_same_priority"] or alloc_specs["user"]["batch_evaluate_same_priority"] is set to True, then all points with the same priority value are given as a batch to the sim.

Workers performing sims will be assigned resources given in H[“resource_sets”] this field exists, else defaulting to one. Workers performing gens are assigned resource_sets given by persis_info[“gen_resources”] or zero.

This is the default allocation function if one is not defined.

tags: alloc, default, batch, priority

See also

test_uniform_sampling.py

Parameters:

  • W (ndarray[tuple[Any, ...], dtype[_ScalarT]])

  • H (ndarray[tuple[Any, ...], dtype[_ScalarT]])

  • sim_specs (dict)

  • gen_specs (dict)

  • alloc_specs (dict)

  • persis_info (dict)

  • libE_info (dict)

Return type:

tuple[dict, dict]

give_sim_work_first.py
 1import time
 2
 3import numpy as np
 4import numpy.typing as npt
 5
 6from libensemble.tools.alloc_support import AllocSupport, InsufficientFreeResources
 7
 8
 9def give_sim_work_first(
10    W: npt.NDArray,
11    H: npt.NDArray,
12    sim_specs: dict,
13    gen_specs: dict,
14    alloc_specs: dict,
15    persis_info: dict,
16    libE_info: dict,
17) -> tuple[dict, dict]:
18    """
19    Decide what should be given to workers. This allocation function gives any
20    available simulation work first, and only when all simulations are
21    completed or running does it start (at most ``gen_specs["num_active_gens"]`` or ``alloc_specs["user"]["num_active_gens"]``)
22    generator instances.
23
24    Allows for a ``gen_specs["batch_mode"]`` or ``alloc_specs["user"]["batch_mode"]`` where no generation
25    work is given out unless all entries in ``H`` are returned.
26
27    Can give points in highest priority, if ``"priority"`` is a field in ``H``.
28    If ``gen_specs["batch_evaluate_same_priority"]`` or ``alloc_specs["user"]["batch_evaluate_same_priority"]`` is set to True, then
29    all points with the same priority value are given as a batch to the sim.
30
31    Workers performing sims will be assigned resources given in H["resource_sets"]
32    this field exists, else defaulting to one. Workers performing gens are
33    assigned resource_sets given by persis_info["gen_resources"] or zero.
34
35    This is the default allocation function if one is not defined.
36
37    tags: alloc, default, batch, priority
38
39    .. seealso::
40        `test_uniform_sampling.py <https://github.com/Libensemble/libensemble/blob/develop/libensemble/tests/functionality_tests/test_uniform_sampling.py>`_ # noqa
41    """
42
43    user = {**gen_specs, **alloc_specs.get("user", {})}
44
45    if "cancel_sims_time" in user:
46        # Cancel simulations that are taking too long
47        rows = np.where(np.logical_and.reduce((H["sim_started"], ~H["sim_ended"], ~H["cancel_requested"])))[0]
48        inds = time.time() - H["sim_started_time"][rows] > user["cancel_sims_time"]
49        to_request_cancel = rows[inds]
50        for row in to_request_cancel:
51            H[row]["cancel_requested"] = True
52
53    if libE_info["sim_max_given"] or not libE_info["any_idle_workers"]:
54        return {}, persis_info
55
56    # Initialize alloc_specs["user"] as user.
57    batch_give = user.get("batch_evaluate_same_priority", False)
58    gen_in = gen_specs.get("in", [])
59
60    manage_resources = libE_info["use_resource_sets"]
61    support = AllocSupport(W, manage_resources, persis_info, libE_info)
62    gen_count = support.count_gens()
63    Work = {}
64
65    points_to_evaluate = ~H["sim_started"] & ~H["cancel_requested"]
66
67    if np.any(points_to_evaluate):
68        for wid in support.avail_worker_ids(gen_workers=False):
69            sim_ids_to_send = support.points_by_priority(H, points_avail=points_to_evaluate, batch=batch_give)
70            try:
71                Work[wid] = support.sim_work(wid, H, sim_specs["in"], sim_ids_to_send, persis_info.get(wid))
72            except InsufficientFreeResources:
73                break
74            points_to_evaluate[sim_ids_to_send] = False
75            if not np.any(points_to_evaluate):
76                break
77    else:
78        for wid in support.avail_worker_ids(gen_workers=True):
79            # Allow at most num_active_gens active generator instances
80            if gen_count >= user.get("num_active_gens", gen_count + 1):
81                break
82
83            # Do not start gen instances in batch mode if workers still working
84            if user.get("batch_mode") and not support.all_sim_ended(H):
85                break
86
87            # Give gen work
88            return_rows = range(len(H)) if gen_in else []
89            try:
90                Work[wid] = support.gen_work(wid, gen_in, return_rows, persis_info.get(wid))
91            except InsufficientFreeResources:
92                break
93            gen_count += 1
94
95    return Work, persis_info

fast_alloc

fast_alloc.give_sim_work_first(W, H, sim_specs, gen_specs, alloc_specs, persis_info, libE_info)

This allocation function gives (in order) entries in H to idle workers to evaluate in the simulation function. The fields in sim_specs["in"] are given. If all entries in H have been given a be evaluated, a worker is told to call the generator function, provided this wouldn’t result in more than gen_specs["num_active_gens"] or alloc_specs["user"]["num_active_gens"] active generators.

This fast_alloc variation of give_sim_work_first is useful for cases that simply iterate through H, issuing evaluations in order and, in particular, is likely to be faster if there will be many short simulation evaluations, given that this function contains fewer column length operations.

tags: alloc, simple, fast

See also

test_fast_alloc.py

fast_alloc.py
 1from libensemble.tools.alloc_support import AllocSupport, InsufficientFreeResources
 2
 3
 4def give_sim_work_first(W, H, sim_specs, gen_specs, alloc_specs, persis_info, libE_info):
 5    """
 6    This allocation function gives (in order) entries in ``H`` to idle workers
 7    to evaluate in the simulation function. The fields in ``sim_specs["in"]``
 8    are given. If all entries in `H` have been given a be evaluated, a worker
 9    is told to call the generator function, provided this wouldn't result in
10    more than ``gen_specs["num_active_gens"]`` or ``alloc_specs["user"]["num_active_gens"]`` active generators.
11
12    This fast_alloc variation of give_sim_work_first is useful for cases that
13    simply iterate through H, issuing evaluations in order and, in particular,
14    is likely to be faster if there will be many short simulation evaluations,
15    given that this function contains fewer column length operations.
16
17    tags: alloc, simple, fast
18
19    .. seealso::
20        `test_fast_alloc.py <https://github.com/Libensemble/libensemble/blob/develop/libensemble/tests/functionality_tests/test_fast_alloc.py>`_ # noqa
21    """
22
23    if libE_info["sim_max_given"] or not libE_info["any_idle_workers"]:
24        return {}, persis_info
25
26    user = {**gen_specs, **alloc_specs.get("user", {})}
27
28    manage_resources = libE_info["use_resource_sets"]
29
30    support = AllocSupport(W, manage_resources, persis_info, libE_info)
31
32    gen_count = support.count_gens()
33    Work = {}
34    gen_in = gen_specs.get("in", [])
35
36    # Give sim work if possible
37    for wid in support.avail_worker_ids(gen_workers=False):
38        persis_info = support.skip_canceled_points(H, persis_info)
39        if persis_info["next_to_give"] < len(H):
40            try:
41                Work[wid] = support.sim_work(wid, H, sim_specs["in"], [persis_info["next_to_give"]], [])
42            except InsufficientFreeResources:
43                break
44            persis_info["next_to_give"] += 1
45
46    # Give gen work if possible
47    if persis_info["next_to_give"] >= len(H):
48        for wid in support.avail_worker_ids(gen_workers=True):
49            if wid not in Work and gen_count < user.get("num_active_gens", gen_count + 1):
50                return_rows = range(len(H)) if gen_in else []
51                try:
52                    Work[wid] = support.gen_work(wid, gen_in, return_rows, persis_info.get(wid))
53                except InsufficientFreeResources:
54                    break
55                gen_count += 1
56                persis_info["total_gen_calls"] += 1
57
58    return Work, persis_info

start_persistent_local_opt_gens

start_persistent_local_opt_gens.start_persistent_local_opt_gens(W, H, sim_specs, gen_specs, alloc_specs, persis_info, libE_info)

This allocation function will do the following:

  • Start up a persistent generator that is a local opt run at the first point identified by APOSMM’s decide_where_to_start_localopt. Note, it will do this only if at least one worker will be left to perform simulation evaluations.

  • If multiple starting points are available, the one with smallest function value is chosen.

  • If no candidate starting points exist, points from existing runs will be evaluated (oldest first).

  • If no points are left, call the generation function.

tags: alloc, persistent, aposmm

See also

test_uniform_sampling_then_persistent_localopt_runs.py

fast_alloc_and_pausing

fast_alloc_and_pausing.give_sim_work_first(W, H, sim_specs, gen_specs, alloc_specs, persis_info, libE_info)

This allocation function gives (in order) entries in H to idle workers to evaluate in the simulation function. The fields in sim_specs["in"] are given. If all entries in H have been given a be evaluated, a worker is told to call the generator function, provided this wouldn’t result in more than alloc_specs["user"]["num_active_gen"] active generators. Also allows for a “batch_mode”.

When there are multiple objective components, this allocation function does not evaluate further components for some point in the following scenarios:

alloc_specs[“user”][“stop_on_NaNs”]: True — after a NaN has been found in returned in some

objective component

alloc_specs[“user”][“stop_partial_fvec_eval”]: True — after the value returned from

combine_component_func is larger than a known upper bound on the objective.

See also

test_uniform_sampling_one_residual_at_a_time.py

only_one_gen_alloc

only_one_gen_alloc.ensure_one_active_gen(W, H, sim_specs, gen_specs, alloc_specs, persis_info, libE_info)

This allocation function gives (in order) entries in H to idle workers to evaluate in the simulation function. The fields in sim_specs["in"] are given. If there is no active generator, then one is started.

See also

test_fast_alloc.py

start_fd_persistent

start_fd_persistent.finite_diff_alloc(W, H, sim_specs, gen_specs, alloc_specs, persis_info, libE_info)

This allocation function will give simulation work if possible, but otherwise start 1 persistent generator. If all points requested by the persistent generator for a given (x_ind, f_ind) pair have been returned from the simulation evaluation, then this information is given back to the persistent generator (where x_ind is in range(n) and f_ind is in range(p))

See also

test_persistent_fd_param_finder.py

persistent_aposmm_alloc

persistent_aposmm_alloc.persistent_aposmm_alloc(W, H, sim_specs, gen_specs, alloc_specs, persis_info, libE_info)

This allocation function will give simulation work if possible, but otherwise start a persistent APOSMM generator. If all points requested by the persistent generator have been returned from the simulation evaluation, then this information is given back to the persistent generator.

This function assumes that one persistent APOSMM will be started and never stopped (until some exit_criterion is satisfied).

See also

test_persistent_aposmm_with_grad.py

give_pregenerated_work

give_pregenerated_work.give_pregenerated_sim_work(W, H, sim_specs, gen_specs, alloc_specs, persis_info, libE_info)

This allocation function gives (in order) entries in alloc_spec[“x”] to idle workers. It is an example use case where no gen_func is used.

See also

test_fast_alloc.py

inverse_bayes_allocf

inverse_bayes_allocf.only_persistent_gens_for_inverse_bayes(W, H, sim_specs, gen_specs, alloc_specs, persis_info, libE_info)

Starts up to gen_count number of persistent generators. These persistent generators produce points (x) in batches and subbatches. The points x are given in subbatches to workers to perform a calculation. When all subbatches have returned, their output is given back to the corresponding persistent generator.

The first time called there are no persis_w 1st for loop is not done