calc_status

The calc_status is similar to an exit code, and is either a built-in integer attribute with a named (enumerated) value and corresponding description, or a user-specified string. They are determined within user functions to capture the status of a calculation, and returned to the manager and printed to the libE_stats.txt file. Only the status values FINISHED_PERSISTENT_SIM_TAG and FINISHED_PERSISTENT_GEN_TAG are currently used by the manager, but others can still provide a useful summary in libE_stats.txt. The user is responsible for determining the status of a user function instance, since a given instance could include multiple application runs of mixed rates of success.

The calc_status is the third optional return value from a user function. Built-in codes are available in the libensemble.message_numbers module, but users are also free to return any custom string.

Example of calc_status used along with Executor in sim_f:

 1  from libensemble.message_numbers import WORKER_DONE, WORKER_KILL, TASK_FAILED
 2
 3  task = exctr.submit(calc_type="sim", num_procs=cores, wait_on_start=True)
 4  calc_status = UNSET_TAG
 5  poll_interval = 1  # secs
 6  while not task.finished:
 7      if task.runtime > time_limit:
 8          task.kill()  # Timeout
 9      else:
10          time.sleep(poll_interval)
11          task.poll()
12
13  if task.finished:
14      if task.state == "FINISHED":
15          print("Task {} completed".format(task.name))
16          calc_status = WORKER_DONE
17      elif task.state == "FAILED":
18          print("Warning: Task {} failed: Error code {}".format(task.name, task.errcode))
19          calc_status = TASK_FAILED
20      elif task.state == "USER_KILLED":
21          print("Warning: Task {} has been killed".format(task.name))
22          calc_status = WORKER_KILL
23      else:
24          print("Warning: Task {} in unknown state {}. Error code {}".format(task.name, task.state, task.errcode))
25
26  outspecs = sim_specs["out"]
27  output = np.zeros(1, dtype=outspecs)
28  output["energy"][0] = final_energy
29
30  return output, persis_info, calc_status

Example of defining and returning a custom calc_status if the built-in values are insufficient:

 1  from libensemble.message_numbers import WORKER_DONE, TASK_FAILED
 2
 3  task = exctr.submit(calc_type="sim", num_procs=cores, wait_on_start=True)
 4
 5  task.wait(timeout=60)
 6
 7  file_output = read_task_output(task)
 8  if task.errcode == 0:
 9      if "fail" in file_output:
10          calc_status = "Task failed successfully?"
11      else:
12          calc_status = WORKER_DONE
13  else:
14      calc_status = TASK_FAILED
15
16  outspecs = sim_specs["out"]
17  output = np.zeros(1, dtype=outspecs)
18  output["energy"][0] = final_energy
19
20  return output, persis_info, calc_status

See forces_simf.py for a complete example. See uniform_or_localopt.py for an example of using FINISHED_PERSISTENT_GEN_TAG.

Available values:

FINISHED_PERSISTENT_SIM_TAG = 11  # tells manager sim_f done persistent mode
FINISHED_PERSISTENT_GEN_TAG = 12  # tells manager gen_f done persistent mode
MAN_SIGNAL_FINISH = 20  # Kill tasks and shutdown worker
MAN_SIGNAL_KILL = 21  # Kill running task - but don't stop worker
WORKER_KILL = 30  # Worker kills not covered by a more specific case
WORKER_KILL_ON_ERR = 31  # Worker killed due to an error in results
WORKER_KILL_ON_TIMEOUT = 32  # Worker killed on timeout
TASK_FAILED = 33  # Calc had tasks that failed
WORKER_DONE = 34  # Calculation was successful

The corresponding messages printed to the libE_stats.txt file:

calc_status_strings = {
    UNSET_TAG: "Not set",
    FINISHED_PERSISTENT_SIM_TAG: "Persis sim finished",
    FINISHED_PERSISTENT_GEN_TAG: "Persis gen finished",
    MAN_SIGNAL_FINISH: "Manager killed on finish",
    MAN_SIGNAL_KILL: "Manager killed task",
    WORKER_KILL_ON_ERR: "Worker killed task on Error",
    WORKER_KILL_ON_TIMEOUT: "Worker killed task on Timeout",
    WORKER_KILL: "Worker killed",
    TASK_FAILED: "Task Failed",
    WORKER_DONE: "Completed",
    CALC_EXCEPTION: "Exception occurred",
    None: "Unknown Status",
}

Example segment of libE_stats.txt:

Worker     1: Calc     0: gen Time: 0.00 Start: 2019-11-19 18:53:43 End: 2019-11-19 18:53:43 Status: Not set
Worker     1: Calc     1: sim Time: 4.41 Start: 2019-11-19 18:53:43 End: 2019-11-19 18:53:48 Status: Worker killed
Worker     2: Calc     0: sim Time: 5.42 Start: 2019-11-19 18:53:43 End: 2019-11-19 18:53:49 Status: Completed
Worker     1: Calc     2: sim Time: 2.41 Start: 2019-11-19 18:53:48 End: 2019-11-19 18:53:50 Status: Worker killed
Worker     2: Calc     1: sim Time: 2.41 Start: 2019-11-19 18:53:49 End: 2019-11-19 18:53:51 Status: Worker killed
Worker     1: Calc     3: sim Time: 4.41 Start: 2019-11-19 18:53:50 End: 2019-11-19 18:53:55 Status: Completed
Worker     2: Calc     2: sim Time: 4.41 Start: 2019-11-19 18:53:51 End: 2019-11-19 18:53:56 Status: Completed
Worker     1: Calc     4: sim Time: 4.41 Start: 2019-11-19 18:53:55 End: 2019-11-19 18:53:59 Status: Completed
Worker     2: Calc     3: sim Time: 4.41 Start: 2019-11-19 18:53:56 End: 2019-11-19 18:54:00 Status: Completed