# Multiprocessing with a Simple Sine

This introductory tutorial demonstrates the capability to perform ensembles of calculations in parallel using libEnsemble with Python’s multiprocessing.

The foundation of writing libEnsemble routines is accounting for at least three components:

A generator function, that produces values for simulations

A simulator function, that performs simulations based on values from the generator function

A calling script, for defining settings, fields, and functions, then starting the run

libEnsemble initializes a *manager* process and as many *worker* processes as the
user requests. The manager (via the allocation function)
coordinates data transfer between workers and assigns units of work to each worker,
consisting of a function to run and accompanying data. These functions perform their
work in-line with Python and/or launch and control user applications with
libEnsemble’s Executors. Workers pass results back to the manager.

For this tutorial, we’ll write our generator and simulator functions entirely in Python without other applications. Our generator will produce uniform randomly sampled values, and our simulator will calculate the sine of each. By default we don’t need to write a new allocation function. All generated and simulated values alongside other parameters are stored in H, the history array.

## Getting started

libEnsemble and its functions are written entirely in Python. Let’s make sure the correct version is installed.

```
$ python --version
Python 3.7.0 # This should be >= 3.7
```

For this tutorial, you need NumPy to perform calculations and (optionally) Matplotlib to visualize your results. Install libEnsemble and these other libraries with

```
$ pip install numpy
$ pip install libensemble
$ pip install matplotlib # Optional
```

If your system doesn’t allow you to perform these installations, try adding
`--user`

to the end of each command.

## Generator function

Let’s begin the coding portion of this tutorial by writing our generator function, or gen_f.

An available libEnsemble worker will call this generator function with the following parameters:

H_in: A selection of the History array, a NumPy structured array for storing information about each point generated and processed in the ensemble. libEnsemble passes a selection of

`H`

to the generator function in case the user wants to generate new values based on previous data.persis_info: Dictionary with worker-specific information. In our case, this dictionary contains NumPy Random Stream objects for generating random numbers.

gen_specs: Dictionary with user-defined fields and parameters for the generator. Customizable parameters such as boundaries and batch sizes are placed within the

`gen_specs['user']`

dictionary, while input/output fields and other specifications that libEnsemble depends on to operate the generator are placed outside`user`

.

Later on, we’ll populate `gen_specs`

and `persis_info`

when we initialize libEnsemble.

For now, create a new Python file named `generator.py`

. Write the following:

```
1import numpy as np
2
3def gen_random_sample(H_in, persis_info, gen_specs, _):
4 # Underscore ignores advanced arguments
5
6 # Pull out user parameters
7 user_specs = gen_specs['user']
8
9 # Get lower and upper bounds
10 lower = user_specs['lower']
11 upper = user_specs['upper']
12
13 # Determine how many values to generate
14 num = len(lower)
15 batch_size = user_specs['gen_batch_size']
16
17 # Create empty array of 'batch_size' zeros. Array dtype should match 'out' fields
18 out = np.zeros(batch_size, dtype=gen_specs['out'])
19
20 # Set the 'x' output field to contain random numbers, using random stream
21 out['x'] = persis_info['rand_stream'].uniform(lower, upper, (batch_size, num))
22
23 # Send back our output and persis_info
24 return out, persis_info
```

Our function creates `batch_size`

random numbers uniformly distributed
between the `lower`

and `upper`

bounds. A random stream
from `persis_info`

is used to generate these values, which are then placed
into an output NumPy array that meets the specifications from `gen_specs['out']`

.

### Exercise

Write a simple generator function that instead produces random integers, using
the `numpy.random.Generator.integers(low, high, size)`

function.

**Click Here for Solution**

```
1import numpy as np
2
3def gen_random_ints(H_in, persis_info, gen_specs, _):
4
5 user_specs = gen_specs['user']
6 lower = user_specs['lower']
7 upper = user_specs['upper']
8 num = len(lower)
9 batch_size = user_specs['gen_batch_size']
10
11 out = np.zeros(batch_size, dtype=gen_specs['out'])
12 out['x'] = persis_info['rand_stream'].integers(lower, upper, (batch_size, num))
13
14 return out, persis_info
```

## Simulator function

Next, we’ll write our simulator function or sim_f. Simulator
functions perform calculations based on values from the generator function.
The only new parameter here is sim_specs, which
serves a purpose similar to the `gen_specs`

dictionary.

Create a new Python file named `simulator.py`

. Write the following:

```
1import numpy as np
2
3def sim_find_sine(H_in, persis_info, sim_specs, _):
4 # underscore for internal/testing arguments
5
6 # Create an output array of a single zero
7 out = np.zeros(1, dtype=sim_specs['out'])
8
9 # Set the zero to the sine of the input value stored in H
10 out['y'] = np.sin(H_in['x'])
11
12 # Send back our output and persis_info
13 return out, persis_info
```

Our simulator function is called by a worker for every work item produced by
the generator function. This function calculates the sine of the passed value,
then returns it so a worker can log it into `H`

.

### Exercise

Write a simple simulator function that instead calculates the *cosine* of a received
value, using the `numpy.cos(x)`

function.

**Click Here for Solution**

```
1import numpy as np
2
3def sim_find_cosine(H_in, persis_info, gen_specs, _):
4
5 out = np.zeros(1, dtype=sim_specs['out'])
6
7 out['y'] = np.cos(H_in['x'])
8
9 return out, persis_info
```

## Calling Script

Now we can write the calling script that configures our generator and simulator functions and calls libEnsemble.

Create an empty Python file named `calling_script.py`

.
In this file, we’ll start by importing NumPy, libEnsemble, and the generator and
simulator functions we just created.

Next, in a dictionary called libE_specs we’ll
specify the number of workers and the type of manager/worker communication
libEnsemble will use. Our communication method, `'local'`

, refers to Python’s
multiprocessing.

```
1import numpy as np
2from libensemble.libE import libE
3from generator import gen_random_sample
4from simulator import sim_find_sine
5from libensemble.tools import add_unique_random_streams
6
7nworkers = 4
8libE_specs = {'nworkers': nworkers, 'comms': 'local'}
```

We configure the settings and specifications for our `sim_f`

and `gen_f`

functions in the gen_specs and
sim_specs dictionaries, which we saw previously
being passed to our functions. These dictionaries also describe to libEnsemble
what inputs and outputs from those functions to expect.

```
1gen_specs = {'gen_f': gen_random_sample, # Our generator function
2 'out': [('x', float, (1,))], # gen_f output (name, type, size)
3 'user': {
4 'lower': np.array([-3]), # lower boundary for random sampling
5 'upper': np.array([3]), # upper boundary for random sampling
6 'gen_batch_size': 5 # number of x's gen_f generates per call
7 }
8 }
9
10sim_specs = {'sim_f': sim_find_sine, # Our simulator function
11 'in': ['x'], # Input field names. 'x' from gen_f output
12 'out': [('y', float)]} # sim_f output. 'y' = sine('x')
```

Recall that each worker is assigned an entry in the
persis_info dictionary that, in this tutorial,
contains a `RandomState()`

random stream for uniform random sampling. We
populate that dictionary here using a utility from the
tools module. We then specify the circumstances where
libEnsemble should stop execution in exit_criteria.

```
1persis_info = add_unique_random_streams({}, nworkers+1) # Worker numbers start at 1
2
3exit_criteria = {'sim_max': 80} # Stop libEnsemble after 80 simulations
```

Now we’re ready to write our libEnsemble libE
function call. This H is the final version of
the history array. `flag`

should be zero if no errors occur.

```
1H, persis_info, flag = libE(sim_specs, gen_specs, exit_criteria, persis_info,
2 libE_specs=libE_specs)
3
4print([i for i in H.dtype.fields]) # (optional) to visualize our history array
5print(H)
```

That’s it! Now that these files are complete, we can run our simulation.

```
$ python calling_script.py
```

If everything ran perfectly and you included the above print statements, you
should get something similar to the following output for `H`

(although the
columns might be rearranged).

```
['y', 'sim_started_time', 'gen_worker', 'sim_worker', 'sim_started', 'sim_ended', 'x', 'allocated', 'sim_id', 'gen_ended_time']
[(-0.37466051, 1.559+09, 2, 2, True, True, [-0.38403059], True, 0, 1.559+09)
(-0.29279634, 1.559+09, 2, 3, True, True, [-2.84444261], True, 1, 1.559+09)
( 0.29358492, 1.559+09, 2, 4, True, True, [ 0.29797487], True, 2, 1.559+09)
(-0.3783986 , 1.559+09, 2, 1, True, True, [-0.38806564], True, 3, 1.559+09)
(-0.45982062, 1.559+09, 2, 2, True, True, [-0.47779319], True, 4, 1.559+09)
...
```

In this arrangement, our output values are listed on the far left with the generated values being the fourth column from the right.

Two additional log files should also have been created.
`ensemble.log`

contains debugging or informational logging output from
libEnsemble, while `libE_stats.txt`

contains a quick summary of all
calculations performed.

Here is graphed output using `Matplotlib`

, with entries colored by which
worker performed the simulation:

If you want to verify your results through plotting and installed Matplotlib
earlier, copy and paste the following code into the bottom of your calling
script and run `python calling_script.py`

again

```
1import matplotlib.pyplot as plt
2colors = ['b', 'g', 'r', 'y', 'm', 'c', 'k', 'w']
3
4for i in range(1, nworkers + 1):
5 worker_xy = np.extract(H['sim_worker'] == i, H)
6 x = [entry.tolist()[0] for entry in worker_xy['x']]
7 y = [entry for entry in worker_xy['y']]
8 plt.scatter(x, y, label='Worker {}'.format(i), c=colors[i-1])
9
10plt.title('Sine calculations for a uniformly sampled random distribution')
11plt.xlabel('x')
12plt.ylabel('sine(x)')
13plt.legend(loc = 'lower right')
14plt.savefig('tutorial_sines.png')
```

Each of these example files can be found in the repository in examples/tutorials/simple_sine.

### Exercise

Write a Calling Script with the following specifications:

Use the

`parse_args()`

function to detect`nworkers`

and auto-populate`libE_specs`

Set the generator function’s lower and upper bounds to -6 and 6, respectively

Increase the generator batch size to 10

Set libEnsemble to stop execution after 160

generationsusing the`gen_max`

keyPrint an error message if any errors occurred while libEnsemble was running

**Click Here for Solution**

```
1import numpy as np
2from libensemble.libE import libE
3from generator import gen_random_sample
4from simulator import sim_find_sine
5from libensemble.tools import add_unique_random_streams
6
7nworkers, is_manager, libE_specs, _ = parse_args()
8
9gen_specs = {'gen_f': gen_random_ints,
10 'out': [('x', float, (1,))],
11 'user': {
12 'lower': np.array([-6]),
13 'upper': np.array([6]),
14 'gen_batch_size': 10
15 }
16 }
17
18sim_specs = {'sim_f': sim_find_sine,
19 'in': ['x'],
20 'out': [('y', float)]}
21
22persis_info = add_unique_random_streams({}, nworkers+1)
23exit_criteria = {'gen_max': 160}
24
25H, persis_info, flag = libE(sim_specs, gen_specs, exit_criteria, persis_info,
26 libE_specs=libE_specs)
27
28if flag != 0:
29 print('Oh no! An error occurred!')
```

## Next steps

### libEnsemble with MPI

MPI is a standard interface for parallel computing, implemented in libraries such as MPICH and used at extreme scales. MPI potentially allows libEnsemble’s manager and workers to be distributed over multiple nodes and works in some circumstances where Python’s multiprocessing does not. In this section, we’ll explore modifying the above code to use MPI instead of multiprocessing.

We recommend the MPI distribution MPICH for this tutorial, which can be found
for a variety of systems here. You also need mpi4py, which can be installed
with `pip install mpi4py`

. If you’d like to use a specific version or
distribution of MPI instead of MPICH, configure mpi4py with that MPI at
installation with `MPICC=<path/to/MPI_C_compiler> pip install mpi4py`

If this
doesn’t work, try appending `--user`

to the end of the command. See the
mpi4py docs for more information.

Verify that MPI has installed correctly with `mpirun --version`

.

### Modifying the calling script

Only a few changes are necessary to make our code MPI-compatible. Modify the top of the calling script as follows:

```
1import numpy as np
2from libensemble.libE import libE
3from generator import gen_random_sample
4from simulator import sim_find_sine
5from libensemble.tools import add_unique_random_streams
6from mpi4py import MPI
7
8# nworkers = 4 # nworkers will come from MPI
9libE_specs = {'comms': 'mpi'} # 'nworkers' removed, 'comms' now 'mpi'
10
11nworkers = MPI.COMM_WORLD.Get_size() - 1
12is_manager = (MPI.COMM_WORLD.Get_rank() == 0) # manager process has MPI rank 0
```

So that only one process executes the graphing and printing portion of our code, modify the bottom of the calling script like this:

```
1 H, persis_info, flag = libE(sim_specs, gen_specs, exit_criteria, persis_info,
2 libE_specs=libE_specs)
3
4 if is_manager:
5 # Some (optional) statements to visualize our history array
6 print([i for i in H.dtype.fields])
7 print(H)
8
9 import matplotlib.pyplot as plt
10 colors = ['b', 'g', 'r', 'y', 'm', 'c', 'k', 'w']
11
12 for i in range(1, nworkers + 1):
13 worker_xy = np.extract(H['sim_worker'] == i, H)
14 x = [entry.tolist()[0] for entry in worker_xy['x']]
15 y = [entry for entry in worker_xy['y']]
16 plt.scatter(x, y, label='Worker {}'.format(i), c=colors[i-1])
17
18 plt.title('Sine calculations for a uniformly sampled random distribution')
19 plt.xlabel('x')
20 plt.ylabel('sine(x)')
21 plt.legend(loc='lower right')
22 plt.savefig('tutorial_sines.png')
```

With these changes in place, our libEnsemble code can be run with MPI by

```
$ mpirun -n 5 python calling_script.py
```

where `-n 5`

tells `mpirun`

to produce five processes, one of which will be
the manager process with the libEnsemble manager and the other four will run
libEnsemble workers.

This tutorial is only a tiny demonstration of the parallelism capabilities of libEnsemble. libEnsemble has been developed primarily to support research on High-Performance computers, with potentially hundreds of workers performing calculations simultaneously. Please read our platform guides for introductions to using libEnsemble on many such machines.

libEnsemble’s Executors can launch non-Python user applications and simulations across allocated compute resources. Try out this feature with a more-complicated libEnsemble use-case within our Electrostatic Forces tutorial.