Communicators and groups

Questions

  • How can we write extensible and safe parallel libraries?

Objectives

The use of software libraries allows programmers to develop new software without having to reinvent the wheel every time one is needed. Software components, if properly packaged and documented, can be easily reused and combined to fit the programmers purposes. How can we write high quality libraries that can efficiently exploit parallelism in the message passing model?

Let us have a look at the basic point-to-point communication routines MPI_Send and MPI_Recv.

../_images/E02-send-recv_step2.svg

Point-to-point communication between two sloths. The tag of the message can be used to match messages between sending and receiving partners.

Both accept a tag parameter, used to match messages sent and received. As a library author, one might decide to reserve a certain range of integers for exclusive use as tags for messages originating from the code. This is however not an optimal solution:

  1. It imposes a restrictive design onto potential users. This is a choice that is not intrinsic to the problem domain and usually suggests a bad design.

  2. The design is not extensible. What if one runs out of tags and need to add a new one to the reserved list? One may break the API or, worse, interfere with existing tagging in client code.

  3. The library is not, most likely, the only one used. How can one guarantee that the choice of reserved tags is unique?

Clearly, using hard-coded tags to distinguish the origin of messages does not scale with the number of components involved in a software project and will reduce the extensibility and maintainability of the codebase.

What might happen if some other component uses your message tag?

  1. The program hangs waiting for the right message

  2. The program crashes

  3. The program continues normally, but perhaps with wrong results

  4. All of the above

Note that MPI_Send and MPI_Recv also take a communicator as parameter and indeed the use of communicators is the solution offered by the MPI standard for the problems faced by library authors. Even if one is not writing parallel libraries, it is a good idea to get acquainted with the concepts of communicators and groups. Communicators come in two flavors:

  1. Intracommunicators. Essentially a collection of processes that can interact through point-to-point and collective message passing routines.

  2. Intercommunicators. These are collection of processes in disjoint intracommunicators that can interact through message passing.

Intracommunicators are the most commonly occuring flavor. We will only deal with intracommunicators in this lesson and, following general practice, we will simply call them communicators from now on. A communicator consists of:

  • A group, i.e. an ordered collection of processes. Each process in the group is assigned a rank, a non-negative integer number. The rank uniquely identifies each process in the group.

  • A context, i.e. a system-defined object that identifies each communicator uniquely. Since the context is unique to the communicator, the same group can span different communicators, without causing issues.

Warm up your communicators and your groups

Time for some practice! We will run with 4 processes, divide them evenly into two groups and create a new communicator out of them.

You can find a scaffold for the code in the content/code/day-1/01_comms-groups-divide-evanly folder. You will have to complete the source code to compile and run correctly: follow the hints in the source file. Compile with:

mpicc -g -Wall -std=c11 comms-groups-divide-evenly.c -o comms-groups-divide-evenly

A working solution is in the solution subfolder.

Calculating \(\pi\)

The computation of high-dimensional integrals with Monte Carlo method can be quite efficient. We will aim at a more modest target here: the calculation of \(\pi\) by Monte Carlo sampling. Given a circle of radius 1, the ratio of randomly drawn points inside and outside the circle will converge to \(\frac{\pi}{4}\). Rather than have every process invoke rand separately, we would like to reserve one single process for that purpose and use all the others to check whether these points are inside or outside the circle. To reduce communication traffic, this single process will fill and send a whole array of random numbers.

You can find a scaffold for the code in the content/code/day-1/02_compute-pi folder.

  1. Only one process will generate random data. Define a checkers_g group including all other processes which will be acting as checkers.

  2. Define a communicator for the checker processes.

  3. The rng process listens for requests and serves them by sending an array of random data. The scaffold defines REQUEST and REPLY tags.

  4. The checkers group goes through the random data and tallies the number of points inside and outside the circle.

  5. Unless we meet the user-provided error threshold, checker processes will request a fresh batch of random data.

Compile with:

mpicc -g -Wall -std=c11 pi-monte-carlo.c -o pi-monte-carlo

A working solution is in the solution subfolder.

See also

  • Chapters 3, 4, and 6 of the Using MPI book by William Gropp et al. show examples of using the functions described in this episode. [GLS14]

  • Chapter 7 of the Parallel Programming with MPI book by Peter Pacheco. [Pac97]

Keypoints

  • You can use tags to differentiate the source of messages, but this approach is neither extensible nor safe.

  • You can create new communicators by splitting or grouping.