Generalized forms of gather

Questions

  • How can I do more complex rearrangements of data?

Objectives

  • Know that gather can be generalized

  • Know the difference between generalized forms of gather

All-gather

An MPI_Allgather call gather the same data from all ranks and provides it to all ranks. It is logically identical to MPI_Gather to a root followed by an MPI_Bcast from that root, but is implemented more efficiently.

../_images/MPI_Allgather.svg

After the call, all ranks have one value from each other rank in the communicator, ordered by rank number.

MPI_Allgather is blocking and introduces collective synchronization into the program. Note that there is no root for this operation.

This can be useful to allow all ranks to collect values from all other ranks in the communicator. For example, all ranks might compute some values, and then all ranks gather that content to use it in a subsequent stage.

All-to-all

An MPI_Alltoall call gathers data from all ranks and provides distinct data to all ranks. It is logically identical to making one call to MPI_Gather for each possible root rank, but is implemented more efficiently.

../_images/MPI_Alltoall.svg

After the call, all ranks have one value from each other rank in the communicator, ordered by rank number.

MPI_Alltoall is blocking and introduces collective synchronization into the program. Note that there is no root for this operation.

This can be useful to allow all ranks to collect values from all other ranks in the communicator. For example, a 3D Fast Fourier Transform often uses an all-to-all operation to redistribute the working data set for each process to a new dimension.

Exercise: all-gather and all-to-all

Use all-gather

You can find a scaffold for the code in the content/code/day-2/01_allgather folder. A working solution is in the solution subfolder. Try to compile with:

mpicc -g -Wall -std=c11 collective-communication-allgather.c -o collective-communication-allgather
  1. When you have the code compiling, try to run with:

    mpiexec -np 4 ./collective-communication-allgather
    
  2. Use clues from the compiler and the comments in the code to change the code so it compiles and runs. Try to get all ranks to report success :-)

Use all-to-all

You can find a scaffold for the code in the content/code/day-2/02_alltoall folder. A working solution is in the solution subfolder. Try to compile with:

mpicc -g -Wall -std=c11 collective-communication-alltoall.c -o collective-communication-alltoall
  1. When you have the code compiling, try to run with:

    mpiexec -np 4 ./collective-communication-alltoall
    
  2. Use clues from the compiler and the comments in the code to change the code so it compiles and runs. Try to get all ranks to report success :-)

Scatterv

An MPI_Scatterv call scatters a buffer in parts to all ranks. It uses sendcounts and displs to determine the number of elements to be sent to each rank.

MPI_Scatterv is blocking and introduces collective synchronization into the program.

This can be useful to when we want to distribute different amount of data to different processes.

Gatherv

An MPI_Gatherv call gathers info with variable size from all ranks. It uses recvcounts and displs to determine the number of elements to be collected from each rank.

MPI_Gatherv is blocking and introduces collective synchronization into the program.

This can be useful to when we want to collect different amount of data from different processes.

Exercise: scatterv and gatherv

Use scatterv and gatherv to compute matrix vector multiplication

In this exercise we compute matrix vector multiplication using MPI_Scatterv and MPI_Gatherv. One would need to scatter the row vectors of the matrix to the individual processes, broadcast the vector, and then calculate the local contribution to the matrix vector product. After that the local contributions are collected to the root process to form the final result.

../_images/mat-vec.png

Matrix vector multiplication in parallel

You can find a scaffold for the code in the content/code/day-2/03_scatterv-and-gatherv folder. A working solution is in the solution subfolder. Try to compile with:

mpicc -g -Wall -std=c11 scatterv-and-gatherv.c -o scatterv-and-gatherv
  1. When you have the code compiling, try to run with different number of processes.

  2. Try to get rank 0 to report success :-)

Final thoughts

There are further generalizations available in MPI (e.g. combined scatter-gather). Check the existing options before rolling your own or giving up!

See also

Keypoints

  • More complex distribution patterns are also optimized in MPI