{
"cells": [
{
"cell_type": "markdown",
"id": "7606d30a",
"metadata": {},
"source": [
"
\n",
"\n",
"### Programming large-scale parallel systems"
]
},
{
"cell_type": "markdown",
"id": "4ac1e5d9",
"metadata": {},
"source": [
"# Intro to MPI (collectives)"
]
},
{
"cell_type": "markdown",
"id": "a341be2e",
"metadata": {},
"source": [
"## Contents\n",
"\n",
"\n",
"In this notebook, we will continue learning the basics of parallel computing using the Message Passing Interface (MPI) from Julia. In particular, we will learn:\n",
"\n",
"- How to use basic collective communication directives\n",
"- How to use MPI communicators\n"
]
},
{
"cell_type": "markdown",
"id": "c87b3c82",
"metadata": {},
"source": [
"## Collective communication\n",
"\n",
"MPI provides a set of routines for communication involving multiple processes. These are called *collective communication* operations. Some usual collective operations are:\n",
"\n",
"\n",
"- `MPI_Barrier`: Synchronize all processes\n",
"- `MPI_Bcase`: Send the same data from one to all processes\n",
"- `MPI_Gather`: Gather data from all processes to one\n",
"- `MPI_Scatter`: Scatter data from one to all all processes\n",
"- `MPI_Reduce`: Reduce data from all processes to a single one\n",
"- `MPI_Scan`: Scan (or prefix) reduction\n",
"- `MPI_Allgather`: Like a `MPI_Gather` but all processes receive the result\n",
"- `MPI_Allreduce`: Like `MPI_Reduce` but all processes receive the result\n",
"- `MPI_Alltoall`: Exchange data from all to all processes\n",
"\n",
"\n",
"We will discuss some of them in greater detail in this notebook."
]
},
{
"cell_type": "markdown",
"id": "4ffa5e56",
"metadata": {},
"source": [
"## Why collective operations?\n",
"\n",
"Point-to-point communication functions provide all the building blocks needed in parallel programs and could be used to implement the collective functions described above. Then, why does MPI provide collective communication functions? There are several reasons:\n",
"\n",
"- Ease of use: It is handy for users to have these functions readily available instead of having to implement them.\n",
"- Performance: Library implementations typically use efficient algorithms (such as reduction trees).\n",
"- Hardware support: Hardware vendors can optimize the MPI library for their machine and even develop hardware specially design to perform MPI operations efficiently. For instance, [IMB Blue Gene](https://en.wikipedia.org/wiki/IBM_Blue_Gene) was equipped with auxiliary networks for MPI collectives.\n"
]
},
{
"cell_type": "markdown",
"id": "53a0f5fa",
"metadata": {},
"source": [
"## Semantics of collective operations\n",
"\n",
"These are key properties of collective operations:\n",
"\n",
"\n",
"- Completeness: All the collective communication directives above are *complete* operations. Thus, it is safe to use and reset the buffers once the function returns.\n",
"- Standard mode: Collective directives are in standard mode only, like `MPI_Send`. Assuming that they block is erroneous, assuming that they do not block is also erroneous.\n",
"- Synchronization: Completion of a call does not guarantee that other processes have completed the operation. A collective operation may or may not have the effect of synchronizing all processes, the only exception is `MPI_Barrier` of course.\n",
"\n",
"\n",
"