seb/XM_40017
1
0 Fork 0
mirror of https://github.com/fverdugo/XM_40017.git synced 2026-11-15 04:57:17 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #38 from fverdugo/francesc

Browse Source 
More changes in notebooks
This commit is contained in:
Francesc Verdugo authored and GitHub committed 2024-08-26 12:58:28 +02:00
2 parents 4e6b7696f0 02968243ba
commit 567534f594
5 files changed
+1104 -45
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/src/getting_started_with_julia.md
+59 -1
View File
@@ -124,6 +124,11 @@ $ julia --version
If this runs without error and you see a version number, you are good to go!
You can also run julia code from the terminal using the `-e` flag:
```
$ julia -e 'println("Hello, world!")'
```
!!! note
In this tutorial, when a code snipped starts with `$`, it should be run in the terminal. Otherwise, the code is to be run in the Julia REPL.
@@ -366,9 +371,62 @@ is equivalent to calling `status` in package mode.
(@v1.10) pkg> status
```
### Creating you own package
In many situations is useful to create your own package, for instance, when working with a large code base, when you want to reduce compilation latency using [`Revise.jl`](https://github.com/timholy/Revise.jl),
or if you want to eventually [register your package](https://github.com/JuliaRegistries/Registrator.jl) and share it with others.
The simplest way of generating a package (called `MyPackage`) is as follows. Open Julia go to package mode and type
```julia
(@v1.10) pkg> generate MyPackage
```
This will crate a minimal package consisting of a new folder `MyPackage` with two files:
* `MyPackage/Project.toml`: Project file defining the direct dependencies of your package.
* `MyPackage/src/MyPackage.jl`: Main source file of your package. You can split your code in several files if needed, and include them in the package main file using function `include`.
!!! tip
This approach only generates a very minimal package. To create a more sophisticated package skeleton (including unit testing, code coverage, readme file, licence, etc.) use
[`PkgTemplates.jl`](https://github.com/JuliaCI/PkgTemplates.jl) or [`BestieTemplate.jl`](https://github.com/abelsiqueira/BestieTemplate.jl). The later one is developed in Amsterdam at the
[Netherlands eScience Center](https://www.esciencecenter.nl/).
You can add dependencies to the package by activating the `MyPackage` folder in package mode and adding new dependencies as always:
```julia
(@v1.10) pkg> activate MyPackage
(MyPackage) pkg> add MPI
```
This will add MPI to your package dependencies.
### Using your own package
To use your package you first need to add it to a package environment of your choice. This is done with `develop path/to/the/package/folder` in package mode. For instance:
```julia
(@v1.10) pkg> develop MyPackage
```
!!! note
You do not need to "develop" your package if you activated the package folder `MyPackage`.
Now, we can go back to standard Julia mode and use it as any other package:
```julia
using MyPackage
MyPackage.greet()
```
Here, we just called the example function defined in `MyPackage/src/MyPackage.jl`.
## Conclusion
We have learned the basics of how to work with Julia. If you want to further dig into the topics we have covered here, you can take a look at the following links:
We have learned the basics of how to work with Julia, including how to run serial and parallel code, and how to manage, create, and use Julia packages.
This knowledge will allow you to follow the course effectively!
If you want to further dig into the topics we have covered here, you can take a look at the following links:
- [Julia Manual](https://docs.julialang.org/en/v1/manual/getting-started/)
- [Package manager](https://pkgdocs.julialang.org/v1/getting-started/)
notebooks/jacobi_method.ipynb
+92 -23
View File
@@ -27,9 +27,9 @@
"\n",
"In this notebook, we will learn\n",
"\n",
"- How to paralleize a Jacobi method\n",
"- How to paralleize the Jacobi method\n",
"- How the data partition can impact the performance of a distributed algorithm\n",
"- How to use latency hiding\n",
"- How to use latency hiding to improve parallel performance\n",
"\n"
]
},
@@ -93,9 +93,12 @@
"id": "93e84ff8",
"metadata": {},
"source": [
"When solving a Laplace equation in 1D, the Jacobi method leads to the following iterative scheme: The entry $i$ of vector $u$ at iteration $t+1$ is computed as:\n",
"When solving a [Laplace equation](https://en.wikipedia.org/wiki/Laplace%27s_equation) in 1D, the Jacobi method leads to the following iterative scheme: The entry $i$ of vector $u$ at iteration $t+1$ is computed as:\n",
"\n",
"$u^{t+1}_i = \\dfrac{u^t_{i-1}+u^t_{i+1}}{2}$"
"$u^{t+1}_i = \\dfrac{u^t_{i-1}+u^t_{i+1}}{2}$\n",
"\n",
"\n",
"This iterative is yet simple but shares fundamental challenges with many other algorithms used in scientific computing. This is why we are studying it here.\n"
]
},
{
@@ -130,6 +133,14 @@
"end"
]
},
{
"cell_type": "markdown",
"id": "432bd862",
"metadata": {},
"source": [
"If you run it for zero iterations, we will see the initial condition."
]
},
{
"cell_type": "code",
"execution_count": null,
@@ -140,14 +151,78 @@
"jacobi(5,0)"
]
},
{
"cell_type": "markdown",
"id": "c75cb9a6",
"metadata": {},
"source": [
"If you run it for enough iterations, you will see the expected solution of the Laplace equation: values that vary linearly form -1 to 1."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b52be374",
"metadata": {},
"outputs": [],
"source": [
"jacobi(5,100)"
]
},
{
"cell_type": "markdown",
"id": "22fda724",
"metadata": {},
"source": [
"In our version of the jacobi method, we return after a given number of iterations. Other stopping criteria are possible. For instance, iterate until the difference between u and u_new is below a tolerance:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "15de7bf5",
"metadata": {},
"outputs": [],
"source": [
"using LinearAlgebra: norm\n",
"function jacobi_with_tol(n,tol)\n",
" u = zeros(n+2)\n",
" u[1] = -1\n",
" u[end] = 1\n",
" u_new = copy(u)\n",
" increment = similar(u)\n",
" while true\n",
" for i in 2:(n+1)\n",
" u_new[i] = 0.5*(u[i-1]+u[i+1])\n",
" end\n",
" increment .= u_new .- u\n",
" if norm(increment)/norm(u_new) < tol\n",
" return u_new\n",
" end\n",
" u, u_new = u_new, u\n",
" end\n",
" u\n",
"end"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "697ad307",
"metadata": {},
"outputs": [],
"source": [
"n = 5\n",
"tol = 1e-9\n",
"jacobi_with_tol(n,tol)"
]
},
{
"cell_type": "markdown",
"id": "6e085701",
"metadata": {},
"source": [
"<div class=\"alert alert-block alert-info\">\n",
"<b>Note:</b> In our version of the jacobi method, we return after a given number of iterations. Other stopping criteria are possible. For instance, iterate until the difference between u and u_new is below a tolerance.\n",
"</div>"
"However, we are not going to parallelize this more complex in this notebook (we will consider it later in this course)."
]
},
{
@@ -156,7 +231,7 @@
"metadata": {},
"source": [
"\n",
"### Where can we exploit parallelism?\n",
"## Where can we exploit parallelism?\n",
"\n",
"Look at the two nested loops in the sequential implementation:\n",
"\n",
@@ -169,8 +244,8 @@
"end\n",
"```\n",
"\n",
"- The outer loop cannot be parallelized. The value of `u` at step `t+1` depends on the value at the previous step `t`.\n",
"- The inner loop can be parallelized.\n",
"- The outer loop over `t` cannot be parallelized. The value of `u` at step `t+1` depends on the value at the previous step `t`.\n",
"- The inner loop is trivially parallel. The loop iterations are independent (any order is possible).\n",
"\n"
]
},
@@ -386,19 +461,11 @@
"source": [
"### Communication overhead\n",
"- We update $N/P$ entries in each process at each iteration, where $N$ is the total length of the vector and $P$ the number of processes\n",
"- Thus, computation complexity is $O(N/P)$\n",
"- We need to get remote entries from 2 neighbors (2 messages per iteration)\n",
"- We need to communicate 1 entry per message\n",
"- Communication/computation ration is $O(P/N)$ (potentially scalable if $P<<N$)\n"
]
},
{
"cell_type": "markdown",
"id": "f6b54b7b",
"metadata": {},
"source": [
"## 1D Implementation\n",
"\n",
"We consider the implementation using MPI. The programming model of MPI is generally better suited for data-parallel algorithms like this one than the task-based model provided by Distributed.jl. In any case, one can also implement it using Distributed, but it requires some extra effort to setup the remote channels right for the communication between neighbor processes."
"- Thus, communication complexity is $O(1)$\n",
"- Communication/computation ration is $O(P/N)$, making the algorithm potentially scalable if $P<<N$.\n"
]
},
{
@@ -457,8 +524,10 @@
"id": "8ed4129c",
"metadata": {},
"source": [
"### MPI Code\n",
"## MPI implementation\n",
"\n",
"We consider the implementation using MPI. The programming model of MPI is generally better suited for data-parallel algorithms like this one than the task-based model provided by Distributed.jl. In any case, one can also implement it using Distributed.jl, but it requires some extra effort to setup the remote channels right for the communication between neighbor processes.\n",
" \n",
"Take a look at the implementation below and try to understand it.\n"
]
},
@@ -749,7 +818,7 @@
"```\n",
"\n",
"- The outer loop cannot be parallelized (like in the 1d case). \n",
"- The two inner loops can be parallelized\n"
"- The two inner loops are trivially parallel\n"
]
},
{
notebooks/julia_mpi.ipynb
+6 -1
View File
@@ -687,7 +687,12 @@
"* `rcvbuf` space to store the incoming data.\n",
"* `source` rank of the sender.\n",
"* `dest` rank of the receiver.\n",
"* `tag`. Might be used to distinguish between different kinds of messages from the same sender to the same receiver (similar to the \"subject\" in an email).\n"
"* `tag`. Might be used to distinguish between different kinds of messages from the same sender to the same receiver (similar to the \"subject\" in an email).\n",
"\n",
"\n",
"<div class=\"alert alert-block alert-info\">\n",
"<b>Note:</b> Note that the C interface provides additional arguments `MPI_Datatype` (type of data to send/receive) and `count` (number of items to send/receive). In Julia, send and receive buffers are usually arrays or references, from which the data type and the count can be inferred. This is true for many other MPI functions.\n",
"</div>"
]
},
{
notebooks/matrix_matrix.ipynb
+46 -20
View File
@@ -29,7 +29,8 @@
"\n",
"- Parallelize a simple algorithm\n",
"- Study the performance of different parallelization strategies\n",
"- Implement them using Julia"
"- Learn the importance of \"grain size\" in a parallel algorithm\n",
"- Implement and measure the performance of parallel algorithms"
]
},
{
@@ -54,10 +55,18 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 8,
"id": "2f8ba040",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"🥳 Well done! \n"
]
}
],
"source": [
"using Distributed\n",
"using BenchmarkTools\n",
@@ -79,7 +88,8 @@
"alg_2_complex_check(answer) = answer_checker(answer, \"b\")\n",
"alg_2_deps_check(answer) = answer_checker(answer,\"d\")\n",
"alg_3_deps_check(answer) = answer_checker(answer, \"c\")\n",
"alg_3_complex_check(answer) = answer_checker(answer, \"b\")"
"alg_3_complex_check(answer) = answer_checker(answer, \"b\")\n",
"println(\"🥳 Well done! \")"
]
},
{
@@ -89,6 +99,8 @@
"source": [
"## Problem Statement\n",
"\n",
"We consider matrix-matrix multiplication as our first algorithm to parallelize. The problem we want so solve is defined as follows.\n",
"\n",
"Given $A$ and $B$ two $N$-by-$N$ matrices, compute the matrix-matrix product $C=AB$. Compute it in parallel and efficiently."
]
},
@@ -113,7 +125,7 @@
"\n",
"- compute the product in parallel using more than one process (distributed implementation)\n",
"- study the performance of different parallelization alternatives\n",
"- implement the algorithms using Julia\n"
"- implement the algorithms using Julia's task-based programming model\n"
]
},
{
@@ -124,7 +136,7 @@
"### Assumptions\n",
"\n",
"- All matrices `A`,`B`, and `C` are initially stored in the master process\n",
"- The result will be overwritten in `C`"
"- The result will be overwritten in `C` (in the master process)"
]
},
{
@@ -148,7 +160,8 @@
"\n",
"- Identify the parts of the sequential algorithm that can be parallelized\n",
"- Consider different parallelization strategies\n",
"- Discuss the (theoretical) performance of these implementations\n"
"- Discuss the (theoretical) performance of these implementations\n",
"- Measure the actual performance of these implementations\n"
]
},
{
@@ -222,7 +235,7 @@
"<b>Note:</b> The matrix-matrix multiplication naively implemented with 3 nested loops as above is known to be very inefficient (memory bound). Libraries such as BLAS provide much more efficient implementations, which are the ones used in practice (e.g., by the `*` operator in Julia). We consider our hand-written implementation as a simple way of expressing the algorithm we are interested in.\n",
"</div>\n",
"\n",
"Run the following cell to compare the performance of our hand-written function with respect to the built in function `mul!`.\n"
"Just to satisfy your curiosity, run the following cell to compare the performance of our hand-written function with respect to the built in function `mul!`.\n"
]
},
{
@@ -272,9 +285,14 @@
"id": "0eedd28a",
"metadata": {},
"source": [
"### Where can we exploit parallelism?\n",
"## Where can we exploit parallelism?\n",
"\n",
"Look at the three nested loops in the sequential implementation:\n",
"\n",
"The matrix-matrix multiplication is an example of [embarrassingly parallel algorithm](https://en.wikipedia.org/wiki/Embarrassingly_parallel). An embarrassingly parallel (also known as trivially parallel) algorithm is an algorithm that can be split in parallel tasks with no (or very few) dependences between them. Such algorithms are typically easy to parallelize.\n",
"\n",
"Which parts of an algorithm are completely independent and thus trivially parallel? To answer this question, it is useful to inspect the for loops, which are potential sources parallelism. If the iterations are independent of each other, then they are trivial to parallelize. An easy check to find out if the iterations are dependent or not is to change their order (for instance changing `for j in 1:n` by `for j in n:-1:1`, i.e. doing the loop in reverse). If the result changes, then the iterations are not independent.\n",
"\n",
"Look at the three nested loops in the sequential implementation of the matrix-matrix product:\n",
"\n",
"```julia\n",
"for j in 1:n\n",
@@ -287,8 +305,12 @@
" end\n",
"end\n",
"```\n",
"- Loops over `i` and `j` are trivially parallelizable.\n",
"- The loop over `k` can be parallelized but it requires a reduction."
"\n",
"Note that:\n",
"\n",
"- Loops over `i` and `j` are trivially parallel.\n",
"- The loop over `k` is not trivially parallel. The accumulation into the reduction variable `Cij` introduces extra dependences. In addition, remember that the addition of floating point numbers is not strictly associative due to rounding errors. Thus, the result of this loop may change with the loop order when using floating point numbers. In any case, this loop can also be parallelized, but it requires a parallel *fold* or a parallel *reduction*.\n",
"\n"
]
},
{
@@ -298,7 +320,7 @@
"source": [
"### Parallel algorithms\n",
"\n",
"All the entries of matrix C can be potentially computed in parallel, but *is it the most efficient solution to solve all these entries in parallel in a distributed system?* To find this we will consider different parallelization strategies:\n",
"Parallelizing the loops over `i` and `j` means that all the entries of matrix C can be potentially computed in parallel. However, *which it the most efficient solution to solve all these entries in parallel in a distributed system?* To find this we will consider different parallelization strategies:\n",
"\n",
"- Algorithm 1: each worker computes a single entry of C\n",
"- Algorithm 2: each worker computes a single row of C\n",
@@ -330,7 +352,7 @@
"source": [
"### Data dependencies\n",
"\n",
"Moving data through the network is expensive and reducing data movement is one of the key points in a distributed algorithm. To this end, we determine which is the minimum data needed by a worker to perform its computations.\n",
"Moving data through the network is expensive and reducing data movement is one of the key points in a distributed algorithm. To this end, we need to determine which is the minimum data needed by a worker to perform its computations. These are called the *data dependencies*. This will give us later information about the performance of the parallel algorithm.\n",
"\n",
"In algorithm 1, each worker computes only an entry of the result matrix C."
]
@@ -380,7 +402,7 @@
"\n",
"Taking into account the data dependencies, the parallel algorithm 1 can be efficiently implemented following these steps from the worker perspective:\n",
"\n",
"1. The worker receives the corresponding row A[i,:] and column B[:,j] from the master process\n",
"1. The worker receives the data dependencies, i.e., the corresponding row A[i,:] and column B[:,j] from the master process\n",
"2. The worker computes the dot product of A[i,:] and B[:,j]\n",
"3. The worker sends back the result of C[i,j] to the master process"
]
@@ -471,19 +493,19 @@
"source": [
"### Performance\n",
"\n",
"Let us study the performance of this algorithm. To this end, we will analyze if algorithm 1 is able to achieve the optimal parallel *speedup*. The parallel speedup on $P$ processes is defined as \n",
"We have a first parallel algorithm, but how efficient is this algorithm? Let us study its performance. To this end we need to consider a performance baseline as reference. In this case, we will use the so-called optimal parallel *speedup*. The parallel speedup on $P$ processes is defined as \n",
"\n",
"$$\n",
"S_P = \\frac{T_1}{T_P},\n",
"$$\n",
"\n",
"where $T_1$ denotes the runtime of the sequential algorithm on one node and $T_P$ denotes the runtime of the parallel algorithm on $P$ processes. If we run an optimal parallel algorithm with $P$ processes we expect it to run $p$ times faster than the sequential implementation. I.e., the *optimal* speedup of a parallel algorithm on $p$ processes is equal to $P$:\n",
"where $T_1$ denotes the runtime of the sequential algorithm and $T_P$ denotes the runtime of the parallel algorithm on $P$ processes. If we run an optimal parallel algorithm with $P$ processes we expect it to run $p$ times faster than the sequential implementation. I.e., the *optimal* speedup of a parallel algorithm on $p$ processes is equal to $P$:\n",
"\n",
"$$\n",
"S^{*}_p = P.\n",
"$$\n",
"\n",
"The ratio of the actual speedup over the optimal one is called the parallel efficiency\n",
"The ratio of the actual speedup over the optimal one is called the parallel efficiency (the closer to one the better).\n",
"\n",
"$$\n",
"E_p = \\frac{S_p}{S^{*}_p} = \\frac{T_1/T_P}{P}.\n",
@@ -754,7 +776,9 @@
"id": "f1f30faf",
"metadata": {},
"source": [
"### Experimental speedup"
"### Experimental speedup\n",
"\n",
"Measure the speedup with the following cell. Is this time better?"
]
},
{
@@ -964,7 +988,9 @@
"\n",
"- Matrix-matrix multiplication is trivially parallelizable (all entries in the result matrix can be computed in parallel, at least in theory)\n",
"- However, we cannot exploit all the potential parallelism in a distributed system due to communication overhead\n",
"- We need a sufficiently large grain size to obtain a near optimal speedup\n"
"- We need a sufficiently large grain size to obtain a near optimal speedup\n",
"- We measured the theoretical parallel performance by computing the complexity of communication over computation\n",
"- We measured the actual performance using the parallel speedup and parallel efficiency\n"
]
},
{
notebooks/mpi_collectives.ipynb
+901
View File
@@ -0,0 +1,901 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "7606d30a",
"metadata": {},
"source": [
"<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/3/39/VU_logo.png/800px-VU_logo.png?20161029201021\" width=\"350\">\n",
"\n",
"### Programming large-scale parallel systems"
]
},
{
"cell_type": "markdown",
"id": "4ac1e5d9",
"metadata": {},
"source": [
"# Intro to MPI (collectives)"
]
},
{
"cell_type": "markdown",
"id": "a341be2e",
"metadata": {},
"source": [
"## Contents\n",
"\n",
"\n",
"In this notebook, we will continue learning the basics of parallel computing using the Message Passing Interface (MPI) from Julia. In particular, we will learn:\n",
"\n",
"- How to use basic collective communication directives\n",
"- How to use MPI communicators\n"
]
},
{
"cell_type": "markdown",
"id": "c87b3c82",
"metadata": {},
"source": [
"## Collective communication\n",
"\n",
"MPI provides collective communication functions for communication involving multiple processes. Some usual collective functions are:\n",
"\n",
"\n",
"- `MPI_Barrier`: Synchronize all processes\n",
"- `MPI_Bcase`: Send the same data from one to all processes\n",
"- `MPI_Gather`: Gather data from all processes to one\n",
"- `MPI_Scatter`: Scatter data from one to all all processes\n",
"- `MPI_Reduce`: Reduce data from all processes to a single one\n",
"- `MPI_Scan`: Scan (or prefix) reduction\n",
"- `MPI_Allgather`: Like a `MPI_Gather` but all processes receive the result\n",
"- `MPI_Allreduce`: Like `MPI_Reduce` but all processes receive the result\n",
"- `MPI_Alltoall`: Exchange data from all to all processes\n",
"\n",
"\n",
"We will discuss some of them in greater detail in this notebook."
]
},
{
"cell_type": "markdown",
"id": "4ffa5e56",
"metadata": {},
"source": [
"## Why collective primitives?\n",
"\n",
"Point-to-point communication primitives provide all the building blocks needed in parallel programs and could be used to implement the collective functions described above. Then, why does MPI provide collective communication directives? There are several reasons:\n",
"\n",
"- Ease of use: It is handy for users to have these functions readily available instead of having to implement them.\n",
"- Performance: Library implementations typically use efficient algorithms (such as reduction trees).\n",
"- Hardware support: Hardware vendors can optimize the MPI library for their machine and even develop hardware specially design to perform MPI operations efficiently. For instance, [IMB Blue Gene](https://en.wikipedia.org/wiki/IBM_Blue_Gene) was equipped with auxiliary networks for MPI collectives.\n"
]
},
{
"cell_type": "markdown",
"id": "53a0f5fa",
"metadata": {},
"source": [
"## Semantics of collective operations\n",
"\n",
"\n",
"- Completeness: All the collective communication directives above are *complete* operations. Thus, it is safe to use and reset the buffers once the function returns.\n",
"- Standard mode: Collective directives are in standard mode only, like `MPI_Send`. Assuming that they block is erroneous, assuming that they do not block is also erroneous.\n",
"- Synchronization: Completion of a call does not guarantee that other processes have completed the operation. A collective operation may or may not have the effect of synchronizing all processes, the only exception is `MPI_Barrier` of course.\n",
"\n",
"\n",
"<div class=\"alert alert-block alert-info\">\n",
"<b>Note:</b> Recent versions of the MPI standard also include non-blocking (incomplete) versions of collective operations (not covered in this notebook). A particularly funny one is the non-blocking barrier `MPI_Ibarrier`.\n",
"</div>"
]
},
{
"cell_type": "markdown",
"id": "e9e2a219",
"metadata": {},
"source": [
"## MPI_Barrier\n",
"\n",
"This function is used to synchronizes a group of processes. All processes block until all have reached the barrier. It is often invoked at the end of for loops to make sure all processes have finished the current loop iteration to move to the next one. We will see an example later in another notebook when studying the traveling sales person problem (TSP).\n",
"\n",
"In Julia:\n",
"```julia\n",
"MPI.Barrier(comm)\n",
"```\n",
"\n",
"In C:\n",
"```c\n",
"int MPI_Barrier(MPI_Comm comm)\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "d9753e4b",
"metadata": {},
"source": [
"### Example\n",
"\n",
"In this example the ranks sleep for a random amount of time and then they call barrier. It is guaranteed that the message \"Done!\" will be printed after all processes printed \"I woke up\" since we used a barrier. Try also to comment out the call to `MPI.Barrier`. You will see that the message can be printed in any order in this case."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9a66c717",
"metadata": {},
"outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
" MPI.Init()\n",
" comm = MPI.COMM_WORLD\n",
" rank = MPI.Comm_rank(comm)\n",
" sleep(rand(1:3))\n",
" println(\"[rank $rank] I woke up\")\n",
" MPI.Barrier(comm)\n",
" println(\"[rank $rank] Done!\")\n",
"end\n",
"run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
]
},
{
"cell_type": "markdown",
"id": "df0223ff",
"metadata": {},
"source": [
"## MPI_Reduce\n",
"\n",
"Combines values provided by different processors according to a given reduction operation. The result is received in a single process (called the root process).\n",
"\n",
"In Julia:\n",
"```julia\n",
"MPI.Reduce!(sendbuf, recvbuf, op, comm; root)\n",
"```\n",
"\n",
"In C:\n",
"```\n",
"int MPI_Reduce(const void *sendbuf, void *recvbuf, int count,\n",
" MPI_Datatype datatype, MPI_Op op, int root,\n",
" MPI_Comm comm)\n",
"```\n",
"\n",
"\n",
"\n",
"<div class=\"alert alert-block alert-info\">\n",
"<b>Note:</b> `MPI_Reduce` (and related functions like `MPI_Allreduce`) use operations that are *commutative* and *associative*, such as addition, multiplication, maximum, minimum, logical and, logical or. Otherwise the result will be undefined as it will depend in which order the operations are done.\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"id": "2fc1aa28",
"metadata": {},
"source": [
"### Example\n",
"Each process generates a random integer. Compute the sum of all these integers. Receive the result on rank 0."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "341b8438",
"metadata": {},
"outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
" MPI.Init()\n",
" comm = MPI.COMM_WORLD\n",
" rank = MPI.Comm_rank(comm)\n",
" value = rand(1:4)\n",
" root = 0\n",
" sndbuf = [value]\n",
" if rank == root\n",
" rcvbuf = [0]\n",
" else\n",
" # This can be anything as only the root process\n",
" # Will receive data\n",
" rcvbuf = nothing\n",
" end\n",
" println(\"[rank $rank] I am contributing with $sndbuf\")\n",
" MPI.Reduce!(sndbuf, rcvbuf, +, comm; root)\n",
" println(\"[rank $rank] The sum is $rcvbuf\")\n",
"end\n",
"run(`$(mpiexec()) -np 3 julia --project=. -e $code`);"
]
},
{
"cell_type": "markdown",
"id": "3ea77a8f",
"metadata": {},
"source": [
"## Reducing multiple values\n",
"\n",
"If more than one item is provided in the send buffers, `MPI_Reduce` does the reduction element by element.\n"
]
},
{
"cell_type": "markdown",
"id": "b26fd0b2",
"metadata": {},
"source": [
"### Example\n",
"\n",
"Each process generates 2 random integers. Take a look to the result of reduce. Note that the values have ben reduced \"element by element\". "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1a145948",
"metadata": {},
"outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
" MPI.Init()\n",
" comm = MPI.COMM_WORLD\n",
" rank = MPI.Comm_rank(comm)\n",
" sndbuf = rand(1:4,2)\n",
" root = 0\n",
" if rank == root\n",
" rcvbuf = [0,0]\n",
" else\n",
" rcvbuf = nothing\n",
" end\n",
" println(\"[rank $rank] I am contributing with $sndbuf\")\n",
" MPI.Reduce!(sndbuf, rcvbuf, +, comm; root)\n",
" println(\"[rank $rank] The sum is $rcvbuf\")\n",
"end\n",
"run(`$(mpiexec()) -np 3 julia --project=. -e $code`);"
]
},
{
"cell_type": "markdown",
"id": "539df34f",
"metadata": {},
"source": [
"## MPI_Allreduce\n",
"\n",
"This is a variant of `MPI_Reduce` in which all processes get the result."
]
},
{
"cell_type": "markdown",
"id": "be0381bb",
"metadata": {},
"source": [
"### Example\n",
"\n",
"This is the same example as above, but all processes get the result."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1595f4f5",
"metadata": {},
"outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
" MPI.Init()\n",
" comm = MPI.COMM_WORLD\n",
" rank = MPI.Comm_rank(comm)\n",
" sndbuf = rand(1:4,2)\n",
" # All ranks allocate a receive buffer\n",
" rcvbuf = [0,0]\n",
" println(\"[rank $rank] I am contributing with $sndbuf\")\n",
" MPI.Allreduce!(sndbuf, rcvbuf, +, comm)\n",
" println(\"[rank $rank] The sum is $rcvbuf\")\n",
"end\n",
"run(`$(mpiexec()) -np 3 julia --project=. -e $code`);"
]
},
{
"cell_type": "markdown",
"id": "97dc2886",
"metadata": {},
"source": [
"## MPI_Gather\n",
"\n",
"Each rank sends a message to the root rank (the root rank also sends a message to itself). The root rank receives all these values in a buffer (e.g. a vector). This function assumes that the amount of data sent from each rank is the same. The root process can be any process and it is rank 0 by default in Julia.\n",
"\n",
"In Julia:\n",
"```julia\n",
"MPI.Gather!(sendbuf, recvbuf, comm; root=0)\n",
"```\n",
"\n",
"In C:\n",
"```c\n",
"int MPI_Gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,\n",
" void *recvbuf, int recvcount, MPI_Datatype recvtype, int root,\n",
" MPI_Comm comm)\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "3d0a1e9d",
"metadata": {},
"source": [
"### Example\n",
"\n",
"Each process sends an integer to rank 0.\n"
]
},
{
"attachments": {
"g13884.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAABFEAAANWCAYAAAA2uxORAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13mGRlnf7/d01iYMg5BxUlKCiggiImDKiYV9c16xpXZY3o6q7u18SuusZVMcvq+jMHRFcRTBgQDICIiGSQnAcmz/3745xmTp+uTtXV/VR3v1/XVVdPV9c5dVd1z3PO+dQTOkiSJM24LAG2AbZufd0I2AxYCmxc/3sRsBWwGNi0vn9p/bPFwJaTfPKh/dwKrANWACuBNcDy+jE31V+X1/evrB+3DrgFuLl+zM317ZbGv2+GzqpJZpIkSbNAp3QASZI0F2QpsAuwM7A7sC0bCiNDRZJtG//erEzOGbOCYUUVbgauB64G/gZcA1wJXFt97dxWKKckSZoEiyiSJGkcWUZVGNmlvu1Wf921vu0MbF8sHqwC7pjkNquBJfW/N6XqnVLSHcBVVEWWdqHlKuBi4BLorC4VUJIkWUSRJEkAZBtg7/p298a/7wps0ecnuwm4ob7d2Pq6iqrXxhrgNjYMoxkaVnMTsLb+2QrorOxfrAwNIVoIbF7fuQWwANiEaqjRkvpnWzZuW7W+b96W0D/rqYorF49yuxI66/v4fJIkqcUiiiRJ80Y2Z0NxpF0s2XoKO14FXEF1gX9Z/fVquhdJbphfF/rZhA0Fle2oevBsT9V7Z0dgp/q2A9Vwp6lYDVzKhqLKRcB5wJ+oerGsm+L+JUma9yyiSJI052QRVYHkAODA+usBVENvJusOqgvzK+vb5fXXK7izcNK5tg+hRTaiKrDs0vi6A7AHsFd924Wqp8xkrQT+XN/OZUNx5a/QWTPl6JIkzRMWUSRJmtWyNXBvNhRKDgD2p1q9ZqJWAhfUt782/n0BdP7W17iaoiymmp9mr1Fuk52bZg3V7/pPjCiwWFyRJKnNIookSbNGtgUOrW8HURVMdpngxuuoCiR/YViRhAuAy6GTvsdVAVnGhoLK3YF9gf3qr5NZCno1cA7wu8bt7P7OQSNJ0uxjEUWSpIGUhcA+wMHAA4HDqS6EJ3LsvgX4I1Wvgj8BvwV+B53JrmCjOSVbUfVS2q/x9S71bSLWUhXhflvfzq2+dm7qf1ZJkgaTRRRJkgZCdqTqYXJY/fUQqhVhxrIWOJ+qx8BZ9e0c6FwxjUE152RbNvRW2Q+4J1VPp4n0XFlHVVgZ6q3yG+BMe6xIkuYqiyiSJBWRPYCHAw8FHkQ1eeiYG1BdrJ4O/Lr+ei50Vk1nSs1nuQtVMaV5224CG64GzgR+Cfyi+urkw5KkucEiiiRJMyLbAQ8BjqQamrPfOBssp+pZchrVheivoXPddCaUxpedqYaYDd2GhgSN5yo2/C2fBvx+fi11LUmaKyyiSJI0LbIF8GCq3iYPo5qDYrTj7nqquUtOB35F1dPkPC8yNTtkR6peKodQDUc7DNhinI1upPpb/0V9OwM6K6YzpSRJ/WARRZKkvsgiqglgH01VNDkYWDjag6km5TwVOAX4GXRunomU0vTLAqqi4eHAA6j+X+w1zkarqIoqp9S3M6CzdjpTSpLUC4sokiT1LMuoCiaPA54A7DDGgy9iw1CG7zn5q+aX7Ajclw0rTR0CbDTGBrdTFVV+VN9+5zLckqRBYBFFkqRJyd2Bo4HHUk0Iu2iUB15J1dOkvnUum5l80myQTdhQVHkAcASw2RgbXMWGnls/gs7l0x5RkiRJkjRZWQR5COS9kPMhGeW2EvIDyCsh9yidWppdsgjyAMi/Qn4CWTXG/7XU/xc/CnlKPf+QJEmSJKmMbAR5POQLkBvHuJC7BvLZ+kJurE/RJU1KNoE8CvKfkN9C1o3x/3A15BTIqyF7l04uSZIkSfNAFkIOh3wQct0YF2wX1o85sp5MVtK0y2b1/7njIGeO00vlQsjxkKMhS0onlyRJkqQ5IgshD6svuK4f5YJsBeT7kJdD9iidWBJAdoc8H/JlyM1jFFRuqh/zbMi2pVNLkiRJ0iyTBXWPkw9DrhpjfpNvQ/4BsmnpxJLGksV1MfR9kL+MUVBZCzkN8kbIvUqnliRJkqQBlgPri6zLx5hX4XuQ50K2LJ1WUq9yd8hrIKfW/6/HGvZzHOSQ0oklSZIkaQBkC8iL60+fu11Erat/dgxk+9JpJfVbltVzoxwP+dsYBZVL67mODod0SqeWJEmSpBmSTt21/wuQO0YpnPy0nuNkh9JpJc2ULIQ8EPJuyLljFFQuhrwHcj8LKpIkSZLmqOwG+de6i363C6M/Q46F7FI6qaRBkP0gb4WcM0ZB5RLIeyGHWlCRJEmSNMtlCeTvqFbPWdvlAug2yGeqLvqSNJrsC/k3yNnjDPn5r6qgIkmSJEmzRvaA/AfkulEudn4BeSFks9JJJc022QfyFshZYxRU/lIXXfYqnVaSJEmSRpGDISdA1nS5qLmhnjzygNIpJc0V2aueeHq0yakDObN+zDal00qSJEma97IU8gLIH7pcvKyFfAfyBMji0kklzWW5O+TNkPNGKabcAfkS5DGQRaXTSpIkSZpXsiPkbaMM2bmZajnSPUunlDQfZX/IcZBrRimoDPWMcz4mSZIkSdPpziE7q7tcmJxfd5tfVjqlJFEtm3xk3WbdPkpB5dy6ILxH6bSSJEmS5oQsgDytnlugfQGyDvItyMNKp5Sk0WUryEvria27FVPWQk6EPK4qvkiSJEnSpGQx5DmjzDFwS90dfp/SKSVpcrI75FjIBaMUVK6shwPZO0WSJEnSeLIx5J8gl3S5uPgT5GUO2ZE0+6UDeRDkk5Bbu7R3a+qedkdVPfIkSZIk6U7ZtJ7T5MouFxO/r3ul2M1d0hyUpZC/Y/Tlkq+oe6fsXjqpJEmSpKKyed21/YYuFw6nQY4unVCSZk4Orocr3jbK3Ckn1wUXi8qSJEnS/JEdIe/pcqGwHvJdyANKJ5SkcrJ5PRnt70fpnXIh5E2QHUonlSRJkjRtshXVkp7tOQDW1atTHFI6oSQNljF7p6yCfAVy/9IpJUmSJPVNNqmH7dzYugBYDTnBlXYkaTzZHPLiMXqnnOZQH0mSJGlWy5J6tZ2ruhRPPu5EiZLUixwG+Z+6J0q7mHIB5FWQzUqnlCRJkjQhWVB/InpBlzlPvgLZu3RCSZr9skM9RPK6LsWUWyEfhOxZOqUkSZKkUeXIUbqbnww5qHQ6SZp7shHVUvBnd2l7h+acOrJ0SkmSJEl3yoMhv+xyAv9zyOGl00nS3JcO5BGQk+qef+32+AzIM6uhlpIkSZIKyF6Qb3Q5Wf895DGl00nS/JR7QD4KWd6lfb4C8hrIpqVTSpIkSfNENqnH4q9onZxfXK8g4QoRklRcNoccA7mkSzHlFshxkK1Lp5QkSZLmqHQgz4Bc3joZvx7yMsji0gklSW1ZBHk65PRRJqF9D2Sn0iklSZKkOST3gfysdfK9BnI8ZNvS6SRJE5HD68lm28WUVZATIHcvnVCSJEmaxbJ1vVTm2tYJ96mQA0qnkyT1Iveuiybttn1oRZ+DSyeUJEmSZpEsquc3ua51gn15tZymJGn2y93qHoVrWm39+rqYcljphJIkSdKAy4Mg57ZOqJdD3gJZWjqdJKnfsle9ok97wvBAToEcWTqhJEmSNGCyRT10Z13rBPpEyB6l00mSplu2r1dfu6lLMeWXkIeXTihJkiQNgDwJcmXrhPn31SSEkqT5JVtB/rXLkM5AfuQwH0mSJM1T2Rny9dYJ8h2QN1TzokiS5q8sg7wGck2XYspJTkArSZKkeSKdaoLY3NA6Kf4p5B6l00mSBkk2gRwDubpLMeVkyH1KJ5QkSZKmSe5WTxTYPAm+qV6Np1M6nSRpUGVTyLFd5kxZD/mKRXhJkiTNIVlUn/y2V184EbJL6XSSpNkim9XHk5tbx5N1dTFl79IJJUmSpCnIAZCzWie7l0IeWzqZJGm2ynaQ99VzaTWPL6shx0N2K51QkiRJmoQsgLwWsrL1SeEHq08SJUmaquwM+XDrWJP6+/dBti6dUJIkSRpHdoOc2jqhPQ9yaOlkkqS5KLtDPln3RGkee26sC/oblU4oSZIkdZFntCb+Ww/5CGTj0skkSXNd7go5oe752CymXAT5eycxlyRJ0oDIFvWJa/Ok9WrI40onkyTNN9kP8t0uyyKfAXlw6XSSJEma1/IAyIWtE9WvQ7YtnUySNJ/lSMgfuhRTTobcs3Q6SZIkzStZDHlbq9v0rZAXl04mSVIlCyB/B7mkVUhZU6/ks2PphJIkSZrzsl+XpYtPg+xVOpkkSSNlGeTfILe1jl23QP4FsknphJIkSZqT8szWSejq+gR0YelkkiSNLTtCPl73RGkWUy6HPK/quSJJkiRNWTaCfKzL0sUHl04mSdLkZD/IiV3mSzkTcmjpdJIkSZrVsjvk160TzW9Uq/JIkjRb5TDIL1rHt/X1inM7lE4nSZKkWSePh9zUOLlcCXl56VSSJPVHOpBnQa5oFVNuhhwDWVQ6oSRJkgZeFnVZfedSuzlLkuambFIf91a2iil/hjyqdDpJkiQNrGwP+VHrJPJEyNalk0mSNL2yN+SkLvOlnAjZo3Q6SZIkDZQ8BPK3xknj2vqTOVcskCTNIzkaclGrkHJ7fUxcWjqdJEmSissb6qLJ0Mni3yBHlE4lSVIZ2bgumtzRKqb8BfKY0ukkSZJURDaCfLZ1gvgTyE6lk0mSVF52rVfs6TbE5y6l00mSJGnGZHvIaa2TwndDFpZOJknSYMmj6olmm8fMFZB/gywpnU6SJEnTKveCXNxavvi5pVNJkjS4srhe+viWLkN8Hlo6nSRJkqZFHg+5rXHydzXksNKpJEmaHbIL5H9bhZR1kI9BtiidTpIkSX2TY+oTvaGTvrMhe5ZOJUnS7JOHQM5vFVOugjyndDJJkiRNSdcJLine truncated
}
},
"cell_type": "markdown",
"id": "4bdf9c02",
"metadata": {},
"source": [
"<div>\n",
"<img src=\"attachment:g13884.png\" align=\"left\" width=\"350\"/>\n",
"</div>"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1f8a70c6",
"metadata": {},
"outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
" MPI.Init()\n",
" comm = MPI.COMM_WORLD\n",
" rank = MPI.Comm_rank(comm)\n",
" msg = 10*(rank+2)\n",
" root = 0\n",
" sndbuf = [msg]\n",
" if rank == root\n",
" nranks = MPI.Comm_size(comm)\n",
" rcvbuf = zeros(Int,nranks)\n",
" else\n",
" # This can be anything as only the root process\n",
" # Will receive data\n",
" rcvbuf = nothing\n",
" end\n",
" println(\"[rank $rank] I am sending $sndbuf\")\n",
" MPI.Gather!(sndbuf, rcvbuf, comm; root)\n",
" println(\"[rank $rank] I received $rcvbuf\")\n",
"end\n",
"run(`$(mpiexec()) -np 3 julia --project=. -e $code`);"
]
},
{
"cell_type": "markdown",
"id": "0f85f08d",
"metadata": {},
"source": [
"## MPI_Allgather\n",
"\n",
"This is a variant of `MPI_Gather` in which all processes get the result."
]
},
{
"cell_type": "markdown",
"id": "4bf2ef34",
"metadata": {},
"source": [
"### Example"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "070e1fc1",
"metadata": {},
"outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
" MPI.Init()\n",
" comm = MPI.COMM_WORLD\n",
" rank = MPI.Comm_rank(comm)\n",
" msg = 10*(rank+2)\n",
" sndbuf = [msg]\n",
" nranks = MPI.Comm_size(comm)\n",
" rcvbuf = zeros(Int,nranks)\n",
" println(\"[rank $rank] I am sending $sndbuf\")\n",
" MPI.Allgather!(sndbuf, rcvbuf, comm)\n",
" # All ranks receive the result\n",
" println(\"[rank $rank] I received $rcvbuf\")\n",
"end\n",
"run(`$(mpiexec()) -np 3 julia --project=. -e $code`);"
]
},
{
"cell_type": "markdown",
"id": "fb6a9034",
"metadata": {},
"source": [
"## MPI_Gatherv\n",
"\n",
"This function is similar to `MPI_Gather`, but it is used when there is a different amount of data sent from each process. There is also `MPI_Allgatherv` for which all processes will receive the result."
]
},
{
"cell_type": "markdown",
"id": "e0d2998c",
"metadata": {},
"source": [
"### Example\n",
"\n",
"Each processes sends `rank+1` integers to rank 0."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "23814278",
"metadata": {},
"outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
" MPI.Init()\n",
" comm = MPI.COMM_WORLD\n",
" rank = MPI.Comm_rank(comm)\n",
" sndbuf = fill(10*rank,rank+1)\n",
" root = 0\n",
" if rank == root\n",
" nranks = MPI.Comm_size(comm)\n",
" # Define the amount of data to be received\n",
" # from each rank\n",
" counts = [ rank+1 for rank in 0:(nranks-1)]\n",
" # Total amount of receive data\n",
" ndata = sum(counts)\n",
" data = zeros(Int,ndata)\n",
" rcvbuf = MPI.VBuffer(data,counts)\n",
" else\n",
" # This can be anything as only the root process\n",
" # Will receive data\n",
" rcvbuf = nothing\n",
" end\n",
" println(\"[rank $rank] I am sending $sndbuf\")\n",
" MPI.Gatherv!(sndbuf, rcvbuf, comm; root)\n",
" if rank == root\n",
" println(\"[rank $rank] I received $data\")\n",
" end\n",
"end\n",
"run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
]
},
{
"cell_type": "markdown",
"id": "25c2fb1e",
"metadata": {},
"source": [
"## How to get the amount of data to be received?\n",
"\n",
"Remember that `MPI_Probe` allows you to query the amount of data to be received with `MPI_Recv`. Unfortunately, this mechanism does not work for MPI collectives. A workaround is to first communicate the amount of data to be gathered, allocate the receive buffer accordingly, and then communicate the actual message.\n"
]
},
{
"cell_type": "markdown",
"id": "12f9e2ca",
"metadata": {},
"source": [
"### Example\n",
"\n",
"Each process sends a random amount of integers to rank 0."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b10430a9",
"metadata": {},
"outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
" MPI.Init()\n",
" comm = MPI.COMM_WORLD\n",
" rank = MPI.Comm_rank(comm)\n",
" # Get number of ints to send\n",
" count = rand(1:4)\n",
" # Generate data to send\n",
" sndbuf = fill(rank,count)\n",
" # Communicate counts\n",
" root = 0\n",
" if rank == root\n",
" nranks = MPI.Comm_size(comm)\n",
" counts = zeros(Int,nranks)\n",
" else\n",
" counts = nothing\n",
" end\n",
" MPI.Gather!([count], counts, comm; root)\n",
" # Allocate receive buffer\n",
" if rank == root\n",
" ndata = sum(counts)\n",
" data = zeros(Int,ndata)\n",
" rcvbuf = MPI.VBuffer(data,counts)\n",
" else\n",
" rcvbuf = nothing\n",
" end\n",
" # Communicate data\n",
" println(\"[rank $rank] I am sending $sndbuf\")\n",
" MPI.Gatherv!(sndbuf, rcvbuf, comm; root)\n",
" if rank == root\n",
" println(\"[rank $rank] I received $data\")\n",
" end\n",
"end\n",
"run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
]
},
{
"cell_type": "markdown",
"id": "8b4254d1",
"metadata": {},
"source": [
"## MPI_Scatter\n",
"\n",
"This is the inverse operation of `MPI_Gather`. The root rank sends a distinct message to each rank.\n",
"\n",
"In Julia:\n",
"```julia\n",
"MPI.Scatter!(sendbuf, recvbuf, comm; root=0)\n",
"```\n",
"\n",
"In C:\n",
"```c\n",
"int MPI_Scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype,\n",
" void *recvbuf, int recvcount, MPI_Datatype recvtype, int root,\n",
" MPI_Comm comm)\n",
"```\n",
"\n",
"<div class=\"alert alert-block alert-info\">\n",
"<b>Note:</b> `MPI_Scatter` sends the same amount of data to each rank. To send a variable amount of data there is `MPI_Scatterv` (`MPI.Scatterv!` in julia), which works similar to `MPI_Gatherv`. \n",
"</div>\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "e55aa353",
"metadata": {},
"source": [
"### Example\n",
"\n",
"Rank 0 sends a unique integer to each rank. "
]
},
{
"attachments": {
"g13389.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAABFEAAANWCAYAAAA2uxORAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13nGV1ff/x190Ou5SlI10FKQpSBaLYsBB7xW6sUSMhxoJGjRiNIbGXRIk1qD8TuwI2FBsWBAsoICC91wUWtu++f3+cM+yZs3fq3pnvnZnX8/G4j9l7555z33Nn9nvO+dxv6SBJkjTpMg/YGtiq9XU+sBmwANik/vccYDEwF1hUP76g/t5cYMsxvvjAfu4C1gLLgRXAauDu+jlL6q9314+vqJ+3FrgTuKN+zh317c7Gv++AzsoxZpIkSVNAp3QASZI0HWQBsBNwH2BXYBvWF0YGiiTbNP69WZmck2Y5g4oq3AHcCtwIXA/cBFwH3Fx97SwtlFOSJI2BRRRJkjSCLKQqjOxU33apv+5c3+4DbFcsHqwElo1xm1XAvPrfi6h6p5S0DLiBqsjSLrTcAFwBXAmdVaUCSpIkiyiSJAmAbA3sWd/2avz7fsAWPX6xJcBt9e321teVVL02VgNLWT+MZmBYzRJgTf295dBZ0btYGRhCNBvYvH5wC2AWsCnVUKN59fe2bNwWt+43b/PonXVUxZUrhrhdB511PXw9SZLUYhFFkqQZI5uzvjjSLpZstRE7XglcS3WBf3X99Ua6F0lum1kX+tmU9QWVbal68GxH1XtnB2DH+rY91XCnjbEKuIr1RZXLgYuAC6l6sazdyP1LkjTjWUSRJGnayRyqAsn+wAH11/2pht6M1TKqC/Pr6ts19ddrubdw0rm5B6FF5lMVWHZqfN0e2A3Yo77tRNVTZqxWAH+ubxewvrjyF+is3ujokiTNEBZRJEma0rIV8GDWF0r2B/ajWr1mtFYAl9a3vzT+fSl0ru9pXG2kzKWan2aPIW5jnZtmNdXv+kI2KLBYXJEkqc0iiiRJU0a2AQ6vbwdRFUx2GuXGa6kKJJcwqEjCpcA10EnP46qALGR9QWUvYB9g3/rrWJaCXgX8Efhd43Z+b+egkSRp6rGIIklSX8psYG/gYOCvgIdSXQiP5th9J/Anql4FFwK/BX4HnbGuYKNpJYupeint2/h63/o2GmuoinC/rW8XVF87S3qfVZKk/mQRRZKkvpAdqHqYHFF/PYRqRZjhrAEupuoxcF59+yN0rp3AoJp2sg3re6vsCzyQqqfTaHqurKUqrAz0VvkNcK49ViRJ05VFFEmSishuwKOBRwIPo5o8dNgNqC5WzwZ+XX+9ADorJzKlZrLcl6qY0rxtO4oNVwHnAr8EflF9dfJhSdL0YBFFkqRJkW2BRwBHUw3N2XeEDe6m6llyFtWF6K+hc8tEJpRGlvtQDTEbuA0MCRrJDaz/Wz4L+P3MWupakjRdWESRJGlCZAvg4VS9TR5FNQfFUMfddVRzl5wN/Iqqp8lFXmRqasgOVL1UDqEajnYEsMUIG91O9bf+i/p2DnSWT2RKSZJ6wSKKJEk9kTlUE8A+nqpocjAwe6gnU03KeSbwI+Bn0LljMlJKEy+zqIqGDwWOpPp/sccIG62kKqr8qL6dA501E5lSkqTxsIgiSdK4ZSFVweSJwFOA7Yd58uWsH8rwHSd/1cySHYBDWb/S1CHA/GE2uIeqqPLD+vY7l+GWJPUDiyiSJI1J9gKeBDyBakLYOUM88Tqqnib1rXP15OSTpoJsyvqiypHAUcBmw2xwA+t7bv0QOtdMeERJkiRJ0lhlDuQRkPdBLoZkiNsKyPchx0EeUDq1NLVkDuRIyNshP4GsHOb/Wur/i/8FeUY9/5AkSZIkqYzMhzwZ8gXI7cNcyN0E+Wx9ITfcp+iSxiSbQh4H+Q/IbyFrh/l/uAryI8jrIHuWTi5JkiRJM0BmQx4K+TDklmEu2C6rn3N0PZmspAmXzer/cydBzh2hl8plkJMhT4LMK51ckiRJkqaJzIY8qr7gunWIC7LlkO9CXgPZrXRiSQDZFfISyP9B7himoLIE8r+QF0C2KZ1akiRJkqaYzKp7nHwUcsMw85t8C/I8yKLSiSUNJ3PrYuj7IZcMU1BZAzkL8mbIg0qnliRJkqQ+lgPqi6xrhplX4TuQF0O2LJ1W0nhlL8g/Qs6s/18PN+znJMjBpRNLkiRJUh/IFpBX1p8+d7uIWlt/73jIdqXTSuq1LKznRjkZcv0wBZWr6rmOHgrplE4tSZIkSZMknbpr/xcgy4YonPy0nuNk+9JpJU2WzIb8FeTfIBcMU1C5AvJeyGEWVCRJkiRNU9kF8va6i363C6M/Q06A7FQ6qaR+kH0h74D8cZiCypWQ90EeYkFFkiRJ0hSXeZBnUa2es6bLBdBSyGeqLvqSNJTsA/lnyPkjDPl5P+Tw0mklSZIkaQyyG+TfIbcMcbHzC8jLIJuVTippqsnekLdBzhumoHJJ3fNt99JpJUmSJGkIORhyCmR1l4ua2+rJI/cvnVLSdJE96omnh5qcOpBz6+dsXTqtJEmSpBkvCyAvhfyhy8XLGsi3IU+BzC2dVNJ0lr0gb4VcNEQxZRnkS5C/hswpnVaSJEnSjJIdICcOMWTnDqrlSHcvnVLSTJT9ICdBbhqioDLQM875mCRJkiRNpHuH7KzqcmFycd1tfmHplJJEtWzy0XWbdc8QBZUL6oLwbqXTSpIkSZoWMgvy7HpugfYFyFrINyGPKp1SkoaWxZBX1RNbdyumDAw/fGJVfJEkSZKkMclcyIuGmGPgzro7/N6lU0rS2GRXyAmQS4coqFxXDweyd4okSZKkkWQTyN9BruxycXEh5NUO2ZE09aUDeRjkk5C7urR3q+uedsdUPfIkSZIk6V5ZVM9pcl2Xi4nf171S7OYuaRrKAsizGHq55Gvr3im7lk4qSZIkqahsXndtv63LhcNZkCeVTihJkycH18MVlw4xd8oZdcHForIkSZI0c2QHyHu7XCisg5wGObJ0QkkqJ5vXk9H+fojeKZdB3gzZrnRSSZIkSRMmi6mW9GzPAbAWcirkkNIJJam/DNs7ZSXky5DDSqeUJEmS1DPZtB62c3vrAmAV5BRX2pGkkWRzyCuH6Z1ylkN9JEmSpCkt8+rVdm7oUjz5hBMlStJ45AjI5+ueKO1iyqWQ4yCblU4pSZIkaVQyq/5E9NIuc558GbJn6YSSNPVl+3qI5C1diil3QT4M2b10SkmSJElDytFDdDc/A3JQ6XSSNP1kPtVS8Od3aXsH5pw6unRKSZIkSffKwyG/7HIC/3PIQ0unk6TpLx3IYyCn1z3/2u3xOZDnV0MtJUmSJBWQPSBf73Ky/nvIX5dOJ0kzUx4A+S/I3V3a52shr4MsKp1SkiRJmiGyaT0Wf3nr5PyKegUJV4iQpOKyOeR4yJVdiim31u34VqVTSpIkSdNUOpDnQq7pcjL+asjc0gklSW2ZAzkWcvYQk9C+F7Jj6ZSSJEnSNJIDIT9rnXyvhpwM2aZ0OknSaOSh9WSz7WLKSsgpkL1KJ5QkSZKmsGxVL5W5pnXCfSZk/9LpJEnjkQfXRZN22z6woo8rqkmSJEmjlzn1/Ca3tE6wr6mW05QkTX25f92jcHWrrV9XF1OOKJ1QkiRJ6nN5GOSC1gn13ZC3QRaUTidJ6rXsUa/o054wPJAfQR5dOqEkSZLUZ7JFPXRnbesE+lTIbqXTSZImWrarV+1Z0qWY8kuLKZIkSRIAeRrkutYJ8++rSQglSTNLFkPe3mVIZyBnOMxHkiRJM1TuA/la6wR5GeRN1bwokqSZKwsh/wi5qUsx5TQnoJUkSdIMkU41QWxua50U/xTygNLpJEn9JJtCjofcOETPlANLJ5QkSZImSO5fTxTYPAleUq/G0ymdTpLUr7IIckKXOVPWQb5sEV6SJEnTSObUJ7/t1RdOhexUOp0kaarIZvXx5I7W8WRtXUzZs3RCSZIkaSNkf8h5rZPdqyBPKJ1MkjRVZVvI++u5tJrHl1WQkyG7lE4oSZIkjUFmQV4PWdH6pPDD1SeJkiRtrNwH8tHWsSb1/fdXq/1IkiRJfS27QM5sndBeBDm8dDJJ0nSUXSGfrHuiNI89t9Wr/MwvnVCSJEnqIs9tTfy3DvIxyCalk0mSprvcD3JK3fOxWUy5HHKsk5hLkiSpT2SL+sS1edJ6I+SJpZNJkmaa7As5rcuyyL+BPLx0OkmSJM1oORJyWetE9WuQbUonkyTNZDka8ocuxZQzIA8snU6SJEkzSuZCTmx1m74L8srSySRJqmQW5FmQK1uFlNX1Sj47lE4oSZKkaS/7dlm6+CzIHqWTSZK0oSyE/DNkaevYdSfkn5y7S5IkSRMkz2+dhK6qT0Bnl04mSdLwsgPkE3VPlGYx5RrIi6ueK5IkSdJGy3zIx7ssXXxw6WSSJI1N9oWc2mW+lHMgDymdTpIkSVNadoX8unWi+fVqVR5JkqaqHAH5Rev4tq5ecW770ukkSZI05eTJkCWNk8sVkNeUTiVJUm+kA3kB5NpWMWUJ5HjInNIJJUmS1Pcyp8vqO1dBDi+dTJKk3sum9XFvRauY8mfIY0unkyRJUt/KdpAftk4iT4VsVTqZJEkTK3tCTu8yX8qpkN1Kp5MkSVJfySMg1zdOGtfUn8y5YoEkaQbJkyCXtwop99THxPml00mSJKm4vKkumgycLF4POap0KkmSysgmddFkWauYcgnkmNLpJEmSVETmQz7bOkH8CWTH0skkSSovO9cr9nQb4nPf0ukkSZI0abId5KzWSeG/QWaXTiZJUn/J4+qJZpvHzOWQt0PmlU4nSZKkCZUHQa5oLV/84tKpJEnqX5lbL318Z5chPo8snU6SJEkTIk+GLG2c/N0IOaJ0KkmSpobsBPl/rULKWsh/QbYonU6SJEk9k+PrE72Bk77zIbuXTiVJ0tSTR0AubhVTboC8qHQySZIkbZSuE8ieDtm8dDJJkqauLine truncated
}
},
"cell_type": "markdown",
"id": "74d5b606",
"metadata": {},
"source": [
"<div>\n",
"<img src=\"attachment:g13389.png\" align=\"left\" width=\"350\"/>\n",
"</div>"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5ea413fd",
"metadata": {},
"outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
" MPI.Init()\n",
" comm = MPI.COMM_WORLD\n",
" nranks = MPI.Comm_size(comm)\n",
" rank = MPI.Comm_rank(comm)\n",
" root = 0\n",
" rcv = Ref(0) \n",
" if rank == root\n",
" snd = [10*(i+1) for i in 1:nranks]\n",
" println(\"[rank $rank] I am sending: $snd\")\n",
" else\n",
" # Only the root rank sends data\n",
" # so this can be anything\n",
" snd = nothing\n",
" end \n",
" MPI.Scatter!(snd,rcv,comm;root)\n",
" println(\"[rank $rank] I have received: $(rcv[])\")\n",
"end\n",
"run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
]
},
{
"cell_type": "markdown",
"id": "ed8da7f9",
"metadata": {},
"source": [
"## MPI_Bcast\n",
"\n",
"Similar to `MPI_Scatter`, but we send the same message to all processes.\n",
"\n",
"In Julia:\n",
"```julia\n",
"MPI.Bcast!(buf, comm; root)\n",
"```\n",
"\n",
"In C:\n",
"```c\n",
"int MPI_Bcast(void *buffer, int count, MPI_Datatype datatype,\n",
" int root, MPI_Comm comm)\n",
"```\n"
]
},
{
"cell_type": "markdown",
"id": "97369713",
"metadata": {},
"source": [
"### Example\n",
"\n",
"Rank 0 sends the same integer to all processes."
]
},
{
"attachments": {
"g1657.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA9MAAANWCAYAAAD9cN8uAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13nC1lYf/xz7n9ApdeLyACgghIUbEhVowVOxqN0RhLjJpgDfYYuybGoPGn2BLRWIgVCyq2SBGlKCBNFOm9t9vv9/fHzMLs7Nm9Z8+W5+zu5/16zWvvPbsz892zu8/MM0/rIElSUekAWwJbNT5uBSwBNgcWAJvW/18KbAIsrD+3EFjW+Nyy+uvHYwtgBbASWAvcXr9+c/3xDmBN/fkVwDrgtvpzt9bbLaNvnTvHmUeSJM0AndIBJEmzTeYB2wM7ATsCOzCystysNG/J7L4erWFkhft64DrgGuAq4Nr643XQubZQTkmSNA6z+eZFkjTpsoiqcrwT91SWd64/7gjci6oiPd7W4cmynqriOh6hapFeCCwGNprsUOO0hqqifRVVZfvqersWuBK4BPgzdG4f7QCSJGnqWZmWJLVkCbBHl+0+VBXpybx23AncBNxYbzc0/n9H/fnVVBXkNVTdq1cBd3FP9+tbqo+TWbnMfKqu5QCbAfOoKtmLgUXAxlTvw+b15zfvYdt48vIB1Xv1Z+6uXA/bLoXOqkk+nyRJarAyLUlzUhYCuwJ71luz0rwz/V8f1lO1pl5B1bJ6GVWL6vXcU2G+6Z6PnZX9fw8zTYbGeW8ObANsR9Wav239cTuqhxXb1/+eP4GTrad6/4cq1xcDFwLnAxdY0ZYkaeKsTEvSrJcdgf3qbf/6430Zf1fstVTdjK9geGX5ynq7HLgaOmsnJ/dclnlUlex2hXs51UOQoW1xHwdfR1XBPpeqcn0+cB5VJfuOCUeXJGmOsDItSbNGFgP7cE+FeajyvNU4DrKeqoJ8UWP7Q/3xEuismcTAmpB0qFqyd2N4BXto25HxtW4HuBS4gKqiPfTx947PliRpJCvTkjQjZTHwAOChwEGMv7X5aqrK0kWt7U92AZ4tsohqQrhdgd2BvYH71duO4zjQeqrfjTMb22+hc/OYe0mSNMtZmZakGSHLgQcCBwOPqP+9pIcd11BVhM6gamU8DzgNOtdMUVDNCNmMakK5fagq2bvV/96LarK1XlxN9Xs1tPl7JUmaU6xMS9LAyUZUleWHAg+rP+7Qw45XAOcAZ9XbOcCFjmFW77IxVYX6ftxT0T6QalK6XlzOPa3XpwO/sgVbkjRbWZmWpOKyDHgk8FjgUVTjnDfUXfsG4NR6+zVwJnRumsqUmsuyDdWwgua2Ww87rqea4OzkejsFOn+cqpSSJE0nK9OSNO2ygKrCfGi9PZJq7eLRrKNa1ugM4CSqSsl50MkUB5XGkE2pxuo/sLH10k38WuA07vl9PmluLZEmSZotrExL0pTLAuBBVC3PjwUeDiwdY4eruafV+VTgdOjcNdUppYnLMqpu4Q+gGqJwMBue7GwFVZfwodbrX0HnxqlMKUnSZLAyLUlTIvcBngI8jqrr9qZjfPG1wM+Bn1Vb509Tn0+aLtmFqlL9cKrJ8/Zl7CW71gNnAz8BfgqcCJ07pzqlJEnjZWVakiZF5gMHAIcBT6Xq8jqaO6hanH9Sb2faZVtzRzah+lsZmpn+YGCLMXZYSzWh3tDfyy+hs3qqU0qStCFWpiWpb9kceAJVBfpJwJajfOFdVGND65ZnzoTOummJKA28zKcaez3Uev1oxp69/nbg/6harX8K/N6HUZIkSdLAy30gr4P8FLIaklG2cyAfgDwSMtbkYpJGyN6Qf4B8G3LLGH9ngVwD+R/I30J2Kp1ckiRJ0t2yP+T9kPPHuKFfBflxXQHYtXRiafbIfMhDIG+tH2KtGOPvcD3kTMh7IA+GbGhmcUmSJEmTK3tD3rWBCvQNkGMhL4JsVjqxNDdkKeTQ+u/zhA30ELnev1FJkiRpymX3uvXrrDFuzs+uW6kfZquXNAiyBeRZkE9BLh/jb3cl5Ed175HdSqeWJEmSZrjcC/JGyGlj3IT/DvIWb8ClmSAHQt4OORWyboy/63MhH4IcUnUllyRJkrQB2RLyGshJ9RjL0W603wm5b+m0kvqVbSEvgXwdctsGhmx8BvIXkAWlU0uSJEkDJPMgj4AcDblzlBvqSyBHVV8naXbJgroM+OAG5kK4CXIM5DCciV+SJElzV+5dT1R06RgV6A9DHlA4qKRplb0h/wQ5cYzu4DdCPgd5EmRh6cSSJEnSFMsSyPOpZvrtdpN8K+TTkIdDOqXTSiotO9RDP/5vjIr1TZDPQ55si7UkSZJmmRwI+Xh909tt/dlfQl4M2bh0UkmDKjtAXg35xQYq1v8NeYot1pIkSZqhsqSeYOiMUW56r4R8ALJn6aSSZppsD3kV5OeQtaOUMdfVcy08qHRaSZIkqQfZvh4LfV2Xm9u1dRfvw201kjQ5shXkRZDvQlaPUrE+ry6Xdi2dVpIkSWrJwyBfGeVm9jzIGyDblk4paTbLdpC/rycv67a83rr6gd6LIJuUTitJkqQ5K4vqVuaTx7hpPczJxCRNv+wMORLyh1Faq++CHFuXUa5hLUmSpOmQ7eqb1Mu73KDeRrVm9P1Kp5SkSh5Yj5++fow5HI6CHFA6qSRJkmal7A35AmRVl5vRC6hm2rXrpKQBlSV1b5rjxhhffSrV5IkblU4rSZKkGS/7Q47pMmvuertyS5qZsgXkFZCTRqlU31r3srG1WpIkSeOVQyDHj9KV+2O4rJWkWSH7QD5Yd/fuVrE+uZ60bGnppJIkSRpoeUTd4ty+obyBanmZLUsnlKTJl3mQQ+uJybqtX31L3Vq9X+mkkiRJGhjp1N21f93lBvLauhK9WemUkjQ9smM90eJlo7RWn153E3dstSRJ0tyUBZAXQs7tcrN4MeSVkMWlU0pSGVkIeQ7kx3Rfu/oGyEcc9iJJkjRn3N0SfXaXm8M/1i0urr0qSXfL7vXY6mu7lJtOyChJkjT75VDIaV1uBn9XT7Izv3RCSRpcWUS1xNYJo7RWXwg5wi7gkiRJs0YeCvlZlxu/30CeWDqdJM082bPu5n1Ll7L1esh7ITuUTilJkqS+5H717LTtFpTz69YVuyRK0oRkk3p4zHldKtWr6jL4oaVTSpIkqSfZuV7Gpb3Ey2WOiZakqXD38lrfHaUL+On1cBrLX0mSpMGTbSEfq1tDmjdx19Xj+JydW5KmXA6A/Bdk5SirJbwOsmnplJIkSaomxXkj5NbWTdutkHdClpVOKElzT7aDvGuUWcBvhXwIsn3plJIkSXNUDoX8vss4vaOrGzlJUll3zwL+61HGVR8D2aN0SkmSpDkie0C+17opW1/flO1UOp0kqZs8dpRx1Wvq8nvv0gklSZJmqWxcdxtsj8U7A3Jw6XSSpF5k37ryvKbLQ9HvOgO4JEnSpEmn7iZ4WevG64Z6crH5pRNKksYr94YcBbmrSxfwkyCHlU4oSZI0g+WBkJNbN1mr6xuwzUqnkyRNVLatex3d0qVSfWb9MLVTOqUkSdIMkS0hn4Osa91Y/QBy39LpJEmTLVvWleobu1Sqz4a8wJ5IkiRJY8phkCtaN1IXVa0TkqTZLRvXQ3gu71KpvhjyCivVkiRJw2TnevKZ5o3THZB/qpZXkSTNHVkM+TvIn0ZpqX6G3b8lSdIcl3mQf4Dc1qVL9y6l00mSSsoCyF9BLuhSqT4N8qTSCSVJkgrIvpBTWjdHN1Xd+CRJGpJ59WRkF3WpVP/K2b8lSdIckYWQIxm5ZvSxkG1Kp5MkDaoshLxolO7fJ0EeUzqhJEnSFMnBkHO7TCrzhNLJJEkzRRbVk5Fd2aVSfQLkoNIJJUmSJkmWQj4GWd+44VkL+TfIRqXTSZJmomwEeRPk+laFej3k25D9SieUJEmagDwAcl7rRue3kAeWTiZJmg2yDPJ2yM2ta806yP9A7l06oSRJ0jhkPuTNkFWNG5uV9WsLSqeTJM022Rzyni4rRKyEfLj6vCRJ0kDLvSG/bN3M/B6yf+lkkqTZLltD/hVyV+s6dAPktdWYa0mSpIGTw+slrppj1452bLQkaXplp/r6s7ZVqb60nhW8UzqhJEkSkM0gX2rdsFwNeXLpZJKkuSz7QL7fZebv30AeWTqdJEma0/I4yOWtm5T/hWxVOpkkSZUcCjmrS6X6u5A9SqeTJElzShZBPtpa8uoWyAtLJ5MkaaTMh7wcclWrQr0aclQ13lqSJGlK5V6QU1s3I7+A7FI6mSRJY8smkHdB7mhdx26GHAlZUjqhJEmalfKkelbUoZuPVZB/gswrnUySpN5lB8hnukxSdjHk6aXTSZKkWSPz6yf56xo3HJdDHl46mSRJ/ctekGO7jKf+CWTv0ukkSdKMlu0gP23dZBzv+DJJ0uyRQyHntK51qyAfqLqGS5IkjUsOgVzZWjv6g3brliTNPlkAOaKeULNZqb7S9aklSVKP0qlvKFY3biauh/xF6WSSJE2tbFXP8L2OkZNt7lc6nSRJGljZFPL11g3ELyHLSyeTJGn65EGQX7Wuh2sgR1cVbkmSpLtlX8ifWt26P1x1fZMkaa7JPMjftVayCOQayEvs+i1JkoA8sTVO7FbIc0qnkiSpvGxRd/1uL6V1OuShpdNJkqRi8obWDcJZkN1Kp5IkabDkQMjJrQr1Osj/g2xWOp0kSZo2WQT5XOum4AfeEEiSNJp0IIdDLmtdP6+uZv2WJEmzXLauJxZr3gh82GWvJEnqRTaF/Hs9KVnzWvptyI6l00mSpCmRPSEXNi78qyB/UzqVJEkzT/aH/LpVob4DciRkful0kiRp0uQJrYnGboA8unQqSZJmrsyDvAJyW6tSfQbkAaXTSZKkCcsrWt3RzoHsWjqVJEmzQ3aEfKtVoV4N+SBkSel0kiRp3LIA8unWxf27kGWlk0mSNPvk+ZBrW9fdC+0JJmmyuMi9NC2yLine truncated
}
},
"cell_type": "markdown",
"id": "26b56c81",
"metadata": {},
"source": [
"<div>\n",
"<img src=\"attachment:g1657.png\" align=\"left\" width=\"350\"/>\n",
"</div>"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4de15781",
"metadata": {},
"outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
" MPI.Init()\n",
" comm = MPI.COMM_WORLD\n",
" nranks = MPI.Comm_size(comm)\n",
" rank = MPI.Comm_rank(comm)\n",
" root = 0\n",
" buffer = Ref(0)\n",
" if rank == root\n",
" buffer[] = 20\n",
" println(\"[rank $rank] I am sending: $(buffer[])\")\n",
" end \n",
" MPI.Bcast!(buffer,comm;root)\n",
" println(\"[rank $rank] I have received: $(buffer[])\")\n",
"end\n",
"run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
]
},
{
"cell_type": "markdown",
"id": "e83be96d",
"metadata": {},
"source": [
"## Communicators\n",
"\n",
"A key component in MPI programming are so-called communicators. A communicator object has to main purposes:\n",
"\n",
"\n",
"1. To provide an isolated communication context\n",
"2. To define a group of processes\n",
"\n",
"\n",
"### Communication context\n",
"\n",
"When you write an MPI program it is very likely that you are going to use libraries that also use MPI to send messages. Ideally, these libraries should not interfere with application messages. Using tags to isolate the messages send by your application does not solve the problem. MPI communicators fix this problem as they provided an isolated communication context. For instance, `MPI_SEND` and `MPI_RECV` specify a communicator. `MPI_RECV` can only receive messages sent to same communicator. The same is also true for collective communication directives. If two libraries use different communicators, their message will never interfere. In particular it is recommended to never use the default communicator, `MPI_COMM_WORLD`, directly when working with other libraries. A new isolated communicator can be created with `MPI_Comm_dup`.\n",
"\n",
"\n",
"### Process groups\n",
"\n",
"On the other hand, imagine that we want to use an MPI communication directive like `MPI_Gather`, but we only want a subset of the processes to participate in the operation. So far, we have used always the default communication `MPI_COMM_WORLD`, which represents all processes. Thus, by using this communicator, we are including all processes in the operation. We can create other communicators that contain only a subset of processes. To this end, we can use function `MPI_Comm_split`.\n"
]
},
{
"cell_type": "markdown",
"id": "556ca223",
"metadata": {},
"source": [
"## MPI_Comm_dup\n",
"\n",
"Duplicate a given communicator, creating a new one with an new communication context.\n",
"\n",
"\n",
"In Julia:\n",
"```julia\n",
"newcomm = MPI.Comm_dup(comm)\n",
"```\n",
"\n",
"In C:\n",
"```c\n",
"int MPI_Comm_dup(MPI_Comm comm, MPI_Comm *newcomm)\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "9acfb792",
"metadata": {},
"source": [
"### Example\n",
"\n",
"Send value 2 from rank 2 to rank 3. Then send value 1 also from rank 2 to rank 3. Note that we are using 2 different communicators for each send receive pair in such a way that we first receive value 1 and then value 2, even if this not was the other in which the data was sent. Try also to define `comm2 = comm` and see how the result will change."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e50d44f6",
"metadata": {},
"outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
" MPI.Init()\n",
" comm = MPI.COMM_WORLD\n",
" # We create a new communicator\n",
" comm2 = MPI.Comm_dup(comm)\n",
" rank = MPI.Comm_rank(comm)\n",
" if rank == 2\n",
" sndbuf = [2]\n",
" MPI.Send(sndbuf, comm2; dest=3, tag=0)\n",
" sndbuf = [1]\n",
" MPI.Send(sndbuf, comm; dest=3, tag=0)\n",
" end\n",
" if rank == 3\n",
" rcvbuf = zeros(Int,1)\n",
" MPI.Recv!(rcvbuf, comm; source=2, tag=0)\n",
" @show rcvbuf\n",
" MPI.Recv!(rcvbuf, comm2; source=2, tag=0)\n",
" @show rcvbuf\n",
" end\n",
"end\n",
"run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
]
},
{
"cell_type": "markdown",
"id": "5e1b13eb",
"metadata": {},
"source": [
"## MPI_Comm_split\n",
"\n",
"Split a given communicator.\n",
"\n",
"\n",
"In Julia:\n",
"\n",
"```julia\n",
"newcomm = MPI.Comm_split(comm, color, key)\n",
"```\n",
"\n",
"In C:\n",
"```c\n",
"int MPI_Comm_split(MPI_Comm comm, int color, int key,\n",
" MPI_Comm *newcomm)\n",
"```\n",
"\n",
"There are two key parameters:\n",
"\n",
"- `color`: all processes with the same color will be grouped in the same communicator.\n",
"- `key`: The processes will be ranked in the new communicator according to key, breaking ties with the rank in the old communicator. \n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "2b128840",
"metadata": {},
"source": [
"### Example\n",
"\n",
"Create two groups of processes, one for processes with rank id even, and another for the rest. Note that the rank id will be function of the given communicator."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b18cb74d",
"metadata": {},
"outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
" MPI.Init()\n",
" comm = MPI.COMM_WORLD\n",
" rank = MPI.Comm_rank(comm)\n",
" color = mod(rank,2)\n",
" key = 0\n",
" newcomm = MPI.Comm_split(comm, color, key)\n",
" newrank = MPI.Comm_rank(newcomm)\n",
" println(\"Hello, I am rank $rank in world and rank $newrank in group $color.\")\n",
"end\n",
"run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
]
},
{
"cell_type": "markdown",
"id": "b22e4bf4",
"metadata": {},
"source": [
"### Example\n",
"\n",
"Processes with odd rank id generate a random integer. Sum all these integers and send them to the first process with odd rank id. Note that we are using a new communicator to perform the reduction."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4c0e504b",
"metadata": {},
"outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
" MPI.Init()\n",
" comm = MPI.COMM_WORLD\n",
" rank = MPI.Comm_rank(comm)\n",
" color = mod(rank,2)\n",
" key = 0\n",
" newcomm = MPI.Comm_split(comm, color, key)\n",
" # Only odd rank ids will execute this\n",
" if color == 1\n",
" newrank = MPI.Comm_rank(newcomm)\n",
" newroot = 0\n",
" sndbuf = [rand(1:4)]\n",
" if newrank == newroot\n",
" rcvbuf = [0]\n",
" else\n",
" rcvbuf = nothing\n",
" end\n",
" println(\"[rank $rank, newrank $newrank] I am contributing with $sndbuf\")\n",
" MPI.Reduce!(sndbuf, rcvbuf, +, newcomm;root=newroot)\n",
" println(\"[rank $rank, newrank $newrank] The sum is $rcvbuf\")\n",
" end\n",
"end\n",
"run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
]
},
{
"cell_type": "markdown",
"id": "5e8f6e6a",
"metadata": {},
"source": [
"# License\n",
"\n",
"This notebook is part of the course [Programming Large Scale Parallel Systems](https://www.francescverdugo.com/XM_40017) at Vrije Universiteit Amsterdam and may be used under a [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) license."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Julia 1.9.0",
"language": "julia",
"name": "julia-1.9"
},
"language_info": {
"file_extension": ".jl",
"mimetype": "application/julia",
"name": "julia",
"version": "1.9.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}