Merge pull request #72 from jvdtoorn/fix/mpi_collectives-deadlock-comm_dup

Change julia kernel version for MPI collectives notebook

docs/make.jl

@@ -13,7 +13,7 @@ EditURL = "https://github.com/fverdugo/XM_40017/blob/main/notebooks/SCRIPT_NAME.
     <div class="admonition-body">
         <ul>
             <li>
-                Download this notebook and run it locally on your machine [recommended]. Click <a href="https://www.francescverdugo.com/XM_40017/dev/SCRIPT_NAME.ipynb" download>here</a>.
+                Download this notebook and run it locally on your machine [highly recommended]. Click <a href="https://www.francescverdugo.com/XM_40017/dev/SCRIPT_NAME.ipynb" download>here</a>.
             </li>
         </ul>
     </div>
@@ -119,13 +119,15 @@ makedocs(;
                          "Julia Basics" => "julia_basics.md",
                          "Asynchronous programming in Julia" => "julia_async.md",
                          "Distributed computing in Julia" => "julia_distributed.md",
-                         "Distributed computing with MPI" => "mpi_tutorial.md",
+                         #"Distributed computing with MPI" => "mpi_tutorial.md",
                          "Matrix-matrix multiplication"=>"matrix_matrix.md",
+                         "MPI (point-to-point)" => "julia_mpi.md",
+                         "MPI (collectives)" => "mpi_collectives.md",
                          "Jacobi method" => "jacobi_method.md",
                          "All pairs of shortest paths" => "asp.md",
                          "Gaussian elimination" => "LEQ.md",
                          "Traveling salesperson problem" => "tsp.md",
-                         "Partial differential equations" => "pdes.md",
+                         #"Partial differential equations" => "pdes.md",
                          ],
           "Solutions" => "solutions_for_all_notebooks.md",
           ],

docs/src/getting_started_with_julia.md

@@ -124,12 +124,23 @@ $ julia --version
 
 If this runs without error and you see a version number, you are good to go!
 
+You can also run julia code from the terminal using the `-e` flag:
+
+```
+$ julia -e 'println("Hello, world!")'
+```
 
 !!! note
     In this tutorial, when a code snipped starts with `$`, it should be run in the terminal. Otherwise, the code is to be run in the Julia REPL.
 
 !!! tip
-    Avoid calling Julia code from the terminal, use the Julia REPL instead! Each time you call Julia from the terminal, you start a fresh Julia session and Julia will need to compile your code from scratch. This can be time consuming for large projects. In contrast, if you execute code in the REPL, Julia will compile code incrementally, which is much faster. Running code in a cluster (like in DAS-5 for the Julia assignment) is among the few situations you need to run Julia code from the terminal.
+    Avoid calling Julia code from the terminal, use the Julia REPL instead!
+    Each time you call Julia from the terminal, you start a fresh Julia session and Julia will need to compile your code from scratch.
+    This can be time consuming for large projects.
+    In contrast, if you execute code in the REPL, Julia will compile code incrementally, which is much faster.
+    Running code in a cluster (like in DAS-5 for the Julia assignment) is among the few situations you need to run Julia code
+    from the terminal. Visit this link ([Julia workflow tips](https://docs.julialang.org/en/v1/manual/workflow-tips/))
+    from the official Julia documentation for further information about how to develop Julia code effectivelly.
 
 ### Running parallel code
 
@@ -199,11 +210,11 @@ To install a package, we need to enter *package* mode. Remember that we entered
 ```julia
 julia> ]
 ```
-At this point, the prompt should have changed to `(@v1.8) pkg>` indicating that we are in package mode. The text between the parentheses indicates which is the active *project*, i.e., where packages are going to be installed. In this case, we are working with the global project associated with our Julia installation (which is Julia 1.8 in this example, but it can be another version in your case).
+At this point, the prompt should have changed to `(@v1.11) pkg>` indicating that we are in package mode. The text between the parentheses indicates which is the active *project*, i.e., where packages are going to be installed. In this case, we are working with the global project associated with our Julia installation (which is Julia 1.11 in this example, but it can be another version in your case).
 
 To install the MPI package, type
 ```julia
-(@v1.8) pkg> add MPI
+(@v1.11) pkg> add MPI
 ```
 Congrats, you have installed MPI!
 
@@ -211,7 +222,8 @@ Congrats, you have installed MPI!
     Many Julia package names end with `.jl`. This is just a way of signaling that a package is written in Julia. When using such packages, the `.jl` needs to be omitted. In this case, we have installed the `MPI.jl` package even though we have only typed `MPI` in the REPL.
 
 !!! note
-    The package you have installed is the Julia interface to MPI, called  `MPI.jl`. Note that it is not a MPI library by itself. It is just a thin wrapper between MPI and Julia. To use this interface, you need an actual MPI library installed in your system such as OpenMPI or MPICH. Julia downloads and installs a MPI library for you, but it is also possible to use a MPI library already available in your system. This is useful, e.g., when running on HPC clusters. See the [documentation](https://juliaparallel.org/MPI.jl/stable/configuration/) of `MPI.jl` for further details.
+    The package you have installed is the Julia interface to MPI, called  `MPI.jl`. Note that it is not an MPI library by itself. It is just a thin wrapper between MPI and Julia. To use this interface, you need an actual MPI library installed in your system such as OpenMPI or MPICH. Julia downloads and installs an MPI library for you, but it is also possible to use an MPI library already available in your system. This is useful, e.g., when running on HPC clusters. See the [documentation](https://juliaparallel.org/MPI.jl/stable/configuration/) of `MPI.jl` for further details.
+
 
 To check that the package was installed properly, exit package mode by pressing the backspace key several times, and run it again
 
@@ -230,7 +242,7 @@ $ mpiexec -np 4 julia hello_mpi.jl
 But it will probably not work since the version of `mpiexec` needs to match with the MPI version we are using from Julia. Don't worry if you could not make it work! A more elegant way to run MPI code is from the Julia REPL directly, by using these commands:
 ```julia
 julia> using MPI
-julia> mpiexec(cmd->run(`$cmd -np 4 julia hello_mpi.jl`))
+julia> run(`$(mpiexec()) -np 4 julia hello_mpi.jl`);
 ```
 
 Now, you should see output from 4 ranks.
@@ -241,9 +253,9 @@ Now, you should see output from 4 ranks.
 
 We have installed the `MPI` package globally and it will be available in all Julia sessions. However, in some situations, we want to work with different versions of the same package or to install packages in an isolated way to avoid potential conflicts with other packages. This can be done by using local projects.
 
-A project is simply a folder in the hard disk. To use a particular folder as your project, you need to *activate* it. This is done by entering package mode and using the `activate` command followed by the path to the folder you want to activate.
+A project is simply a folder in your file system. To use a particular folder as your project, you need to *activate* it. This is done by entering package mode and using the `activate` command followed by the path to the folder you want to activate.
 ```julia
-(@v1.8) pkg> activate .
+(@v1.11) pkg> activate .
 ```
  The previous command will activate the current working directory. Note that the dot `.` is indeed the path to the current folder.
 
@@ -253,7 +265,7 @@ The prompt has changed to `(lessons) pkg>` indicating that we are in the project
     You can activate a project directly when opening Julia from the terminal using the `--project` flag. The command `$ julia --project=.` will open Julia and activate a project in the current directory. You can also achieve the same effect by setting the environment variable  `JULIA_PROJECT` with the path of the folder you want to activate.
 
 !!! note
-    The active project folder and the current working directory are two independent concepts! For instance,  `(@v1.8) pkg> activate folderB` and then `julia> cd("folderA")`, will activate the project in `folderB` and change the current working directory to `folderA`.
+    The active project folder and the current working directory are two independent concepts! For instance,  `(@v1.11) pkg> activate folderB` and then `julia> cd("folderA")`, will activate the project in `folderB` and change the current working directory to `folderA`.
 
 At this point all package-related operations will be local to the new project. For instance, install the `DataFrames` package.
 
@@ -271,7 +283,7 @@ Now, we can return to the global project to check that `DataFrames` has not been
 ```julia
 (lessons) pkg> activate
 ```
-The prompt is again `(@v1.8) pkg>`
+The prompt is again `(@v1.11) pkg>`
 
 Now, try to use `DataFrames`.
 
@@ -295,13 +307,13 @@ In other words, `Project.toml` contains the packages relevant for the user, wher
 You can see the path to the current `Project.toml` file by using the `status` operator (or `st` in its short form) while in package mode
 
 ```julia
-(@v1.8) pkg> status
+(@v1.11) pkg> status
 ```
 
 The information about the `Manifest.toml` can be inspected by passing the `-m` flag.
 
 ```julia
-(@v1.8) pkg> status -m
+(@v1.11) pkg> status -m
 ```
 
 ### Installing packages from a project file
@@ -325,7 +337,7 @@ julia> mkdir("newproject")
 
 To install all the packages registered in this file you need to activate the folder containing your `Project.toml` file
 ```julia
-(@v1.8) pkg> activate newproject
+(@v1.11) pkg> activate newproject
 ```
 and then *instantiating* it
 ```julia
@@ -339,12 +351,12 @@ The instantiate command will download and install all listed packages and their
 You can get help about a particular package operator by writing `help` in front of it
 
 ```julia
-(@v1.8) pkg> help activate
+(@v1.11) pkg> help activate
 ```
 
 You can get an overview of all package commands by typing `help` alone
 ```julia
-(@v1.8) pkg> help
+(@v1.11) pkg> help
 ```
 
 ### Package operations in Julia code
@@ -357,12 +369,65 @@ julia> Pkg.status()
 ```
 is equivalent to calling `status` in package mode.
 ```julia
-(@v1.8) pkg> status
+(@v1.11) pkg> status
 ```
 
+### Creating you own package
+
+In many situations, it is useful to create your own package, for instance, when working with a large code base, when you want to reduce compilation latency using [`Revise.jl`](https://github.com/timholy/Revise.jl),
+or if you want to eventually [register your package](https://github.com/JuliaRegistries/Registrator.jl) and share it with others.
+
+The simplest way of generating a package (called `MyPackage`) is as follows. Open Julia, go to package mode, and type
+
+```julia
+(@v1.11) pkg> generate MyPackage
+```
+
+This will crate a minimal package consisting of a new folder `MyPackage` with two files:
+
+* `MyPackage/Project.toml`: Project file defining the direct dependencies of your package.
+* `MyPackage/src/MyPackage.jl`: Main source file of your package. You can split your code in several files if needed, and include them in the package main file using function `include`.
+
+!!! tip
+    This approach only generates a very minimal package. To create a more sophisticated package skeleton (including unit testing, code coverage, readme file, licence, etc.) use
+    [`PkgTemplates.jl`](https://github.com/JuliaCI/PkgTemplates.jl) or [`BestieTemplate.jl`](https://github.com/JuliaBesties/BestieTemplate.jl). The later one is developed in Amsterdam at the
+    [Netherlands eScience Center](https://www.esciencecenter.nl/).
+
+You can add dependencies to the package by activating the `MyPackage` folder in package mode and adding new dependencies as always:
+
+```julia
+(@v1.11) pkg> activate MyPackage
+(MyPackage) pkg> add MPI
+```
+
+This will add MPI to your package dependencies.
+
+### Using your own package
+
+To use your package you first need to add it to a package environment of your choice. This is done by changing to package mode and typing `develop ` followed by the path to the folder containing the package. For instance:
+
+```julia
+(@v1.11) pkg> develop MyPackage
+```
+
+!!! note
+    You do not need to "develop" your package if you activated the package folder `MyPackage`.
+
+Now, we can go back to standard Julia mode and use it as any other package:
+
+```julia
+using MyPackage
+MyPackage.greet()
+```
+
+Here, we just called the example function defined in `MyPackage/src/MyPackage.jl`.
+
+
 ## Conclusion
 
-We have learned the basics of how to work with Julia. If you want to further dig into the topics we have covered here, you can take a look at the following links:
+We have learned the basics of how to work with Julia, including how to run serial and parallel code, and how to manage, create, and use Julia packages.
+This knowledge will allow you to follow the course effectively!
+If you want to further dig into the topics we have covered here, you can take a look at the following links:
 
 - [Julia Manual](https://docs.julialang.org/en/v1/manual/getting-started/)
 - [Package manager](https://pkgdocs.julialang.org/v1/getting-started/)

docs/src/index.md

@@ -10,7 +10,7 @@ Welcome to the interactive lecture notes of the [Programming Large-Scale Paralle
 This page contains part of the course material of the Programming Large-Scale Parallel Systems course at VU Amsterdam.
 We provide several lecture notes in jupyter notebook format, which will help you to learn how to design, analyze, and program parallel algorithms on multi-node computing systems.
 Further information about the course is found in the study guide
-([click here](https://studiegids.vu.nl/EN/courses/2023-2024/XM_40017#/)) and our Canvas page (for registered students).
+([click here](https://studiegids.vu.nl/en/vakken/2025-2026/XM_40017#/)) and our Canvas page (for registered students).
 
 !!! note
     Material will be added incrementally to the website as the course advances.

docs/src/solutions_for_all_notebooks.md

@@ -2,19 +2,6 @@
 
 ## Julia Basics
 
-### NB1-Q1
-
-In the first, line we assign a variable to a value. In the second line, we assign another variable to the same value. Thus,we have 2 variables associated with the same value. In line 3, we associate `y` to a new value (re-assignment). Thus, we have 2 variables associated with 2 different values. Variable `x` is still associated with its original value. Thus, the value at the final line is `x=1`.
-
-### NB1-Q2
-
-It will be `1` for very similar reasons as in the previous questions: we are reassigning a local variable, not the global variable defined outside the function.
-
-### NB1-Q3
-
-It will be `6`. In the returned function `f2`, `x` is equal to `2`. Thus, when calling `f2(3)` we compute `2*3`.
-
-
 ### Exercise 1
 
 ```julia
@@ -40,94 +27,33 @@ ex2(f,g) = x -> f(x) + g(x)
 ### Exercise 3
 
 ```julia
-using GLMakie
-max_iters = 100
 n = 1000
 x = LinRange(-1.7,0.7,n)
 y = LinRange(-1.2,1.2,n)
-heatmap(x,y,(i,j)->mandel(i,j,max_iters))
+values = zeros(n,n)
+for j in 1:n
+    for i in 1:n
+        values[i,j] = surprise(x[i],y[j])
+    end
+end
+using GLMakie
+heatmap(x,y,values)
 ```
 
 ## Asynchronous programming in Julia
 
-### NB2-Q1
-
-Evaluating `compute_π(100_000_000)` takes about 0.25 seconds. Thus, the loop would take about 2.5 seconds since we are calling the function 10 times.
-
-### NB2-Q2
-
-The time in doing the loop will be almost zero since the loop just schedules 10 tasks, which should be very fast.
-
-### NB2-Q3
-
-It will take 2.5 seconds, like in question 1. The `@sync` macro forces to wait for all tasks we have generated with the `@async` macro. Since we have created 10 tasks and each of them takes about 0.25 seconds, the total time will be about 2.5 seconds.
-
-### NB2-Q4
-
-It will take about 3 seconds. The channel has buffer size 4, thus the call to `put!`will not block. The call to `take!` will not block neither since there is a value stored in the channel. The taken value is 3 and therefore we will wait for 3 seconds. 
-
-### NB2-Q5
-
-The channel is not buffered and therefore the call to `put!` will block. The cell will run forever, since there is no other task that calls `take!` on this channel. 
-
 ## Distributed computing in Julia
 
-### NB3-Q1
-
-We send the matrix (16 entries) and then we receive back the result (1 extra integer). Thus, the total number of transferred integers in 17.
-
-### NB3-Q2
-
-Even though we only use a single entry of the matrix in the remote worker, the entire matrix is captured and sent to the worker. Thus, we will transfer 17 integers like in Question 1.
-
-### NB3-Q3
-
-The value of `x` will still be zero since the worker receives a copy of the matrix and it modifies this copy, not the original one.
-
-### NB3-Q4
-
-In this case, the code `a[2]=2` is executed in the main process. Since the matrix is already in the main process, it is not needed to create and send a copy of it. Thus, the code modifies the original matrix and the value of `x` will be 2. 
-
-## Distributed computing with MPI
-
 ### Exercise 1
 
-```julia
-using MPI
-MPI.Init()
-comm = MPI.Comm_dup(MPI.COMM_WORLD)
-rank = MPI.Comm_rank(comm)
-nranks = MPI.Comm_size(comm)
-buffer = Ref(0)
-if rank == 0
-    msg = 2
-    buffer[] = msg
-    println("msg = $(buffer[])")
-    MPI.Send(buffer,comm;dest=rank+1,tag=0)
-    MPI.Recv!(buffer,comm;source=nranks-1,tag=0)
-    println("msg = $(buffer[])")
-else
-    dest = if (rank != nranks-1)
-        rank+1
-    else
-        0
-    end
-    MPI.Recv!(buffer,comm;source=rank-1,tag=0)
-    buffer[] += 1
-    println("msg = $(buffer[])")
-    MPI.Send(buffer,comm;dest,tag=0)
-end
-```
-
-### Exercise 2
-
 ```julia
 f = () -> Channel{Int}(1)
-chnls = [ RemoteChannel(f,w) for w in workers() ]
-@sync for (iw,w) in enumerate(workers())
+worker_ids = workers()
+chnls = [ RemoteChannel(f,w) for w in worker_ids ]
+@sync for (iw,w) in enumerate(worker_ids)
     @spawnat w begin
         chnl_snd = chnls[iw]
-        if w == 2
+        if iw == 1
             chnl_rcv = chnls[end]
             msg = 2
             println("msg = $msg")
@@ -145,21 +71,26 @@ chnls = [ RemoteChannel(f,w) for w in workers() ]
 end
 ```
 
-This is another possible solution.
+This is another possible solution that does not use remote channels.
 
 ```julia
-@everywhere function work(msg)
+@everywhere function work(msg,iw,worker_ids)
     println("msg = $msg")
-    if myid() != nprocs()
-        next = myid() + 1
-        @fetchfrom next work(msg+1)
+    if iw < length(worker_ids)
+        inext = iw+1
+        next = worker_ids[iw+1]
+        @fetchfrom next work(msg+1,inext,worker_ids)
     else
-        @fetchfrom 2 println("msg = $msg")
+        @fetchfrom worker_ids[1] println("msg = $msg")
     end
+    return nothing
 end
 msg = 2
-@fetchfrom 2 work(msg)
+iw = 1
+worker_ids = workers()
+@fetchfrom worker_ids[iw] work(msg,iw,worker_ids)
 ```
+
 ## Matrix-matrix multiplication
 
 ### Exercise 1
@@ -209,62 +140,306 @@ end
 end
 ```
 
+## MPI (Point-to-point)
+
+### Exercise 1
+
+```julia
+function matmul_mpi_3!(C,A,B)
+    comm = MPI.COMM_WORLD
+    rank = MPI.Comm_rank(comm)
+    P = MPI.Comm_size(comm)
+    if  rank == 0
+        N = size(A,1)
+        myB = B
+        for dest in 1:(P-1)
+            MPI.Send(B,comm;dest)
+        end
+    else
+        source = 0
+        status = MPI.Probe(comm,MPI.Status;source)
+        count = MPI.Get_count(status,eltype(B))
+        N = Int(sqrt(count))
+        myB = zeros(N,N)
+        MPI.Recv!(myB,comm;source)
+    end
+    L = div(N,P)
+    myA = zeros(L,N)
+    if  rank == 0
+        lb = L*rank+1
+        ub = L*(rank+1)
+        myA[:,:] = view(A,lb:ub,:)
+        for dest in 1:(P-1)
+            lb = L*dest+1
+            ub = L*(dest+1)
+            MPI.Send(view(A,lb:ub,:),comm;dest)
+        end
+    else
+        source = 0
+        MPI.Recv!(myA,comm;source)
+    end
+    myC = myA*myB
+    if rank == 0
+        lb = L*rank+1
+        ub = L*(rank+1)
+        C[lb:ub,:] = myC
+        for source in 1:(P-1)
+            lb = L*source+1
+            ub = L*(source+1)
+            MPI.Recv!(view(C,lb:ub,:),comm;source)
+        end
+    else
+        dest = 0
+        MPI.Send(myC,comm;dest)
+    end
+    C
+end
+```
+
 ### Exercise 2
 
-At each call to @spawnat we will communicate O(N) and compute O(N) in a worker process just like in algorithm 1. However, we will do this work N^2/P times on average at each worker. Thus, the total communication and computation on a worker will be O(N^3/P) for both communication and computation.  Thus, the communication over computation ratio will still be O(1) and thus the communication will dominate in practice, making the algorithm inefficient.
+```julia
+using MPI
+MPI.Init()
+comm = MPI.COMM_WORLD
+rank = MPI.Comm_rank(comm)
+nranks = MPI.Comm_size(comm)
+buffer = Ref(0)
+if rank == 0
+    msg = 2
+    buffer[] = msg
+    println("msg = $(buffer[])")
+    MPI.Send(buffer,comm;dest=rank+1,tag=0)
+    MPI.Recv!(buffer,comm;source=nranks-1,tag=0)
+    println("msg = $(buffer[])")
+else
+    dest = if (rank != nranks-1)
+        rank+1
+    else
+        0
+    end
+    MPI.Recv!(buffer,comm;source=rank-1,tag=0)
+    buffer[] += 1
+    println("msg = $(buffer[])")
+    MPI.Send(buffer,comm;dest,tag=0)
+end
+```
+## MPI (collectives)
+
+### Exercise 1
+
+```julia
+function matmul_mpi_3!(C,A,B)
+    comm = MPI.COMM_WORLD
+    rank = MPI.Comm_rank(comm)
+    P = MPI.Comm_size(comm)
+    root = 0
+    if  rank == root
+        N = size(A,1)
+        Nref = Ref(N)
+    else
+        Nref = Ref(0)
+    end
+    MPI.Bcast!(Nref,comm;root)
+    N = Nref[]
+    if  rank == root
+        myB = B
+    else
+        myB = zeros(N,N)
+    end
+    MPI.Bcast!(myB,comm;root)
+    L = div(N,P)
+    # Tricky part
+    # Julia works "col major"
+    myAt = zeros(N,L)
+    At = collect(transpose(A))
+    MPI.Scatter!(At,myAt,comm;root)
+    myCt = transpose(myB)*myAt
+    Ct = similar(C)
+    MPI.Gather!(myCt,Ct,comm;root)
+    C .= transpose(Ct)
+    C
+end
+```
+
+This other solution uses a column partition instead of a row partition.
+It is more natural to work with column partitions in Julia if possible since matrices are
+in "col major" format. Note that we do not need all the auxiliary transposes anymore.
+
+```julia
+function matmul_mpi_3!(C,A,B)
+    comm = MPI.COMM_WORLD
+    rank = MPI.Comm_rank(comm)
+    P = MPI.Comm_size(comm)
+    root = 0
+    if  rank == root
+        N = size(A,1)
+        Nref = Ref(N)
+    else
+        Nref = Ref(0)
+    end
+    MPI.Bcast!(Nref,comm;root)
+    N = Nref[]
+    if  rank == root
+        myA = A
+    else
+        myA = zeros(N,N)
+    end
+    MPI.Bcast!(myA,comm;root)
+    L = div(N,P)
+    myB = zeros(N,L)
+    MPI.Scatter!(B,myB,comm;root)
+    myC = myA*myB
+    MPI.Gather!(myC,C,comm;root)
+    C
+end
+```
 
 ## Jacobi method
 
 ### Exercise 1
 
 ```julia
-@everywhere workers() begin
-    using MPI
-    comm = MPI.Comm_dup(MPI.COMM_WORLD)
-    function jacobi_mpi(n,niters)
-        nranks = MPI.Comm_size(comm)
-        rank = MPI.Comm_rank(comm)
-        if mod(n,nranks) != 0
-            println("n must be a multiple of nranks")
-            MPI.Abort(comm,1)
+function jacobi_mpi(n,niters)
+    u, u_new = init(n,comm)
+    load = length(u)-2
+    rank = MPI.Comm_rank(comm)
+    nranks = MPI.Comm_size(comm)
+    nreqs = 2*((rank != 0) + (rank != (nranks-1)))
+    reqs = MPI.MultiRequest(nreqs)
+    for t in 1:niters
+        ireq = 0
+        if rank != 0
+            neig_rank = rank-1
+            u_snd = view(u,2:2)
+            u_rcv = view(u,1:1)
+            dest = neig_rank
+            source = neig_rank
+            ireq += 1
+            MPI.Isend(u_snd,comm,reqs[ireq];dest)
+            ireq += 1
+            MPI.Irecv!(u_rcv,comm,reqs[ireq];source)
         end
-        n_own = div(n,nranks)
-        u = zeros(n_own+2)
-        u[1] = -1
-        u[end] = 1
-        u_new = copy(u)
-        for t in 1:niters
-            reqs = MPI.Request[]
-            if rank != 0
-                neig_rank = rank-1
-                req = MPI.Isend(view(u,2:2),comm,dest=neig_rank,tag=0)
-                push!(reqs,req)
-                req = MPI.Irecv!(view(u,1:1),comm,source=neig_rank,tag=0)
-                push!(reqs,req)
-            end
-            if rank != (nranks-1)
-                neig_rank = rank+1
-                s = n_own+1
-                r = n_own+2
-                req = MPI.Isend(view(u,s:s),comm,dest=neig_rank,tag=0)
-                push!(reqs,req)
-                req = MPI.Irecv!(view(u,r:r),comm,source=neig_rank,tag=0)
-                push!(reqs,req)
-            end
-            for i in 3:n_own
-                u_new[i] = 0.5*(u[i-1]+u[i+1])
-            end
-            MPI.Waitall(reqs)
-            for i in (2,n_own+1)
-                u_new[i] = 0.5*(u[i-1]+u[i+1])
-            end
-            u, u_new = u_new, u
+        if rank != (nranks-1)
+            neig_rank = rank+1
+            u_snd = view(u,(load+1):(load+1))
+            u_rcv = view(u,(load+2):(load+2))
+            dest = neig_rank
+            source = neig_rank
+            ireq += 1
+            MPI.Isend(u_snd,comm,reqs[ireq];dest)
+            ireq += 1
+            MPI.Irecv!(u_rcv,comm,reqs[ireq];source)
         end
-        return u
+        # Upload interior cells
+        for i in 3:load
+            u_new[i] = 0.5*(u[i-1]+u[i+1])
+        end
+        # Wait for the communications to finish
+        MPI.Waitall(reqs)
+        # Update boundaries
+        for i in (2,load+1)
+            u_new[i] = 0.5*(u[i-1]+u[i+1])
+        end
+        u, u_new = u_new, u
     end
+    return u
+end
+```
+
+### Exercise 2
+
+```julia
+function jacobi_mpi(n,niters,tol,comm) # new tol arg
+    u, u_new = init(n,comm)
+    load = length(u)-2
+    rank = MPI.Comm_rank(comm)
+    nranks = MPI.Comm_size(comm)
+    nreqs = 2*((rank != 0) + (rank != (nranks-1)))
+    reqs = MPI.MultiRequest(nreqs)
+    for t in 1:niters
+        ireq = 0
+        if rank != 0
+            neig_rank = rank-1
+            u_snd = view(u,2:2)
+            u_rcv = view(u,1:1)
+            dest = neig_rank
+            source = neig_rank
+            ireq += 1
+            MPI.Isend(u_snd,comm,reqs[ireq];dest)
+            ireq += 1
+            MPI.Irecv!(u_rcv,comm,reqs[ireq];source)
+        end
+        if rank != (nranks-1)
+            neig_rank = rank+1
+            u_snd = view(u,(load+1):(load+1))
+            u_rcv = view(u,(load+2):(load+2))
+            dest = neig_rank
+            source = neig_rank
+            ireq += 1
+            MPI.Isend(u_snd,comm,reqs[ireq];dest)
+            ireq += 1
+            MPI.Irecv!(u_rcv,comm,reqs[ireq];source)
+        end
+        MPI.Waitall(reqs)
+        # Compute the max diff in the current
+        # rank while doing the local update
+        mydiff = 0.0
+        for i in 2:load+1
+            u_new[i] = 0.5*(u[i-1]+u[i+1])
+            diff_i = abs(u_new[i] - u[i])
+            mydiff = max(mydiff,diff_i)
+        end
+        # Now we need to find the global diff
+        diff_ref = Ref(mydiff)
+        MPI.Allreduce!(diff_ref,max,comm)
+        diff = diff_ref[]
+        # If global diff below tol, stop!
+        if diff < tol
+            return u_new
+        end
+        u, u_new = u_new, u
+    end
+    return u
+end
+```
+
+## All pairs of shortest paths
+
+### Exercise 1
+
+```julia
+function floyd_iterations!(myC,comm)
+    L = size(myC,1)
+    N = size(myC,2)
+    rank = MPI.Comm_rank(comm)
+    P = MPI.Comm_size(comm)
+    lb = L*rank+1
+    ub = L*(rank+1)
+    C_k = similar(myC,N)
+    for k in 1:N
+        if (lb<=k) && (k<=ub)
+            # If I have the row, fill in the buffer
+            myk = (k-lb)+1
+            C_k[:] = view(myC,myk,:)
+        end
+        # We need to find out the owner of row k.
+        # Easy since N is a multiple of P
+        root = div(k-1,L)
+        MPI.Bcast!(C_k,comm;root)
+        # Now, we have the data dependencies and
+        # we can do the updates locally
+        for j in 1:N
+            for i in 1:L
+                myC[i,j] = min(myC[i,j],myC[i,k]+C_k[j])
+            end
+        end
+    end
+    myC
 end
 ```
 
 
 
 
+

notebooks/LEQ.ipynb

@@ -22,7 +22,7 @@
     "In this notebook, we will learn\n",
     "\n",
     "- How to parallelize Gaussian elimination\n",
-    "- How to fix static load imbalance"
+    "- What is load imbalance, and how to fix it in this algorithm"
    ]
   },
   {
@@ -45,24 +45,69 @@
     "using Printf\n",
     "function answer_checker(answer,solution)\n",
     "    if answer == solution\n",
-    "        \"🥳 Well done! \"\n",
+    "        \"🥳 Well done!\"\n",
     "    else\n",
     "        \"It's not correct. Keep trying! 💪\"\n",
     "    end |> println\n",
     "end\n",
+    "function ge_par_why()\n",
+    "    msg = \"The outer loop of the algorithm is not parallelizable, since the iterations depend on the results of the previous iterations. However, we can extract parallelism from the inner loops.\"\n",
+    "    println(msg)\n",
+    "end\n",
     "ge_par_check(answer) = answer_checker(answer, \"a\")\n",
-    "ge_dep_check(answer) = answer_checker(answer, \"b\")"
+    "ge_dep_check(answer) = answer_checker(answer, \"b\")\n",
+    "function ge_lb_answer()\n",
+    "    msg = \"It is a form of static load balancing. We know in advance the load distribution and the partition strategy does not depend on the actual values of the input matrix\"\n",
+    "    println(msg)\n",
+    "end\n",
+    "println(\"🥳 Well done!\")"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "8dcee319",
+   "id": "8ad95aa5",
    "metadata": {},
    "source": [
     "## Gaussian elimination\n",
     "\n",
     "\n",
-    "[Gaussian elimination](https://en.wikipedia.org/wiki/Gaussian_elimination) is a method to solve systems of linear equations, e.g.\n",
+    "[Gaussian elimination](https://en.wikipedia.org/wiki/Gaussian_elimination) is a method to solve systems of linear equations, like this one:\n",
+    "\n",
+    "$$\n",
+    "\\left[\n",
+    "\\begin{matrix}\n",
+    "1 & 3 & 1 \\\\\n",
+    "1 & 2 & -1 \\\\\n",
+    "3 & 11 & 5 \\\\\n",
+    "\\end{matrix}\n",
+    "\\right]\n",
+    "\\left[\n",
+    "\\begin{matrix}\n",
+    "x \\\\\n",
+    "y \\\\\n",
+    "z  \\\\\n",
+    "\\end{matrix}\n",
+    "\\right]\n",
+    "=\n",
+    "\\left[\n",
+    "\\begin{matrix}\n",
+    "9 \\\\\n",
+    "1 \\\\\n",
+    "35  \\\\\n",
+    "\\end{matrix}\n",
+    "\\right].\n",
+    "$$\n",
+    "\n",
+    "This is just a small example with three unknowns, but practical applications need to solve linear equations with large number of unknowns. Parallel processing is needed in these cases.\n",
+    "\n",
+    "### Problem statement\n",
+    "\n",
+    "Let us consider a system of linear equations written in matrix form $Ax=b$, where $A$ is a nonsingular square matrix, and $x$ and $b$ are vectors. $A$ and $b$ are given, and $x$ is unknown. The goal of Gaussian elimination is to transform the system $Ax=b$, into a new system $Ux=c$ such that\n",
+    "- both systems have the same solution vector $x$,\n",
+    "- the matrix $U$ of the new system is *upper triangular* with unit diagonal, namely $U_{ii} = 1$ and $U_{ij} = 0$ for $i>j$.\n",
+    "\n",
+    "\n",
+    "For the particular system shown above, the transformed one is the following:\n",
     "\n",
     "$$\n",
     "\\left[\n",
@@ -87,51 +132,97 @@
     "35  \\\\\n",
     "\\end{matrix}\n",
     "\\right]\n",
+    "\\longrightarrow\n",
+    "\\left[\n",
+    "\\begin{matrix}\n",
+    "1 & 3 & 1 \\\\\n",
+    "0 & 1 & 2 \\\\\n",
+    "0 & 0 & 1 \\\\\n",
+    "\\end{matrix}\n",
+    "\\right]\n",
+    "\\left[\n",
+    "\\begin{matrix}\n",
+    "x \\\\\n",
+    "y \\\\\n",
+    "z  \\\\\n",
+    "\\end{matrix}\n",
+    "\\right]\n",
+    "=\n",
+    "\\left[\n",
+    "\\begin{matrix}\n",
+    "9 \\\\\n",
+    "8 \\\\\n",
+    "4  \\\\\n",
+    "\\end{matrix}\n",
+    "\\right]\n",
     "$$\n",
     "\n",
-    "The steps of the Gaussian elimination will transform the system into an upper triangular matrix. The system of linear equations can now easily be solved by backward substitution. \n",
+    "The most challenging part of solving a system of linear equations is to transform it to upper triangular form. Afterwards, the solution vector can be obtained easily with a backward substitution. For this reason, we will study here the triangulation step only."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "980cb36f",
+   "metadata": {},
+   "source": [
     "\n",
     "\n",
+    "### Augmented system matrix\n",
+    "\n",
+    "In practice, vector $b$ is added as an additional column to A forming the so-called *augmented* matrix $A^* = [A  b]$. The augmented matrix in the example above is\n",
+    "\n",
     "$$\n",
     "\\left[\n",
     "\\begin{matrix}\n",
-    "1 & 3 & 1 & 9 \\\\\n",
+    "1 & 3 & 1 \\\\\n",
+    "1 & 2 & -1 \\\\\n",
+    "3 & 11 & 5 \\\\\n",
+    "\\end{matrix}\n",
+    "\\right]\n",
+    "\\left[\n",
+    "\\begin{matrix}\n",
+    "x \\\\\n",
+    "y \\\\\n",
+    "z  \\\\\n",
+    "\\end{matrix}\n",
+    "\\right]\n",
+    "=\n",
+    "\\left[\n",
+    "\\begin{matrix}\n",
+    "9 \\\\\n",
+    "1 \\\\\n",
+    "35  \\\\\n",
+    "\\end{matrix}\n",
+    "\\right]\\longrightarrow\n",
+    "A^*=\n",
+    "\\left[\n",
+    "\\begin{matrix}\n",
+    "1 & 3 & 1 & 9  \\\\\n",
     "1 & 2 & -1 & 1 \\\\\n",
-    "3 & 11 & 5 & 35 \\\\\n",
+    "3 & 11 & 5 & 35\\\\\n",
     "\\end{matrix}\n",
-    "\\right]\n",
-    "\\rightarrow\n",
+    "\\right].\n",
+    "$$\n",
+    "\n",
+    "With this new notation, the goal of Gaussian elimination is to find the augmented matrix containing $U$ and $c$, namely $U^*= [U  c]$, given the augmented matrix $A^* = [A  b]$. These are $A^*$ and $U^*$ in our example:\n",
+    "\n",
+    "$$\n",
+    "A^*=\n",
+    "\\left[\n",
+    "\\begin{matrix}\n",
+    "1 & 3 & 1 & 9  \\\\\n",
+    "1 & 2 & -1 & 1 \\\\\n",
+    "3 & 11 & 5 & 35\\\\\n",
+    "\\end{matrix}\n",
+    "\\right]\\longrightarrow\n",
+    "U^*=\n",
     "\\left[\n",
     "\\begin{matrix}\n",
     "1 & 3 & 1 & 9 \\\\\n",
-    "0 & -1 & -2 & -8 \\\\\n",
-    "0 & 2 & 2 & 8 \\\\\n",
+    "0 & 1 & 2 & 8\\\\\n",
+    "0 & 0 & 1 & 4\\\\\n",
     "\\end{matrix}\n",
-    "\\right]\n",
-    "\\rightarrow\n",
-    "\\left[\n",
-    "\\begin{matrix}\n",
-    "1 & 3 & 1 & 9 \\\\\n",
-    "0 & 1 & 2 & 8 \\\\\n",
-    "0 & 2 & 2 & 8 \\\\\n",
-    "\\end{matrix}\n",
-    "\\right]\n",
-    "\\rightarrow\n",
-    "\\left[\n",
-    "\\begin{matrix}\n",
-    "1 & 3 & 1 & 9 \\\\\n",
-    "0 & 1 & 2 & 8 \\\\\n",
-    "0 & 0 & -2 & -8 \\\\\n",
-    "\\end{matrix}\n",
-    "\\right]\n",
-    "\\rightarrow\n",
-    "\\left[\n",
-    "\\begin{matrix}\n",
-    "1 & 3 & 1 & 9 \\\\\n",
-    "0 & 1 & 2 & 8 \\\\\n",
-    "0 & 0 & 1 & 4 \\\\\n",
-    "\\end{matrix}\n",
-    "\\right]\n",
+    "\\right].\n",
     "$$\n",
     "\n"
    ]
@@ -142,9 +233,13 @@
    "metadata": {},
    "source": [
     "### Serial implementation\n",
-    "The following algorithm computes the Gaussian elimination on a matrix which represents a system of linear equations.\n",
-    "- The first inner loop in line 4 divides the current row by the value of the diagonal entry, thus transforming the diagonal to contain only ones. \n",
-    "- The second inner loop beginning in line 8 substracts the rows from one another such that all entries below the diagonal become zero. "
+    "\n",
+    "\n",
+    "The following algorithm computes the Gaussian elimination on a given augmented matrix `B`, representing a system of linear equations. The result is given by overwriting `B`, avoiding the allocation of an additional matrix.\n",
+    "\n",
+    "- The outer loop is a loop over rows.\n",
+    "- The first inner loop in line 4 divides the current row by the value of the diagonal entry, thus transforming the diagonal to contain only ones. Note that we skip the first entries in the row, as we know that these values are zero at this point. The cells updated in this loop at iteration $k$ are depicted in red in the figure below.\n",
+    "- The second inner loop beginning in line 8 subtracts the rows from one another such that all entries below the diagonal become zero. The entries updated in this loop are depicted in blue in the figure below."
    ]
   },
   {
@@ -172,13 +267,28 @@
     "end"
    ]
   },
+  {
+   "attachments": {
+    "g27851.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAzAAAAJsCAYAAAAiM0YjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15nBx1nf/xV/X0zOS+LxIIwYRDDkFuFBFUUFbE+wQPFBRXXQVcL1xss3ggrgfeeKKuq7grIh4/BeUSOQxBbgEFQgIkAZJJyDVX1++PmoGeTnfPVKe7q2rm9Xw8+pGu6u936t09k5n+9Lfq+w2QJI12eeCsku2/A5cO3O8EXgG8GdgD2BkoAvcDvwMuAB6u8nWfBZwKPB94xsDXWgvcBPwQ+GUjn4QkSZKksWE8EJbcfjqwf1/g7rLHym8bgdeUfb1xwNeB/mH6/hqY0KTnJEmSJGmUqlTAHAxspnYBMnjrA44b+FrjgD+OsF8I/BYImvz8JEmSJI0i5QXMH4AHBu5vA75NNMpyBHAM8FFgdVmfFQNf57sl+24hOjXtRcBhwOuITk0rL2JObv5TlCRJkjRalBcwg7d7gd2r9Jk78Hhp+x8P/FsE/h3IVen74bJ+tzTiSUiSJEkaGyoVMOuAXYbpd3yFfiGwdJh+AXBzWZ8968wuSZIkaYypVMB8YAT9cmx/KtlKoH0EfT9Q1u9NsVNLklRBteF/SdLo1Q/8YATtisDysn0/AnpH0PevZduOwEiSGsICRpLGnjuArhG2XVW2fV2d/aaNsJ8kSTVZwEjS2FNtYcpKNpVtPzLCfpvLtifHOKYkSVVZwEjS2FNelNRSrLNveT/XgpEkNYQFjCSNPWHSASRJqpcFjCRJkqTMsICRJEmSlBkWMJIkSZIywwJGkiRJUmZYwEiSJEnKDAsYSZIkSZlhASNJkiQpMyxgJEmSJGWGBYwkSZKkzLCAkSRJkpQZ+aQDSJKaLgTWl2xvjtF3a1nf/jqPuSXGMSVJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkqTWGA+cDFwGtCecRZIkxTcTuBJ4M9HfdUkadQLgKOASYDWwDfjfRBNJkqQdcS3R3/M1wKXA84FcookkqQGWAF8AVgKPAeHAbQ2wc4K5JEnSjjmQoX/b1wGPAl8B9kgwlyTFNh14L3An0WhLH0//chu83ZFYOkmS1CgPsf3f+CLwBHAv8AFgRmLpNOoFSQdQprUDxwPnEH3qMo7a17f8GfhRC3JJkqTm+QDwzBqP9wFbgduBzwG/BXpbkEtjhAWM6nU48AeigmVcwlkkSVI69RBdM3MocE/CWTRKWMCoXjngGOBM4LnAJKBtmD73E30aI0mSsuswYN4wbXqB9cBvgAuAvzU7lCTFMQ54PfAXYC3RL63yc2ND4LqkAkqSpIa5j8p/57cSXdB/PfA6XDZBUkbsDCwFHmToLCUh0cX9ExNLJkmSdtR8or/ng3/b+4lmGb0DeDcwNblokrTjDia6aH81sIHoHNgPJ5pIkiTtiG8SFS2PEy2XcD7wjEQTSVITtAMvB64mmmLRIWVJkrJnJtF1LT8nuv7Va6kljQmdSQeQJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSpOFMB9aV3L6WbBxJktIhn3QASVJFAVERM2hCUkEkSUqTXNIBJEmSJGmkLGAkSZIkZYYFjCRJkqTMsICRJEmSlBkWMJIkSZIywwJGkiRJUmY4jbIkjT47A3PL9t0BdCeQRZIkSdIYMAMIS27fH2G/Y4ENJf2KQKEJ+SRJkiTpKfUUMG8Dekr6dANvalI+SZIkSXpKnAImIBplKW2/Hji6qQklSZIkacBIC5g8cGFZ24eB/VuQUZIkSZKAkRUwk4DflrW7HdilRRklSZIkCRi+gNkJWF7W5gpgagszSpIkSRJQu4DZB1hR9vhFQHuLM0qSJEkSUL2AOYboAv3Sx75MdCG/JEmSJCWiUgHzWmBryb4+4N1JBZQkSZKkQeUFzKNEi1IObm8CXppYOkmSJEkqUV7AlN8uBzoSSydJkiRJJcoLmOuIRmFK9/0GGJdUQEmSJEkaVOkamN2A+8v2XwVMTiaiJEmSJEWqzUK2ELiv7LFrgSkJZJQkSZIkoPY6MPOA28seXwbMbHFGSZIkSQJqFzAAc4C/lbW5E9iphRklSZIkCRi+gAGYDtxQ1u5uYEGLMkqSJEkSMLICBmAq8OeytvcTXfAvSZIkSS0x0gIGYALRujCl7VcAS5qcUZIkSZKAeAUMQCfwy7I+jwL7NjGjJEmSJAHxCxiADuB/y/qtAfZvUkZJkiRJAuorYADagB+U9V0HHNb4iJIktV6QdABJUkU5YFHJ9pPAYyPs2wbsWrZvC7B6x2NJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJktKrHZiRdAhJkiRJ6ZRPOkCZdwGvAF6UdBCNVLgbsEfSKSRJUqr9GYLNSYfQ6BAkHaDMXcBcYAmwPuEsGpHww8Bnk04hSZJSbR8I7ko6hEaHXNIBSiwBZgETgZMTzqJhheMhnAK/PCbpJJIkKe0+enz0viFsTzqJsi9NIzBfAM4YuP8A8IwEs2hY4aXAiUmnkCRJmXIaBN9JOoSyLS0jMAHw2pLtqXhdhSRJkqQyabmI/3nA+JLt6UQX9J+VTBzVYTnwlqRDbO/A38DCi+GXFyWdpLIXfRQ27gs3nZR0ksq+sRC+/gu4/eCkk1T3rBvh9W+Fs/+edJLKDvsBTHgQriwkHKSK170O7nsH3PLipJNUt9+tcPv+Saeo7oAr4Jlfg/+5JOkklT3/09A9G244LekklZ29P1z2Lbjt8KSTVLffzfAfx8Pr1iadpLJDfgrTl8EfPp90kjKTgeuTDiE1yyVAWHZbRbpOcdMQ4aUQhiW3a5NOVMW9wJlJh6jhAtL9y30JkPZZYzYCByQdooargAuTDlHD6USn7aZZb9IBhvEQcErSIWq4CLg86RA1HA50JR1iGFuBBUmHqOEm4PykQ2wvnFb2XiGE8NSkUyn70nAK2TjgiAr7JwBHtjiLJEmSpBRLQwHzGqKZx8pNBz7Q4iySJEmSUiwNBcz7gUlVHnsu0UiMJEmSJCVewMwFdqnx+CTglS3KIkmSJCnlkp6F7FSiU8WqmUg0QvPfrYmjhM0F9gbaBrbvAVYmF0eSJElpk3QBcxrRKNAaopmOdgY6gPuJipdJwO7APGB1QhnVPM8FjgEOBg4i+v6Xeh/w1VaHkiRJUnoleQrZvkSFyReB/YHFwGMDj70W2BV4L9APpHSNDO2gTwD/Cbyc7YsXSZIkaTtJjsDcD+wGPFrl8W7gB8DFwNQWZVIyeoDbgJuJCteXJBtHkiRJaZXkCMwWqhcv9bRT9nye6NSxScAhRAvq3ZBoIkmSJKVa0tfAaGz7Q9IBJEmSlC1JT6MsSZIkSSNmASNJkiQpMyxgJEmSJGWGBYwkSZKkzLCAkSRJkpQZFjCSJEmSMsMCRpIkSVJmWMBIkiRJygwLGEmSJEmZYQEjSZIkKTMsYCRJkiRlhgWMJEmSpMywgJEkSZKUGRYwkiRJkjLDAkZJejkQlt0KJY9/pcLjR7Y2oiRJktLEAkaSJElSZuSTDqAx7S7gIzH7PNiEHJIkScoICxgl6T7gvKRDSJIkKTs8hUySJElSZljASJIkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFjCRJkqTMsICRJEmSlBmuA6MGWTUPOD3pFBXMAJ4PbEk6SBXPBhaQztcOYC7QTnrzAXQCrwcOTzpIFQuBcaT3NXwBMJX05oPow7Y055sCHEv0s5hG+5Du7/ESotcurfkger/0VmBd0kGq2Ak4hNS9hruPj5Z8kxorSDpAmVVEb+YOApYnnEU1hZcCJz69fdM2OGx1YnGq2wXYBKxPOkgVs4j+cD+cdJAq2oH5wIqkg9SwK7Aa6E46SBXzgT5gbdJBqpg6cHso6SA1LAIeTDhDLQuBLmBj0kGqmAu0AY8kHaSKcUQZ0/x7ZhHRe5S+hHNUs4Dod+DjSQcZakYOnlhYtvM0CL6TSBypSVYBIXBg0kE0nPBSCMOS27VJJ6riXuDMpEPUcAFwfdIhalgCbE46xDA2AgckHaKGq4ALkw5Rw+nAA0mHGEZv0gGG8RBwStIhargIuDzpEDUcTlQAptlWoiIhrW4Czk86xPbCaWXvFUIIT006lbLPa2AkSZIkZYYFjCRJkqTMsICRJEmSlBkWMJIkSZIyw2mUpeHtDOzB0wX/DUQzm0mSJKnFLGCk7b2UaD79g4mm9J5X9vgBwK2tDiVJkiQLGKmS/yVal0CSJEkp4zUwUnVbiU4X+xouJSxJkpQKjsBI23sX8DfgLp5edflyYPfEEkmSJAmwgJEq+WHSASRJklSZp5BJkiRJygwLGEmSJEmZYQEjSZIkKTMsYCRJkiRlhgWMJEmSpMywgJEkSZKUGRYwkiRJkjLDAkaSJElSZljASJIkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFjCRJkqTMsICRJEmSlBn5pANIKbQUeG/Zvkkl968B+ku2Q2Bms0NJkiTJAkaqZDwwvcbjU8q2wyZmkSRJUgkLGGl7VzJ0hGU4FjCSJEktYgEjbe+3AzdJkiSljBfxS5IkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFLine truncated
+    }
+   },
+   "cell_type": "markdown",
+   "id": "cc6ce65e",
+   "metadata": {},
+   "source": [
+    "<div>\n",
+    "<img src=\"attachment:g27851.png\"  align=\"left\" width=\"300\"/>\n",
+    "</div>"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "3763b000",
    "metadata": {},
    "source": [
     "<div class=\"alert alert-block alert-info\">\n",
-    "<b>Note:</b> This algorithm is not correct for all matrices: if any diagonal element <code>B[k,k]</code> is zero, the computation in the first inner loop fails. To get around this problem, another step can be added to the algorithm that swaps the rows until the diagonal entry of the current row is not zero. This process of finding a nonzero value is called <b>pivoting</b>.   \n",
+    "<b>Note:</b> This algorithm is not correct for all nonsingular matrices: if any diagonal element <code>B[k,k]</code> is zero, the computation in the first inner loop fails. To get around this problem, another step can be added to the algorithm that swaps the rows until the diagonal entry of the current row is not zero. This process of finding a nonzero value is called <b>pivoting</b>. We are not going to consider pivoting here for simplicity.  \n",
     "</div>"
    ]
   },
@@ -203,6 +313,16 @@
     "gaussian_elimination!(B)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "a7d6f90a",
+   "metadata": {},
+   "source": [
+    "### Complexity of the algorithm\n",
+    "\n",
+    "The number of operations of the algorithm is $O(N^3)$, where $N$ is the number of unknowns in the system. Intuitively, this makes sense as there are three nested loops. However, the length of the loops is not equal to $N$. In any case, it can be proven that the total number of operations is still $O(N^3)$. The actual proof is a bit challenging and it is not discussed here."
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "39f2e8ef",
@@ -262,72 +382,50 @@
    ]
   },
   {
-   "cell_type": "markdown",
-   "id": "14d57c52",
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1169c86e",
    "metadata": {},
+   "outputs": [],
    "source": [
-    "### Two possible data partitions"
+    "ge_par_why()"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6b17aee4",
+   "id": "74491f64",
    "metadata": {},
    "source": [
-    "The outer loop of the algorithm is not parallelizable, since the iterations depend on the results of the previous iterations. However, we can extract parallelism from the inner loops. Let's have a look at two different parallelization schemes. \n",
+    "### Data partition\n",
     "\n",
-    "1. **Block-wise partitioning**: Each processor gets a block of subsequent rows. \n",
-    "2. **Cyclic partitioning**: The rows are alternately assigned to different processors. "
+    "Let's start considering a row-wise block partition, as we did in previous algorithms.\n",
+    "\n",
+    "In the figure below, we use different colors to illustrate which entries are assigned to a CPU. All entries with the same color are assigned to the same CPU."
    ]
   },
   {
    "attachments": {
-    "fig-asp-1d-partition.png": {
-     "image/png": "iVBORw0KGgoAAAANSUhEUgAABL4AAAI0CAYAAAATGeCIAAAACXBIWXMAAB7CAAAewgFu0HU+AAAAGXRFWHRTb2Z0d2FyZQB3d3cuaW5rc2NhcGUub3Jnm+48GgAAIABJREFUeJzs3XmcXFWZ//HvU1XdnU5nIyEhCYRVGBYBWRwQERFk2JRVUBxEXAYXXEcUB3QE11FREcF12BQFQWRRGJBNVFBQdhAwkIXsezq9d1Xd5/fHrfzsVN1Oeq17cvN5v155Kae2p+69ffv09557jrm7AGy+zOxsSVdXNX/N3S+oYw3Nkjqrmue4+y71qmFzYmbtklr6NLW5+4S06gEAAEA2mdmnJX2jqvk8d/9WGvUAaSikXQC2XGZ2uqRJfZpud/elI/Teh2tgx3e7pF5JayV1S+p097UjUQMAAACyzcxOkbR1n6Y73X3hCL33oZLGDOCpHZJ6FPdnexT3Z9eMRA3YcpnZREn/lfDQbe7+53rXAwwHwRdSYWY5SVdpw1Evz0kakeBL0i3aMFQbMDNbJ+kFSX+X9Lyk+yQ97gyPBAAAwIZ+KGlqn/8+RtKIBF+SfiFp1lBeaGZtkl7UP/uz90v6K/1ZDMJ4SecntC+URPCFzQrBF9JyqDYMvUIyQdK/Vv6tt8DMbpf0fXf/ezplAQAAIBRmtr82DL1CMl7SgZV/6y02s99I+oG7P5VOWQBQf7m0C8CWx8yaJH0t7ToGaZakcyU9Y2a/NLM90y4IAAAA6TCzBtXOmxS6mZI+IOkJM7vFzPZNuyAAqAeCL9SVmc2U9FtJh6RdyxDlJJ0u6XEz+5SZWdoFAQAAoH7MbKqkX0s6Mu1ahsgknSTpb2Z2oZnl0y4IAEYTtzpi1JjZBMWTfU6TtI+kNyn+JTuQSTpHw/skJQ3rbpHUKGmypB0k7SLp1ZIOUv8/I02SLpH0ejM7w917Rr5cAAAApMnMxkuaImkbxf3DwyWdrPSm7PiokudXGqu4f7qV/tmf3UvSwZIa+nmvgqQvSzrUzE519+oVurFl65D044T25+pdCDBcBF8YFWY2W9Kr0q6jyovu/thAn1xZyeTNkj6k/q/onSzpDjN7q7t3jUCNAAAACICZPSHpNWnXUWX2IPuz4xX3Y89RPPF+0t0Kx0j6nZkd6+5tI1MmNneVlUE/kHYdwEjgVkeMls3+2HL3Vne/2d3fLOm1ild3THKkpCu57REAACBTstCfbXP3W939OEn7Sbqzn6e+XtJ1lZXXASBTOLEBA+Duf5N0lKSPSColPOUMSZ+oa1EAAADAALn7U+5+vOLpP5Km6ThB0oX1rQoARh+3OmK03KB4fq/+nKR47q/Nhru7pCvMbK7iCU2bqp7yZTO71d3n1r+6gTOzaZK2U7yyT0HSMknz3H1JqoX1ozLh6s6Kj5epklzSCklLFdcd1bGOrSv/xkhaK2k5twQAAJBZN0v6y0YeP17StnWqZcS4+1WV/uxvFc8N1tfnzOxX7v58CqVtkpmNUdwXmyqpLGm1pBVZnG/XzJr1z75nJr+rmTXqn98xr39+R+abw4gi+MKocPeNXi0ys320mQVf67n7nWb2QUlXVz00VtK3JJ1S/6o2rhJ2fVjxyLTd+nnO3yT9QtIP3L27juUl1VJQHI6erHjeicn9PHWZmd2hOIi8sxJOjmQdhyju1B6t+PaAmlGyZjZPcaf4Pkm3uPuqkaxhE/Vto3ikYfVttt2SvkGnAQCAoXP3L27scTO7R5th8CVJ7v6AmZ0t6ZfasB/RKOm7kv4tjbqqVYKuExTXc7Tii7fVSmb2tOJJ/++QdK+7FxPe61TF05f09aK7V/fph1Ln8ZLeUNX8F3e/dRDv0awNv2vSsVXs811/K+n+pO86UszsI6rd5l9z99Yhvl9e8Xc7VvH3TPq7JDKz5yU9Iun/FPfx6dNiWAi+gCFw92vM7C2STq166EQz+xd3fzGNuqqZ2ThJn5J0nqRxm3j6gZV/55jZe909acWgUWdmR0u6XANbHGEbSe+t/HvSzD7g7o+OQA2vl/QVSW8cwNN3rPx7h6QfmNndkr7r7vcMt46NMbOtJd2reIWpvroknUQHAQAAbIy731QJbN5d9dBRZra/uz+eRl2SZGYNivt3n1Ny2NVXQdL+lX/nSlptZtcp7o/N6fO8LknnV722s3LHxpph1JqTdJniOxT66m9xrOrXNyq+/fRCbTpIbZB0QOXfRyStMrOfSbpslO46ebfivw/6ulzSoIKvylzIp0r6oqQ9NvH0nOIVSfdSfAx0mNmvJF3q7k8O5nOB9ZjjCxi6/5RUfYUlJ+ljKdRSw8wOlvS0pIu06dCrr90l3W9mR4xGXRtjZp9RfGVnKCuCvkbSn8zsXcP4/LyZXSrpTxpY6FWtoHiE2O/M7KKh1rEpZraVpHuUHHqd6O6/G63PBgAAmXK+4v5DtY/Xu5D1zGxbxaPpf6hNh15JJivuj//DzPbt036XpJernjtW0tlD+Iy+jlFt6PWcpAc29UIzm6V4ZNP3NbTRg1MUj/6fbWbV/cIgmNkExaPTbtKmQ68kLYoDuCfM7OSRrA1bDoIvYIjc/RXFt9hVOymAFR5PlvRHSTsN8fVjJN1mZnVbwtvMviHp60peZluK5zZYqnhOsv5uaWyQdK2ZDXqhgcpy37dr5Dp6LSP0Phsws4mS7lbt8uqdkk4Y7ZFmAAAgO9x9meKpLqq9tTL1RF2Z2f6SHlU8emu48pKa1/9HZV7YHyY874PD7Lufm9B2+aam4DCzAxWHXiPR384r7r8Hxcx2kPSQpONG6C0HczEf+P8IvoDh+XlC20yNzC/r4dhdtbcyr5V0naTPSnqXpLMq//9+xaFStXGSflKPZa3N7CxJn054qFvSpZJeJ6nR3We4+3TFAdfhkn6i2lU2TdK3zOyoQXx+QdKvlPxLuaz4CtW/S9pT8VXEqZL2lnS64rnelg/0s4ajcuvqnaqdn2J96HVvPeoAAACZktSf3UrS6+tZhJntJul3ivvS1ZZL+rbieaF2kjRB8WiwgxQHT3creaXKalepdoTbbpLePMSad1Y84quvtZJ+tonX7a74u85IeHiZ4nmDj1I8ncb673qwpI9WXhf8BPdmNkXJdyhIUrukHyue02w3xd9xhuJbON+jeGGJ9vpUii0Bc3wBw/N7xbc7NlS1HyzpsbpXk+yvkr6qeGLI3oTHv14ZBn6dan8xHSjp/Yp/MY0KM9tJ0vcSHnpa0inuXj0kXe5elvSgpAfN7PuSbpW0Q5+n5CRdY2b7DHCy+c8peRLXRyS9392fTXhspaRnJd1UmYfi3xUHif8ygM8bNDMbq3iY+CFVD3VKequ73z8anwsAADLvIUkdqh2tfrDi/taoq8xz9UvFt+5Vu0zSfydMqN4maZHiEWLfryz685+SPiRpfNLnuPtqM7te8dxRfX1YcUgzWB9S7WCSq9y9o78XmFmTpBsVh4vVLpX0BXdfV9W+/rs+IulyM5uueB7fDyrcUVBXSto1of1Xkj7u7our2tsU393xuOJ+/HjF++U/tZkuioZwMOILGAZ3b5OUNJH9nvWuJcFixZOc/6u739pP6CVJcvenJB0hKWlSzNGe4+Griq/y9PWCpCOTQq9qlUkuD5e0pOqhmYonCd0oM3uVpAsSHvqdpCP6Cb2qayi6+zWK9/snlTxXxpBVVjS6TbXzjnVIeguhFwAAGKpKH/GZhIfq2Z/9mJJv+fuku398IKsIuvsydz9f8XxbN27kqVcktL2lMt/WgFVWYXxPVXOkeL6ujfmk4jsH+nJJH3P3TyaEXjXcfam7f1rSLopHRwXFzE6UdGLCQ5dLOj0h9Krh7m3u/nXFI/y+q3jbAkNC8AUM3+yEtqSleevpAUmvdvfbBvoCd1+h+KpVtT0r8y2MuMrkpdUrY0aKR1mtHOj7uPs8JS8q8N7K1aKN+ZxqR+wtlPT2wa6M6O6Ru18qaT/FV+SGrXIF9GbVDsHvkHS8u29y4lQAAIBNSOrPJo3WGXGVAOmzCQ9dU+lXDYq7r3T3t0t6u6Sakf+V1SqrVy8vSDpnkB/1DtWOULtzYxduzaxF0mcSHrrS3ZPugNgod1/u7m+T9E5JQ16ZchRcnND2R8UjvTY691k1d+9090+o/4v0wCYRfAHDtyyhbVLdq9jQX4ayLLO7363kWzTfNvySEp2j2tDpN+7+0GDfyN1/pfi2zr4mKp7PLJGZTVLcKar2CXdfO9ga+tTyYqWeYancQvlL1c491i7pOHevy+0HAAAg85L6s0m34o2GU1UbIK1VvFrhkLn7je6eFOhJyaOy3l+54DhQH05o21R4dZpqt+tqxbctDpm7Xz+QOyXqobKy/L5VzesvbA951Ja7P+jufxpWcdhiEXwBw5c08eKmRhmFLGmlygNG6bOS5tW6bhjvl/TajU1yf7hqV8BZovi2wlSZWV7StZJOqnqoQ/FE9n+of1UAACCjkuakqld/9tiEtp8P5PbGYbhRtWHfdEmnDOTFlXDnwKrmf2jT84QlfdefDeT2xs1I9WT/knSPu/+j7pUAFQRfwPAlrYi4OS8ckbQy4Egss7yByrxV+1U1u+JVeYbqjoS2QzayRHX1nFmS9Et3r14psq4qK2leI+mMqofaJB3D7Y0AAGCEJfV96tWfPSyh7YbR/MDKvGZXJjyUNO1HkqTnXTaA2/jq/l1TkPQdr697FUAfBF/A8FWvgCNJm/NVmxcS2qZVliQeSa+R1FTVNreyYMBQzVEcDvU1TfGkmEkOTWh7dBifP2yVkO4nks6seqhN0rEM8QYAAKMglf5sZXXv7aqaS4pX9httP1LtBezDzKx6lfMNmNlUSadXNa+T9NNNvG5XxaPK+uqV9MSmS908VKbpODjhoerpSIC6IvgChi9p/oPRHJo9qipDrVckPDTS8zzMTGhLCt0GrHKVLWmVzW37eUl1R0uSnhpODcNRCb2uUO0S2+sUj/Qa9NxnAAAAA5BWfzapLine truncated
+    "g31767.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAAArUAAAHoCAYAAABXWBf1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13nB11vf/x1/ecs9l0UiCEhC5FShAISBGkiHAFERW7KAq2a7lYsF69ev1ZLhYUwYIoYi+ABRVFQAWU3gnSpJsQCKRnk83uOd/fH7PBzdnZzfaZb3g9H4/zgP3Ome98dmZ289453/lO4GnxauAAJEmS8p0D4W1FFyHlqXT9dyIsm1hoJZIkqeQemgFUi65CyhMg/h2WTYLa7jAhFF2QJEkqq0URpt4AtXshvKHoaqTuAsRYdBGSJCkp8yDMKboIqbtac8PWu8EWzyqilL7d9If/iXNf9OnSXkm+7bIv1HfY94TqhE1mFV1Krntv/AGTp23HzO0PLrqUXIsX3M7CB/5e3/Wg/yztx1plPwdv//OX6tvt+YrqpGnbFl1Krvtv+Rmt4zdtbLnzEZUNv3v0LXvyPh658+LGnENOKWV9kJ2De//Hp0II5Szxjr9+pb71bi+ubrLZjkWXkuuhO35FqLQ0ttntxaXcgWvXLGfelV9r7H3kx0tX3wO3wKJHi65C6luPULvFDjDn0AIq2YBbLzuLOYd9uugyenXX389p7LjvkdXps8sZah9/+CJmbncQux9SzlD74G33snzxTxpzDitvqL310jNLfQ7ec+13G8/a+8Dq5tttW3QpuRY/9gcmT98hzjnsiKJLybXgvkdY9MgPGnMOK2+ovfXSs5hz6Ccpaablvhu/19huzznV2TuVM9SuWHwZ1ZbxjTmHlTPUti1byT3XfjPOOezjRZfSw5KFhlqVXyl/sCVJkqSBMNRKkiQpeYZaSZIkJc9QK0mSpOQZaiVJkpQ8Q60kSZKSZ6iVJElS8gy1kiRJSp6hVpIkSckz1EqSJCl5hlpJkiQlz1ArSZKk5BlqJUmSlDxDrSRJkpJnqJUkSVLyDLWSJElKnqFWkiRJyTPUSpIkKXmGWkmSJCXPUCtJkqTkGWolSZKUPEOtJEmSkmeolSRJUvIMtZIkSUqeoVaSJEnJM9RKkiQpeYZaSZIkJc9QK0mSpOQZaiVJkpQ8Q60kSZKSZ6iVJElS8gy1kiRJSp6hVpIkSckz1EqSJCl5hlpJkiQlz1ArSZKk5AWIsXtDreV/YrXl60XV06vOtatCbcyEuOF3FqPR6AwhVGII5fw7oXNtW6hUW6hUW0q5Dxv1DhqNjlBrGV/K+gA6O1aHWsu40tYX6x2BSiWGUC26lFydHatDpVKlUh1Tyn3YqHfQqK8t9e+Zemd7qNZaS1tfo94RQqgQKtVS1ljvWE0I1VCplfMcJEZirIdQqZWuvs61Z4V652u7N82DMKeoeqQ8PULtfsfBnEMLqqYPP/zvafENn10ciq6jN7/4zI4dL3jz+S3TZ+9ZdCm5LjvvFczc7iB2P+S9RZeS68HbLmDelWd0HPueq1qKrqU3P/zY1PiGzy0p7Tl4wed36Tj4Nd9t2Xy7A4suJdcVPzmRydN3qO911CdKmboX3Hc51130oc6XfeCmWtG19OaHH5sWT/jMohAqpdyF/PKLe3Tsd9zpLbN3OqLoUnJdfeG7qLaM79zvJV8s5TFuW7aAX5++b/11/zu/dAf4yp/Cvdev12SoVemU87KiJEmSNACGWkmSJCWvlB/BSJIkDU4cDxwFHAzsBWwHzADGAZ3Akq7XP4Frul5XQVg7gG0cBWzTzzevBJYBS4E7ICzv/3ae3t5cYG5T468hPDHwvtbr9zBgx6bGH0BYM7R+ByvOBvYBNm9acBeEqza0tqFWkiRtBOIuwAeA1wATenlTDdis67UTcHRX+yKIPwLOgXBXPzb2TuAlgyiyAfEe4LfAtyHc38/1jgU+2dR2OzDEUMtJwAlNbb8CRiHUxi3Igvo+3f47s5c3fxvYYKh1+IEkSUpYnAjxa8AdwMn0Hmj7shnwPmAexG9DnDGcFXZTAXYBPgTcC/GbEAdTb8LisRDnAwvIwv0ngRfTe6DtN0OtJElKVNwZuB54D9DXrBGdwELgIaCtj/dVgLeSBc5xw1RkX9t6B3ADxM1GeFtlsgUwayQ6dviBJElKUJwDXEY2XrbZCuAC4BLgCggLm9adCOwKvIjsKuFcoPuUjZvQd0jO81Egb47hiWRjRPcH8qZB2wX4PcTnFzeWtVTagdbBrGiolSRJiYmzgT/QM9B2At8C/l/fN1GFlWRXeK8H/hfifsBngRcMoajTIGzgwRlxLvB1YL+mBfuSjdM9fQjbT9Ea4Dbgpq7XjcCjwOLBdGaolSRJCYkV4OfA7KYFbcArIVw88D7DdcAREP8D+D75V3+HQbgpuyLL74AXNi38CMSzBjYLQ5JuBd5CFmLvhNCx/uI4abAdO6ZWkiSl5F3A85ra2oEjBxdouwt/JLsL//oNvXMI21hLNuPAqqYFm5ENUdjIheshfBfCrT0D7dAYaiVJUiLiJOATOQs+CeHvw7ON8CjwfGD18PSXu40nyK42Nzts5La58TPUSpKkVJxIdkWzu5uBLw3vZkI7hPrw9tnDFTltW47wNjdqhlpJkpSKk3LazhiFADoSFuS0bTrqVWxEDLWSJCkBcSuyx952txg4v4BiVEKGWkmSlIK86bYuhzCCY19HVPPsDQBPjnoVGxFDrSRJSsHeOW03jHoVw+f5OW3/GvUqNiKGWkmSlIJn5bQlGmrjDODVOQv+OsqFbFQMtZIkKQXTctrmj3oVQxZbgB8CE5oWPAVcM/r1bDwMtZIkKQXTc9qWjXoVQxL3Aa4CjsxZ+H/PgKeJjSgfkytJklKQd6W2TKH2wxBjTvt4YCZwADCnl3VvBr4+UoU9UxhqJUlSqvJCZFE+P8j17gOOSXgWh9Jw+IEkSUrB4py2TUa9iuETge8CcyEsLLqYjYFXaiVJUgryQu0UYNFoFzIEEbgX+B3wbQj3FlzPRsVQK0mSUpAXarcg+/i+DN5B/nCINrKxv0uAOyCUaRxwKLqA4WSolSRJKXggp21f4MrRLqQX34YwUmN86zlt44ah37w+Ooeh30I4plaSJKXg5py25456FcVYmtM2cRj6ndT0dQNYPgz9FsJQK0mSUnB5TtvhEMeOeiWjLy/U5s3bO1CbNn29HELeVeEkGGolSVICwsPA7U2NmwKvKKCY0fZETttuQ+syVoCd+7GdZBhqJUlSKs7NaXtPV0DbmN1Iz5vQ9hpin88mezBEdzcMsc9CbewngSRJ2nicBzzV1PZc4JTh3UwcU66gHJ4C7mlqfD7E5uEDA/HynLarh9Bf4Up0wCRJkvoSlgGfzVnwWYj7DM824izgz/S8ilm0PzZ93QKcNLiuYivw5qbGBnDJ4PorB0OtJElKydeA65raxgGXQ3zB0LqOLwBuAp43tH5GxFn0nNrrExC3HURfHwW2b2r7LYT7B1NYWRhqJUlSQkIdeDXweNOCycDFEL8IcdrA+ox7Q7wYuAyYORxVDr9wP3BBU+NE4K8Qd+lfHzFA/BDwP00LGsAXhlhg4Xz4giRJSkx4GOIxwJ+A7gF2DHAq8BaIPyf7OP2vEJasv34cS3bn/zFdrwNI4+la7ya7irxlt7ZtgFshngv8GLgewtr1V4ubA0cA7wf2zun3NAijOJ42ngLs2svClpy2gyGe3UeHX4Nwp6FWkiQlKNwE8RDgV8AOTQunAG/vegFxNfAk2SNrN+9a3pfF5D/Fq2DhSYivIhtfO7nbgjFkj+l9B7AG4kKyG+payeaz3aKPTi+h55XbkXY0cOQA3r9L16s3vwLudPiBJElKVJgHzAXOJvsIvTfjgK3Irs72FWg7ga8DO0FYPVxVDq9wDbAfPWdDWGcssC3ZftmdvgPtOcBxEJJ9NG53hlpJkpSwsBzCO4A9gR8CawbRyRPAl4HdILy7awqtEgt3A/sAHwQeG0QHfwIOgfA2CO3DWlqBHH4gSZI2AuEO4I0Q3wX8B3AQ2QMKtie7WrnuQt5SsuEF9wLXAteQjbtd29xjH74B/L5p+80PRxhhYSXwJYhnko2XPZjse96KbJzxRLKr14u7XncAVwKXQ7hzdGvt4QzgwmHs706AAHG9gzBu0un1sRO/V7pxJKuWPDpmwtStBnLCjaq2ZQtaWsdPa1RbxpZu3wG0LV/YUmsZy5hxUzqKriVPR/uKSkf7yur4yVuUsj5I4BxcvqBl7PhNG5XamFKeg6uW/Ks2ZtwmoWXspFIe4441Kyod7Stq4zeZVdpjvGrJo2MmTNmqg9DjyUKl0LZsQcuYcVMatTHjS3kOrlmxqBYqldA6YXopz8HYaFRWL3+sOn7K7NLVt2rpZ2prVx/X/dPdeRDmFFbQoMQa0IDQ1zCFjUhsgVC6c2kk9bhSu+2cl1S323PPahHF9OWyc1/OAS87Y0zRdfTmyp+e1Njj8A9VJ2/6rNLtO4CbL/k007bYnW33eHkp9+HC+6/iodsvbOz/sq+Wsj6Ay753fLnPwZ+d3Njt4PdUp8zctbTn4OTp28Ud9nlDKffhokeu597rvtco8zG+7HvHx/1f9pWWEMo5cuzv5/9n/dkHvq06ffZepTwH77zya1RrYxvPPvBtpTzG7W2LuebC98QynoO3/3kX/nV30VUM1cYxbrT/nlmBFnJC7eTNdmDWjs03ERYvVCpx1o6Hl3a6jUqlpb7Z1vtWps/es+hScv3j799g8qY7MGvHw4suJVd722Lm33dZfdaOh5fzX2uyjzXKfA7WamPr07fcu7L5dgcWXUqu+274PpOmbd+YtePhpQw8EHng1vMb5T4HA7N2OIxQKecurLa0NqbNek61rL9nHrr9Qqot40t7jNuWLYBQKeXPyD9vLLoCacNK+YMtSZIkDYShVpIkSckz1EqSJCl5hlpJkiQlz1ArSZKk5BlqJUmSlDxDLine truncated
     }
    },
    "cell_type": "markdown",
-   "id": "c518f863",
+   "id": "7460b0af",
    "metadata": {},
    "source": [
     "<div>\n",
-    "<img src=\"attachment:fig-asp-1d-partition.png\"  align=\"left\" width=\"400\"/>\n",
+    "<img src=\"attachment:g31767.png\"  align=\"left\" width=\"250\"/>\n",
     "</div>"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "102d6fa2",
+   "id": "3ebddda1",
    "metadata": {},
    "source": [
-    "## What is the work per process at iteration k?\n",
-    "To evaluate the efficiency of both partitioning schemes, consider how much work the processors do in the following example. \n",
-    "In any iteration k, which part of the matrix is updated in the inner loops? \n",
+    "### Load imbalance\n",
     "\n",
-    "### Block-wise partition"
-   ]
-  },
-  {
-   "attachments": {
-    "fig-asp-data-updated.png": {
-     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtcAAAJNCAYAAAD6c1l4AAAACXBIWXMAAB7CAAAewgFu0HU+AAAAGXRFWHRTb2Z0d2FyZQB3d3cuaW5rc2NhcGUub3Jnm+48GgAAIABJREFUeJzs3XeYJFXVx/Hv6ZnZvLC7LGFZwrLkDIpECRJEEMVXl/waEQyor6IoBhAFEwZEBUFUVEREEQwgIkEyIkGQKJkl7i5h8+ykPu8ft4btrqmeTtVTNTO/z/P0A32n6tbZ6urqU7du3Wvujohkx8w6gXHR2053n5BlPKOFmc0GHi8pusvdd8ggjo2B24GpJcW3APu6+4qhjkdkuDCz04AvlhQd5+5nZxVP3pjZGGBtYDxQBJYBL7p7bw3rfhU4KVb8Pnf/ZeqBjkDtWQdQjZl9B5g8yCLdhANmafT/c4GHgYf1wyQieWZmqwB/pDyxvhk4sNL5y8zWAy6NFX/C3W9tTZTSKmb2e2CDkqJL3f3rWcUjw5uZjQPeARwI7AasD7TFFltuZr90948OVpe7n2xmHcCJJcXnmNmD7n5HmnGPRLlProF3A2s0sF7RzO4B/kQ4Yd2fbljpMrMdgXeWFN3g7ldmFY+Mbmb2JmD/kqK/uvuNWcUzgp0DbFHy/r/A2919ySDrjANeHytbtdLCZjaZ8ta9+e7+vXoDlXJmtivw9pKia9396jqr2YLyz//OpgOTUcfMJgL/BxwPrFZl8QnAZrXU6+6fN7OZhDwMwrnnEjPbxt0XNRrvaDAckutGFYDXRa+vmNkVwCnunteT13bA52JlSq4lK7tSfjy+BCi5TpGZHQIcUVK0kJBYv5rypiZR/lk+CCi5bt7rKd+vXUC9ybVIU8xsF+DXwOwWbeJYYBNgp+j9eoTzx9Et2t6IMByT6yVAaX+hVRh42yPJW4EDzOx04GR372lFcCIi1ZjZNOCsWPHH3P2RLOIRkeHHzOYQEuuxsT8VgbuAfwILCF1n1wVmAXvWsw13X2FmRwL3sLKL7gfM7HfuflXj0Y9swzG5PsDdbyktMDMDpgBrARsRWhTeDOxIeeJdIPQf2tbM5rj78qEJWUSkzMnA6iXvL3H3C2tc91lgv1jZv1OJSobaB4GJJe+fyyoQGV7MbF/gN0BHSXEn4aL92+4+v8J6HcDG9WzL3Z8ws+OB80qKv2dm29bycORoNByT6wE8DHnyavR6CPgLcIqZbQR8BvgA5QfgAcAfzexAHRgiMpTMbBOg9GGi5cAna10/ahS4Ju24ZOi5+21ZxyDDj5mtxcDE+ingIHd/YLB1o7v2Dzaw2Z8RuoLsHL3fgtBlRKOzJChkHUAruftj7v5h4A2EpLvUfsC3hj4qERnlTqT8R/EMd1eLpYjU6kzK73w9A+xaLbFuRtSI+ZlY8YlRS7jEjIiW62rc/V4z2w34O1A6ju0nzez37v7PWusys3ZC36X1CE/ljgH6CK1P8whDAC5NLfgmmNkkwjBPMwl9pdoIsb4CPA086e592UWYPTPbENgSWIewb54HbnP3l1KqfzPCFf46QA/h7sp/CMdJMYX6OwjH4rrAdMJ3uo8wNOWL0XY6m91OmsysDdieMEzUmoSL/JeBJwhjTTd9NynqKrY94UGcGYTv5wLg3+7+ZLP1NxHX2sCRJUULgdMzCicVZrYusA3hx34K4RhfQPgepf1wZt2ic3bpd6SD8B1Zxspz9rLsImyemW0AbEX4DCYT9v+zwL9aNSRttM2tCfvVCeeb29z9hVZsTwIz2wk4pKSoB5gzFPvd3W8xs8uBg6KidYHDgQtave3hZlQk1wDu/qqZHUzom9g/tF8B+C5hPMiKoocG9iK0gG9HSKgHW/4e4DLgR+7+yiDLTQRuiN5Oj/35vVGfqkqucvfS4bUws9WA90dxvoHy8VOTdJrZ1YQvxh98CGYUMrNbWbn/utx90H0fW/cA4NSSot+6+3cGWf47hM+t37HufneUkB4NHEMYTSau18yuAk5w9/gdj1ribAc+RLj1v0WFxZ43s4uBHzRQ/3sIx+wbCD+og7UcFM3sLuD3wI8Hu/Azs3UIYy5DSEhL/Z+ZHT7Idgb9LKL6NyIMCXcQA4/3fovM7BLgNHd/arD6KmxjInAC4XuwXoVlHiE8BPS3eutPwbGUP3z0C3dfXG8lZnYl5S1Xu7l7V2yZnxCO7/jxMdvMBhs1abm771Fl++OA44D3EBLrJH1m9k/gW+7+l8HqK6n357H65pQeB9Ht8P8Bdif8sI8jjGRzm7t/tWS5w4E9CN+RbRn8O+Jm9m/C2OFnufvCQeKbysoRQeJDxB5jZm8dZDuXu/spsfpOBOaUFH2x1ofEoglCjiN0e9yqwmLLomPlVHf/Ty31RnX/kdAg0G9vd18cndveTzi/xYeDhLAvrwc+l/exkM3sy8DbSoqcEPd1GYVUq88AVvL+Z+7+ryHc/pmsTK4BPoGS64HcPdcvQsuCl7x2a7K+98fqc2CXKus8kLBOLa+XCT8OlepdpcF6HfhNQn07N1HfLcDaQ/B5dpZsc3md6/5vLOYzqyx/WWz5PQkPuj5W4z7pBP63zhg3A+6rY7/3EJ7srnmfAPMb/IyfB/YbpN4Nmzh+Kn4WhDsmpxMmeaq1vi7gA3Xu+zcRWutq3UY8njuH4Ph/pGR7fcBGDdbzXCz28QnLXN/gZ7mkyrb3I9z5qqfOPwOTavh33RJbb7OofE3Cg1qVjqG7Y/U83uC/fT5w8CDxrd5gvQ6cn1DfWbFljqjx89+B2s9jThhh6/tAW431x+ueBuwTO36rfX8PG4Lv02mx7X60xvW+HFuvSJjdsaXxpvDvnRb7DvQC6wxxDEbos126/zbOet/k7TWi+1xXcAHhB7jUUS3a1jTgt2Z2aIvqT9OuwI1R6/dI9XlCa+WGNS4/Dji/SmvUa8zs9cBNVG5FStJOeStEK80A/lLljkiqopbkywityfHWw2eBewnPQ8QnJBgD/MzMPlHjdt5BGBd+Zh3hDWlfQTPbgfKn9G9298eGMoZmmdn7gSsYeFdgEeFzvIeB51cILYRXm1l8yLBatnkoYXKdj9L6z2x1wiQZB7d4Ow2LJni6juTzmBO6QMW1ESYZ+X0jnwHwVUK3ylpHmRgD/NLMtm5gWy1lZp8DTikpcsIwmPGhMfPorZR/B25y96TvW8t4yLDjU6AfNpQxDAejpltIP3fvNbPfUt4x/8AaV19OaFm5mdA6+QRh3O0xhOmL1yXManckITGDcFI7y8yudfeXY/V1sfKhym2Bt5T87XZCy1Ml91aJ9RngH8C/CC3v86LtTSIk/VsRLip2LllnwyieD1ape7gqnXGwmzDiwnWEZKBA+PcfSuhH2K8d+KmZbeqD3L6Pxi3+IwO7O9xIeKr7fkL/2tUIx8mmhBNlUreUWi2J6r8lqv9pQl/rsYTjcQPCsX0IK0/IY4HzzGxLHzgU5UJWHo+7EG6r9+s/liqpNPX2eZTfel1I6Ir1S3d/pr8w6oe9I6HbSOnFzLfN7FYfZPInM9sCuJDy7hZO+DwuJbTCLSFcXMwgdO16O2HYzqH05tj7mrpKNOEiwji3kwjdB/q9RHjyv5LupEIz2x/4KeUPwl9K+Dxv95LnN6K+5R8kTLIyISremXB81TwyCmGyigPqWD5uGeF83X/Oforyc/b6rDxn93dXaydM83y9D5yFbjkrvyOvB0ovVPt/Gyq5q+F/RcTM1ifs88klxYuB7xC6fj3q7n1mtgrhTt3/EVqc+/0P8E3gU3VuuvT46SV0jbmG8DvTRvguHUZ5w8JY4NuU/65lysw+Sfj393NCi/WPq6zXRjhWWqnTq/eb3jH2vuwcEnUV2pxwkTiVcNH7KvCgp/tcwZ8p349vJtxFkH5ZN53XcAsi1W4hUZ1vYeBtrOmDLP8V4GBgbI31b0N4oKS0/o9XWefY2PLfbODftTHwNWCbOtb5Umy73cCUFn6eWXYLccKP7enA6hXWKRASvPh6p1XZ1sWx5Tup4bYoIdErvc1XS7eQ/h+sjhr3266EpLs0vsOrrBPfB59p4LM+OlbHf4ENq6xjhGStdL2rBlm+jdBaWrr8i9XOE9F25sTWa2m3EEKLb+n2NmmirqrdQkqWnRFb9oEGtrcW5d2ReoH317DeTtF3rn+9HmD9QZaPdwspfT1F+AHfi3AhvD4hmTsIOCpWz9cIF3W1nrNfR3jQu3R7x1ZZ5+Ox5b/SwH6tq1sIA7v6PEyVbgGEu0al6xSBfaqsk9TlpAv4ETCjwjptwI9j6/QN9nmn8J2quVsI8BHKu+AVgQ/XuJ3VBjku03pdU0McN8fW2ZfQkPceQkNRZ4W6+wh3VT9AjV2Daojl0ZL6O2v9ro2WV+YB1PABtiK5Xjfh4Ns95bg/GKv/r1WWbzq5biLWW2PbPqSF28oyub6FGvuGJfxIPFvppES4mIqftN82FPukjm18Pvbv+VWV5ZtKrgmtVs+XrL+QKol1ybrtlCfMRWDzCsseFotzGbBVjduZHVu3Zck1IZl/tWRbzzVZ31An199p9HgAPh1b91uDLJuUXM8DPgy0t+rzibZ9XGy7f6yy/JAm14SW6NJllwKzatzOj2Lr3lBl+XhyfRewZQ3biX/vHXh3Cz+zmpJrQlIZP0d/qI7tZJ5cExp+lsTW+TL1PWfihHNrTefIKvGcF6t30GfXRttrNPa5hnAwxodBWzPlbfwx9r7ayB1Z+lPs/awsghgCX3D3R2tc9mSgdAirmYQRCpLEn96+0GscHWEIxY/HWS3e3pGUjzjyNXd/vJYVPQzF98OSImPgjIT9Toi9P83d7685yqGzBmGYun7VunXlhpmtShhZp9/9wBl1VHEu5f2A491jBnMV4U7cOd76Cb/yfh78WOz9d732EXW+QOg+0m8PM9u2jm3v7zWMoexhxJrLY8WVRpMZEmZ2FCER7D9HO6HF+tzsomrIaoQuXqVOob7nTCB0Qb3JzAYdEagG8XPYpk3WN6KMuj7XLine truncated
-    }
-   },
-   "cell_type": "markdown",
-   "id": "a67e0aad",
-   "metadata": {},
-   "source": [
-    "<div>\n",
-    "<img src=\"attachment:fig-asp-data-updated.png\"  align=\"left\" width=\"250\"/>\n",
-    "</div>"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "d9d29899",
-   "metadata": {},
-   "source": [
-    "It is clear from the code that at any given iteration `k`, the matrix is updated from row `k` to `n` and from column `k` to `m`. If we look at how that reflects the distribution of work over the processes, we can see that CPU 1 does not have any work, whereas CPU 2 does a little work and CPU 3 and 4 do a lot of work. "
+    "Each CPU will be on charge of updating the entries stored locally. However, this algorithm is a bit different than the previous ones. At iteration $k$ only a subset of the entries in the matrix `B` are updated, i.e. only the entries in the bottom right block  `B[k:end,k:end]` are updated. This block is illustrated in a darker color in the next figure for $k=6$. "
    ]
   },
   {
@@ -347,34 +445,16 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6409890d",
+   "id": "e341fda6",
    "metadata": {},
    "source": [
-    "### Load imbalance\n",
+    "In this particular example CPU 1 will not have any work to do at iteration $k=6$ and it will be waiting, while other CPUs are computing. In general, CPUs containing rows previous to row $k$ will have less work to do than CPUs containing rows after row $k$. This will worsen as the value of $k$ increases. At some point, only the last CPU will have work to do and the others will idle. \n",
     "\n",
-    "The block-wise partitioning scheme leads to load imbalance across the processes: CPUs with rows $<k$ are idle during any iteration $k$. The bad load balance leads to bad speedups, as some CPUs are waiting instead of doing useful work. \n",
-    "                    \n",
-    "### Data dependencies\n",
-    "                    \n",
-    "<div class=\"alert alert-block alert-success\">\n",
-    "<b>Question:</b>  What are the data dependencies in the block-wise partitioning?\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "<b>Definition:</b> *Load imbalance*: is the problem when work is not equally distributed over all processes and consequently some processes do more work than others.\n",
     "</div>\n",
     "\n",
-    "    a) CPUs with rows >k need all rows <=k\n",
-    "    b) CPUs with rows >k need part of row k\n",
-    "    c) All CPUs need row k \n",
-    "    d) CPUs with row k needs all rows >k  \n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "e0565e92",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "answer = \"x\" # replace x with a, b, c, or d \n",
-    "ge_dep_check(answer)"
+    "Having processors waiting for others is a waste of computational resources and affects negatively parallel speedups. The optimal speedup (speedup equal to the number of processors) assumes that the work is perfectly parallel and that it is evenly distributed. If there is load imbalance, the last assumption is not true anymore and the speedup will be suboptimal.\n"
    ]
   },
   {
@@ -382,9 +462,11 @@
    "id": "51498a44",
    "metadata": {},
    "source": [
-    "### Cyclic partition\n",
+    "### Fixing load imbalance\n",
     "\n",
-    "In contrast, if we look at how the work is balanced for the same example and cyclic partitioning, we find that the processes have similar work load. "
+    "In this application, is relatively easy to fix the load imbalance problem. We know in advance which data is going to be processes at each CPU and we can design a more clever data partition.\n",
+    "\n",
+    "We can consider row-wise cyclic partition to fix the problem. See figure below. In this case, the CPUs will have less work as the value of $k$ increases, but amount of work will be better distributed than with the previous row block partition."
    ]
   },
   {
@@ -404,58 +486,123 @@
   },
   {
    "cell_type": "markdown",
-   "id": "866824c6",
+   "id": "1c9e4c8a",
    "metadata": {},
    "source": [
-    "## Conclusion\n",
-    "Cyclic partitioning tends to work well in problems with predictable load imbalance. It is a form of **static load balancing** which means using a pre-defined load schedule based on prior information about the algorithm (as opposed to **dynamic load balancing** which can schedule loads flexibly during runtime). The data dependencies are the same as for the 1d block partitioning.\n",
+    "### Static vs dynamic load balancing\n",
     "\n",
-    "At the same time, cyclic partitioning is not suitable for all communication patterns. For example, it can lead to a large communication overhead in the parallel Jacobi method, since the computation of each value depends on its neighbouring elements."
+    "Load balancing is the process of distributing work to processors uniformly with the aim to efficiently exploit a parallel system. There are two key forms of load balancing: Static and dynamic load balancing.\n",
+    "\n",
+    "- **Static load balancing**  the work distribution strategy is based on prior information of the algorithm and it does not depend on runtime values.\n",
+    "- **Dynamic load balancing** the work distribution strategy is based on runtime values.\n",
+    "\n",
+    "Static load balancing is often used in algorithms for which the load distribution is known in advance and it does not depend on runtime values. On the other hand, dynamic load balancing is often needed in problems in which the work distribution cannot be predicted in advance and depends on runtime values.\n",
+    "\n",
+    "\n",
+    "\n",
+    "<div class=\"alert alert-block alert-success\">\n",
+    "<b>Question:</b>  Using a cyclic partition for Gaussian elimination, is a form of static or dynamic load balancing?\n",
+    "</div>\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a6741a25",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ge_lb_answer()"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "20982b04",
+   "id": "e151fb8d",
    "metadata": {},
    "source": [
-    "## Exercise\n",
-    "The actual implementation of the parallel algorithm is left as an exercise. Implement both 1d block and 1d cyclic partitioning and compare their performance. The implementation is closely related to that of Floyd's algorithm. To test your algorithms, generate input matrices with the function below (a random matrix is not enough, we need a non singular matrix that does not require pivoting). "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "a65cf8e6",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "function tridiagonal_matrix(n)\n",
-    "    C = zeros(n,n)\n",
-    "    stencil = [(-1,2,-1),(-1,0,1)]\n",
-    "    for i in 1:n\n",
-    "        for  (coeff,o) in zip((-1,2,-1),(-1,0,1))\n",
-    "            j = i+o\n",
-    "            if j in 1:n\n",
-    "                C[i,j] = coeff\n",
-    "            end\n",
-    "        end\n",
+    "### Data dependencies\n",
+    "\n",
+    "Using a cyclic partition, we managed to distribute the work uniformly. But we still need to study the data dependencies in order to implement this algorithm in parallel efficiently.\n",
+    "\n",
+    "Look again to the algorithm\n",
+    "\n",
+    "```julia\n",
+    "n,m = size(B)\n",
+    "for k in 1:n\n",
+    "    for t in (k+1):m\n",
+    "        B[k,t] =  B[k,t]/B[k,k]\n",
     "    end\n",
-    "    C\n",
-    "end"
+    "    B[k,k] = 1\n",
+    "    for i in (k+1):n \n",
+    "        for j in (k+1):m\n",
+    "            B[i,j] = B[i,j] - B[i,k]*B[k,j]\n",
+    "        end\n",
+    "        B[i,k] = 0\n",
+    "    end\n",
+    "end\n",
+    "```\n",
+    "\n",
+    "Note that all updates on the loop over i and j we do the following: \n",
+    "\n",
+    "```julia\n",
+    "B[i,j] = B[i,j] - B[i,k]*B[k,j]\n",
+    "```\n",
+    "\n",
+    "As we are using row-wise partitions, the CPU that updates `B[i,j]` will also have entry `B[i,k]` in memory (both are in the same row). However, `B[k,j]` is in another row and it might be located on another processor. We might need to communicate `B[k,j]` for `j=(k+1):m`. This corresponds to the cells marked in red in the figure below.  These red entries are the data dependencies of this algorithm. The owner of these entries has to send them to the other processors. This is very similar to the communication pattern seen in previous notebook for Floyd's algorithm. There is a key difference however. In the current case, we do not need to send the full row, only the entries beyond column $k$ (the red cells in the figure).\n"
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "31d8586a",
+   "attachments": {
+    "g27851.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAzAAAAJsCAYAAAAiM0YjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15nBx1nf/xV/X0zOS+LxIIwYRDDkFuFBFUUFbE+wQPFBRXXQVcL1xss3ggrgfeeKKuq7grIh4/BeUSOQxBbgEFQgIkAZJJyDVX1++PmoGeTnfPVKe7q2rm9Xw8+pGu6u936t09k5n+9Lfq+w2QJI12eeCsku2/A5cO3O8EXgG8GdgD2BkoAvcDvwMuAB6u8nWfBZwKPB94xsDXWgvcBPwQ+GUjn4QkSZKksWE8EJbcfjqwf1/g7rLHym8bgdeUfb1xwNeB/mH6/hqY0KTnJEmSJGmUqlTAHAxspnYBMnjrA44b+FrjgD+OsF8I/BYImvz8JEmSJI0i5QXMH4AHBu5vA75NNMpyBHAM8FFgdVmfFQNf57sl+24hOjXtRcBhwOuITk0rL2JObv5TlCRJkjRalBcwg7d7gd2r9Jk78Hhp+x8P/FsE/h3IVen74bJ+tzTiSUiSJEkaGyoVMOuAXYbpd3yFfiGwdJh+AXBzWZ8968wuSZIkaYypVMB8YAT9cmx/KtlKoH0EfT9Q1u9NsVNLklRBteF/SdLo1Q/8YATtisDysn0/AnpH0PevZduOwEiSGsICRpLGnjuArhG2XVW2fV2d/aaNsJ8kSTVZwEjS2FNtYcpKNpVtPzLCfpvLtifHOKYkSVVZwEjS2FNelNRSrLNveT/XgpEkNYQFjCSNPWHSASRJqpcFjCRJkqTMsICRJEmSlBkWMJIkSZIywwJGkiRJUmZYwEiSJEnKDAsYSZIkSZlhASNJkiQpMyxgJEmSJGWGBYwkSZKkzLCAkSRJkpQZ+aQDSJKaLgTWl2xvjtF3a1nf/jqPuSXGMSVJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkqTWGA+cDFwGtCecRZIkxTcTuBJ4M9HfdUkadQLgKOASYDWwDfjfRBNJkqQdcS3R3/M1wKXA84FcookkqQGWAF8AVgKPAeHAbQ2wc4K5JEnSjjmQoX/b1wGPAl8B9kgwlyTFNh14L3An0WhLH0//chu83ZFYOkmS1CgPsf3f+CLwBHAv8AFgRmLpNOoFSQdQprUDxwPnEH3qMo7a17f8GfhRC3JJkqTm+QDwzBqP9wFbgduBzwG/BXpbkEtjhAWM6nU48AeigmVcwlkkSVI69RBdM3MocE/CWTRKWMCoXjngGOBM4LnAJKBtmD73E30aI0mSsuswYN4wbXqB9cBvgAuAvzU7lCTFMQ54PfAXYC3RL63yc2ND4LqkAkqSpIa5j8p/57cSXdB/PfA6XDZBUkbsDCwFHmToLCUh0cX9ExNLJkmSdtR8or/ng3/b+4lmGb0DeDcwNblokrTjDia6aH81sIHoHNgPJ5pIkiTtiG8SFS2PEy2XcD7wjEQTSVITtAMvB64mmmLRIWVJkrJnJtF1LT8nuv7Va6kljQmdSQeQJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSpOFMB9aV3L6WbBxJktIhn3QASVJFAVERM2hCUkEkSUqTXNIBJEmSJGmkLGAkSZIkZYYFjCRJkqTMsICRJEmSlBkWMJIkSZIywwJGkiRJUmY4jbIkjT47A3PL9t0BdCeQRZIkSdIYMAMIS27fH2G/Y4ENJf2KQKEJ+SRJkiTpKfUUMG8Dekr6dANvalI+SZIkSXpKnAImIBplKW2/Hji6qQklSZIkacBIC5g8cGFZ24eB/VuQUZIkSZKAkRUwk4DflrW7HdilRRklSZIkCRi+gNkJWF7W5gpgagszSpIkSRJQu4DZB1hR9vhFQHuLM0qSJEkSUL2AOYboAv3Sx75MdCG/JEmSJCWiUgHzWmBryb4+4N1JBZQkSZKkQeUFzKNEi1IObm8CXppYOkmSJEkqUV7AlN8uBzoSSydJkiRJJcoLmOuIRmFK9/0GGJdUQEmSJEkaVOkamN2A+8v2XwVMTiaiJEmSJEWqzUK2ELiv7LFrgSkJZJQkSZIkoPY6MPOA28seXwbMbHFGSZIkSQJqFzAAc4C/lbW5E9iphRklSZIkCRi+gAGYDtxQ1u5uYEGLMkqSJEkSMLICBmAq8OeytvcTXfAvSZIkSS0x0gIGYALRujCl7VcAS5qcUZIkSZKAeAUMQCfwy7I+jwL7NjGjJEmSJAHxCxiADuB/y/qtAfZvUkZJkiRJAuorYADagB+U9V0HHNb4iJIktV6QdABJUkU5YFHJ9pPAYyPs2wbsWrZvC7B6x2NJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJktKrHZiRdAhJkiRJ6ZRPOkCZdwGvAF6UdBCNVLgbsEfSKSRJUqr9GYLNSYfQ6BAkHaDMXcBcYAmwPuEsGpHww8Bnk04hSZJSbR8I7ko6hEaHXNIBSiwBZgETgZMTzqJhheMhnAK/PCbpJJIkKe0+enz0viFsTzqJsi9NIzBfAM4YuP8A8IwEs2hY4aXAiUmnkCRJmXIaBN9JOoSyLS0jMAHw2pLtqXhdhSRJkqQyabmI/3nA+JLt6UQX9J+VTBzVYTnwlqRDbO/A38DCi+GXFyWdpLIXfRQ27gs3nZR0ksq+sRC+/gu4/eCkk1T3rBvh9W+Fs/+edJLKDvsBTHgQriwkHKSK170O7nsH3PLipJNUt9+tcPv+Saeo7oAr4Jlfg/+5JOkklT3/09A9G244LekklZ29P1z2Lbjt8KSTVLffzfAfx8Pr1iadpLJDfgrTl8EfPp90kjKTgeuTDiE1yyVAWHZbRbpOcdMQ4aUQhiW3a5NOVMW9wJlJh6jhAtL9y30JkPZZYzYCByQdooargAuTDlHD6USn7aZZb9IBhvEQcErSIWq4CLg86RA1HA50JR1iGFuBBUmHqOEm4PykQ2wvnFb2XiGE8NSkUyn70nAK2TjgiAr7JwBHtjiLJEmSpBRLQwHzGqKZx8pNBz7Q4iySJEmSUiwNBcz7gUlVHnsu0UiMJEmSJCVewMwFdqnx+CTglS3KIkmSJCnlkp6F7FSiU8WqmUg0QvPfrYmjhM0F9gbaBrbvAVYmF0eSJElpk3QBcxrRKNAaopmOdgY6gPuJipdJwO7APGB1QhnVPM8FjgEOBg4i+v6Xeh/w1VaHkiRJUnoleQrZvkSFyReB/YHFwGMDj70W2BV4L9APpHSNDO2gTwD/Cbyc7YsXSZIkaTtJjsDcD+wGPFrl8W7gB8DFwNQWZVIyeoDbgJuJCteXJBtHkiRJaZXkCMwWqhcv9bRT9nye6NSxScAhRAvq3ZBoIkmSJKVa0tfAaGz7Q9IBJEmSlC1JT6MsSZIkSSNmASNJkiQpMyxgJEmSJGWGBYwkSZKkzLCAkSRJkpQZFjCSJEmSMsMCRpIkSVJmWMBIkiRJygwLGEmSJEmZYQEjSZIkKTMsYCRJkiRlhgWMJEmSpMywgJEkSZKUGRYwkiRJkjLDAkZJejkQlt0KJY9/pcLjR7Y2oiRJktLEAkaSJElSZuSTDqAx7S7gIzH7PNiEHJIkScoICxgl6T7gvKRDSJIkKTs8hUySJElSZljASJIkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFjCRJkqTMsICRJEmSlBmuA6MGWTUPOD3pFBXMAJ4PbEk6SBXPBhaQztcOYC7QTnrzAXQCrwcOTzpIFQuBcaT3NXwBMJX05oPow7Y055sCHEv0s5hG+5Du7/ESotcurfkger/0VmBd0kGq2Ak4hNS9hruPj5Z8kxorSDpAmVVEb+YOApYnnEU1hZcCJz69fdM2OGx1YnGq2wXYBKxPOkgVs4j+cD+cdJAq2oH5wIqkg9SwK7Aa6E46SBXzgT5gbdJBqpg6cHso6SA1LAIeTDhDLQuBLmBj0kGqmAu0AY8kHaSKcUQZ0/x7ZhHRe5S+hHNUs4Dod+DjSQcZakYOnlhYtvM0CL6TSBypSVYBIXBg0kE0nPBSCMOS27VJJ6riXuDMpEPUcAFwfdIhalgCbE46xDA2AgckHaKGq4ALkw5Rw+nAA0mHGEZv0gGG8RBwStIhargIuDzpEDUcTlQAptlWoiIhrW4Czk86xPbCaWXvFUIIT006lbLPa2AkSZIkZYYFjCRJkqTMsICRJEmSlBkWMJIkSZIyw2mUpeHtDOzB0wX/DUQzm0mSJKnFLGCk7b2UaD79g4mm9J5X9vgBwK2tDiVJkiQLGKmS/yVal0CSJEkp4zUwUnVbiU4X+xouJSxJkpQKjsBI23sX8DfgLp5edflyYPfEEkmSJAmwgJEq+WHSASRJklSZp5BJkiRJygwLGEmSJEmZYQEjSZIkKTMsYCRJkiRlhgWMJEmSpMywgJEkSZKUGRYwkiRJkjLDAkaSJElSZljASJIkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFjCRJkqTMsICRJEmSlBn5pANIKbQUeG/Zvkkl968B+ku2Q2Bms0NJkiTJAkaqZDwwvcbjU8q2wyZmkSRJUgkLGGl7VzJ0hGU4FjCSJEktYgEjbe+3AzdJkiSljBfxS5IkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFLine truncated
+    }
+   },
+   "cell_type": "markdown",
+   "id": "0050b6c4",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "n = 12\n",
-    "C = tridiagonal_matrix(n)\n",
-    "b = ones(n)\n",
-    "B = [C b]\n",
-    "gaussian_elimination!(B)"
+    "<div>\n",
+    "<img src=\"attachment:g27851.png\"  align=\"left\" width=\"300\"/>\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cf189776",
+   "metadata": {},
+   "source": [
+    "### Parallel implementation\n",
+    "\n",
+    "\n",
+    "At iteration $k$,\n",
+    "\n",
+    "1. The CPU owning row $k$ does the loop over $t$ to update row $k$.\n",
+    "2. The CPU owning row $k$ sets $B_{kk} = 1$.\n",
+    "2. This CPU sends the red cells in figure above to the other processors.\n",
+    "3. All processors receive the updated values in row $k$ and do the loop over i and j locally (blue cells).\n",
+    "\n",
+    "\n",
+    "As you probably see, the parallel implementation of this method is closely related to Floyd's algorithm. But there are some differences. \n",
+    "\n",
+    "1. The process that owns row $k$ updates its values before sending them.\n",
+    "2. We do not send the full row $k$, only the entries beyond column $k$.\n",
+    "3. We need a cyclic partition to balance the load properly.\n",
+    "\n",
+    "A key similarity between the two algorithms, however, is that they both suffer from synchronization problems. We need to make sure that the rows arrive in the right order. The strategies discussed for Floyd's algorithm also apply in this current case.\n",
+    "\n",
+    "The actual implementation of the parallel algorithm is left as an open exercise."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "866824c6",
+   "metadata": {},
+   "source": [
+    "## Conclusion\n",
+    "\n",
+    "\n",
+    "We studied the parallelization of an algorithm that leads to load imbalance. We fixed the problem using a cyclic partition. This is a form of static load balancing since we were able to distribute the load in advance without using runtime values. This is opposed to dynamic load balancing which can schedule loads flexibly during runtime. Note however that cyclic partitioning is not suitable for all communication patterns. For example, it can lead to a large communication overhead in the parallel Jacobi method, since the computation of each value depends on its neighboring elements.\n"
    ]
   },
   {
@@ -469,27 +616,19 @@
     "\n",
     "This notebook is part of the course [Programming Large Scale Parallel Systems](https://www.francescverdugo.com/XM_40017) at Vrije Universiteit Amsterdam and may be used under a [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) license."
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "8ab22f67",
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Julia 1.9.1",
+   "display_name": "Julia 1.11.6",
    "language": "julia",
-   "name": "julia-1.9"
+   "name": "julia-1.11"
   },
   "language_info": {
    "file_extension": ".jl",
    "mimetype": "application/julia",
    "name": "julia",
-   "version": "1.9.1"
+   "version": "1.11.6"
   }
  },
  "nbformat": 4,

notebooks/asp.ipynb

@@ -47,13 +47,33 @@
     "\n",
     "function answer_checker(answer,solution)\n",
     "    if answer == solution\n",
-    "        \"🥳 Well done! \"\n",
+    "        \"🥳 Well done!\"\n",
     "    else\n",
     "        \"It's not correct. Keep trying! 💪\"\n",
     "    end |> println\n",
     "end\n",
     "floyd_check(answer) = answer_checker(answer,\"c\")\n",
-    "floyd_impl_check(answer) = answer_checker(answer, \"d\")"
+    "floyd_impl_check(answer) = answer_checker(answer, \"d\")\n",
+    "function q1_answer(bool)\n",
+    "    bool || return\n",
+    "    msg = \"\"\"\n",
+    "    Then we can change the loop order over i and j without changing the result. Remember:\n",
+    "    \n",
+    "    C[i,j] = min(C[i,j],C[i,k]+C[k,j])\n",
+    "    \n",
+    "    if we substitute j=k, we get\n",
+    "    \n",
+    "    C[i,k] = min(C[i,k],C[i,k]+C[k,k]).\n",
+    "    \n",
+    "    Since C[k,k]=0, thus, C[i,k] = min(C[i,k],C[i,k]), and C[i,k] = C[i,k].\n",
+    "    \n",
+    "    In other words, the value of C[i,k] will not be updated at iteration k.\n",
+    "    \n",
+    "    The same is true for i=k.\n",
+    "    \"\"\"\n",
+    "    println(msg)\n",
+    "end\n",
+    "println(\"🥳 Well done!\")"
    ]
   },
   {
@@ -63,7 +83,7 @@
    "source": [
     "## The All Pairs of Shortest Paths (ASP) problem\n",
     "\n",
-    "Let us start by presenting the all pairs of shortest paths (ASP) problem and its solution, the [Floyd–Warshall algorithm](https://en.wikipedia.org/wiki/Floyd%E2%80%93Warshall_algorithm).\n",
+    "In this notebook, we will parallelize the [Floyd–Warshall algorithm](https://en.wikipedia.org/wiki/Floyd%E2%80%93Warshall_algorithm) used to find a solution of the  all pairs of shortest paths (ASP) problem.\n",
     "\n",
     "### Problem statement\n",
     "\n",
@@ -74,27 +94,36 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ade31d26",
+   "id": "1faddbfa",
    "metadata": {},
    "source": [
-    "We represent the distance table as a matrix, where $C_{ij}$ is the distance from node $i$ to node $j$. The next figure shows the input and solution (output) of the ASP problem for a simple 4-node directed graph. Note that the minimum distance from node 2 to node 3, $C_{23}=8$, is highlighted in the figure."
+    "We represent the distance table as a matrix $C$, where $C_{ij}$ is the distance from node $i$ to node $j$ via a direct connection (a single hop in the graph). If there is no direct connection from $i$ to $j$, this is represented using a large value in $C_{ij}$ representing infinity. \n",
+    "The next figure shows a simple directed graph with 4 nodes an its corresponding distance matrix (labeled as \"input\")."
    ]
   },
   {
    "attachments": {
-    "g37385.png": {
-     "image/png": "iVBORw0KGgoAAAANSUhEUgAABGgAAAGgCAYAAAD2CVFOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15fBT1/cfx1+zm4iYcAipCCIfggZUqihYFxIpnrYr1bNVqb6+qJEF/jRZyeFXb2npWrUetZ+utXFqtWhUVxQOBBBTkkpsQcuzO749vwu7ObpLdZHdnd/N+Ph77MDPz3ZnPhDg7+5nv9/O1EBEREZFY9QMuClpeAfzTnVBERCQOxgAnAscAewF7AL2BDU2vL4CXgReBdS7FKBnOcjsAERERkTS0H7A4aHkOcKxLsWSa8cDYoOX7AJ9LsYhI5hsP3AwcGWV7H3A/8H/AmkQF1U4jgaODlp8GvnUnFFedCOzZ9PMm4EkXYxERERGRBNsPsINer7obTkapIPR3m+duOCKSobKAvwB+Qq850b62A2cnPerW/YTQGA92NRr3zCPwO/jE5VhikuV2ACIiIiIiIiJJ1AX4B3CKY/1W4FnMUKavgR1AX2A4cCowmcB36O7AQ5ihULclPmQREREREYlEPWgSRz1oRCTRHif0OuMH7gH6tPG+A4C3CO9N8+OERRqbn6AeNJDGPWg8bgcgIiIiIiIikiQXA2cELdvApU3rN7Xx3k+Ao4B/O9bfAYyKV4DSeSlBIyIiIiIiIp3BHsAfHOvKgT/HsI8GYDrwXtC6bph6NiIdoho0IiIiIol3KNCz6ectwPtB2wqB84HRmKldGzC1DxZgZp7YFuUxxgH5TT/vAN4J2vYd4EfACGBg0/aVwHOYWgv1UR7je0Bu08/riL7r+ADM0IBm7xJ6Xl2BCU0/D3W8d3Ir8S0G1kYZg4jILzDJlGafAze0Yz/1mB437xP4Tj0ZM6TogxbeczCBIVTOa3RrugOHBS1/SvjsUVMwMzSPcawfT8vDtqqB5Y51kwBv08/fAJ81/expOsYpwBBMomsDsBR4CvgvpidSW3IxnyPNlmI+i6IxounYzeY6tg8E9m/6Oficu2OmTm/JW8DOKGMQERERkRQUaw2a94Pavtm0rgemwGRrM4isA06PMqYFQe9b1LRuEOYmtrWZSJYSOi1ra1YFve8fUb4HTHIo+JiHOLYPbyPGll5nxRCDiHRuuZiEbjyvIY869vf3VtrOCWr3UQzHGOs4xvkR2tQT+/Xz+gj72RG0/d6mdQcDS9rY1+uYBEpb9nS877dRvKfZzY73Op3VRowtvYbHEEPCaYiTiIiISPL1x/QiORfz1LMlewD/xCQ4YjUU8yR3ShvthmOSOO05hohIuhiP6c3XbBPwTAf3eZ9j+UQy6zv2VEzvmJFttJsI/A/TW1M6QEOcRERERJIrG3gC2Bfz9O4V4EXgK0yy5gDgIgJduT3AX4HXiH44Tw4msTOw6RhPAY8BVZhZkQ4EfkbgZtqLefK7GnijvSfWARuBoqafp2GKcDa7Fmhs4X2xPIUWkc7tSMfyHGBXB/e5ANiO6REJZpjpfiR/5qCZmM+KcYQWQP4T5roeyX/b2OdewMOYz4xdmGTU85jhVT0xv89fAIOb2udjPs8OxJ2hpx8R+By5GDN8GExPVGfdoWAbExmUiIiIiCReR4Y4Nb/WEKi74tQFUxsmuP3v2jhG8BCn5tcuzBPdSLKAWx3tv6D1aa0TNcQpmKbZFpFEeI7Qa8tVcdrva479/qyFdokc4tTsJ462sU6zHTzEqfm1GvOZF0kv4CVH+yda2X8ihzgF0zTbIiIiIhK1WuBYTHHClrafj3ky2+zMdhznl5gnnpE0Ym6Og7v4jwLOa8dxRERS3d6O5cVx2q9zP87jpLNGTJL/0xa2b8XUSVsStO504KAEx5WxlKARERERSb7baPup3nrCkyc9W2gbybvA/W20sYHLCJ0l6ZIYjiEiki6csxltjtN+N7VxnHR2N/BhG21qgKsd6/Q50k5K0IiIiIgkn7OwZEveDvrZQ2BMfTT+RtvdwMFM6f1K0PJ3CS2kKSKSCfIdy1vjtF/nfjIpQRPtZ9ULhNa6OSEBsXQKStCIiIiIJNcaYHmUbVc5lnvHcJw5MbR11tBprUaMiEg68juWW5tBLxZex7IvTvt120bMTIDR8GNmA2y2DzAo7hF1AkrQiIiIiCRXLLNb1DiWu0X5vh1AdQzHcQ63GhXDe0VE0oFzSFMsCe/WOIeexmvolNtiLa7rbD8iXoF0JkrQiIiIiCTXzhjaOocoRfvE99sI723NeseycyiAiEi6c9aK6Run/fZv4zjpakOM7fU5EgdK0IiIiIhknliSQGB63ATrEa9ARERSxArH8tg47de5n1h6L6YyfY64QAkaERERkczTJcb2zqFTzhttEZF091/HcjxqbeUQnqBxHidddY2xvT5H4kAJGhEREZHM05fYCmA6u+hHU0Mhlv3HeqMvIhJvbzqWpwK9OrjPE4C8oOW1wNIo3pcO189Yh4C153MkFp3ic0QJGhEREZHM0xMYEkP7/R3LX7bQLrjLeyw3ywNjaCsikggLCR1+1BU4u4P7vNix/FQrbdt7/XRrNqQDOtg+UqLKOWwqHX4PSaUEjYiIiEhmmhJD22Mcy++20G5r0M97xbD/CTG0bXQs58bwXhGRlviAPzrWXUf7Z3M6BpgWtOwHbm+lffD1cxDRfxc/PIaYGhzLHbl+9gcOjLKtRehnzirgmwjtthE63Xm0nyMWcFiUbSH0cyStPkOUoBERERHJTBdE2W4Qppt+s49oeSrw4Cei+xFdEcjBwLFRxgLhdQv6xfBeEZHW3Efo7ESDMEmVWIYcgZmh6K+OdU/R+vCm4G3diK6HSi5wbgxxxfv6Ge3nyFRgn6Dll1po5weqgpajTT6dSGw9MbcH/ZxWnyFK0IiIiIhkpiOAM6NodxOhTxjvaqXte0E/5wLntbFvC7gDyI4ijmZfO5YLYniviEhrtgMXAXbQuvOB24g+SdMbeBUYHrRuHfCbNt73nmP5p1Ec6/fElphwXj+HxvDeSH4BjG6jTS7mcyTY3a20D/49HEjbxZp7ALe20cZpVdDPvdGU3yIiIiIZbT/MDX7z69U22r8f1NZZqLI1RzuOc1IrbRc42tqYp6mTW2jvAW5wtF9G6zUBBmO6jje33wQc1ELb7sBDEWKyaf2GfJSj7SuEzw4iItIRtxN+XXodGNPG+04BvnK8rxE4Lopj5gDrg97XgOkZEkk25vrsjxDn+a0cIxdT56W57WfEluDZEeF4VUBhC+27Ak872v+7jWOc4mi/GNijhbZ7A+9EiMluoX2zcxxtbwG8bbxHRERERNJUqidolmCGKtmYugsPYmol7IvpVv9j4C3HvhuJrm6NM+lSC/wZ+CEwqem/NwJrmrbXA3c63tPWE9P/Odpvx9TF+TfweNDriCjiFRFx8mDq0Ti/9Psx159STE+bM4BfY3pwVEdoX4u55kWrxPF+H/AIcBbm+nkS8DtMofbmNn9yvKe1BA2Y631w+13Ah8DzhF4/p0d4b3CCZg4mCd+c7L8JOArTc+i7wGWYpH7wsTZjkiqtyQI+cbxvPSYhdULT7+Ec4B6gpmn72qaYo03Q9MDUu3Ee401MQin496Ai9iIiIiJpLtUTNIuAEcBGx/tbevmIvtZAP8wT1Wj3ezHwI8f6thI044j8JNf5OivKmEVEIrmCQBIg1tcq4MgYj5eN+QyI9hg3A2Md69pK0AwCVkex7+sjvDf4unsv5jOnPspYtxF9Id/vYhLv0ex3OyYZf7NjfVvOx3wGtbX/4S3twA2qQSMiIiKSmZZiEh3OugdOX2O62d8f5X6/BSbSdqJpPabXzj1R7jfYQuBg4GHCi16KiMTLH4CRmOtfXZTv+RbT22MMsSXcwQxrOg54oo12OzE9eK6Kcf9gei9+B9PzZkMbbdvyHGaY7Ko22r2Hma3vnSj3+z6msPCyNtp9gkno/zfK/Qb7O6ZX6EuEz26VsmKtVi0iIiIipiZK8OwTm4APWml/KNCz6ectmJvTaORjkizNPsJ8OYhkAabHDcDHmKeuYO73JmIKBhcCe2KedK7A3Hw/i/ky0B7HYuoJ7A/0xXRvXwm8iOlGvqup3QBCZyx5tymGaHiAYU376OLYtpiWZ5wSEYlFN8zU2VMwMxLtgblub8RcZ77EfNn/H6ZnRkcdCpyG6U2yByYZvQqYD/wD81kBpp5XcM+UTzFJmGgNxgw7ctbyqgaWO9btCGp3H4FCxrmY4UcnA0MwPSk3AZ8DzwBzad/vJAfzOzgec53vhSm4XIUZfjSPwLTcI5qO3WxuDMfJxiTi+hFetP4t2v8ZKCIiIiISkXOIk4iISCycQ5wkyTTESURERERERETEZRriJCIiItKyHsAbbgcRpUJMV3gwQ4uWuBiLG2ZgpuQWEWlLLmaokoQ6gEAnjk2YGmWdwVQ6Xq8nLrLcDkBEREQkhXkJ1HJJJ3mkZ9wdke92ACKSNjx0vmtkrPo0vToDZ10a1yhBIyIiItKyHZgna+mgeTpWMAUWf+ZiLG74xO0ARCRt1JE+1/Zkeg6T4AdTEPlWF2NJppaK74tIgnXBVO8eBhyEqQa+J4GZNURERNKVigSLiEhHqEiwy9SDRjLZYGASZhrUkcAoYK9W2m/DTJ/3JfAh8FrTf+MxjZ6IiIiIiIhIi1QkWDLN/sD5wKnA8Djsbwum4ODDwMtAYxz2KSIikgiXYHqIAqwFbnMxFhERST83ADlNP78HPOViLCKSprIxSZmFBLrkJeK1DigHBiTntEREREREREREUl8WpgBiNYlNzDhfO4E/AgMTf4oiIiIiIiLSGWiIk6SrCcBfaHt6PBszq8N7mNoySzBz3NcA24FcoBumcPBQTK2ascBhQNc29r0V+D/gDlSnRkREREREREQ6kS6YxIyflnu41AFPA9OB/u08Tg4wETO13JpWjmUDLine truncated
+    "g32490.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAABGgAAAGgCAYAAAD2CVFOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15fBT1/cfx12xOwhkOueQI4RBE8SdaFE9ArPctaq1atV6t9apKgtpGC0nwqtVa7/uot61H1aKgxVtR8UJEElCRS25IyLE7vz++SXZ3dpPsJrs7u5v38/HYh5mZ7858JobZ2c98v5+vhYiIiIhEqzdwdsDyMuBJd0IREZEYGAMcARwEDAR2AHoAaxte3wCvAv8BVrsUo6Q5y+0ARERERFLQzsCXActzgINdiiXdTADGBSzfB3hdikVE0t8E4EZg3wjbe4EHgD8BK+MVVBuNBA4MWH4O+NmdUFx1BDCg4ef1wDMuxiIiIiIicbYzYAe8/utuOGllNsG/21x3wxGRNJUJ/APwEXzNifS1BfhVwqNu2W8IjnF3V6Nxzxv4fwdfuBxLVDLdDkBEREREREQkgToBTwBHOdZvAl7ADGX6AdgK9AKGA8cCk/F/h+4CPIIZCnVL/EMWEREREZFw1IMmftSDRkTi7WmCrzM+4B6gZyvv2wV4l9DeNGfELdLo/Ab1oIEU7kHjcTsAERERERERkQQ5FzghYNkGLgLOwdQrackXwAHAvx3rbwdGxSpA6biUoBEREREREZGOYAfgZse6MuDvUeyjDjgJ+ChgXWdMPRuRdlENGhEREZH4+wXQreHnjcDHAdsKgdOB0cCOQC2m9sE8zMwTmyM8xnggv+HnrcD7Adv+DzgZGAH0a9i+HHgRU2uhNsJj7AfkNPy8msi7jvfFDA1o9CHB55UHTGz4eajjvZNbiO9LYFWEMYiIXIBJpjT6BriuDfupwfTE+Qj/d+rJmCFFnzTznt3xD6FyXqNb0gXYK2D5K0Jnj5qCmaF5jGP9BJoftlUJLHWsmwRkNPz8E/B1w8+ehmMcDQzBJLrWAkuAZ4F3MD2RWpOD+RxptATzWRSJEQ3HbvS6Y3s/YGzDz4Hn3AUzdXpz3gWqIoxBRERERJJQtDVoPg5o+3bDuq6YApMtzSCymuCu+C2ZF/C+hQ3r+mNuYluaiWQJwdOytuTHgPf9M8L3gEkOBR5zT8f2Ea3E2NzrlChiEJGOLQeT0I3lNeRxx/4ebqHtnIB2n0VxjHGOY5wepk0t0V8/rw2zn60B2+9tWLc7sLiVfb2FuY63ZoDjfX+M4D2NbnS81+mUVmJs7jU8ihjiTkOcRERERBKvD6YXya8xTz2bswPwJCbBEa2hmCe5U1ppNxyTxGnLMUREUsUETG++RuuB59u5z/scy0eQXt+xp2J6x4xspd3+wAeY3prSDhriJCIiIpJYWZgZRHbCPL17DfgP8D0mWbMLcDb+rtwe4A7gTSIfzpONSez0azjGs5gpZSswsyLtCpyH/2Y6A/PkdwUwv60n1g4/A0UNPx+KKcLZ6Gqgvpn3RfMUWkQ6tn0dy3OA7e3c5zxgC6ZHJJhhpjuT+JmDrsJ8VowHTgxYfxvmuh7OO63scyDwKOYzYzsmGfUSZnhVN8zv8wJgUEP7fMzn2a64M/T0M/yfI+dghg+D6Yn61xbety6eQYmIiIhI/LVniFPjayX+uitOnTC1YQLb/7mVYwQOcWp8bcc80Q0nE1MsM7D9N7Q8rXW8hjgF0jTbIhIPLxJ8bbk8Rvt907Hf85ppF88hTo1+42gb7TTbgUOcGl8rMJ954XQHXnG0f7qF/cdziFMgTbMtIiIiIhGrBg7GFCdsbvvpmCezjU5qw3F+h3niGU495uY4sIv/KOC0NhxHRCTZ7ehY/jJG+3Xux3mcVFaPSfJ/1cz2TZg6aYsD1p0A7BbnuNKWEjQiIiIiiXcLrT/VW0No8qRbM23D+RB4oJU2NnAxwbMknRvFMUREUoVzNqMNMdrv+laOk8ruBj5tpc024ArHOn2OtJESNCIiIiKJ5yws2Zz3An724B9TH4n7ab0bOJgpvV8LWN6D4EKaIiLpIN+xvClG+3XuJ50SNJF+Vr1McK2bw+MQS4egBI2IiIhIYq0ElkbY9kfHco8ojjMnirbOGjot1YgREUlFPsdySzPoRSPDseyN0X7dtg4zE2AkfJjZABsNBvrHPKIOQAkaERERkcSKZnaLbY7lzhG+bytQGcVxnMOtRkXxXhGRVOAc0hRNwrslzqGnsRo65bZoi+s624+IVSAdiRI0IiIiIolVFUVb5xClSJ/4/hzmvS1Z41h2DgUQEUl1zloxvWK03z6tHCdVrY2yvT5HYkAJGhEREZH0E00SCEyPm0BdYxWIiEiSWOZYHhej/Tr3E03vxWSmzxEXKEEjIiIikn46RdneOXTKeaMtIpLq3nEsx6LWVjahCRrncVJVXpTt9TkSA0rQiIiIiKSfXkRXANPZRT+SGgrR7D/aG30RkVh727E8Fejezn0eDuQGLK8ClkTwvlS4fkY7BKwtnyPR6BCfI0rQiIiIiKSfbsCQKNqPdSx/20y7wC7v0dws94uirYhIPCwgePhRHvCrdu7zHMfysy20bev1063ZkHZpZ/twiSrnsKlU+D0klBI0IiIiIulpShRtD3Isf9hMu00BPw+MYv8To2hb71jOieK9IiLN8QK3OtZdQ9tnczoIODRg2Qf8rYX2gdfP/kT+XXzvKGKqcyy35/rZB9g1wrYWwZ85PwI/hWm3meDpziP9HLGAvSJsC8GfIyn1GaIEjYiIiEh6OjPCdv0x3fQbfUbzU4EHPhHdmciKQA4CDo4wFgitW9A7iveKiLTkPoJnJ+qPSapEM+QIzAxFdzjWPUvLw5sCt3Umsh4qOcCvo4gr1tfPSD9HpgKDA5ZfaaadD6gIWI40+XQE0fXE3BLwc0p9hihBIyIiIpKe9gFOiqDdDQQ/YbyrhbYfBfycA5zWyr4t4HYgK4I4Gn3vWC6I4r0iIi3ZApwN2AHrTgduIfIkTQ/gv8DwgHWrgT+08r6PHMu/jeBYfyG6xMQPjuWhUbw3nAuA0a20ycF8jgS6u4X2gb+HXWm9WHNX4OZW2jj9GPBzDzTlt4iIiEha2xlzg9/4+m8r7T8OaOssVNmSAx3HObKFtvMcbW3M09TJzbT3ANc52n9HyzUBBmG6jje2Xw/s1kzbLsAjYWKyafmGfJSj7WuEzg4iItIetxJ6XXoLGNPK+47GJJED31cPHBLBMbOBNQHvqyO492KgLMz12RcmztNbOEYOps5LY9uviS7BszXM8SqAwmba5wHPOdr/u5VjHO1o/yWwQzNtdwTeDxOT3Uz7Rqc62t4EZLTyHhERERFJUcmeoFmMGapkY+ouPISplbATplv9GcC7jn3XE1ndGmfSpRr4O3AcMKnhv9cDKxu21wJ3Ot7T2hPTDxztt2Dq4vwbeCrgtU8E8YqIOHmA2wj90u/DXH9KMD1tTgQuxPTgqAzTvhpzzYvUDMf7vcBjwCmY6+eRwJ8xhdob2zjjbClBA+Z6H9h+O/Ap8BLB189pYd4bmKCZg0nCNyb7bwAOwPQc2gO4GJPUDzzWBkxSpSWZwBeO963BJKQOb/g9nArcA2xr2L6qIeZIEzRdMfVunMd4G5NQCvw9qIi9iIiISIpL9gTNQmAEsM7x/uZeXiKvNdAb80Q10v2eA5zsWN9agmY84Z/kOl+nRBiziEg4l+FPAkT7+hHYN8rjZWE+AyI9xo3AOMe61hI0/YEVEez72jDvDbzu3ov5zKmNMNbNRF7Idw9M4j2S/W7BJONvdKxvzemYz6DW9j+8uR24QTVoRERERNLTEkyiw1n3wOkHTAHGByLc78/A/rSeaFqD6bVzT4T7DbQA2B14lNCilyIisXIzMBJz/auJ8D0/Y3p7jCG6hDuYYU2HAE+30q4K04Pn8ij3D6b34v9het6sbaVta17EDJP9sZV2H2Fm63s/wv1+jCks/F0r7b7AJPTfiXC/gR7G9Ap9hdDZrZJWtNWqRURERMTURAmcfWI98EkL7X8BdGv4eSPm5jQS+ZgkS6PPMF8OwpmH6XED8DnmqSuY+739MQWDC4EBmCedyzA33y9gvgy0xcGYegJjgV6Y7u3Lgf9gupFvb2jXl+AZSz5siCESHmBYwz46ObZ9SfMzTomIRKMzZursKZgZiXbAXLfXYa4z32K+7H+A6ZnRXr8Ajsf0JtkBk4z+EZgL/BPzWQGmnldgz5SvMEmYSA3CDDty1vKqBJY61m0NaHcf/kLGOZjhR0cBQzA9KdcDizDX+jdo2+8kGzgBk8wfBnTHFFyuwAw/egP/tNwjGo7d6PUojpOFScT1JrRo/bu0/TNQRERERCQs5xAnERGRaDiHOEmCaYiTiIiIiIiIiIjLNMRJREREpHldgfluBxGhQkxXeDBDixa7GIsbpmOm5BYRaU0OZqiSBNsFfyeO9ZgaZR3BVNpfrycmMt0OQERERCSJZeCv5ZJKcknNuNsj3+0ARCRleOh418ho9Wx4dQTOujSuUYJGREREpHlbMU/WUkHjdKxgCiye52IsbvjC7QBEJGXUkDrX9kR6EZPgB1MQ+WYXY0mk5orvi0icdcJU7x4G7IapBj4A/8waIiIiqUpFgkVEpD1UJNhl6kEj6WwQMAkzDepIYBQwsIX2mzHT530LfAq82fDfWEyjJyIiIiIiItIsFQmWdDMWOB04Fhgeg/1txBQcfBR4FaiPwT5FRETi4VxMD1GAVcAtLsYiIiKp5zogu+Hnj4BnXYxFRFJUFnAGsAB/l7x4vFYDZUDfxJyWiIiIiIiIiEjyy8QUQKwkvokZ56sKuBXoF/9TFBERERERkY5AQ5wkVU0E/kHr0+PZmFkdPsLUllmMmeN+G7AFyAE6YwoHD8XUqhkH7AXktbLvTcCfgNtRnRoRERERERER6UA6YRIzPprv4VIDPAeciEm8tEU2sD9marmVLRzLBj4BLine truncated
     }
    },
    "cell_type": "markdown",
-   "id": "7605a53f",
+   "id": "39db374f",
    "metadata": {},
    "source": [
     "<div>\n",
-    "<img src=\"attachment:g37385.png\"  align=\"left\" width=\"450\"/>\n",
+    "<img src=\"attachment:g32490.png\"  align=\"left\" width=\"450\"/>\n",
     "</div>"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "ade31d26",
+   "metadata": {},
+   "source": [
+    "The ASP problem consists in computing the minimum distance between any two pair of nodes $i$ and $j$ in the graph. All these values can be also represented as a matrix (labeled as \"output\" in the figure above). For instance, the minimum distance from node 2 to node 3 is 8 as highlighted in the figure. You can understand both input and output matrices as distance tables. The key difference is that the input contains the distance using direct connections only, whereas the output contains the (minimum) distance allowing indirect connections."
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "d9979e39",
@@ -131,7 +160,7 @@
    "id": "b27707b9",
    "metadata": {},
    "source": [
-    "You can check that this function computes the solution of the small ASP problem in the figure above by executing next cell. Note that we used a large number to represent that there is no direct connection between two nodes."
+    "You can check that this function computes the solution of the small ASP problem in the figure above by executing next cell."
    ]
   },
   {
@@ -151,6 +180,21 @@
     "floyd!(C)"
    ]
   },
+  {
+   "attachments": {
+    "g32490.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAABGgAAAGgCAYAAAD2CVFOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15fBT1/cfx12xOwhkOueQI4RBE8SdaFE9ArPctaq1atV6t9apKgtpGC0nwqtVa7/uot61H1aKgxVtR8UJEElCRS25IyLE7vz++SXZ3dpPsJrs7u5v38/HYh5mZ7858JobZ2c98v5+vhYiIiIhEqzdwdsDyMuBJd0IREZEYGAMcARwEDAR2AHoAaxte3wCvAv8BVrsUo6Q5y+0ARERERFLQzsCXActzgINdiiXdTADGBSzfB3hdikVE0t8E4EZg3wjbe4EHgD8BK+MVVBuNBA4MWH4O+NmdUFx1BDCg4ef1wDMuxiIiIiIicbYzYAe8/utuOGllNsG/21x3wxGRNJUJ/APwEXzNifS1BfhVwqNu2W8IjnF3V6Nxzxv4fwdfuBxLVDLdDkBEREREREQkgToBTwBHOdZvAl7ADGX6AdgK9AKGA8cCk/F/h+4CPIIZCnVL/EMWEREREZFw1IMmftSDRkTi7WmCrzM+4B6gZyvv2wV4l9DeNGfELdLo/Ab1oIEU7kHjcTsAERERERERkQQ5FzghYNkGLgLOwdQrackXwAHAvx3rbwdGxSpA6biUoBEREREREZGOYAfgZse6MuDvUeyjDjgJ+ChgXWdMPRuRdlENGhEREZH4+wXQreHnjcDHAdsKgdOB0cCOQC2m9sE8zMwTmyM8xnggv+HnrcD7Adv+DzgZGAH0a9i+HHgRU2uhNsJj7AfkNPy8msi7jvfFDA1o9CHB55UHTGz4eajjvZNbiO9LYFWEMYiIXIBJpjT6BriuDfupwfTE+Qj/d+rJmCFFnzTznt3xD6FyXqNb0gXYK2D5K0Jnj5qCmaF5jGP9BJoftlUJLHWsmwRkNPz8E/B1w8+ehmMcDQzBJLrWAkuAZ4F3MD2RWpOD+RxptATzWRSJEQ3HbvS6Y3s/YGzDz4Hn3AUzdXpz3gWqIoxBRERERJJQtDVoPg5o+3bDuq6YApMtzSCymuCu+C2ZF/C+hQ3r+mNuYluaiWQJwdOytuTHgPf9M8L3gEkOBR5zT8f2Ea3E2NzrlChiEJGOLQeT0I3lNeRxx/4ebqHtnIB2n0VxjHGOY5wepk0t0V8/rw2zn60B2+9tWLc7sLiVfb2FuY63ZoDjfX+M4D2NbnS81+mUVmJs7jU8ihjiTkOcRERERBKvD6YXya8xTz2bswPwJCbBEa2hmCe5U1ppNxyTxGnLMUREUsUETG++RuuB59u5z/scy0eQXt+xp2J6x4xspd3+wAeY3prSDhriJCIiIpJYWZgZRHbCPL17DfgP8D0mWbMLcDb+rtwe4A7gTSIfzpONSez0azjGs5gpZSswsyLtCpyH/2Y6A/PkdwUwv60n1g4/A0UNPx+KKcLZ6Gqgvpn3RfMUWkQ6tn0dy3OA7e3c5zxgC6ZHJJhhpjuT+JmDrsJ8VowHTgxYfxvmuh7OO63scyDwKOYzYzsmGfUSZnhVN8zv8wJgUEP7fMzn2a64M/T0M/yfI+dghg+D6Yn61xbety6eQYmIiIhI/LVniFPjayX+uitOnTC1YQLb/7mVYwQOcWp8bcc80Q0nE1MsM7D9N7Q8rXW8hjgF0jTbIhIPLxJ8bbk8Rvt907Hf85ppF88hTo1+42gb7TTbgUOcGl8rMJ954XQHXnG0f7qF/cdziFMgTbMtIiIiIhGrBg7GFCdsbvvpmCezjU5qw3F+h3niGU495uY4sIv/KOC0NhxHRCTZ7ehY/jJG+3Xux3mcVFaPSfJ/1cz2TZg6aYsD1p0A7BbnuNKWEjQiIiIiiXcLrT/VW0No8qRbM23D+RB4oJU2NnAxwbMknRvFMUREUoVzNqMNMdrv+laOk8ruBj5tpc024ArHOn2OtJESNCIiIiKJ5yws2Zz3An724B9TH4n7ab0bOJgpvV8LWN6D4EKaIiLpIN+xvClG+3XuJ50SNJF+Vr1McK2bw+MQS4egBI2IiIhIYq0ElkbY9kfHco8ojjMnirbOGjot1YgREUlFPsdySzPoRSPDseyN0X7dtg4zE2AkfJjZABsNBvrHPKIOQAkaERERkcSKZnaLbY7lzhG+bytQGcVxnMOtRkXxXhGRVOAc0hRNwrslzqGnsRo65bZoi+s624+IVSAdiRI0IiIiIolVFUVb5xClSJ/4/hzmvS1Z41h2DgUQEUl1zloxvWK03z6tHCdVrY2yvT5HYkAJGhEREZH0E00SCEyPm0BdYxWIiEiSWOZYHhej/Tr3E03vxWSmzxEXKEEjIiIikn46RdneOXTKeaMtIpLq3nEsx6LWVjahCRrncVJVXpTt9TkSA0rQiIiIiKSfXkRXANPZRT+SGgrR7D/aG30RkVh727E8Fejezn0eDuQGLK8ClkTwvlS4fkY7BKwtnyPR6BCfI0rQiIiIiKSfbsCQKNqPdSx/20y7wC7v0dws94uirYhIPCwgePhRHvCrdu7zHMfysy20bev1063ZkHZpZ/twiSrnsKlU+D0klBI0IiIiIulpShRtD3Isf9hMu00BPw+MYv8To2hb71jOieK9IiLN8QK3OtZdQ9tnczoIODRg2Qf8rYX2gdfP/kT+XXzvKGKqcyy35/rZB9g1wrYWwZ85PwI/hWm3meDpziP9HLGAvSJsC8GfIyn1GaIEjYiIiEh6OjPCdv0x3fQbfUbzU4EHPhHdmciKQA4CDo4wFgitW9A7iveKiLTkPoJnJ+qPSapEM+QIzAxFdzjWPUvLw5sCt3Umsh4qOcCvo4gr1tfPSD9HpgKDA5ZfaaadD6gIWI40+XQE0fXE3BLwc0p9hihBIyIiIpKe9gFOiqDdDQQ/YbyrhbYfBfycA5zWyr4t4HYgK4I4Gn3vWC6I4r0iIi3ZApwN2AHrTgduIfIkTQ/gv8DwgHWrgT+08r6PHMu/jeBYfyG6xMQPjuWhUbw3nAuA0a20ycF8jgS6u4X2gb+HXWm9WHNX4OZW2jj9GPBzDzTlt4iIiEha2xlzg9/4+m8r7T8OaOssVNmSAx3HObKFtvMcbW3M09TJzbT3ANc52n9HyzUBBmG6jje2Xw/s1kzbLsAjYWKyafmGfJSj7WuEzg4iItIetxJ6XXoLGNPK+47GJJED31cPHBLBMbOBNQHvqyO492KgLMz12RcmztNbOEYOps5LY9uviS7BszXM8SqAwmba5wHPOdr/u5VjHO1o/yWwQzNtdwTeDxOT3Uz7Rqc62t4EZLTyHhERERFJUcmeoFmMGapkY+ouPISplbATplv9GcC7jn3XE1ndGmfSpRr4O3AcMKnhv9cDKxu21wJ3Ot7T2hPTDxztt2Dq4vwbeCrgtU8E8YqIOHmA2wj90u/DXH9KMD1tTgQuxPTgqAzTvhpzzYvUDMf7vcBjwCmY6+eRwJ8xhdob2zjjbClBA+Z6H9h+O/Ap8BLB189pYd4bmKCZg0nCNyb7bwAOwPQc2gO4GJPUDzzWBkxSpSWZwBeO963BJKQOb/g9nArcA2xr2L6qIeZIEzRdMfVunMd4G5NQCvw9qIi9iIiISIpL9gTNQmAEsM7x/uZeXiKvNdAb80Q10v2eA5zsWN9agmY84Z/kOl+nRBiziEg4l+FPAkT7+hHYN8rjZWE+AyI9xo3AOMe61hI0/YEVEez72jDvDbzu3ov5zKmNMNbNRF7Idw9M4j2S/W7BJONvdKxvzemYz6DW9j+8uR24QTVoRERERNLTEkyiw1n3wOkHTAHGByLc78/A/rSeaFqD6bVzT4T7DbQA2B14lNCilyIisXIzMBJz/auJ8D0/Y3p7jCG6hDuYYU2HAE+30q4K04Pn8ij3D6b34v9het6sbaVta17EDJP9sZV2H2Fm63s/wv1+jCks/F0r7b7AJPTfiXC/gR7G9Ap9hdDZrZJWtNWqRURERMTURAmcfWI98EkL7X8BdGv4eSPm5jQS+ZgkS6PPMF8OwpmH6XED8DnmqSuY+739MQWDC4EBmCedyzA33y9gvgy0xcGYegJjgV6Y7u3Lgf9gupFvb2jXl+AZSz5siCESHmBYwz46ObZ9SfMzTomIRKMzZursKZgZiXbAXLfXYa4z32K+7H+A6ZnRXr8Ajsf0JtkBk4z+EZgL/BPzWQGmnldgz5SvMEmYSA3CDDty1vKqBJY61m0NaHcf/kLGOZjhR0cBQzA9KdcDizDX+jdo2+8kGzgBk8wfBnTHFFyuwAw/egP/tNwjGo7d6PUojpOFScT1JrRo/bu0/TNQRERERCQs5xAnERGRaDiHOEmCaYiTiIiIiIiIiIjLNMRJREREpHldgfluBxGhQkxXeDBDixa7GIsbpmOm5BYRaU0OZqiSBNsFfyeO9ZgaZR3BVNpfrycmMt0OQERERCSJZeCv5ZJKcknNuNsj3+0ARCRleOh418ho9Wx4dQTOujSuUYJGREREpHlbMU/WUkHjdKxgCiye52IsbvjC7QBEJGXUkDrX9kR6EZPgB1MQ+WYXY0mk5orvi0icdcJU7x4G7IapBj4A/8waIiIiqUpFgkVEpD1UJNhl6kEj6WwQMAkzDepIYBQwsIX2mzHT530LfAq82fDfWEyjJyIiIiIiItIsFQmWdDMWOB04Fhgeg/1txBQcfBR4FaiPwT5FRETi4VxMD1GAVcAtLsYiIiKp5zogu+Hnj4BnXYxFRFJUFnAGsAB/l7x4vFYDZUDfxJyWiIiIiIiIiEjyy8QUQKwkvokZ56sKuBXoF/9TFBERERERkY5AQ5wkVU0E/kHr0+PZmFkdPsLUllmMmeN+G7AFyAE6YwoHD8XUqhkH7AXktbLvTcCfgNtRnRoRERERERER6UA6YRIzPprv4VIDPAeciEm8tEU2sD9marmVLRzLBj4BLine truncated
+    }
+   },
+   "cell_type": "markdown",
+   "id": "c0804b07",
+   "metadata": {},
+   "source": [
+    "<div>\n",
+    "<img src=\"attachment:g32490.png\"  align=\"left\" width=\"450\"/>\n",
+    "</div>"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "b9eeb64d",
@@ -158,7 +202,7 @@
    "source": [
     "### The algorithm explained\n",
     "\n",
-    "The main idea of the algorithm is to perform as many iterations as nodes in the graph. At iteration $k$, we update the distance matrix $C$ by finding the shortest paths between each pair of nodes, allowing indirect paths via nodes from 1 to $k$. At the last iteration, it is allowed to visit all nodes and, thus, the distance table will contain the solution of the ASP problem.\n",
+    "The main idea of the  Floyd–Warshall algorithm is to perform as many iterations as nodes in the graph. At iteration $k$, we update the distance matrix $C$ by finding the shortest paths between each pair of nodes, allowing indirect paths via nodes from 1 to $k$. At the last iteration, it is allowed to visit all nodes and, thus, the distance table will contain the minimum possible distances, i.e. the solution of the ASP problem.\n",
     "\n",
     "This process is cleverly done with three nested loops:\n",
     "\n",
@@ -173,7 +217,7 @@
     "end\n",
     "```\n",
     "\n",
-    "At each outer iteration $k$, we do a loop over the distance matrix $C$. For each pair of nodes  $i$ and $j$ we compare the current distance $C_{ij}$ against the distance via node $k$, namely $C_{ik}+C_{kj}$, and update  $C_{ij}$ with the minimum."
+    "At each outer iteration $k$, we do a loop over the distance matrix $C$. For each pair of nodes  $i$ and $j$ we compare the current distance $C_{ij}$ against the distance via node $k$, namely $C_{ik}+C_{kj}$, and update  $C_{ij}$ with the minimum. I.e., at iteration $k$ one checks if it is beneficial to visit node $k$ to reduce the distance between nodes $i$ and $j$."
    ]
   },
   {
@@ -196,62 +240,78 @@
    "id": "722e330c",
    "metadata": {},
    "source": [
-    "The update of the distance matrix at each iteration is illustrated in the next figure for a small ASP presented above. We highlight in green the distances that are updated in each iteration. Note that adding values to the table can be interpreted as adding new connections in the graph, as illustrated in the figure."
+    "The update of the distance matrix at each iteration is illustrated in the next figure for the small ASP presented above. We highlight in green the distances that are updated in each iteration. Note that some distances that were initially infinity (i.e., no connection) will be updated with a finite value in this process. You can understand this as adding new edges in the graph as illustrated in the figures below."
    ]
   },
   {
    "attachments": {
-    "g38106.png": {
-     "image/png": "iVBORw0KGgoAAAANSUhEUgAABGgAAAGhCAYAAAA9VYLrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13nBT1/cfx114vcPQOAkoURFEpKgooaGLvoljQkKixYie/2DUaRU0sMTE2ogY1auyKIHbRaBARaSKKICC9Hndc3fn98d29nZ3bu9u9292Z3X0/H495sDM3u/eZO2525r3fAiIiIiIiIiLSHPcDlm3xuVuOpLIctwsQERERSUG7Ai/Y1r8ALnGpFhERkWRpCwwBhgb+3dX2tRrgQDeKShcKaERERERiV4i5MA3a7FYhaeg84HTb+jFAlUu1iIgI/Ab4FeZ9r18j+1Unp5z0pYBGRERERLykH3C4bT3LrUJERASAS4H93C4iE+gNT0RERERERERitQq1cIwrtaARERERERERkcasAebYltnAWmA50Nu9stKLAhoRERERERERachoYJvbRWQCdXESERER8Y4OQKsEf4+OQEGCv4eIiKQPhTNJohY0IiIiIol3PbBL4PEPwN2Bx22AcwPLnoSCk1LgXeBhYGaU3+NqYPfA49XAbYHH3TEzI40DdgPyAtvXAW8AfwfmRvk9JmOmWAXTvP3xKJ83LFBD0J2YZvFBnYE/Bh4f4HjuQ0BtA6/7eKAOEZFU4gMmYs77Qcsx51i/GwWJiIiIiKSqgYBlW95pYv8vbfvOCmw7EHNBbjWx/JPoPlT7wPaceYFtJ2CmAG/s9WswNwXRtKxeZXvec1HsHzTO8T2HOb7er4kaG1rOiKEGEZFEuJ/w85Kvif1zgEccz5kH9ExgjYm0nNBxaMDgFlILGhEREZHkOwh4H8iPYt9fAzuAy2L8HscArxC6WagB1mNa0HS07ZcNTApsOw9zkS0iIvHXCngeONq27V3gFGC7KxWJp2gMGhEREZHk6gy8hAlnVgCXArsCuYFtg4EpjudcDAyN4Xt0BJ7ChDNrMSFPW6AH0AnohelmVGN7zm8CtbhhJeb4hmLqtjvI9jXnMiOJNYqItEQ34CPCw5knA+sKZ0REREREmqklXZyCy0waHxD4Jsf+TY33Yu/iFFx+ALo28pwjMU3Sg/tvx4Q4DUlUFye7uxz7akBjEfGyaLo4DcQE8sF9/MDNUbx2B0w31UQuT8R2uPUsR12cRERERMRFLQ1olmEGCG5MDvCj7Tlrm9jfGdDUAkOaeA7AjY7n/bGRfRXQiIiEayqgGQ1ssX29Grggytfu6HjtRCz/ju1w61luey0FNC2kLk4iIiIiyfcnmp62tAbTFSqoC2ZGpmi9CMyJYr97gE229d/Q9CCXIiLStFOBaYRmv9uBGbz9UdcqEk/TIMEiIiIiyeUnPHhpzNeO9V7Az1E+99ko96sI1BP8RLc78AvguyifLyIi9f0eM9ZXMPBegxm8fW4Mr1EKnBbnupxWJvj1JQYKaERERESS60dMc/dobHSsN9Utyu6zGPb9lPAm9/ujgEZEpDmyMd2eLrFtW4QZDHhFjK9ViWkNKRlCAY2IiIhIcm2OYd8Kx3o003KDCXac4U5jFjvWe8bwXBERCXkZOM62/jFwItEH85LBFNCIiIiIJFcyBlHc2sL928WrEBGRDGMPZ9ZgWs6UuVSLpBgFNCIiIiLppzrG/Z2hUbQtdUREJNwSYI/A427Ak8BZNC+c9xEaYDhRqlCA5BkKaERERETST6sY92/tWI+1BU5TNCuUiGSKQ4B3gEGB9VOBYuAUYGeMr9UB2BC/0iJ6HhiX4O8hUdI02yIiIiLppzOQG8P+zjFnNkXcy8xAFZQdw+uXxLCviEgqWw8cCnxh23YU8Daxh+eSYRTQiIiIiKSffGDPGPbfz7H+TQP77bA9juVGo28M+4qIpLotwBGEz6Z3CCakUWAtDVIXJxEREZH0dDwwL8p9T7A9rgHmNLCffQaqfjHUcngM+1Y61ltRfzYrERGv24YJad7AtKgBGIHp/nQU0c3qtBUYmojibDS7lIcooBERERFJT+cB99L0mAcHAvvb1qcR3lLGbj5wcOBxP6A3sKKJ1x8FDGliH7vtjvVOxDZluIiIV+wAjgRewITmAAcA7wO/ounxZRoLzCUNqYuTiIiISHraBbi9iX2Kgb8TPojvPxrZ/1PbYx9wdROv3wF4ool9nL5zrO8b4/NFRLykEhgLvGzbti/wMdDdlYrEs9SCRkRERCT9VGMGCb4q8Pgm6k/x2hOYSvj4M9MxYyQ05BVMC5fgGAqXAD8CDxA+gDDASOBxTEubcqAoyto/t9UPcCemCf5XmME3RURSTRVmpqSnCc2Y1B/4ANMFdKVLdUWrHQ23hCywPfbRcJfWBcDaeBYlIiIiIgIwELBsyztN7P+lbd9ZMXyfQx3f57hG9v3Att88zI1AcH0l8FfgcuD3wHNAmeO112K6LDXlWsfzLGAp8BhwV+Df+bavrQKudOw/rInv8UyE72FhxqLZbFtOiaJeEZFEup/w85SvkX2zMa0K7fsvJ7YxvdxwKJHPybEsZyS76FSkFjQiIiIi6ekiTOAyCtNa5tJG9l2P+dSzqfFkAP4CDAdOsm3rR+QbjNWYcRd2j+J17SYCe1D/E9v8wBKUF+Prioi4qRYzPlgpJjAHc57+BHMOXuhSXeIRGoNGREREJD2VAYcBd2BmAomkBtPNaS9M8/No1AKnATdgbjIiqcYMijkY0zUpVpswIdBZwEvAD418LxGRVGJhWhX+2batK2bg4EGuVCSe0VjzKxERERFJHR8Qmsr1G2Af29cKMDOG7IoZlHIHZuyYt2nZDEnFwBhgb8wYBVsxrXDeQePFiIiIiIiIiEgGco5BIyIiIilEY9CIiIiINMyHaXqeCuzjseQC3dwqxCVbMIMIi4i4pSvqpbIV2Ol2ESIiIiKSftrS8pkrtCRnCU5dKyLillrcPxe6vUxo8U8xg6kFjYiIiEjDLMDvdhFR8hH+yW2q1B0vmXa8IuI9tW4X4AE6F4uIiIhIxvuA0CeYGoNGREQkxagFjWSKfGB3zOwVrTGzTrQFyjHTkJYCq4ElmKk9RURERERERJJGAY2kq92A0YFlONAbyIryuZuAucCHwPvAbKAm/iWKiIiIiIiIiKSfrsBVwNfEd6CrDcCDwLDkHYqIiEjMdgeGBJY9Xa5FRERERDLQQGAqUE3iRyX/CjiV6FvjiIiIiIiIiIiktd2AlzAjhSd7+rhFwAmJP0QREREREREREW8qAG7CDPCb7GDGubwB9E3s4YqIiIiIiIiIeMuewALcD2bsSxkwIZEHLSIiIiIiIiLiFb8GduB+INPQ8hRm+m4RERERERERkbTjA/6E+wFMNMsXQMfE/BhERERERERERNyRDTyG+8FLLMu3QO9E/DBERERERERERJLNBzyO+4FLc5afgF7x/5GIiIiIiIiIiCTXZNwPWlqyzAPaxv2nIiIiIiIiIiKSJL/D/YAlHst7mG5aIiIiIiIiIg3SjaN40d7Af4BctwuJg76YoOZDl+sQEREREREREYlaK2AJ7rd8iedSC4yO5w9JRERERERERCSR7sH9QCURy/dAQRx/TiIiIiIiIpJGfG4XIGIzEJhLnLs2dezYkaFDhzJkyBCGDBnC4MGDKSkpCdvn2GOP5bPPPovnt43kFuDWRH8TERERERERST05bhcgYvMgcQhnunbtyoQJExgyZAhDhw6ld+/eTT4nJycpfwr/B0wBVibjm4mIiIiIiIiIxOpA4tSd6IgjjrBiNWrUqGR1dXowMT8+ERERERERSWVZbhcgEnCj2wUkyXlAV7eLEBEREREREW9RFyfxggHAUYl6cb/fz9KlS/nyyy+ZM2cOAH/5y18S9e2aUghcBNzsVgEiIiIiIiLiPQpoxAvOIY4DVm/dupVnn32WOXPmMGfOHObOncv27dvrvj5y5Mh4favmGo8ZMNhyuQ4REREREREREcB0s/uJJE55PXLkSDfHoAkuo+L3IxQREREREZFUpzFoxG0HAr3cLsIFp7pdgIiIiIiIiHiHAhpx2xi3C3DJYW4XICIiIiIiIt6hgEbclqkBzQCgm9tFiIiIiIiIiDcooBE3ZWG6OGUiH3CQ20WIiIiIiIiINyigETf1xkw7nan6u12AiIiIiIiIeIMCGnHTHm4X4LLd3S5AREREREREvEEBjbhpN7cLcFk/twsQERERERERb1BAI25q63YBLmvjdgEiIiIiIiLiDQpoxE2t3C7AZSVuFyAiIiIiIiLekON2AZLRMj2gae12ASIikjBZwC+AAUBHzOx9W4DvgQVAjXultUhbYDhmFsZBQKfA0hEoxxzjPOAj4AVguztliogkVBtgH6AXUAzsBDYCc4D1LtbVElmY96wDA0svoCvm/F4AbAOWA18CLwNfuFKliCTMfYCV7GXkyJGW06hRo5JeB7AuXj9IEZEU0g04DrgFeBNYBWy2LUe6VlnLtQV+i7lw3UbD5/9S4DHMhXAqeRjwE/373A7g96TXB4KdgB8J/z8729WKRLwjFxgMXAA8ggkr7H8rn7hXWlwMAyYDX9P4ufBr4HeYn0eq6AtsJbZ7mc8wQX06uY7w/7ObgXGuViSSRLeR2QHN9/H6QYqIpIC7gdU0fW48zq0CW+hCzCeosbwPVABXu1FsM71E897vXgfyXag3EZ6n/vEtcbUiEfftAfwPc05r7Fwwz60CWygXWETs5745pM6kIHvSvPN7OXCEC/UmwgAiv4+fm8wi0ukTDUk9pW4X4LJMP34RySzDgO5uF5FAu2GagNuVA3Mxzd2rgF2AIUBe4Ov5wL1AEfDH5JQZF0uBdzHN25dijs+PaeY/CBgLHI/p1gUmdLsduDbplcbX8cBpbhch4kEdMef4dJVN5BaPy4DvMC0mCzDn9562rw8GPgBGACsSXGO87MS0dPoA0x13Line truncated
+    "g33734.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAABGgAAAGhCAYAAAA9VYLrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13fBR1/sfx16YnkITQq4CgUgQFRUVBxYq9oij2Xu6w40/Fgqc/9fTO8tPzFE/Fs3c9FWwgiu0EsVAUlaIgRGoIqZvs/v74ZrO7s7vJbrI7s7t5Px+PfZCZzO5+hiSzM+/5FhARERERERGRlrgP8AY8XM6WY6tewJ7AIcCuQGdnyxERERGRtmh7YH7A4yFnyxEREYe0tYCmPXADsJTg/fY9PgPOBzKdKlBERERE2pahBJ+QvudsOWnlfOD9gEeOs+WIiDSpLQU0ewI/Ez6YsT4+BrZzpszUleV0ASIiIiIiAQYCBwUsZzhViIiINBqOCc0LLet/B/4AehPcxWks8CGwF7DRjgLTgT7wRERERERERCSSYuA/BIczc4ARmHFoRgBdgIOBHwK2GQi8SHq3KoorBTQiIiIiIiIiEslVBHdXmgmMB76xbPcBMIbgkOYA4NiEVpdGFNCIiIiIiIiISDgdgcsDlrcC5wG1EbbfCJyLGYfG5y+JKS39KKARERERSR6dMDNkJFJnIC/B7yEiIunhGIK7Ns3AjDvTlM+ATwKWh2K6QUkzNEiwiIiISOLdgL95+C/AXxu+LgbObHgMwR+clGOaij+MGZQxGlcBOzZ8vQa4teHrnpi7nROBAfhnRSrFjCnwD2BhlO9xF9Ch4euvgMeifN6ohhp87gBWBix3xX+HdU/Lcx8E6iO87mMNdYiIpBIXMBlz3PdZiTnGepwoqAlHW5b/HeXzngT2DVg+hug/a0REREREohbrNNvzA7ad17BuL8wJeXNTlT5BdDfV5gQ859uGdccAm5p5/TrMRUE0LatXBzzvuSi295loec9Rlu/v0EyNkR6nxFCDiEgixDrNdjbwuOU5C4AeCayxpVyYLk2+OrcCmVE+d3uC93Fe05sLqIuTiIiIiBP2Bj4C+kax7VnAvS14jyOA14CShuU6TLP0DZbtMoEpwHQ004aISCIVAm8BZwesmwXsB6x1pKKm9Sa4e9MCIrdotFqOmX7bZ0ikDcVPAY2IiIiIvboCrwC5wCrgT5g7jdkN60Zi7q4GugTYPYb36IwZJ8AFrMOEPB0w06F2AfpguhnVBTznnIZanPArZv92B56yfG/vgO9ZH+/aWKOISGv0wATzhwSsexLThWibA/VEwxqqrIjx+SsDvi4BurWqmjZAY9CIiIiI2GuHhn8/AI4j9MR8IWYGjFXAtIZ1GcBFBI/j0pSeDf8uB/bBhDSBVgPXAx8Db2LCIYDbgVcxY9jYqQZzZxZCa10IVNtbjohIXA0F3sE/FpkXc3yfFvEZRifgpwTWBaal5bkRvmdt5bk6xtf+DdgjYLk/ZvwziUABjYiIiIj9VgAn0vRd0//FNIPv17B8ZIzv4QFOIjTwCDQLMzivb0DhQkwQdGOM7yUiIuGNwwTfvgHW64BLgUejeK4LfzfVRGnXxPcKLcubY3xt6/aJnqUw5amLk4iIiIj9/hcoa2abOkxXKJ9u+FvGROMl/K1SmnI3sDFg+Rw0Fo2ISDyciGk54wtntmEGb48mnEkG1kClKsbnW7dXQNMMtaARERERsZeH4OClKd9YlvtgBvqNxrNRblfdUM8FDcs9Md2wlkX5fBERCXUtZqwvX+C9FjN4eyxTTZdjWkIm0m9NfC/fslwT42tbA5qmWusICmhERERE7LaC6JuJW2dcKo7hfT6LYdtP8Qc0YMYMUEAjIhK7TMzU25cGrFsCHI4ZWywWNZjWkE6ptSxnh90qshzLcqwBT5ujgEZERETEXpti2NY6OG5ulM/bQGi405SlluXeMTxXRET8XgWOClj+GDiW2MdvSQbWcdKsLWqaY90+WWerShoKaERERETsZb0jmQhbWrl9ogelFBFJV4HhzFpMy5kKh2ppLWugEusYMtbtFdA0QwGNiIiISPpxx7i9NTSKtqWOiIgE+xHYqeHrHsAM4FRaFs678A8wnCi1RA6Q1lqWe8X42tbWmNbXEwsFNCIiIiLpJ9a7nNapVGNtgdMczQolIm3F/sB7wLCG5RMwA7FPILTbanM6AevjVll4LwATI3zvB8ty3xhfO3D7KmBljM9vczTNtoiIiEj66Upsgzla73JuDLuVmYHKJzOG1y+KYVsRkVRWCuwHfBmw7khgFqk3zfTPBLfIHBnDczsC2wUsLwPq41FUOlNAIyIiIpJ+coEhMWw/wrL8XYTtAscPiOVCo38M24qIpLrNwKEEz6a3H/AOoS0Wk5kb+CpguTuwfZTPHUNw3hDLzIJtlro4iYiIiKSno4Fvo9z2mICv64AFEbYLnIFqYAy1HBTDttZpWNsTe7cAERGnlWFCmjeBcQ3rxgLvA4cR3axOW4DdE1KdX3N1vAHsHbA8Abgritc9ybL8eixFiYiIiIhEayjgDXi818z28wO2nRfD++xveZ+jmth2jmXbVUQ3JepemK5Lvue90cS2Dwds5yG68Qj2tdTlBUY1sf3Vlm0HR/EeIiJOuY/gY5Z1zK18YKZlm6+BLjbW2Bo7YLom+Wr/Bchp5jk9MWPO+J6zPornCOriJCIiIpKutgNua2abdsA/CL6g+GcT238a8LULuKqZ1+8E/KuZbayWWZZ3jfH5IiLJpAo4luAWJCMwoXoPRyqKzU/AiwHL2wM3NbG9CxPm5wWsu5uWzWIlIiIiItKsZG9BUxvw9Z2Ev3PZG/jI8vozm6mnHabZvm/7euAKwt/0G4uZbtaLmcI12hY0XS31rwTGN6wXEUk2zbWg8ckCnrNsuwzoY0ONrbUjpqtpYAvKWwj9bCkCniZ4H38DCuwqVERERETanmQPaL4FniL4BPn/gMuAazEXCdbQZB3RdVm6xvI8L+YO63RMGDQd+D7ge6sxIU60AQ3AM2Hew4u5QNgU8DghinpFRBIp2oAGzOx3j1u2X0H0A+866XSCu8N6gTXADEwLmecJDvB94Xxzx3sRERERkVZJhYCmHTCX8EGH9VEK7BxlTZnAq1G+7mrMtKwTLeubO2HvRPD/WaTHKVHWLCKSKLEENDR8/yFCj5U7JbDGeLmM4BaOTT02YgZDlhhoDBoRERGR9FQBHAjcjpkJJJw6THP0nYFFUb5uPWZ2jqlAeYRt3JgxC0ZiBsOM1UZgNDAJeBn4GXNnVkQk1XmBPwH3BKzrhQnUhzlSUfTuxwwsPwfTmiacGszxfxea7zYrFs2leyIiIiKSGuZgWtwAfIc5OfbJAw7BNKPvCWzDNKufCWxoxXu2Aw7AXFSUYIKgVZgWRX+04nVFRCS59cFMH94HKMYc83/DHP+jmUJcRERERCRtWbs4iYiISArJcroAERERkSTmAro7XUSUAmfTyCY1pm+Np82YQYRFRJzSHfVS2YKZWlxEREREJK46EN1giHo4/5gY4WcoImKXepw/Fjr9OLvV/4ttmFrQiIiIiETmJfJAiMnGRfCd21SpO17a2v6KSPKpd7qAJKBjsYiIiIi0eXPw38HUGDQiIiIpRi1opK3IBXYEBgDtMbNOdAAqMdOQlgOrgWWYqT1FREREREREbKOARtLVAMy0b+OA0UBfICPK524EFgIfAbOBr4C6+JcoIiIiIiIiIpJ+ugNXYpp1x3Ogq/XAA8Ao+3ZFREQkZjsCuzU8hjhci4iIiIi0QTsDTwNuEj8q+dfAiUTfGkdEREREREREJK0NAF7FjBQeS8iyFfgd+AVYBWwi9inxlgDHJH4XRURERERERESSUx5wE1BF0yFKPTAfuBuYAOwKFER4zUxge2A8MAV4BzNwcHNBzX+A/nHePxERERERERGRpDYEWETToclnwMVAp1a+VzZwGPAsZranSO9XAZzdyvcSEREREREREUkJZwHbCB+S1AHPAcMT9N4lmFY7GyK8vxeYgZm+W0REREREREQk7biA/yVyMDILGGRTLe2B24DqCLV8CXS2qRYREREREREREVtkAtMJH4aUYsaWccJOwNwIdf0A9HWoLhERERERERGRuHIBjxE+BJkL9HKuNMCER7dguldZ6/sV6ONYZSIiIiIiIiIicXIX4cOZ/8OEI8nicMKPjfMt0MHBukREREREREREWuVCwoczU50sqgl7AusJrfdDkitMEhERERERERGJyjDCT2t9nZNFRWFnYBOhdd/iYE0iIiIiIiIiIjFrDywjNOS418miYrAfUEVw7fXAOCeLEhERERERERGJxT2kfjehcwndh5+BPCeLEhERERERERGJxlCgluBgYx3QvZWv2xkYD9wAvAqsxHRFCnzs3cr3sJpBaEhzc5zfQ0REREREREQk7mYTGmqMb8HrdMeMV/MyJowJN9iw9bFv60oPUQj8ZnmPKjT1toiIiIiIiIgksdGEhiYvtfC1xod5LbsDGoCTwrzPAwl4HxERERERERGRuHiH+LU2SZaABkJbBVXS+i5bIiIiIiIikmYynC5ABBgCHGZZ9wSmi1A8eIAfgWeAKxsedplmWc4HLrbx/UVEREREREREonInwa1M3ED/VrzeXvjDmP0w48EEGot9LWgAPrG813LAlcD3ExERERERERGJSQbwK8EBxhsJfk+7A5pTbX4/ERERERERSTHq4iROG03oWDP/dqKQBHod2GpZd6IThYiIiIiIiEhyUkAjTjvAslwO/MeJQhKoEhPSBDrQiUJEREREREQkOSmgEadZA5qPgRonCkmwDyzLg4EeThQiIiIiIiIiyUcBjTgpEzOgb6A5ThRig9mWZRewtxOFiIiIiIiISPJRQCNO6gvkWdZ94UQhNlhD6LThg5woRERERERERJKPAhpx0k5h1v1gexX2+dGyvKMjVYiIiIiIiEjSUUAjThpgWd7Y8EhX1oBmLine truncated
     }
    },
    "cell_type": "markdown",
-   "id": "50b782df",
+   "id": "deb384be",
    "metadata": {},
    "source": [
+    "\n",
     "<div>\n",
-    "<img src=\"attachment:g38106.png\"  align=\"left\" width=\"450\"/>\n",
+    "<img src=\"attachment:g33734.png\"  align=\"left\" width=\"450\"/>\n",
     "</div>"
    ]
   },
   {
    "attachments": {
-    "g38354.png": {
-     "image/png": "iVBORw0KGgoAAAANSUhEUgAABGgAAAGhCAYAAAA9VYLrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13nBT1/cfx1+xVDjh6BxFFBRUsiIoBFTQRS+zYwRBLLBFbxK6oMYAaNcaYGJWIsUSNvaHYRaM/RURRFJWiICC9Xb+d3x/fXXZ2du9u9253Zsv7+Xjsg5vvze5+hrub3X3Pt1iIiIiIiIiISHPcCVzo2A4Atk+1eKUN0B/ojDneDcBSYJmfReWCQr8LEBEREclC2wFPOLY/As73qRYREZF0KgV+BYwGRgE7YIIZtx+B54F7gK88q05ERERE8toumCuk4dtr/paTU84EZjpuxf6WIyLSqDuJfj2w/C0npYqBe4F1RB9jU7d64HaglfclZzf1oBERERGRTNIfONixHe8qrYiIpF8ZcHYz7hcALgYGA0cAVaksKpfpBU9EREREREREmrIM02PoGGAbzNAnC+gNjAU+c+1/EHCHlwWKiIiISP7REKf0mUL0/22pv+WIiDQql4c4tccc01vAoUBBE/sXAf8kdrjTLmmsMaeoB42IiIiIiIiIuFVjJgUeCbyCCVsaUwucA7zvaAsAJ6WluhykgEZEREQkc3TCLF+aTp1RrxQREWlaJab3TDKCwG2utl+kppzcp0mCRURERNLvasx4fYDvgVtCX7cDTg/ddiYSnGwCXgf+jlnJKBGXAjuGvl4G3Bj6uidmZaSTgO2JrIq0EngBsxzqnASfYyqmyzvAx8D9Cd5vaKiGsMnAYsd2V+Cm0Nf7uO57Nw1ftb0/VIeISDaxgAmY837YYsw5NuhHQSn2kWu7hy9ViIiIiEheSHYOmk8c+84Kte2LeUPe1HKl/yKxi2pvOe4zN9R2FLC2icevw3woSKRn9VLH/R5LYP+wk1zPOdT1/f5N1NjQ7eQkahARSYdk56ApAqa57jOb3AoxuhB9fJ/7W072UA8aEREREe/tB7wJlCSw72+AzcAFST7H4cAzRD4s1AE/Y3rQdHbsVwBMDLWdiXkzLSIiqdcW+C/wK0fbDGAM5jyfK3Zybf/oSxVZSHPQiIiIiHirK/AUJpxZAvwe2A5zVbUE2BNzddXpPGCvJJ6jMzAdE86swIQ87YFemCubfTDDjOoc9/ltqBY//Ig5vr0wdTvt5/ie+/aqhzWKiLRED+BtosOZB4Ejya1wBmInBX7dlypEREREJC+0ZIhT+DaTxicEvs61f1PzvTiHOIVv3wPdG7nPaKDGsf9GTIjTkHQNcXLSMtsikk0SGeK0CyaQD+8TBK5P4LE7YYappvP2QHKH26RtMYFT+Fgrgd4pfg4RERERka1aGtAsxEwQ3JhCYJHjPiua2N8d0NQDQ5q4D8C1rvvd1Mi+CmhERKI1FdAcCKxzfL8GGJ/gY3d2PXY6bv9J6mgbF8D0bHQ+vntFJ2mEhjiJiIiIeO9PwIYm9qnDDIUK64ZZkSlRT2ImnmzKrcAax/ZvaXqSSxERadrxwCtEVr/bDByNmfw9F00iegjX90RWFJQEaJJgEREREW8FiQ5eGvOZa7sP8FOC9300wf2qQvWcHdruCewALEjw/iIiEutyzFxf4cB7OWby9jlJPMYm4IQU1+WWqgl8TwCucWzXYHpPbkzR4+cFBTQiIiIi3lqE6e6eiNWu7aaGRTl9kMS+7xMJaAD2RgGNiEhzFGCGPZ3vaPsKOAwzD00yqjG9ITPdIcBDRMIoGzgXM7xXkqCARkRERMRba5PYt8q1nciy3GCCHXe405j5rm1N6Cgi0jxPA792bL+LGdaUaDCfbX4JPEvk9SkczrhXI5QEKKARERER8VaNB8+xvoX7d0hVISIiecYZzizH9JzZ4lMt6bY/JpxxTuY+EbjXn3KynwIaERERkdxTm+T+7tAo0Z46IiIS7Rtgp9DXPYDpwCk0L5y3iEwwnC41NC9A2hd4EShztN2AVm1qEQU0IiIiIrmnTZL7t3VtJ9sDpylaFUpE8sWBwGvAoND2cZiJ2McQO2y1KZ2AVSmrLL7HMZP5JmMIZnUq52vHZMwqTtICWmZbREREJPd0BYqS2N8958yauHuZFajCCpJ4/PIk9hURyWYrgQOAjxxtRwAzSD48z0S7Aa8S3bPnTuAqf8rJLQpoRERERHJPCbBzEvvv4dr+vIH9Nju+TuaDRr8k9hURyXbrMCsbOVfTOwDT6ySbA+tBwOuYnj1hdwEX+1NO7tEQJxEREZHcdCQwN8F9j3J8XQfMbmA/5wpU/ZOo5eAk9q12bbch+WEBIiJ+24AJaZ4HRobahmOGPx1KYqs6rQf2Skt1EYmuLrUzJpzp7Gj7B3BRyisSEREREUnCLpilRMO315rY/xPHvrOSeJ4DXc/z60b2fcu17xKgVQLPsS9m6FL4fs81su/fHfsFgb4JPP7+rrpsYGgj+1/q2ndgAs8hIuKXO4k+Z7nn3CrBnFed+8wBunhYY0vtCPxE9DFMQyNyUk7/oSIiIiK5aRvgj03s0xq4h+gPFP9oZP/3HV9bmDClMZ2BB5rYx22Ba3v3JO8vIpJJqjETBD/taNsdeBfo5UtFydkBcwGgh6PtIeBMouclkxTQECcRERGR3FOLmST4ktDX1xG7xGtv4GGi55+ZgZkjoSHPABuJzKHwe2AR8Bdi36jvD9yPGQpVQfRSrI350FE/mJVB1gGfAj8n+BgiIpmkBjgR+DeRFZMGYIKPg4AffaqrKT2BN0P/hn0FPAqMSvKx3gLqU1SXiIiIiMhWmT7EaS7mCmd4+0fgr8CFwOXAY8AW12OvILEhS5cRO2TpW+A+YEro33mO7y3FTCCZ6BAngEfiPIeNmYtmreN2XAL1ioikU1NDnJwKML0K3cNRk5nTy0ujiH8ubs4t0ZBeRERERCQp2RDQtAbeIbE3ziuBXROsqQDTVT+Rx10K7Im5YpxMQNOJ6P+zhm4nJ1iziEi6JBPQEPq++z7LMa8rmUYBjcc0B42IiIhIbtqC6Tp/M2YlkHjqMMOcdsX0eklEPXACcA2wqYF9aoEnMOHMpwk+rtMaYBhwKvBf4DvMiigiItnOxqx85JwjrDtmKNFgXyqSjNFUuiciIiIi2eEtTI8bgM+B3RzfKwV+BWyHmUtgM2bumFeA1S14ztaYK6yDgA6YIGgJpkeR5osRERERERERkbzjHuIkIiIiWUSrOImIiIg0zMJ0Pc8GxY6vi4heEjUfrMNMIiwi4pfuaJTKeqDS7yJEREREJPe0J3UTJOqW3lt46VoREb/U4/+50O/b+Bb/L+Yx9aARERERaZgNBP0uIkEW0Vdus6XuVMm34xWRzFPvdwEZQOdiEREREcl7bxG5gqk5aERERLKMetBIvigBdsSsXtEWs+pEe6ACswzpJmAZ8A1maU8RERERERERzyigkVy1PTAydBsG9AUCCd53DTAHeBt4E/gYqEt9iSIiIiIiIiIiuac7cAnwGamd6GoVcBcw1LtDERERSdqOwJDQbWefaxERERGRPLQL8DBQS/pnJf8UOJ7Ee+OIiIiIiIiIiOS07YGnMDOFJxOybASWA98DS4C1JL8k3lfAUek/RBERERERERGRzFQKXIeZ4LexEKUemA3cCpwA7A6UNfCYBZgJhEcDE4GXMRMHNxXUvAD0S/HxiYiIiIiIiIhktJ2BeTQemnwAnAt0auFzFQGHAo/SeBi0BRjfwucSEREREREREckKvwE2Ez8kqQMeAwan6bk7YHrtrG7g+W1gOmb5bhERERERERGRnGMBf6LhYGQGsJNHtbQB/ghUNVDLR0Bnj2oREREREREREfFEAXAf8cOQlcAYn+raCXingbq+Bvr6VJeIiIiIiIiISEpZwP3ED0HeAXr5VxpgwqNJmOFV7vp+APr4VpmIiIiIiIiISIpMJX4481dMOJIpDiP+3DhzgfY+1iUiIiIiIiIi0iK/I344c42fRTViH2AVsfW+QWaFSSIiIiIiIiIiCRlE/GWtr/SzqATsCqwltu5JPtYkIiIiIiIiIpK0NsACYkOOO/wsKgkHAJVE114PjPSzKBERERERERGRZNxG9g8TOoPYY/gOKPWzKBERERERERGRROwC1BAdbKwAurfwcTsDo4GrgaeBxZihSM7bfi18DrfpxIY016f4OUREREREREREUu5NYkON0c14nO6Y+Wr+iwlj4k027L7t37LSY7QFfnQ9RyVaeltEREREREREMtgwYkOTJ5v5WIfEeSyvAxqAE+I8z11peB4RERERERERkZR4mdT1NsmUgAZiewVV0PIhWyIiIiIiIpJjAn4XIALsDBzqavsXZohQKgSBb4BHgEtCN6/c4NpuBZzr4fOLiIiIiIiIiCRkCtG9TGqBfi14vH2IhDEHAOWu74/Aux40AO+5nmshYKXx+UREREREREREkhIAfiA6wHguzc/pdUBzisfPJyIiIiIiIllGQ5zEb8OInWvm334UkkbPAhtdbcf7UYiIiIiIiIhkJgU04rdRru1NwAt+FJJGFZiQxukgPwoRERERERGRzKSARvzmDmjeBar9KCTNXndtDwR6+FGIiIiIiIiIZB4FNOKnAmBfV9tbfhTigTdd2xawnx+FiIiIiIiISOZRQCN+6guUuto+9KMQDywjdtnwAX4UIiIiIiIiIplHAY34aac4bV97XoV3vnFt7+hLFSIiIiIiIpJxFNCIn7Z3ba8J3XKVO6Dp70sVIiIiIiIiknEU0Iif2ru2V/hShXdWurbb+VKFiIiIiIiIZBwFNOKnNq7tzb5U4Z2Nru1yX6oQERERERGRjFPodwGS1/ItoNnk2m7rSxUiIuKFALADMBDojFm9bx3wHTAPqPOvtBZpDwzDrMI4GOgSunUGKjDHOBd4B3iC2IsTIiK5oB2wG9AHaA1UAquB2cDPPtbVEgHMLine truncated
+    "g32779.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAABGgAAAGhCAYAAAA9VYLrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13fBR1/sfx1256QhJCr4LKoVhQqqKAYjnFdpbDhvUsZztOT8WfXU/v1FPPcp6eDQUFFc/GKWJv2EUUKYqKoCCElgRI3WTn98d3N1uySXaT3Z3Zzfv5eMyD7Ozs7mc2ZHbmvd8CIiIiIiIiItIWdwNW0OKytxxJZZl2FyAiIiKSgnYAZgfd/hS40KZaREREkqUzMAIY6ft3h6D76oG97SgqXSigEREREYldHubE1G+zXYWkoXOA44NuHw7U2VSLiIjAH4DfYj73BrWwnSc55aQvBTQiIiIi4iSDgIOCbrvtKkRERAC4CBhmdxEdgT7wRERERERERCRWq1ELx7hSCxoRERERERERaclaYEHQ8jmwDlgFbGdjXWlFAY2IiIiIiIiINGcCUGF3ER2BujiJiIiIOEdXoFOCX6MbkJvg1xARkfShcCZJ1IJGREREJPGuJtAE/EfgH76fi4HTfcsuBIKTrcCbwAPAG1G+xqXAYN/Pa4C/+n7uA5wNnAjsCGT71pcC/wPuBxZG+Rq3YaZYBdO8/ZEoHzfKV4PfLcDKoNs9gJt8P+8V9tj7gIZmnvcRXx0iIqnEBUzBHPf9VmKOsV47ChIRERERSVW7AlbQ8nor238RtO1837q9MSfkVivLY0T3pdo7QY/52rfud5gpwFt6/nrMRUE0LatXBz3uqSi29zsx7DVHhd3/m1ZqbG45KYYaREQS4W5Cj0uuVrbPBB4Me8zXQL8E1phIqwjshwYMbie1oBERERFJvn2At4GcKLY9A9gG/CnG1zgceIHAxUI9sB7TgqZb0HYZwFTfurMxJ9kiIhJ/nYBngMOC1r0JHAdssaUicRSNQSMiIiKSXD2A5zDhzCrgImAHIMu3bjgwLewxFwAjY3iNbsB0TDizDhPydAb6At2B/phuRvVBj/mDrxY7/IzZv5HAjLD79gm6L3x5LYk1ioi0R2/gPULDmcd9txXOiIiIiIi0UXu6OPmXN2h5QODrwrZvbbyX4C5O/uVHoFcLjzkU0yTdv/0WTIjTnER1cQp2W9i2GtBYRJwsmi5OuxLaFcgLXB/Fc3fFdFNN5PJobLvbhLo4iYiIiIit2hvQrMAMENySTOCnoMesa2X78ICmARjRymMArg173E0tbKuARkQkVGsBzQSgLOh+D3BulM/dLey5E7E8HdvuNqGAJo7UxUlEREQk+f5O69OW1mO6Qvn1xMzIFK1ngQVRbHc7sCno9h9ofZBLERFp3e+BuQRmv9uGGbz9IdsqEkfTIMEiIiIiyeUlNHhpyVdht/sDv0b52FlRblfjq8f/jW4fzKxKy6N8vIiINHUFZqwvf+C9FjN4+8IYnmMrcHyc6wr3S4KfX2KggEZEREQkuX7CNHePxsaw2611iwr2UQzbfkhok/vRKKAREWmLDEy3pwuD1i3FDAa8KsbnqsW0hpQOQgGNiIiISHJtjmHbmrDb0UzLDSbYCQ93WrIs7Ha/GB4rIiIBzwNHBt1+Hzia6IN56cAU0IiIiIgkVzIGUSxv5/Yl8SpERKSDCQ5n1mJazlTaVIukGAU0IiIiIunHE+P24aFRtC11REQk1HfATr6fewPTgZNpWzjvIjDAcKLUoQDJMRTQiIiIiKSfTjFuXxh2O9YWOK3RrFAi0lHsB7wODPXdPg540fdvdYzP1RXYEL/SInoGODHBryFR0jTbIiIiIumnB5AVw/bhY85siriVmYHKLyOG5y+KYVsRkVS2Htgf+DRo3UTgVWIPz6WDUUAjIiIikn5ygF1i2H5Y2O1FzWy3LejnWC40to9hWxGRVFcGHELobHr7YUIaBdbSLHVxEhEREUlPRwFfR7nt74J+rgcWNLNd8AxUg2Ko5aAYtq0Nu92JprNZiYg4XQUmpJkDTPCtG4vp/jSR6GZ1KgdGJqS6AM0u5SAKaERERETS09nAHbQ+5sHewOig23MJbSkT7BtgX9/Pg4ABwKpWnn88MKKVbYJtCbvdndimDBcRcYptmDBmNiY0B9gLeBv4La2PL9NSYC5pSF2cRERERNLTdsDNrWxTANxP6CC+/2lh+w+DfnYBl7by/N2AR1vZJtzysNt7xvh4EREnqQUmAc8HrdsTeB/oa0tF4lhqQSMiIiKSfjyYQYL/4vv5OppO8doPeJLQ8WfmYcZIaM4LmBYu/jEULgJ+Au4hdABhMC1nHsG0tKkC8qOs/ZOg+gFuwTTB/xIz+KaISKqpA04AniAwY9LOwDvAgcAvNtUVrRKabwmZG/Szi+a7tC4G1sWzKBERERERgF0BK2h5vZXtvwjadn4Mr7N/2Osc2cK27wRt9zUwI+j2L8C/gD8DVwBPAZVhz70O02WpNZeHPc4CvgceBm71/bs46L7VwCVh249q5TVmRngNCzMWzeag5bgo6hURSaS7CT1OuVrYNgPTqjB4+1XENqaXHfYn8jE5luWkZBeditSCRkRERCQ9nY8JXMZjWstc1MK26zHferY2ngzAP4ExwDFB6wYR+QJjDWbchcFRPG+wKcBONP3GNse3+GXH+LwiInZqwIwPthUTmIPpjvoB5hi8xKa6xCE0Bo2IiIhIeqrENJ3/G2YmkEjqMd2cdsO0eolGA3A8cA3mIiMSD2ZQzOGYrkmx2oQJgSYD/wV+wMyIIiKS6izgYkLHCOuFGTh4qC0ViWO01PxKRERERFLHO5hm6ACLgD2C7svFzBiyA9AHM7PIT5jxZtozQ1IBcACwO2aMgnJMK5zX0XgxIiIiIiIiItIBhY9BIyIiIilEY9CIiIiINM+FaXqeCoLHY8kCettViE3KMIMIi4jYpRfqpVIOVNtdhIiIiIikn860f+YKLclZ/FPXiojYpQH7j4V2L2e2+13swNSCRkRERKR5FuC1u4gouQj95jZV6o6Xjra/IuI8DXYX4AA6FouIiIhIh/cOgW8wNQaNiIhIilELGukocoHBmNkrioB8oBjTP7IKM3Xnr8B3tG82CxEREREREZGYKaCRdDUImOBbxgDbAe4oH7sZWAi8C7wNfAbUx79EEREREREREZH00xu4FNOsO54DXW0E/gWMTt6uiIiIxGwwMMK37GJzLSIiIiLSAe0GzAQ8JH5U8oXAJKJvjSMiIiIiIiIiktZ2BJ7HjBSe7OnjlgFHJ34XRUREREREREScKRe4HjPAb7KDmfDlZczAwyIiIiIiIiIiHcYuwGLsD2aCl0rgzETutIiIiIiIiIiIU5yJCUPsDmSaW2YABQnbexERERERERERG7mAW7A/gIlm+Qzonpi3QURERERERETEHhnAI9gfvMSyfAcMTMB7ISIiIiIiIiKSdC5SL5zxLz8D/eP/loiIiIiIiIiIJNc/sD9oac+yCOgc93dFRERERERERCRJzsP+gCUey1uYbloiIiIiIiIizdKFozjR7sB/gSy7C4mD7TFBzbs21yEiIiIiIiIiErVOwHLsb/kSz6UBOCCeb5KIiIiIiIiISCLdif2BSiKWH4DcOL5PIiIiIiIikkZcdhcgEmQ34Evi3LWpW7dujBw5khEjRjBixAiGDx9OUVFRyDZHHHEEH330UTxfNpIbgBsT/SIiIiIiIiKSejLtLkAkyL+IQzjTq1cvzjzzTEaMGMHIkSMZMGBAq4/JzEzKn8L/AY9hpuAWEREREREREXGcMcSpO9EhhxxixWr8+PHJ6up0b2LePhEREREREUllbrsLEPG5zu4CkuRsoJfdRYiIiIiIiIizqIuTOMEuwKGJenKv18v333/PF198wYIFCwD45z//maiXa00ecAEdJ5ASERERERGRKCigESc4LZ5PVl5ezqxZs1iwYAELFixg4cKFbNmypfH+cePGxfPl2uJU4HpMlycREREREREREdu5MYPmJm3K63Hjxtk5Bo1/GR+/t1BERERERERSncagEbuNAfrbXYQNJtldgIiIiIiIiDiHAhqx2wF2F2CTjrrfIiIiIiIiEoECGrFbRw0qhgC97S5CREREREREnEEBjdgpA9jb7iJs4gL2sbsIERERERERcQYFNGKnAUCu3UXYaGe7CxARERERERFnUEAjdtrJ7gJsNtjuAkRERERERMQZFNCInXa0uwCbDbK7ABEREREREXEGBTRip852F2CzYrsLEBEREREREWdQQCN26mR3ATYrsrsAERERERERcYZMuwuQDq2jBzSFdhcgIiIJ4wZ+AwwBumFm7ysDfgAWA/X2ldYuJZgZGPcGhgLdfUs3oAqzj18D7wGzgS32lCkiklDFwB5Af6AAqAY2AguA9TbW1R5uzGeW/xjfH+iFOb7nAhXASuAL4HngU1uqFJGEuRuwkr2MGzfOCjd+/Pik1wGUxuuNFBFJIb2BI4EbgJeB1cDmoOVQ2yprv87AWZgT1wqaP/5vBR7GnAinkv8AXqL/nNsGXEF6fSHYHfiJ0P+zn9takYhzZAHDgXOBBzFhRfDfygf2lRYXo4DbgK9o+Vj4FfBHzPuRKrYHyontWuYjTFCfTq4i9P/sZuBEWysSSaKb6NgBzQ/xeiNFRFLAP4A1tH5sPNKuAtvpPMw3qLF8DtQAl9pRbBs9T9s+714CcmyoNxGeoen+fWdrRSL22wn4DHNMa+lY8LVdBbZTFrCU2I99C0idSUF2oW3H9yrgEBvqTYQhRP4cPz2ZRaTTNxqSerbaXYDNOvr+i0jHMgroY3cRCbQjpgl4sCpgIaa5ex2wHTACyPbdnwPcAeRjvrRIFd8Db2Kat3+P2T8vppn/UGAScBSmWxe+n28Cpia90vg6Cjje7iJEHKgb5hifrjKI3OJxBbAc02IyF3N87xd0/3DgHWBf4OcE1xgv1ZiWTu9guuOuLine truncated
     }
    },
    "cell_type": "markdown",
-   "id": "eb5e46a2",
+   "id": "4157df08",
    "metadata": {},
    "source": [
     "<div>\n",
-    "<img src=\"attachment:g38354.png\"  align=\"left\" width=\"450\"/>\n",
+    "<img src=\"attachment:g32779.png\"  align=\"left\" width=\"450\"/>\n",
     "</div>"
    ]
   },
   {
    "attachments": {
-    "g38620.png": {
-     "image/png": "iVBORw0KGgoAAAANSUhEUgAABGgAAAGhCAYAAAA9VYLrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13mFTV/cfx991lC73YQUQWCzZU1EQSLKhR7C0omlhjN7Ggwi5qJEbZJRpbmj1qNCqJvaEo9vKzERVFUFhQUBBEkLrsMuf3x5lh7tyZ3Z1675TP63nmce/dO3e+s7J3Zz7zPec4iIiIiIiIiEg6bgIudG2XASagWvzSBdg4fOsGrAWWADOApgDrKngdgi5AREREpADVABNd2/8HnB9QLSIiIrlSBfwUGBK+7Qls0sqxzcAHwAPAfcCPfhQoIiIiIqVtB+wnpJHbC8GWU1TOACa7bpXBliMi0qabiP174ARbTtYdSOzzS/Y2Hzg8gHoLmjpoRERERCSfbAUc4NouC6oQERFJW2/gMWzofk+wpRQO/cETERERERERkbYsBh4GzsIOedoY2y3kYOehGQI0AMtd9ykHbgW287VSERERESkpGuKUOw3E/myrgy1HRKRNxT7EqTcwmOSbO/phJwt2/0zuz01pxUcdNCIiIiIiIiKSyDfAh0AoyePnAid79h2Bsoek6IckIiIikj82wC5fmksboq4UERHJnf8DvnRtdwU2CqiWgqJJgkVERERy73Jgi/DXs4A/hb/uDpyK/bRxe6LByXLgReAf2JWMknEJsE346/nA1eGve2MnaRwJDCC6KtJC4Cng78DUJB9jAtAj/PV7wJ1J3m+PcA0R9cAc1/bGwB/DX//Uc9+/AutaOe+d4TpERAqJA1yAve5HzMFeY5PtVMl387GTvkd0CqoQERERESluqc5B877r2DfC+4ZgW8HbW6r0nyT3odrLrvt8FN53JLCknfO3YN8UJNNZPc91vweTOD5ipOcx9/B8f6t2amztdkIKNYiI5EKqc9BUAHd77vMBsFkOawzCZ0Sf31qgY7DlFAZ10IiIiIj47+fAS0BVEseeCqwAfpfiYxyKXeI08mahBfgO20Gzoeu4cmB0eN8Z2BfTIiKSfV2B/wIHuvZNAkZgr/PFYntgoGv7NWB1QLWIiIiISJHLpINmJvBt+Os5wPlAf+wHZ5XArsBdnvOvA3Zv5zHcHTTzsUuimvBjnQJ0dh27OTAeaPY8TnshUK46aKqA3cK3ezzHDnF9z3vrlUINIiK5kGwHTW/scFL3sXdRfE0TvbAdQZHnGAKGBVqRiIiIiBS1TAKayG0ybU8I/HvP8e3N9+IOaCK3WcCmbdxnOLb1PHL8j0CfNo7PVUDjpmW2RaSQJBPQ7EDskNYQcFUS594AO0w1l7e7Unu6CXXDfohwBXZ+M/fP44osnF9EREREpFWZBjSzsRMEt6UD0Oi6z4J2jvcGNOuwXSbtudJzvz+2cawCGhGRWO0FNPsCP7i+vxY4Lclzb+g5dy5uD6X0bGG7JM/bCByb4rlLnpbZFhEREfHfeGBZO8e0AI+4tjfBtsgn6z/YNvP2XAd879o+nfYnuRQRkfadiJ1jJrL63XLgcOzk78XKAPdiO2oeaedY8Si28W4iIiIi+S5E8i9a/+fZ7gt8k+R9/53kcWvC9ZwV3u4NbI2dK0dERNIzBqgnGnh/i528fWoK51gOHJflury+zvL5HOy8Z0cDf8b+DJqz/BhFSwGNiIiIiL8ase3uyVjs2W5vWJTbWykc+ybRgAbgJyigERFJRzlwC3Cua99nwCHYeWhS0YTthswnjcROWt8J+7dpR2wANTS8vxvwB2Bv4AhglY81FiwFNCIiIiL+WpLCsWs828ksyw022PGGO22Z7tnePIX7iohI1KPYYUwRrwFHkXwwn+/WkHj47NPYOcT2Bu7HdnwC7I8NrM7wpboCpzloRERERPy11ofHWJrh8T2zVYiISIlxhzPfYjtniiWcScZrwD7E/l05HdthI+1QB42IiIhI8Ul1vL83NEq2U0dERGLNALYNf70ZdsLcE0kvnHeITjCcK2uBlVk+ZyNwPXBNeNsBfgXUZflxio4CGhEREZHi0yXF47t6tlPtwGmPVoUSkVKxL/ACsFN4+1jsROwjiB+22p4NgEVZqyyxh4GROTjvU0QDGoidt0ZaoSFOIiIiIsVnY6AiheO9c858n/AouwJVRHkK5++WwrEiIoVsITCM2HlaDsMGFp0CqSgY8z3bGwRSRYFRQCMiIiJSfKqA7VM4flfP9setHLfC9XUqXTr9UzhWRKTQfY+dHNe9mt4BwCTiOxaLlXfVwWWBVFFgNMRJREREpDgdAXyU5LFHur5uIfEKHRC7AtVWKdRyQArHNnm2u5D6sAARkaAtAw4CnsR21ADsBUwGDia5iYOXkvuhQbmawHiIZ9vbUSMiIiIikhU7AMZ1e6Gd4993HftGCo+zr+dxDm/j2Jc9x84FOibxGHtihy5F7vdEG8f+w3VcCOiXxPn39tRlgD3aOP4Sz7HbJfEYIiJBuYnYa5Z3zq2OwHOeYz4ENvKxRr+VA+8R+5xPDbKgQqEhTiIiIiLFaQtiJ2hMpDPwd2LfUNzaxvFvur52sGFKWzYE7mrnGK+Znu1dUry/iEg+WQ0cBTzu2rcrNlTfLJCKkrcR9oOBVCZ67wDcRmznz4/YOXhERERERLIu3zto1rq+bgAqExy/OfCK5/zPtVNPZ2zbvruL5mISf+i3NzZsMdglXJPtoNnIU/8cYDh24mMRkXzTXgdNRAfgQc+xM4G+PtSYrq2wdX4CXA7sTOsT0HcDTgKmEd81eXHOKxURERGRkpXvAc1HwH2u7a+BvwAXAmOwbxK8ockCkhuydBnxL76/AO7AhkF3EPsCfR72xXmyAQ3AAwkew2Dnolniuh2bRL0iIrmUbEADdujP3Z7jG4GaHNeYrkhA4741YSeSfx07n87rwJfAugTHGuzz1cgdEREREcmZQghoOgOvkvgFs/e2ENgxyZrKgUeTPO88YDAw0rO/vYBmA2J/Zq3dTkiyZhGRXEkloCH8/b8Rf63cNoc1pitRQJPsbTW260bhTAr0wxIREREpTiuxy7xei10JJJEW4H5sODMtyfOuA44DrgCWt3JMMzARG858mOR53b7HrgDyK+C/2E9ntUSriBQDA/wWuN61rw82UN8pkIpa9xU2YH8Q22WZjIXAn7EfZFyLHQorSUplsh8RERERyV8vYztuwLaf7+z6XjVwILaNvjewAttW/xywOIPH7Azsh31T0RMbBM3FdhR9l8F5RUQk//THXu9rsPOCVWGHnv6IvfZPxQ55VSgjIiIiIiXNO8RJRERECkiHoAsQERERyWMOsGnQRSTJvVJTBfm/fGu2/YD9JFdEJCibolEqS7Hzz4iIiIiIZFUP0p8gUTd/byNb+X8oIuKX1lYyKqXbaRn/FEuYOmhEREREWmconLH0DrGf3BZK3dlSas9XRPLPuqALyAO6FouIiIhIyXuZ6CeYmoNGRESkwKiDRkpFNbANdsbxbkAnoDt2fOQq7NKd3wAzyGw1CxEREREREZGUKaCRYrUVMCx8GwJsAZQled8l2CXiXgGmAO8CLdkvUURERERERESk+GwGXIJt687mRFeLgb8AP/HvqYiIiKRsG2C38G37gGsRERERkRK0I/AA0EzuZyWfCowg+W4cEREREREREZGiNgB4FDtTeCohyzJgPjALaMQOaWpJ8RzTgaNy/xRFRERERERERPJTNXAVdoLftkKUFuB94E/YrpedgY6tnLMM6A8cBFwKPAMsb+f8BngaO/GwiIiIiIiIiEjJ2B6YRtuhyZvAOUCvDB+rAzAcO3xqVRuPtxI4LcPHEhEREREREREpCKdhw5DWumX+DQzK0WP3AK4EFrXy+Aa4D+ico8cXEREREREREQmUA9TTejDyHHblCj90Bq4G1rRSy7vARj7VIiIiIiIiIiLii3LgThKHIQuBXwZU1zbAK63UNQPYMqC6RERERERERESyyqH1cOYVoHdglVllwBgSrwD1FdA3uNJERERERERERLLjTyQOZ27Bdtbki4OBFcTX+TF27hoRERERERERkYJ0DvGBRwi4PMii2vATEk8g/BL5FSaJiIiIiIiIiCRlJxIva10bZFFJ2BFYQnzd4wKsSUREREREREQkZV2AmcSHHDcEWVQK9gZWE1v7OmC/IIsSEREREREREUnFn0m8jHZZkEWl6HTin8OXQHWQRYmIiIiIiIiIJGNHYC2xwca3wKYZnndDYDh2/ppHgTnYoUju288yfAyve4kPaa7K8mOIiIiIiIiIiGTdy8SHGgelcZ5NgTrgv9gwJtFKUN7b3pmVHqcr8LXnMVYDW2T5cUREREREREREsmYI8aHJxDTPdVCCc/kd0ACMSPA4t+TgcUREREREREREsuI54rtNNk/zXPkS0IBdZtv9OKvIfMiWiIiIiIiIFJlCmnhVitf22Dli3P4JzMvS+UPADOABYFT45pc/eLY7Auf5+PgiIiIiIiIiIklpILbLpBnYMoPz/ZRoGLMP0M3z/b3wr4MG4DXPYzUCTg4fT0REREREREQkJWXAV8QGGE/k+DH9DmhO8PnxREREREREpMBoiJMEbQjQ17PvX0EUkkOPAz969o0IohARERERERHJTwpoJGj7ebaXA08FUUgOrcaGNG7e5y0iIiIiIiIlTAGNBM0bVLwGNAVRSI696NneDtgsiEJEREREREQk/yigkSCVA3t69r0cRCE+mOLZdoCfBVGIiIiIiIiI5B8FNBKkfkC1Z987QRTig/nA1559A4MoRERERERERPKPAhoJ0rYJ9n3uexX+meHZ3iaQKkRERERERCTvKKCRIA3wbH8fvhUrb0CzVSBViIiIiIiISN5RQCNB6uHZXhBIFf5Z6NnuHkgVIiIiIiIikncU0EiQuni2VwRShX9+9Gx3C6QKERERERERyTsdgi5ALine truncated
+    "g33077.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAABGgAAAGhCAYAAAA9VYLrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13nBN1/sfx12TJ7rLAUqUjYqMJFuROFFSsWM7eFdTTsx/WQz09xVNPOQvqed7Ps5zds5y9oNjFcqeAKEWxgSK9w/bdzO+Pb7KZTLK7yW4yk2Tfz8cjD3a+O0k+s8tOkvd8i4WIiIiIiIiINMcdwIWO7QBg+1SLV9oD2wLdMMe7AVgC/OJnUfmgjd8FiIiIiOSgrYGnHdv/Bc73qRYREZFMKgYOAMYB+wDbYYIZt5+Bl4B7gPmeVSciIiIirdpQzBXSyO1Nf8vJK78Dpjtuhf6WIyLSqDuIfT2w/C0nrQqBe4F1xB5jU7c64Hagrfcl5zb1oBERERGRbLItsJ9jO9FVWhERybwS4Kxm3C8AXAwMBw4FKtNZVD7TC56IiIiIiIiINOUXTI+hI4AtMUOfLKAvMB74wrX/vsBULwsUERERkdZHQ5wyZwqxP9tif8sREWlUPg9x6oQ5pneBg4CCJvYPAv8kfrjT0AzWmFfUg0ZERERERERE3KowkwKPBV7HhC2NqQHOAT5ytAWAEzJSXR5SQCMiIiKSPbpili/NpG6oV4qIiDStAtN7JhUh4FZX2x7pKSf/aZJgERERkcy7CjNeH+B74K/hrzsCp4ZvQ4gGJ5uAt4B/YFYySsalwPbhr38B/hz+ujdwJuYK5jZEV0VaAbyMWQ51dpLPMQXT5R3gM+D+JO83MlxDxE3AIsd2d+D68Ne/dt33bhq+ant/uA4RkVxiARMx5/2IRZhzbMiPgtLsv67tXr5UISIiIiKtQqpz0Hzu2HdGuG03zBvyppYr/RfJXVR713GfOeG2w4G1TTx+LeZDQTI9q5c47vdkEvtHnOB6zpGu72/XRI0N3U5MoQYRkUxIdQ6aIPCg6z4zya8QYwtij+9Lf8vJHepBIyIiIuK93YF3gKIk9j0N2Az8PsXnOAR4nuiHhVpgJaYHTTfHfgXApHDbmZg30yIikn4dgGeBAxxt04BjMef5fDHQtf2zL1XkIM1BIyIiIuKt7sB/MOHMYuACYGvMVdUiYBfM1VWn84BdU3iObsDDmHBmOSbk6QT0wVzZ7IcZZlTruM9vw7X44SfM8e0KPOL63u6O77lvb3hYo4hIS/QC3iM2nHkIOIz8CmcgflLgt3ypQkRERERahZYMcYrcptP4hMDXuPZvar4X5xCnyO17oGcj9xkHVDv234gJcRqSqSFOTlpmW0RySTJDnIZiAvnIPiHg2iQeuytmmGombw+kdrhN2goTOEWOtQLom+bnEBERERGp19KA5gfMBMGNaQP86LjP8ib2dwc0dcCIJu4D8CfX/a5vZF8FNCIisZoKaPYG1jm+Xw2cnuRjd3M9diZu/07paBsXwPRsdD6+e0UnaYSGOImIiIh47y/Ahib2qcUMhYrogVmRKVnPYCaebMotwBrH9m9pepJLERFp2jHA60RXv9sMHIGZ/D0fTSZ2CNf3RFcUlCRokmARERERb4WIDV4a84Vrux+wNMn7PpHkfpXhes4Kb/fGrKq0MMn7i4hIvMsxc31FAu9lmMnbZ6fwGJuA49Jcl1u6JvA9DrjasV2N6T25MU2P3yoooBERERHx1o+Y7u7JWO3abmpYlNPHKez7EdGABuBXKKAREWmOAsywp/MdbfOBgzHz0KSiCtMbMtsdiJngPRJG2cC5mOG9kgIFNCIiIiLeWpvCvpWu7WSW5QYT7LjDncYscG1rQkcRkeZ5DviNY/sDzLCmZIP5XLM/8ALR16dIOONejVCSoIBGRERExFvVHjzH+hbu3zldhYiItDLOcGYZpudMmU+1ZNqemHDGOZn7JOBef8rJfQpoRERERPJPTYr7u0OjZHvqiIhIrG+AgeGvewEPAyfRvHDeIjrBcKZU07wAaTfgFaDE0XYdWrWpRRTQiIiIiOSf9inu38G1nWoPnKZoVSgRaS32Bt4EhoW3j8ZMxH4s8cNWm9IVWJW2yhJ7CjOZbypGYFancr523IRZxUlaQMtsi4iIiOSf7kAwhf3dc86sSbiXWYEqoiCFxy9NYV8RkVy2AtgL+K+j7VBgGqmH59loR+ANYnv23AH80Z9y8osCGhEREZH8UwQMSWH/nV3bXzaw32bH16l80BiQwr4iIrluHWZlI+dqenthep3kcmA9DHgL07Mn4i7gYn/KyT8a4iQiIiKSnw4D5iS57+GOr2uBmQ3s51yBatsUatkvhX2rXNvtSX1YgIiI3zZgQpqXgLHhttGY4U8HkdyqTuuBXTNSXVSyq0sNwYQz3Rxt/wdclPaKRERERERSMBSzlGjk9mYT+3/u2HdGCs+zt+t5ftPIvu+69l0MtE3iOXbDDF2K3O/FRvb9h2O/ENA/icff01WXDYxsZP/LXPsOTuI5RET8cgex5yz3nFtFmPOqc5/ZwBYe1thS2wNLiT2GB9GInLTTD1REREQkP20J3NDEPu2Ae4j9QPF/jez/keNrC7i0icfvBjzQxD5uC13bO6V4fxGRbFKFmSD4OUfbTsAHQB9fKkrNdpgLAL0cbY8AZxI7L5mkgYY4iYiIiOSfGswkwZeEv76G+CVe+wKPETv/zDTMHAkNeR7YSHQOhfOBH4E7iX+jvidwP2YoVDmxS7E25lNH/WBWBlkHzAJWJvkYIiLZpBo4HniU6IpJgzDBx77Azz7V1ZTewDvhfyPmA08A+6T4WO8CdWmqS0RERESkXrYPcZqDucIZ2f4Z+BtwIXA58CRQ5nrs5SQ3ZOkPxA9Z+ha4D7g5/O9cx/eWYCaQTHaIE8DjCZ7DxsxFs9ZxOzqJekVEMqmpIU5OBZhehe7hqKnM6eWlfUh8Lm7OLdmQXkREREQkJbkQ0LQD3ie5N84rgB2SrKkA01U/mcddAuyCuWKcSkDTldifWUO3E5OsWUQkU1IJaAh/332fZZjXlWyjgMZjmoNGREREJD+VYbrO34hZCSSRWswwpx0wvV6SUQccB1wNbGpgnxrgaUw4MyvJx3VaA4wCTgaeBb7DrIgiIpLrbMzKR845wnpihhIN96UiyRpNpXsiIiIikhvexfS4AfgS2NHxvWLgAGBrzFwCmzFzx7wOrG7Bc7bDXGEdBnTGBEGLMT2KNF+MiIiIiIiIiLQ67iFOIiIikkO0ipOIiIhIwyxM1/NcUOj4OkjskqitwTrMJMIiIn7piUaprAcq/C5CRERERPJPJ9I3QaJumb1Flq4VEfFLHf6fC/2+nd7in2Irph40IiIiIg2zgZDfRSTJIvbKba7UnS6t7XhFJPvU+V1AFtC5WERERERavXeJXsHUHDQiIiI5Rj1opLUoArYHtgHaY1ad6ASUY5Yh3QQsARZilvYUERERERER8YwCGslX2wBjw7dRQH8gkOR91wCzgfeAd4DPgNr0lygiIiIiIiIikn96ApdgunWnc6KrVcBdwEjvDkVERCRl2wMjwrchPtciIiIiIq3QDsBjQA2Zn5V8FnAMyffGERERERERERHJa9sAz2FmCk8lZNkILAW+BxYDa0l9Sbz5wOGZP0QRERERERERkexUDFwDVNB4iFIHfA7cAhwL7ASUNPCYBcDWwDhgEvAaZuLgpoKal4EBaT4+EREREREREZGsNgSYS+OhycfAuUDXFj5XEDgIeAKz2lNDz1cGnN7C5xIRERERERERyQmnAZtJHJLUAk8CwzP03J0xvXZWN/D8NvAwZvluEREREREREZG8YwF/oeFgZBowyKNa2gM3AJUN1PJfoJtHtYiIiIiIiIiIeKIAuI/EYcgKzNwyfhgIvN9AXV8D/X2qS0REREREREQkrSzgfhKHIO8DffwrDTDh0WTM8Cp3fT8B/XyrTEREREREREQkTaaQOJz5GyYcyRYHk3hunDlAJx/rEhERERERERFpkbNJHM5c7WdRjfg1sIr4et8mu8IkEREREREREZGkDCPxstZX+llUEnYA1hJf92QfaxIRERERERERSVl7YCHxIcdUP4tKwV5ABbG11wFj/SxKRERERERERCQVt5L7w4TOIP4YvgOK/SxKRERERERERCQZQ4FqYoON5UDPFj5uN2AccBXwHLAIMxTJedu9hc/h9jDxIc21aX4OEREREREREZG0e4f4UGNcMx6nJ2a+mmcxYUyiyYbdtz1bVnqcDsDPrueoQEtvi4iIiIiIiEgWG0V8aPJMMx9rXILH8jqgATguwfPclYHnERERERERERFJi9dIX2+TbAloIL5XUDktH7IlIiIiIiIieSbgdwEiwBDgIFfbvzBDhNIhBHwDPA5cEr555TrXdlvgXA+fX0REREREREQkKTcT28ukBhjQgsfbjWgYsxdmPhinMXjXgwbgQ9dz/QBYGXw+EREREREREZGUBICfiA0wXszwc3od0Jzk8fOJiIiIiIhIjtEQJ/HbKOLnmnnUj0Iy6AVgo6vtGD8KERERERERkeykgEb8to9rexPwsh+FZFA5JqRx2tePQkRERERERCQ7KaARv7kDmg+AKj8KybC3XNuDgV5+FCIiIiIiIiLZRwGN+KkAM6Gv07t+FOKBd1zbFrC7H4WIiIiIiIhI9lFAI37qDxS72j71oxAP/EL8suGD/ChEREREREREso8CGvHTwARtX3tehXe+cW1v70sVIiIiIiIiknUU0IiftnFtrwnf8pU7oNnWlypEREREREQk6yigET91cm0v96UK76xwbXf0pQoRERERERHJOgpoxE/tXdubfanCOxtd26W+VCEiIiIiIiJZp43fBUir1toCmk2u7Q6+VCEiIl4IANsBg4FumNX71gHfAXOBWv9Ka5HOmBUYdwOGA1uEb92AcswxzgHeB54m/uKEiEg+6AjsCPQD2gEVwGpgJrDSLine truncated
     }
    },
    "cell_type": "markdown",
-   "id": "d02606e0",
+   "id": "7ff1e8df",
    "metadata": {},
    "source": [
     "<div>\n",
-    "<img src=\"attachment:g38620.png\"  align=\"left\" width=\"450\"/>\n",
+    "<img src=\"attachment:g33077.png\"  align=\"left\" width=\"450\"/>\n",
+    "</div>"
+   ]
+  },
+  {
+   "attachments": {
+    "g33443.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAABGgAAAGhCAYAAAA9VYLrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13fBR1/sfx16SHDqKgolJEQRQrv5MTVNRT7BUsd/bePT2BRDxzKiScnu1Oz95776Io9nIqdkURElBRutSQuvP747ubnZ3dJLvJ7szu5v18PPZBZnZ25jNJmOx+5vP9fC1EREREREREpC1uAC50LOcAtk+xeKULsFHw0Q2oA1YAPwC1PsaV8fL8DkBEREQkAw0EHncs/w8416dYREREUqUQ+AMwMvjYFejTzLb1wCzgIeB+YLUXAYqIiIhIxzYMc4c09HjN33CyyunADMejwN9wRERadAORfw8sf8NJun2JPL94HwuBg32IN6OpgkZERERE0smWwD6O5Ry/AhERkTbbBHgGOA24199QMof+4ImIiIiIiIhIS5YBjwFnYIY8bYSpFrIwfWhGAhXAGsdrcoFbgaGeRioiIiIiHYqGOKXONCK/t0X+hiMi0qJsH+K0CbAT8Rd3bIFpFuz8njyYmtCyjypoRERERERERCSWX4HPgECc2y8ATnCtOxjlHuKib5KIiIhI+tgAM31pKvVGVSkiIpI6/wPmOpa7Yf72SCvUJFhEREQk9S4DNg9+PQ/4Z/Dr7sCJwcc2hBMna4DXgf9iZjKKxyXAVsGvFwJXBr/eBNOk8RhgEOFZkRYDLwC3AJ/HeYxpQI/g158Ad8b5uhHBGELKgfmO5Y2Aq4Jf/8H12v8Ajc3s985gHCIimcQCLsBc90PmY66x8VaqpLuFmKbvIZ38CkREREREsluiPWg+dWz7XnDdrpg35K1NVXoP8d1Ue9Pxmi+D6w4FVrSy/wbMh4J4Kqt/cbzukTi2DznGdcwRrucHtxJjc49jE4hBRCQVEu1Bkw/c7XrNLGDjFMboh+8In18tqtyMiypoRERERLz3R2AmUBjHticBa4HzEzzGgZgpTkMfFhqAJZgKGmepeS4wIbjuNMybaRERSb6uwJPAvo5104FxmOt8ttgGGOJYfgeo8SkWEREREcly7amgmQP8Fvx6PnAuMABz46wA2BG4y7X/RmCXVo7hrKBZiJkS1Q4e60Sgs2PbfsBUoN51nNaSQKmqoCkEdg4+7nNtO9LxnPvRK4EYRERSId4Kmk0ww0md295F9hVN9MJUBIXOMQCM8TUiEREREclq7UnQhB4zaLkh8N9d27fW78WZoAk95gF9W3jNWKDOsf1qYNMWtk9VgsZJ02yLSCaJJ0EzDDO7kTNpcUUc+94AM0w1lY+7EjvdmLphbiJMxvQ3c34/Jidh/yIiIiIizWpvgqYS0yC4JXlAleM1i1rZ3p2gacRUmbTmctfrrmphWyVoREQitZag2RP43fF8HXBynPvu7dp3Kh6PJnS2MDTO/c4Hjkpw3x2eptkWERER8d5UYFUr2zQATzmW+2BK5OP1BKbMvDXXAMsdy6fQepNLERFp3XGYHjOh2e/WAAdjmr9nKxszVHVnTL8dSUC2jXcTERERSXcBIhMvLfnCtbwZ8Gucr304zu1qgvGcEVzeBDOr0pw4Xy8iItEmAuWEE96/YZq3f57APtYA45Mcl9vPSd6fhel7djhwLVCB6XcmcVCCRkRERMRbVZhy93gscy23NizK6YMEtn2fcIIG4P9QgkZEpC1ygZuAsx3rvgMOwPShSUQtphoynVQR2bS+E+Zv07aYBNSo4PpuwJXAHpiqofUexpixlKARERER8daKBLZ1T0saz7TcYBI77uROS2a7lvsl8FoREQl7GpOQCHkHOIz4E/PprobYw2dfxFTL7A48iKn4BNgb+DdwmifRZTj1oBERERHxVp0Hx1jZzu17JisQEZEOxpmc+Q1TOZMtyZl4vIOpmnH+XTkFU2EjrVAFjYiIiEj2SXS8vztpFG+ljoiIRPoB2Dr49caYhrnH0bbkvEW4wXCq1AHrkrzPKkz/mauDyxbwZ6AkycfJOkrQiIiIiGSfLglu39W1nGgFTms0K5SIdBR7Aq8B2wWXj8Q0Yh9H9LDV1mwALE1aZLE9BhyTgv2+QDhBA5F9a6QZGuIkIiIikn02AvIT2N7dc2Z5zK3MDFQhuQnsv1sC24qIZLLFwBgi+7QchElYdPIlIn8sdC1v4EsUGUYJGhEREZHsUwhsk8D2O7qWv2pmu7WOrxOp0hmQwLYiIpluOaY5rnM2vX2A6URXLGYr96yDq3yJIsNoiJOIiIhIdjoE+DLObQ91fN1A7Bk6IHIGqi0TiGWfBLatdS13IfFhASIiflsF7Ac8j6moARgNzAD2J77GwStJ/dCgVDUwHuladlfUiIiIiIgkxTDAdjxea2X7Tx3bvpfAcfZ0HefgFrZ907XtAqA4jmPsihm6FHrdcy1s+1/HdgFgizj2v7srLhsY0cL2f3NtOzSOY4iI+OUGIq9Z7p5bxcArrm0+Azb0MEav5QKfEHnOJ/kZUKbQECcRERGR7LQ5kQ0aY+kM3ELkB4pbW9j+fcfXFnBJK/vfALirlW3c5riWd0jw9SIi6WQ9cBjwrGPdjpik+sa+RBS/DTE3BhJp9J4H3EZk5c9qTA8eEREREZGkS/cKmjrH1xVAQYzt+wFvufb/SivxdMaU7Ye2bwT+SuybfqMx083amClc462g2cgV/3xgbHC9iEi6aa2CJiQPeMS17RxgMw9ibKstMXF+DVwGbE/zDei7AccD3xBdNfnXlEcqIiIiIh1WuidovgTudyz/DPwbuBCYiPmQ4E6aLCK+IUuXEv3m+0fgDkwy6A7Mm/nQc79g3pzHm6ABeCjGMWxML5oVjseRccQrIpJK8SZowAz9udu1fRUwMMUxtlUoQeN81GIayb+L6afzLjAXk7CPdd2+G43cEREREZEUyoQETWfgbWK/YXY/FgPbxhlTLvB0nPv9BdgJOMa1vrUEzQZEfs+aexwbZ8wiIqmSSIKG4PM3E32t3DqFMbZVrARNvI/1mKobJWcSoG+WiIiISHZah5nmdQpmJpBYGoAHMcmZb+LcbyMwHpgMrGlmm3rgcUxy5rM49+u0HDMDyJ+BJzF3ZzVFq4hkAxs4D7jWsW5TTEJ9O18iat5PmAT7I5gqy3gsBv6FuZExBdNQXuKUSLMfEREREUlfb2IqbsCUn2/veK4I2BdTRr8JsBZTVv8KsKwdx+wM7IX5UNETkwhagKkoWtKO/YqISPoZgLneD8T0BSvEDD1djbn2f44Z8qqkjIiIiIh0aO4hTiIiIpJB8vwOQERERCSNWUBfv4OIk3OmpnzSf/rWZPsdcydXRMQvfdEolZWY/jMiIiIiIknVg7Y3SNTD28cxzfwMRUS80txMRh3pcXK7v4sdmCpoRERERJpnkzlj6S0i79xmStzJ0tHOV0TST6PfAaQBXYtFREREpMN7k/AdTPWgERERyTCqoJGOohDYChgEdMHMOtEDqMZMQ7oG+AWYg5naU0RERERERMQzStBIthoEjAk+RgJbADlxvnY5Zoq4t4CZwCdAQ/JDFBERERERERHJPn2BizFl3clsdLUUuAkY4d2piIiIJGwrYOfgYxufYxERERGRDmhb4EGgntR3Jf8MOIr4q3FERERERERERLLaIOBpTKfwRJIsq4FfgXnAAmAFiU+J9x1waOpPUUREREREREQkPRUBfwfW03ISpRH4FLgGGAfsAHRqZp+5wEBgLDABeBnTOLi1RM0LwIAkn5+IiIiIiIiISFrbBviGlpMmHwBnAxu081j5wP7Aw5jZnpo73jrg5HYeS0REREREREQkI5wErCV2kqQBeAQYnqJj98RU7Sxr5vg2cB9m+m4RERERERERkaxjAVNpPjEyHRjiUSxdgKuBmmZi+R/Q26NYREREREREREQ8kQvcQexkyGJMbxk/bA283Uxc3wNb+BSXiIiIiIiIiEhSWcCdxE6CvA1s6l9ogEkelWGGV7nj+wnYzLfIRERERERERESSZBqxkzP/xiRH0sUBxO6N8yXQw8e4RERERERERETa5UxiJ2cm+xlUC/4ALCU63jdIr2SSiIiIiIiIiEhctiP2tNYlfgYVh22BFUTHXeZjTCIiIiIiIiIiCesCzCE6yXG9n0ElYA9gPZGxNwJj/AxKRERERERERCQR15L5w4ROJfoc5gJFfgYlIiIiIiIiIhKPYUAdkYmNRUDfdu63NzAWuAx4GpiPGYrkfPyxncdwu4/oJM0VST6GiIiIiIiIiEjSzSQ6qTG2Dfvpi+lX8yQmGROr2bD7sXv7Qo/SFfjZdYz1aOptEREREREREUljI4lOmjzRxn2NjbEvrxM0AONjHOemFBxHRERERERERCQpXiZ51SbpkqCB6Kqgato/ZEtERERERESyTI7fAYgA2wD7u9bdgxkilAwB4AfgIeDi4MMr/3AtFwNne3h8EREREREREZG4VBBZZVIPDGjH/nYlnIzZA9MPxmk03lXQALzrOlYlYKXweCIiIiIiIiIiCckBfiIygfFcio/pdYLmOI+PJyIiIiIiIhlGQ5zEbyOJ7jXzgB+BpNCzwGrXuqP8CERERERERETSkxI04re9XMtrgBf8CCSFqjFJGqe9/QhERERERERE0pMSNOI3d4LmHaDWj0BS7HXX8lBgYz8CERERERERkfSjBI34KRfT0NfpTT8C8cBM17IF/NGPQERERERERCT9KEEjftoCKHKt+8iPQDywkOhpw4f4EYiIiIiIiIikHyVoxE9bx1j3vedReOcH1/JWvkQhIiIiIiIiaUcJGvHTINfy8uAjW7kTNFv6EoWIiIiIiIikHSVoxE89XMuLfInCO4tdy919iUJERERERETSjhI04qcuruW1vkThndWu5W6+RCEiIiIiIiJpJ8/vAKRD62gJmjWu5a6+RCEiIl7IAQYDQ4HeLine truncated
+    }
+   },
+   "cell_type": "markdown",
+   "id": "15095a9a",
+   "metadata": {},
+   "source": [
+    "<div>\n",
+    "<img src=\"attachment:g33443.png\"  align=\"left\" width=\"450\"/>\n",
     "</div>"
    ]
   },
   {
    "attachments": {
     "g38913.png": {
-     "image/png": "iVBORw0KGgoAAAANSUhEUgAABGgAAAGhCAYAAAA9VYLrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13fBR1/sfx12xICL1KRzooKiLq2UDBCvaGevZer6onB+rp3SlFvVPvflctZ69nOQuiIKjYFREbSgtFlN4TCEl2fn98s9nZ2U2ym+zMbHk/H499kJ3M7nw2JLO77/1+P18QERERERERkYa4B7AdFyvYciSbNQm6ABEREZEs1Bd4xnH9I+CagGoREREJ2qXAla5t04HxAdSStRTQiIiIiKSuGbCv4/qGoArJQZcCZzquHwfsDKgWERGpX2/gbqCla/ti/0vJbgpoRERERCST9AeOdFwPBVWIiIjUywL+RXw4Iw2gJzwRERERERERaYhLgaOrv94aZCG5QAGNiIiIiIiIiKSqG3BH9ddh4NbgSskNCmhEREREREREJFX/ANpWf/034L0Aa8kJCmhEREREMkcHvJ/H3xEo9vgYIiKS2y4ATqz+ejlwY4C15Aw1CRYRERHx3o3ArtVfLyY6JLwN5kXuBcBgosHJVmAG5tPJ6Uke4zpgYPXXK4E/VH/dDdMj4CygH1BUvX018DLwd2BukseYQvTT0k+A+5O83f7VNURMApY6rncC/lj99QGu2/4fUFXL/d5fXYeISDaxgF9gzvsRSzHn2HAQBaWoC/Bnx/WrUf8ZEREREQnIHoDtuLxRz/6fOvZ9t3rbgZgX5HY9l/+Q3Idqsxy3mVe97STMEuB13X8l5k1BMiOrv3fc7skk9o84y3XM/V3f719PjbVdfppCDSIiXriH2POSVc/+TTCrHjlvMw/o4WGN6fYc0dqfcGw/gNjH9bT/pWU3jaARERER8d/BwEygaRL7XghsA36e4jGOA14g+mahEliDGUHT0bFfAXBD9bZLMS+qRUQk/VpiQotjHdtmAKcBWwKpKHVnAqdWf70B+HWAteQc9aARERER8VcnzKePTYFlwM+AvkBh9bZhwIOu21wN7JfCMToCD2PCmVWYkKct0B3YBeiJmWZU6bjNxdW1BGEF5vHth6nb6WDH99yX132sUUSkMboCbxMbzjxUfT1bwpkOwL2O69dipsuKiIiIiASmMVOcIpfp1N0Q+Heu/evr9+Kc4hS5LMb0CqjNaGCnY/8tmBCnNl5NcXKa7NpXDY1FJJMlM8VpD0wgH9knDNySxH13wIxS8fLyQAqP9Qlin8Pcj1VTnERERETEd40NaJZgGgTXpQlQ4rjNqnr2dwc0VcC+9dwG4GbX7f5Yx74KaEREYtUX0IwCNjq+XwFcnuR9d3TdtxeXp5Ks5XjHbUoxTefdFNA0kqY4iYiIiPhvIrC5nn0qMVOhIjpjVmRK1rPAnCT2uxNY77h+MfU3uRQRkfqdDkwluvrdNkzz9n8HVlHDtAH+6bh+C2aEpqSZmgSLiIiI+CtMbPBSl89d13sCPyR52yfq3wWAHdX1RD7R7QYMABYkeXsREYk3DtPrKxJ4/4hp3j43hfvYCpyR5rrcViSxz91Ep7/OwYwaEg8ooBERERHxVwlmuHsy1rmu1zctyun9FPZ9j9gh9z9BAY2ISEMUYAKMaxzbvsE0A16W4n2VY0ZDBukITKN5MCM7Lye2wbykkQIaEREREX9tSGHfHa7rySzLDSbYcYc7dZnvut4jhduKiEjU88AJjuvvACeTfDCfSVphVhWMjAL6E/BZcOXkPgU0IiIiIv7a6cMxNjVy/3bpKkREJM84w5kfMSNnSgOqpbHuAHat/noR8PsAa8kLCmhEREREck9Fivu7Q6NkR+qIiEis74BB1V93BR4GzqZh4bxFtMGwV3aSOEDaDbii+mu7+uvtHteS9xTQiIiIiOSelinu38p1PdUROPXRqlAiki8OA94AhlRfPw14sfrfVAOODsDa9JWW0NPAWQm2tyN67raANxtw32cQ2+T4feCQBtxP3tAy2yIiIiK5pxNQmML+7p4z6xPuZVagiihI4f5bp7CviEg2WwOMBD5ybBsDvEbq4bnkGQU0IiIiIrmnKTA4hf33cV3/opb9tjm+TuWNRp8U9hURyXYbgWOIXU3vMExIo8BaaqUpTiIiIiK56URgXpL7nuT4uhKYU8t+zhWo+qdQy5Ep7Fvuut6S+NWsREQy3WZMSPMSMKp623DM9KcxJLeq0yZgP0+qi6qtji8acOw9gYcc16cD4x3XtyF1UkAjIiIikpsuBe6i/p4HBwI/cVyfSu0vor8k2j+gP9ALWFbP/R8K7FvPPk5bXNd3IbUlw0VEMsU2TBjzDCY0BzgAmAkcTf39ZeoKzL1W2oBju/OFjQ24j7ymKU4iIiIiuWlX4LZ69mkB/J3YJr7/rGP/9xxfW8B19dx/R+CBevZxW+C6PjTF24uIZJJyYCzwvGPbUOAdoHsgFUnGUkAjIiIiknsiy2xfC0wGihLs0wN4ldj+M9MwPRJq8wKxI1yuAX5N4teUh2L6L/QHypKq2viQ2GXCJwGjMY2PRUSy0U7MakaPOLbtBsxGPbpERERERBplD8B2XN6oZ/9PHfu+m8JxRrqOc0Id+85y7DcP80Ygcn0F8Ffgl8A44EnM8HXnfa/CTFmqz29ct7OBhcB9mDDoPuArx/e+x4Q4zv33r+cYjyc4ho3pRbPBcTktiXpFRLx0D7HnKauOfQswowqd+y8jtZ5emewAYh/b08GWIyIiIiL5IBsCmhbA2yQOOtyX1ZjmjskowAzVT+Z+vweGAWe5ttcX0HQg9mdW2+WnSdYsIuKVVAIaqr/vvs2PmOeVbKeAppE0xUlEREQkN5UCRwC3Y1YCSaQSeAwTznyV5P1WYYbq3wRsrWWfCkxTzGHAZ0ner9N64CDgHOC/wCLMiigiItnOBn5FbI+wLpjGwUMCqUgyRn3pnoiIiIhkh1mYETdglkfd2/G9YsyKIX2BbpiVRUow/WYas0JSC+BwYC+gHSYIWoYZUbSmEfcrIiIiIiIiIpKV3FOcREREJIu41ykXERERkSgLM/Q8GzhXaioEugZVSEA2YpoIi4gEpQuapbIJ2B50ESIiIiKSe9qSXDNcXYK/nFXL/6GIiF+qCP5cGPTlokb/FPOYRtCIiIiI1M4GwkEXkSSL2E9us6XudMm3xysimacq6AIygM7FIiIiIpL3ZhH9BFM9aERERLKMRtBIvmgKDMSsXtEKs+pEW6AMswzpVmAl8B1maU8RERERERER3yigkVzVDxhVfTkI6AWEkrztemAu8BYwE/gEqEx/iSIiIiIiIiIiuacLcC3wOeltdLUW+Auwv38PRUREJGUDgX2rL4MDrkVERERE8tAewGNABd53Jf8MOJ3kR+OIiIiIiIiIiOS0fsBzmE7hqYQsW4AfgcXAMmADqS+J9w1wkvcPUUREREREREQkMxUDv8M0+K0rRKkC5gB3AmcAQ4HmtdxnAaaB8GjgBmAqpnFwfUHNy0CfND8+EREREREREZGMNhj4irpDk/eBq4AOjTxWITAGeIK6w6BS4KJGHktEREREREREJCtcCGwjcUhSCTwJDPHo2O0wo3bW1XJ8G3gYs3y3iIiIiIiIiEjOsYCJ1B6MTAMG+VRLS+A2YEcttXwEdPSpFhERERERERERXxQA95E4DFkNjA2orkHA27XU9S3QK6C6RERERERERETSygLuJ3EI8jbQPbjSABMe3YqZXuWubznQM7DKRERERERERETSZAqJw5m/YsKRTHEsiXvjzAPaBliXiIiIiIiIiEijXEHicOamIIuqwwHAWuLrfZPMCpNERERERERERJKyF4mXtR4fZFFJ2BPYQHzdtwZYk4iIiIiIiIhIyloCC4gPOe4OsqgUHAZsJ7b2KmBUkEWJiIiIiIiIiKTiLrJ/mtAlxD+GRUBxkEWJiIiIiIiIiCRjD2AnscHGKqBLI++3IzAauBF4HliKmYrkvBzcyGO4PUx8SHNLmo8hIiIiIiIiIpJ2M4kPNUY34H66YPrV/BcTxiRqNuy+HNq40uO0Ala4jrEdLb0tIiIiIiIiIhnsIOJDk2cbeF/HJLgvvwMagDMSHOcvHhxHRERERERERCQtppK+0SaZEtBA/KigMho/ZUtERERERERyTCjoAkSAwcAY17b/YKYIpUMY+A54HLi2+uKX37uuNwOu8vH4IiIiIiIiIiJJmUzsKJMKoE8j7u8AomHMYUBr1/dH4N8IGoDZrmMtASwPjyciIiIiIiIikpIQsJzYAON/Hh/T74DmbJ+PJyIiIiIiIllGU5wkaAcR32vm0SAK8dCLwBbXttODKEREREREREQykwIaCdrhrutbgZeDKMRDZZiQxumIIAoRERERERGRzKSARoLmDmjeAcqDKMRjM1zXdwe6BlGIiIiIiIiIZB4FNBKkAuBA17ZZQRTig5mu6xZwcBCFiIiIiIiISOZRQCNB6gUUu7Z9GEQhPlhJ/LLhuwVRiIiIiIiIiGQeBTQSpEEJtn3rexX++c51fWAgVYiIiIiIiEjGUUAjQernur6++pKr3AFN/0CqEBERERERkYyjgEaC1NZ1fVUgVfhntet6m0CqEBERERERkYyjgEaC1NJ1fVsgVfhni+t660CqEBERERERkYzTJOgCJK/lW0Cz1XW9VSBViIiIH0LAAGB3oCNm9b6NwCLgK6AyuNIapS1wEGYVxiHALtWXjkAZ5jHOA94GniH+wwkRkVzQBtgb6Am0ALYD64A5wJoA62qMEOY568DqS0+gC+b8XgxsBpYCnwLPAx8FUqWIeOYewHZc3vbpuCNcx7WBQ3047pWuY7qnPImI5IOuwAnArcArwPfABsdldGCVNV5b4BLMC9fNxD/XRC5bgfswL4SzyT+AMLU/LvdlGzCO3PpAcBeghNjf2U8CrUgkcxQCw4DLgX9hwgrn38rs4EpLLine truncated
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAABGgAAAGhCAYAAAA9VYLrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13fFvV/f/x15VnnD3JIGSRQQIZjLIymWHPAGWWVVYnKySBFihZQFvafsuvlFE2YdMCIayEvUMIK2QnJIEMsmPHjm3d3x/Hsq6uZFuydXU13s/HQ49YV1e6H8X2tfTWOZ8DIiIiIiIiItIYdwG242L5W45ksny/CxARERHJQL2BpxzXPwau8qkWERERv10CXO7a9jowwYdaMpYCGhEREZHENQP2c1zf5FchWehS4AzH9eOAXT7VIiIiDesJ/BVo4dq+NPWlZDYFNCIiIiKSTvYEjnBcD/hViIiINMgC7iE6nJFG0B88EREREREREWmMS4Cjar7e7mch2UABjYiIiIiIiIgkqitwe83XQeBm/0rJDgpoRERERERERCRR/w9oU/P1P4H3fawlKyigEREREUkf7fF+Hn8HoNjjY4iISHa7ADix5uvvgUk+1pI11CRYRERExHuTgD1qvl5KeEh4a8yL3AuAgYSDk+3AG5hPJ1+P8xjXAP1qvl4D3FrzdVdMj4CzgD5AYc32dcCLwN3AvDiPMZ3wp6WfAvfFeb8DamoImQqscFzvBPyp5usDXff9P6C6jse9r6YOEZFMYgG/wZz3Q1ZgzrFBPwpKUGfgL47rV6L+MyIiIiLik0GA7bi81sD+nzn2fa9m20GYF+R2A5f/EN+HanMc95lfs+0kzBLg9T1+FeZNQTwjq1c77vdEHPuHnOU65gGu2/s2UGNdl58nUIOIiBfuIvK8ZDWwfz5m1SPnfeYDu3tYY7I9S7j2xx3bDyTyeT2Z+tIym0bQiIiIiKTeIcBsoCiOfX8B7AB+neAxjgOeJ/xmoQpYjxlB08GxXx5wfc22SzAvqkVEJPlaYEKLYx3b3gBOA7b5UlHizgROrfl6E/B7H2vJOupBIyIiIpJanTCfPhYBK4FfAb2Bgppt+wIPuO5zJbB/AsfoADyECWfWYkKeNkA3oCPQHTPNqMpxn4tqavHD95jntz/wsOu2Qxy3uS+vprBGEZGm6AK8TWQ482DN9UwJZ9oDf3NcvxozXVZERERExDdNmeIUurxO/Q2B/+Dav6F+L84pTqHLUkyvgLqMBXY59t+GCXHq4tUUJ6fprn3V0FhE0lk8U5wGYQL50D5B4I9xPHZ7zCgVLy/3J/BcHyfyb5j7uWqKk4iIiIikXFMDmmWYBsH1yQeWO+6ztoH93QFNNbBfA/cBuMl1vz/Vs68CGhGRSA0FNGOAzY7bK4FfxvnYHVyP7cVlRpy1HO+4Tymm6bybApom0hQnERERkdSbAmxtYJ8qzFSokN0wKzLF62lgbhz73QFsdFy/iIabXIqISMNOB2YSXv1uB6Z5+799q6hxWgP/clz/I2aEpiSZmgSLiIiIpFaQyOClPl+4rncHfojzvo83vAsA5TX1hD7R7YpZVWlRnPcXEZFo4zG9vkKB94+Y5u3zEniM7cAZSa7LbVUc+/yV8PTXuZhRQ+IBBTQiIiIiqbUcM9w9Hj+5rjc0LcrpgwT2fZ/IIfc/QwGNiEhj5GECjKsc277FNANemeBjVWBGQ/rpcEyjeTAjO39JZIN5SSIFNCIiIiKptSmBfctd1+NZlhtMsOMOd+qzwHV99wTuKyIiYc8BJziuvwOcTPzBfDppiVlVMDQK6M/A5/6Vk/0U0IiIiIik1q4UHGNLE/dvm6xCRERyjDOc+REzcqbUp1qa6nZgj5qvlwC3+FhLTlBAIyIiIpJ9KhPc3x0axTtSR0REIi0E+td83QV4CDibxoXzFuEGw17ZRewAaQBwWc3Xds3XOz2uJecpoBERERHJPi0S3L+l63qiI3AaolWhRCRXjAJeAwbXXD8NeKHm30QDjvbAhuSVFtOTwFkxtrclfO62gDcb8dhnENnk+APg0EY8Ts7QMtsiIiIi2acTUJDA/u6eMxtj7mVWoArJS+DxWyWwr4hIJlsPjAY+dmw7BniFxMNzyTEKaERERESyTxEwMIH9h7muf1nHfjscXyfyRqNXAvuKiGS6zcDRRK6mNwoT0iiwljppipOIiIhIdjoRmB/nvic5vq4C5taxn3MFqj0TqOWIBPatcF1vQfRqViIi6W4rJqT5HzCmZttwzPSnY4hvVactwP6eVBdWVx1fNuLYewMPOq6/DkxwXN+B1EsBjYiIiEh2ugS4k4Z7HhwE/MxxfSZ1v4j+inD/gD2BHsDKBh5/JLBfA/s4bXNd70hiS4aLiKSLHZgw5ilMaA5wIDAbOIqG+8vUF5h7rbQRx3bnC5sb8Rg5TVOcRERERLLTHsBtDezTHLibyCa+/6pn//cdX1vANQ08fgfg/gb2cVvkuj40wfuLiKSTCmAc8Jxj21DgHaCbLxVJ2lJAIyIiIpJ9QstsXw1MAwpj7LM78DKR/WdmYXok1OV5Ike4XAX8ntivKUdi+i/sCZTFVbXxEZHLhE8FxmIaH4uIZKJdmNWMHnZsGwC8i3p0iYiIiIg0ySDAdlxea2D/zxz7vpfAcUa7jnNCPfvOcew3H/NGIHR9FfAP4LfAeOAJzPB152OvxUxZash1rvvZwGLgXkwYdC/wteO21ZgQx7n/AQ0c47EYx7AxvWg2OS6nxVGviIiX7iLyPGXVs28eZlShc/+VJNbTK50dSORze9LfckREREQkF2RCQNMceJvYQYf7sg7T3DEeeZih+vE87mpgX+As1/aGApr2RP6f1XX5eZw1i4h4JZGAhprb3ff5EfN3JdMpoGkiTXESERERyU6lwOHAZMxKILFUAY9iwpmv43zcasxQ/RuB7XXsU4lpirkv8Hmcj+u0ETgYOAd4BliCWRFFRCTT2cDviOwR1hnTOHiwLxVJ2mgo3RMRERGRzDAHM+IGzPKoQxy3FWNWDOkNdMWsLLIc02+mKSskNQcOA/YB2mKCoJWYEUXrm/C4IiIiIiIiIiIZyT3FSURERDKIe51yEREREQmzMEPPM4FzpaYCoItfhfhkM6aJsIiIXzqjWSpbgJ1+FyEiIiIi2acN8TXD1cX/y1l1fA9FRFKlGv/PhX5fLmzy/2IO0wgaERERkbrZQNDvIuJkEfnJbabUnSy59nxFJP1U+11AGtC5WERERERy3hzCn2CqB42IiEiG0QgayRVFQD+gD9ACs+pEG6AMswzpdmA1sAiztKeIiIiIiIhIyiigkWzVBxhTczkY6AEE4rzvRmAe8BYwG/gUqEp+iSIiIiIiIiIi2aczcDVmWHcyG11tAP4OHJC6pyIiIpKwfsB+NZeBPtciIiIiIjlob+BRoBLvu5J/DpxO/KNxRERERERERESyWh/gOUyn8ERClm3AD8BSYCWwicSXxPsWOMn7pygiIiIiIiIikp6KgT8AO6k/RKkGPgPuAMYBQ4GSOh4zD+gNjAWuB2ZiGgc3FNS8CPRK8vMTEREREREREUlrA4GvqT80+QC4AmjfxGMVAMcAj2NWe6rreKXAhU08loiIiIiIiIhIRvgFsIPYIUkV8AQw2KNjt8WM2vmpjuPbwEOY5btFRERERERERLKOBUyh7mBkFjAgRbW0AG4Dyuuo5WOgQ4pqERERERERERFJiTzgXmKHIeswvWX80B94u466vgN6+FSXiIiIiIiIiEhSWcB9xA5B3ga6+VcaYMKjmzHTq9z1fQ90960yEREREREREZEkmU7scOYfmHAkXRxL7N4484E2PtYlIiIiIiIiItIklxE7nLnRz6LqcSCwgeh63yS9wiQRERERERERkbjsQ+xlrSf4WVQc9gY2EV33zT7WJCIiIiIiIiKSsBbAIqJDjr/6WVQCRgE7iay9GhjjZ1EiIiIiIiIiIom4k8yfJnQx0c9hCVDsZ1EiIiIiIiIiIvEYBOwiMthYC3Ru4uN2AMYCk4DngBWYqUjOyyFNPIbbQ0SHNH9M8jFERERERERERJJuNtGhxthGPE5nTL+aZzBhTKxmw+7LyKaVHqUlsMp1jJ1o6W0RERERERERSWMHEx2aPN3Ixxob47FSHdAAnBHjOH/34DgiIiIiIiIiIkkxk+SNNkmXgAaiRwWV0fQpWyIiIiIiIpJlAn4XIAIMBI5xbfsPZopQMgSBhcBjwNU1l1S5xXW9GXBFCo8vIiIiIiIiIhKXaUSOMqkEejXh8Q4iHMaMwvSDcRpB6kbQALzrOtYywPLweCIiIiIiIiIiCQkA3xMZYPzX42OmOqA5O8XHExERERERkQyjKU7it4OJ7jXziB+FeOgFYJtr2+l+FCIiIiIiIiLpSQGN+O0w1/XtwIt+FOKhMkxI43S4H4WIiIiIiIhIelJAI35zBzTvABV+FOKxN1zX9wK6+FGIiIiIiIiIpB8FNOKnPExDX6c5fhSSArNd1y3gED8KERERERERkfSjgEb81AModm37yI9CUmAN0cuGD/CjEBEREREREUk/CmjET/1jbPsu5VWkzkLX9X6+VCEiIiIiIiJpRwGN+KmP6/rGmku2cgc0e/pShYiIiIiIiKQdBTTipzau62t9qSJ11rmut/alChEREREREUk7CmjETy1c13f4UkXqbHNdb+VLFSIiIiIiIpJ28v0uQHJargU0213XW/pShYiIpEIA6AvsBXTArN63GVgCfA1U+Vdak7TFrMB4EDAY6Fhz6QCUYZ7jfOBt4CmiP5wQEckGrYEhQHegObAT+AmYC6z3sa6mCGD+ZoXO8d2BzpjzezGwFVgBfAY8B3zsS5Ui4pm7ANtxeTtFxx3hOq4NjEzBcS93HdM95UlEJBd0AU4AbgZeAlYDmxyXsb5V1nRtgIsxL1y3Ev23JnTZDtyLeSGcSf4FBKn7ebkvO4DxZNcHgh2B5UT+zH7qa0Ui6aMA2Bf4JXAPJqxwLine truncated
     }
    },
    "cell_type": "markdown",
-   "id": "0aca29d3",
+   "id": "53a05c72",
    "metadata": {},
    "source": [
     "<div>\n",
@@ -266,7 +326,7 @@
    "source": [
     "### Serial performance\n",
     "\n",
-    "This algorithm is memory bound, meaning that the main cost is in getting and setting data from the input matrix `C`. In this situation, the order in which we traverse the entries of matrix `C` has a significant performance impact.\n",
+    "Before starting to parallelize this code, we want to make sure that we have an efficient sequential implementation.  In this algorithm, the order in which we traverse the entries of matrix `C` has a significant performance impact.\n",
     "\n",
     "The following function computes the same result as for the previous function `floyd!`, but the nesting of loops over i and j is changed.\n"
    ]
@@ -347,7 +407,23 @@
     "```\n",
     "\n",
     "- The outer loop cannot be parallelized since the operations at iteration $k$ require the result of iteration $k-1$.\n",
-    "- The inner loops over $i$ and $j$ can be parallelized at a given iteration $k$."
+    "- The inner loops over $i$ and $j$ can be parallelized at a given iteration $k$.\n",
+    "\n",
+    "\n",
+    "<div class=\"alert alert-block alert-success\">\n",
+    "<b>Question (hard):</b>  Can we really parallelize the loops over `i` and `j` ? To compute `C[i,j]` at iteration `k`, we first need to compute `C[i,k]` and `C[k,j]`. In order words, it seems that the order of the loops over `i` and `j` cannot be arbitrary. However,  this is not really true, why?\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8d05b686",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "uncover = false\n",
+    "q1_answer(true)"
    ]
   },
   {
@@ -411,7 +487,7 @@
    "id": "db9a3294",
    "metadata": {},
    "source": [
-    "At each iteration, the processor needs the input values `C[i,j]`, `C[i,k]` and `C[k,j]`. Since we split the data row-wise, the process already has values `C[i,j]` and `C[i,k]`. However, `C[k,j]` may be stored in a different process. "
+    "At each iteration, the processor needs the input values `C[i,j]`, `C[i,k]` and `C[k,j]`. Since we split the data row-wise, the process already has values `C[i,j]` and `C[i,k]`. However, `C[k,j]` may be stored in a different process. See figure below."
    ]
   },
   {
@@ -481,7 +557,7 @@
    "metadata": {},
    "source": [
     "<div class=\"alert alert-block alert-success\">\n",
-    "<b>Question:</b>  How much data is communicated in each iteration in this parallel algorithm?\n",
+    "<b>Question:</b>  How much data is send from the owner of row k in each iteration in this parallel algorithm?\n",
     "</div>\n",
     "\n",
     "    a) O(N²/P)\n",
@@ -506,7 +582,7 @@
    "id": "eba45ea4",
    "metadata": {},
    "source": [
-    "### Efficiency"
+    "### Theoretical performance analysis"
    ]
   },
   {
@@ -514,7 +590,9 @@
    "id": "a15bc34e",
    "metadata": {},
    "source": [
-    "**Computation cost:** Each process updates $N^2/P$ entries per iteration. "
+    "### Computation complexity\n",
+    "\n",
+    "Each process updates $N^2/P$ entries per iteration. The computation complexity per iteration is $O(N^2/P)$."
    ]
   },
   {
@@ -537,8 +615,9 @@
    "id": "d89d2b45",
    "metadata": {},
    "source": [
-    "**Communication cost:** \n",
-    "- One process broadcasts a message of length $N$ to $P-1$ processes per iteration. Thus, the **send cost** is $O(N P)$ per iteration (if we use send/receive instead of broadcast).\n",
+    "### Communication complexity\n",
+    "\n",
+    "- One process broadcasts a message of length $N$ to $P-1$ processes per iteration. Thus, the **send cost** is $O(N P)$ per iteration.\n",
     "- $P-1$ processes receive one message of length $N$ per iteration. Hence, the **receive cost** is $O(N)$ per iteration at each process. "
    ]
   },
@@ -553,7 +632,7 @@
    "metadata": {},
    "source": [
     "<div>\n",
-    "<img src=\"attachment:fig-asp-efficiency-comm-2.png\" align=\"left\" width=\"300\"/>\n",
+    "<img src=\"attachment:fig-asp-efficiency-comm-2.png\" align=\"left\" width=\"250\"/>\n",
     "</div>"
    ]
   },
@@ -562,7 +641,13 @@
    "id": "6993b9d0",
    "metadata": {},
    "source": [
-    "In summary, the send/computation ratio is $O(P^2/N)$ and the receive/computation ratio is $O(P/N)$. The algorithm is potentially scalable if $P<<N$."
+    "The communication over computation cost is\n",
+    "\n",
+    "- On the send side: $O(NP)/O(N^2/P) = O(P^2/N)$\n",
+    "- On the receive side $O(N)/O(N^2/P) = O(P/N)$\n",
+    "\n",
+    "\n",
+    "In summary, the send/computation ratio is $O(P^2/N)$ and the receive/computation ratio is $O(P/N)$.  The algorithm is potentially scalable if $P^2<<N$. Note that this is worse than for matrix-matrix multiplication, which is scalable for $P<<N$. I.e., you need a larger problem size in the current algorithm than in matrix-matrix multiplication."
    ]
   },
   {
@@ -572,15 +657,7 @@
    "source": [
     "## Parallel Implementation\n",
     "\n",
-    "Now, we will explore the parallel implementation of Floyd's algorithm using MPIClusterManagers."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0b451628",
-   "metadata": {},
-   "source": [
-    "### Creating MPI workers"
+    "Now, we will explore the parallel implementation of Floyd's algorithm. We will use MPI."
    ]
   },
   {
@@ -590,7 +667,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "] add MPI, MPIClusterManagers"
+    "] add MPI"
    ]
   },
   {
@@ -600,171 +677,344 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "using MPIClusterManagers\n",
-    "using Distributed\n",
-    "if procs() == workers()\n",
-    "    nranks = 3\n",
-    "    manager = MPIWorkerManager(nranks)\n",
-    "    addprocs(manager)\n",
-    "    @everywhere using MPI; MPI.Init()\n",
-    "end"
+    "using MPI"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "680e56cf",
+   "id": "4c2e0010",
    "metadata": {},
    "source": [
-    "### Code\n",
+    "We are going to implement the method in a function with the following signature\n",
     "\n",
-    "We split the code into two functions. The first function is called on the main process (the process running this notebook). It splits the input matrix into blocks of rows. Then, we use a remotecall to compute Floyd's algorithm in each worker with its corresponding block of rows.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "22234b2f",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "function floyd_dist!(C)\n",
-    "    m = size(C,1)\n",
-    "    @assert mod(m,nworkers()) == 0\n",
-    "    nrows_w = div(m,nworkers())\n",
-    "    @sync for (i,w) in enumerate(workers())\n",
-    "        rows_w = (1:nrows_w) .+ (i-1)*nrows_w\n",
-    "        Cw = C[rows_w,:]\n",
-    "        @async C[rows_w,:] = remotecall_fetch(floyd_worker!,w,Cw,rows_w)\n",
-    "    end\n",
-    "    C\n",
-    "end"
+    "```julia\n",
+    "    floyd_mpi!(C,comm)\n",
+    "```\n",
+    "\n",
+    "The signature is similar to the sequential function `floyd!(C)`, but there are some important differences\n",
+    "\n",
+    "- This function is going to be called on several MPI ranks, but only one rank (rank 0 in the implementation below) will receive the input matrix, the other ranks will receive an empty matrix. It would not be memory efficient if all ranks receive a copy of the input matrix as this one could be very large.\n",
+    "\n",
+    "- On output, only the value of C will be correct on rank 0. It will not be efficient to try to write the result on all ranks.\n",
+    "\n",
+    "- The parallel function takes an MPI communicator object. This allows the user to decide which communicator to use. For instance, `MPI.COMM_WORLD` directly or a duplicate of this one (which is the recommended approach)."
    ]
   },
   {
+   "attachments": {
+    "g3642.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAABRQAAAMuCAYAAABo148ZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15fFTV/f/xV0IChMiusijIorK4oagoQusC1rbgtyp8tdaqSLUFQRCqBCWTZMKqYLW41VYK9VurFtpfv9J+bWMVC9a6gLsiqICyibIq+zK/Pz6MTiaT5NzJ3Nnyfj4e82i5nHPvBySfuffccz4HRDLLQ0Ao4jM3pdHUTWOgC9ARaJjiWEQkOeZSOYc9mNJoRETi979UzmdTUhtOXJoDnYEeQDugILXhiIjUC9Oo/P3x/1IbjsQrL9UBiNQjucAPgB8D/YAjo35/PbAImAP8M6mRiYi4yQdOBs4Eukb93h6gNNkBiYg4ygPOAb5z+H97UfVeDGAF8DLwZ2zQ9GCyAhQRkbRyPPATD+2/BLYD7wOvHf7/6aAB0B27f++GjUtEKgN2JzsokVTI1BmKvYBlVI69ps+zwHEpiVRE/DSXzJmh2AA4CbgWuA9YAuyi+ry1NTVhikiKZMIMxRzsJe4jwBbc78PCnzXAVUmPWkQku2XKDMUBeP/eCH8OAf8GrgMaJTHmXGzW/TXAvcBi4KtaYm0R78U0Q1HEf98F5gNNPPS5CHgVuBh4w4+gRERq8TLQO9VBiIjUwY+Ax+rQvyPwB+BS4HpgXwJiEhGR7JcDnHv483NslWIynuv/CFyehOsAGlAU8duZwAKq1uTZBfwV+Aj7OewGXIItJww7Cvg79kC/1vdIRUQq0z2CiGS6BjX83kFgFbAJuy9rDvQECmO0/SFW7/pKtARaRES8ORmbrXgxtuLHT0m9f9fDgoh/GmJvtaMHEx8DxgFfRB0/Fvg1NrAYdjS2RHKAPyGKiHh2AHgXWx6h0gwikkk2AY9jL2xfxOpdRcoHvofVg+0V9XtXACOB2f6GKCIiaewO4B8xjjcFWmOTgYZi9RcjFQB/wb5bPvUzwGocAj7AlkR3S8H1RdJCJtVQHEXV+gQPYdOfq5OPJZroft/xNVIRSZa5ZE4NxTeA/djg4e+AMVgtsnD5ht+iGooi9Vkm1FC8DovtZeAyKq8EqUkjLO/FqhXbPPFhiojUK5lcQ9Glrm4ucAuwN0b/3/gS6Tf+gs2kfx+byDQW6I8NeIK9MEtYDUWRTJMpA4qFwAYqx7oStzqKbbDZi5F9X6PmgUgRyQxzyZwBxZ5UnWEdSQOKIvVbJgwo9gcGxdk3D1uaFv3gdXViQhMRqbeyfUAx7IYY/fcDRyY2zEpO4JvBw1hKY8QU94Bi9HbRIpIYg4G2UccmYzV6avMZ8MuoY72BMxIQl4iIq/eA3akOQkSkDhYDC+PsewBb2hbtkhjHREREos3BaidGysM2YPXLSqqW8/CNaiimTg5VR4J3ULnQcz5wHtAH6AS0PNzmXeC+Ws5/JDYI1RPoDByBLd/YevjzFnaTtTHO+POoOvIdPTulAPth6QMcg9UU/AJYAzyDTcNNheZUHUzfQ2IfnC+N+vU2bMclV3OAYir/jP4XsLSOcYkkSi5Vl31tx+pzhDXEZoecxTc5bBvwJrXPzDsKy2EnHe57xOHzbcFyzZtYDtsUZ/zROSx0OLZITbA3k32A9of7fIEV8X8GWBHntesqVg7bjeUxEfGuAdAs6tg2LC+ENQa+jW221hHLZ5uxpbRzazl/O765J+uI5bM8vslnS7F6fpvjjD//8DnDDmH5OFJTrBj8mYfjycXy2YdYPvs4zmvXVQuqrsDYhS0TSwdLsP9GLSOOdUxRLCKSnlpG/fpL7IVEWAPgHKAv9lzeEstzK4HptZy7BXYf3QPoiuX6Ar55pn8PeAFYF2fsLt9/DYELsN2KO2Dfh19gdQD/gY0rpEJTqo5n7cVtAk8y/Q/23z7SOcCTKYhFskgrqk41DRfHzANuBdbHaBPCBuRi6Q3cBbyN3UzG6hv9+RdWfNqrPjHOFX7AzQcmUnXZbvTnxcPn8aKuS56LqPp3s5SqswnrIh9L8JHX8DKYGPafqHOkKlmLxHIMVX+mOxz+vYbABGy2bayf/epeJpwNzMJemrjmsH8CA+OI//yo80S+UGgEBKj6cxz9WYT3mcNzo87hZclzDrZMIfrv5j/YAGyyacmzZItTqPrzHR6gK8B+7jbHaBOi6swDsPuhftjmHR9U0y/6cwj46+F+Xl0eda7IB8tC7IH1y1qu/3/YgKcXdVnynAv8IkYcz5J+NQrfoHKMb6c2HBFJI7lUzWN9In5vBPbsHivvRr/4CTsFy6fLsMlGLt8h4TqxXlegdotxrlaHf68BVj97Yy3XXoq9cPOirkueR2CDtpHneI/EbxZY1yXPEHvc5PcJjNGr0hjxqIZiBqpuQPEo7Oa0ph/aWLsC/bSWPrV9fk/NtbKiVTeg2AZLaK7X3Ye3H8p4BxQbYA/usW6gj6ihXzxOjXGdW+M4z8yocxyi5noIIslU3YBiO6zmZ00/9x/EON/YWvrU9nkUGwh0dX5U//CA4jHA6x6uuwf4gYfrzo3q7zqgmI/NXI6+/l9wq83qh99GxaIBRclU1Q0odsYeUGrKAS/FOF9ZLX1q+/wCb6uIqhtQ7Aos93DdL4ELPVw33gHFxsD8GNd/DHshlW4+onKc/0ptOCKSRqobUGyOvSCpLedGu7KWPrV9/oK358XqBhRbAs97uO5B4EYP1413QDEnRt9wXm5VQ794JWJA8YQY5/hbAmP0qjRGPKqhmCUKsQGuc2tpF2tzjpp2rdvPN8tqqnM19ma8QS3XrknB4XOc7aFPPjYN+Jw6XLc2TYA/Y28yIs3Bah1+leDrdY9x7NU4zhPdJ6eac4uki2bY0ofetbTzmsP2YTkseklypBuwn/O6fK8dgeXgXh76NAKe8tjHq6bA08CwqOMPYgMJ6ba0QyQbHI3NgO5RSzuv+Wwvls+qm5kC9oJlXi3XrU1roIJvVr+4OALLNV3reO2atMYesq+IOj4NuBbL9+nkCKoucf4kFYGISMZoiL1wqa1OntfvjwN880wfqqbNpdh3l5eX7LFi+DP28t1VLvArrLSGXxpi4wZFUcf/ePi6W3y8dl3EmryU6PGHlNGAYnqZyjcP4gewH5ghwMnYzJ/TgR8Dz9VwjkPYUuLbsSTQEvvha42N2jfGBu/upurD+QVY3b54/Toi/o3AnYd/3QwbqGwP/Ah4J6pfA2z7dD/+PR6NvV0ZHHW8DPgJletbJEqsh4/VcZwnVp/aHmxEUmkWlq/AXmTMxZZfnITlsDOA67Dah9U5iL1lHA98C3tj1gjLYS2xFxd9gXuo+mb3u8Btccaegz3An3L41+uwG5ZeWA7Lw2YvXovN+ImUj+U/P7THauN8J+JYCCsrcTOV6+6KSOI8iM1QBJuJ/Ah2L9ETy2dnYfcRNb0wPIANnt2C1cRuit2HtcZyWyGW5x6kah3nq/E22yNSHvaiIxz/KmAclp+bYjmrw+Hzr4rq2wT/dp/vjN2jnhdx7CD2wvcOqn9ATqX/pups0YpUBCIiGeNOLLeDPZv/EZvVdgqWe087/OuaZqmFgFew3DgA+97I55tn+kZYTdzJWJmxSGcBM+oQ/y/4ZgnzZuy5uQ/2vZWLlQobiq1IipSDDSr6MdO8BVbv9+oYsV5FetcRjzUh6POkRyFZJ9aS5/DnQ755qHU1DLgXb4Wij6Hq8updWKKqTawlz+HPfGpeRtwQewMe3c9ltyMvS55PwP4uI9vvx2Yy+emxqGvuI76Zn+2o+ncUTFCMInUVa8lz+PM+3mfT/gxb5n+Mhz4dqbq8egdVi0vHcj7Vx/97al5GXIA9UEb3iy64HMvcqD41Pbj3xF4sRLbfi72YSQe/pXJsWvIsmSrWkufwZynfDMy5uh0ox15qujqRqsurN+D2YBa95Dk6x9R0jmbYsu3ofi453MuS5zOpWodrJ1Vf+KaTPKx+dfR98pGpDEpE0kqsJc/hz1pscM+LIcDDwPEe+hxF1eXV+/mmtnlNYi15Dn+eoealsA2Ax2P0i56BHouXJc8dsNq1ke0PYrP5/ZaIJc9PxDjHzxIYo1elMeJRDcUMVN2A4md4e6Cuq+bY2+nIGG536FfdgOLfcJtp2Iqqm7Y86tDPdUDxXGzkP7Ltl8AlDteoq+gb7Hh33cql6uYLte3uLZIs1Q0oforVUk2WVtjPWGQMox36nU/s+P9M7CUo0drwzS544c/9Dv3mRvWpbkDx29jSjci22/BW38xvGlCUbFHdgOJyknuTfQxVf+5dXiBUN6A4x/G6XbAZkpF9XV5gug4ofh9b3hXZdhPeN+ZLtglU/Tuty6wfEck+1Q0obsPboGBdNcFWAUbGMNWhX3UDikuoefl1WCF27x/Zd4FDP9cBxdOwgdnItruxgddkqOuA4llU3TwmRHL/bUQrjRGPaihmkbHEPwAVj+3YNOZI8ez6DPbWdhg2CFabLdhMvkheai/W5DKsdkTkG+SN2AP6Mwm6Rk0Ko34dvYzJ1SGqTt/WpiyS7kZhL0aSZQtVH2K/G+e5dgDDsS/W2nyGvXGMlKgcdhXwd2yJd9haoD81l7wQkcS6kZprtybaOqycQ6R489kG3F6uAHyMrRyJlKh8dhP2oBh5b/QhNqP75QRdww+9gZKoY5uwh2ARkdpMxHJdsuwCJkUdi/eZ/gBW4me/Q9udWOmySIn6/hiIlUGKnGy15fDx+Qm6hp96Y99/0SsV/0Vy/234SgOK6WUtVvMm2aJvIs/E2+6CYY/hbSDhH1G/7k7dCsiC3Tz/kco7Vi/HZiwuq+O5XUUv965LTYfowchE70gtkkgfYjNWki06h/XBbZZhtDl4K+gcncNOoe7fq7dhy0cic+HbWA57u47nFhF3r1JzvVe/ROezeDetewh70HMVnc9Oi/O6YTlYba9fUfmeLine truncated
+    }
+   },
    "cell_type": "markdown",
-   "id": "9fc3be11",
+   "id": "99554811",
    "metadata": {},
    "source": [
-    "The second function is the one that runs on the workers. Note that we use MPI for communication in this case."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "d5a8d3a8",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "@everywhere function floyd_worker!(Cw,rows_w)\n",
-    "    comm = MPI.Comm_dup(MPI.COMM_WORLD)\n",
-    "    rank = MPI.Comm_rank(comm)\n",
-    "    nranks = MPI.Comm_size(comm)\n",
-    "    m,n = size(Cw)\n",
-    "    C_k = similar(Cw,n)\n",
-    "    for k in 1:n\n",
-    "        if k in rows_w\n",
-    "            # Send row k to other workers if I have it\n",
-    "            myk = (k-first(rows_w))+1\n",
-    "            C_k .= view(Cw,myk,:)\n",
-    "            for proc in 0:(nranks-1)\n",
-    "                if rank == proc\n",
-    "                    continue\n",
-    "                end\n",
-    "                MPI.Send(C_k,comm;dest=proc,tag=0)\n",
-    "            end\n",
-    "        else\n",
-    "            # Wait until row k is received\n",
-    "            MPI.Recv!(C_k,comm,source=MPI.ANY_SOURCE,tag=0)\n",
-    "        end\n",
-    "        for j in 1:n\n",
-    "            for i in 1:m\n",
-    "                @inbounds Cw[i,j] = min(Cw[i,j],Cw[i,k]+C_k[j])\n",
-    "            end\n",
-    "        end\n",
-    "    end\n",
-    "    Cw\n",
-    "end"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "bc81a83d",
-   "metadata": {},
-   "source": [
-    "<div class=\"alert alert-block alert-success\">\n",
-    "<b>Question:</b>  Which of the following statements is true?\n",
+    "<div>\n",
+    "<img src=\"attachment:g3642.png\" align=\"left\" width=\"300\"/>\n",
     "</div>"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "4ec6718c",
+   "id": "075ee09d",
    "metadata": {},
    "source": [
-    "    a) The processes are synchronized in each iteration due to the blocking send and receive of row k.\n",
-    "    b) Receiving processes may overwrite the data in row k, which can lead to incorrect behavior.\n",
-    "    c) The sending process can only continue the computation after the data are received in every other process.\n",
-    "    d) The receiving process does not know the source of the received data."
+    "The implementation of the parallel function is given in the following code snipped:"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "4f4a57de",
+   "id": "37863d46",
    "metadata": {},
    "outputs": [],
    "source": [
-    "answer = \"x\" # replace x with a, b, c or d\n",
-    "floyd_impl_check(answer)"
+    "code1 = quote\n",
+    "    function floyd_mpi!(C,comm)\n",
+    "        myC = distribute_input(C,comm)\n",
+    "        floyd_iterations!(myC,comm)\n",
+    "        collect_result!(C,myC,comm)\n",
+    "    end\n",
+    "end;"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c624722a",
+   "id": "c6cdae12",
    "metadata": {},
    "source": [
-    "### Testing the implementation"
+    "In the following cells, we discuss the implementation of the helper functions `distribute_input`, `floyd_iterations!`, and `collect_result!`."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "627378c3",
+   "metadata": {},
+   "source": [
+    "### Distributing the input matrix\n",
+    "\n",
+    "Since only rank 0 receives the input matrix C, this rank needs to split it row-wise and\n",
+    "send the pieces to all other ranks. This is done in the function below. We start by communicating the problem size N to all ranks (at the start only rank 0 knows the problem size). This is trivially done with an `MPI.Bcast!`. Once all ranks know the problem size, they can allocate space for their local part of `C` , called `myC` in the code. After this, rank 0 sends the pieces to all other ranks. We do it here with `MPI.Send` and `MPI.Recv!`. This can also be done with `MPI.Scatter!`, but it is more challenging since we are using a row partition and Julia stores the matrices in column major order. Note that this algorithm can also be implemented using a column partition. In this case, using `MPI.Scatter!` would probably be the best option.\n"
+   ]
+  },
+  {
+   "attachments": {
+    "g5958.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAABSkAAATJCAYAAAA7PJMlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15eJTlof7x70wSdgFxAXFDLSouVQ9WRQhbCbYq2ipUjwdBxZ/WDaypqEcTIJFqtWkLlVo90IpaTy3QcyxqjwYECWrdagWVKlWxyC4oKGvIvL8/RiOEQN6EzLxJ5vu5rvkj7zzPzA0ql94+C0iSJEmSJEmK2hFAUOXVKdJEaZQddQBJkiRJkiRJTcKBwKnAKcA+Vd77C/D87iZaUkqSJEmSJEmqrXbAiUD3HV7dgNhuxn+KJaUkSZIkSZKkevIdkisj6028Pj9MkiRJkiRJUpNX7wsfLSklSZIkSZIk1Zc1wLzaTnK7tyRJkiRJkqS6WA+8Bby+w+sd4CBgWW0+yJJSkiRJkiRJUm28BHQBPqqvD7SklCRJkiRJklQba7981RtLSkmSJEmSJNW31kCzHX7eCmyqMuYIoD9wHLA/ybtTVgIPAov38NnZwDeBE4Gjv5zbDvgC+BRYCrwA/B2oqGP+fdi5N9sMbKkyphswADgG2A/4HFhNcpXhHHb99aZDc6BVNc8/TXcQSZIkSZIkKWqPAMEOr/t2eO9UYFaV93d8fb+az+sAXAk8DWzYw9wdX58AY7+cW1vzqnzW6B3eO62a96u+1gO3kSwNwzqims/pVIv5RwHvVZlfTvL3Ld06s+uv5ZYIckiSJEmSJCmD7a6kvAnYzp4Lvguq+bxNNczZ02s50KOW+XdXUt4YIv+Or9lAm5DfuTcl5enAqipzPwfODjm/vtW6pIynIZQkSZIkSZJ0LVACZNUwrrq+KmcP49cD60huKa/OQSS3X/esKWANrgB+Qc35d9Qf+O1efm9NzgOeAw7c4dlKoC/JlaeSJEmSJElSRqq6knIOyTMdv/p5IfBjklunDwO6kiz0fgHkVfN55Xy9KvLXwGDgG0CsyrjDgeHAX9l1Jd9HwL4h81ddSTljh/wJ4I/A90gWoNkkz+A8DZi0Q9YdX+eE+M66rKS8ll1Xdi768rOi5HZvSZIkSZIkRa5qSfnVK0Fy63RtViMCzAUGEX5XcIzk1uyKKt9/R8j5uztzci3Qp4a5ZwPbqsz7vxDfWZuSMgbcXc34+SQv8YmaJaUkSZIkSVItHUL1NyKr7nZXUuanOUdBle//mJ1v7d6d6krKrSQv/Qnjp1XmVrDzduzqhC0pmwOPVTN2OtAiZL5Us6SUJEmSJEkK6UKSW2O/KqDup+YiSeFUV1K+QPrvR2kGrKiSo3uIedWVlONq8b2HsOsqzpousQlTUrYnuXW+6rhf0rDunrGklCRJkiRJqkE3kttvq1vp9wUwluRqNdVddSXl4IiyPFQlx7Uh5lQtKbcAB9TyexdW+YzbahhfU0l5GPBWlfcrSN6Y3tB4u7ckSZIkSdJudAAmkCyPztrNmNbAmC/HDElTrkywFZgZ0Xd/WOXn4+vwGa8Ca2o55+0qP9e25NzRycBL7Jx9C3Ax8PO9+NwGI8wefEmSJEmSpMYsB7gcGA/sH3JOV5I3OM8meQHLW6mJljHeIllU1pc2wDeBE0lurW795as6J1X5OewN3zt6ow5z1lX5uV0dPgNgIMnzJvep8tnfA8rq+JkNjiWlJEmSJElqygYAvwBOqOP8b5MsqH4L3A58Uk+5Mk3V1Yx1dTbw/4DvUvct+e3rMGdtHeZsqvJzyzp8xuUkz8LM2eHZEpK/D4vq8HkNltu9JUmSJElSU3QM8BRQSt0Lyq9kA1eRLIWuAbL28vMy0fq9nH8Ayb+eT5FcQbg3Z4bWZe7mOswJ6jCnqp+wc0H5d+BMmlhBCa6klCRJkiRJTUt7oAC4gZ3Lnd069dRTeffdd/n8889rGro/8Gvgh8CPgOf2Imem2bIXcw8keaP1cbt5fwXwMcki9AugvMr7x5DcGt4U7A+0JflrliRJkiRJUgMTB4YBK6n+1u5dXp07dw4mTJgQLF++PHj77beDESNGBFlZWaHmfvmaCRyZvl9io1L1du/79uKz/sCuv/fzgB8Q7jKa0VXmzg4xp+rt3qNrnRrurfIZj9UwvrrbvccDiSrPVrD3q4NTrda3e0uSJEmSJDV2fUieGxmqXGzVqlWQn58ffPjhh8Hy5ct3ej377LPBGWecUZuicivJG8Pbpu+X2yjUV0l5Mrv+nv8EiNXiM+6sMr8xlZSdgCuBiirP1wD/VodM6VLrktIzKSVJkiRJUmN1CPAwya3AJ9c0OBaLMWjQIObNm0d+fj7Nm+96NOEJJ5zAn/70J6ZOncqhhx4aJkMzYCTwD5LnVtq11K8Lqvy8ALiD2p33eGD9xYnEZJKrhLfv8Gx/kmXrGZEkSgH/wZEkSZIkSY1NK2As8B5wKSFW1Z188sn8+c9/5oEHHqBz5841fkFeXh7PP/88t99+O23atAmT6SDgAeAVoFeYCQrlW1V+nkFy+3NtnFpPWaL0e+AiYNsOz9oDz5JcSdzoWVJKkiRJkqTGIgYMAd4BxgAta5rQsWNHJkyYwFNPPUX37t1r9WUtWrTguuuuY968eQwePJhYLNQO4+4ktwr/ETi8Vl+o6lRdBfmvWs4/iKZzac6fSK4s3fESon2Ap4GBkSSqR5aUkiRJkiSpMegOlBGy/PuqYCwrK2PIkCFhC8ZqderUiYkTJ9am6NyxTB1LiDJVu5Vd5ecWtZz/QyCrnrI0BE8B5wIbd3jWCvgzMCiSRPXEklKSJEmSJDVkO26j7hlmQl5eHnPnzq3NVu1QvtoyPmHCBA48MNQxh61Irvh8j+SZgnVvSjPXmio/n16Lud8A8usxS0MxG/gOsGGHZ81JboUfEkmiemBJKUmSJEmSGqIcYBS1uJBmx0tvDjvssJSEisViDBkyhBdffHG3l+9U4xBgKiEv+NFOXq3y88Uky8eaHEiytGtd74kahvnAAGDdDs9ygP8meU5ro1N1yawkSZIkSVLUBgG/AI4KM3jfffflRz/6EZdffjlZWenZ2duqVSvy8/O58MILueuuu5g5c2aYaX2A10legnIzsCqVGZuI/wFu3eHnFsBfSJ7NuHA3cwYAv+Hrv3++AOpvSW3D8SrQn+TlOV8t7c0CHiJ5xMCDKf7+09j90Qv7VvPsRPaw0tOSUpIkSZIkNRTHkiwnvxNmcE5ODsOGDePmm2+mbdu2qU22G126dOGBBx7g0ksvZcyYMSxatKimKXGSK92+D5QAdwFbUxyzMXsFeJLkOYxf+QbwBsmyci6wguSKyS7Ad4FTdhj7AcnVhbenPmok3gT6ArOAr66tj5MsaVsCE1L43ddTu1Wb//Hlq1pu95YkSZIkSVHrQLJMWUjIgjI3N5dnn32W4uLiyArKHfXq1YtnnnmGe+65h/322y/MlDYkz6tcSCM+RzBNRgBLqjzLIllc/ozkytQHgf9k54JyJXA2sD71ESO1COgNfLTDsxjwS3ZehdqgWVJKkiRJkqSoZJM8b/JdYCQhdnweeeSRPPzwwzz++OMcc8wxqc5XK9nZ2QwdOpT58+czYsQIsrNDbWDtSvLG8lLghJQGbLxWA7nAC7WYM4fkduR3U5Ko4XmfZFH5zyrP7wLGpT9O7XmrlCRJkiRJisK3SW7tPjHM4LZt23LDDTdw1VVXkZOTk9pk9eT9999n7NixzJ49O+yUcuB3JLcmf5KyYOlxETuvanwJeGIvPzMOnE9yZWVvYJ8d3guA5cDzwMMkz2kMvnyvN8kVlV95H/ivGr7rhyS3j3/lL19+dm2c/eV3f+VNklvPd6cDMLrKs/HA57X4zoNIFv5VO7+pJFdc1qeqf433iiWlJEmSJElKp28APyHkFud4PM4FF1xAYWEh+++/f2qTpUhZWRl33HEHixcvDjtlHVAE3AdUpCxY45YFHEDygpYNJC8h2h5pIu0VS0pJkiRJkpQOrUneaH0r0DzMhDPPPJOioiKOO+64lAZLh/LycqZOncq9997L55+HXhi3CPgR8EzqkkkNgyWlJEmSJElKpTgwFLgH6BhmQufOnbnlllsYMqTp3Sezbt06fvGLX/DQQw9RURF6keSTwCiSN1VLTZIlpSRJkiRJSpXTSN7afUaYwS1btuTaa6/l+uuvp3nzUIstG62FCxdSWFjIyy+/HHbKNuA3wB3U7oxCqVGwpJQkSZIkSfXtYJK3Cg8lRPcQi8U499xzKSws5OCDD055uIaktLSUO+64g6VLl4adspzkbc2TgUTKgklpZkkpSZIkSZLqSyvgBpKr/dqEmXDSSSdRXFzMqaeemtJgDdnmzZv57W9/yy9/+Us2btwYdtprJLeAv5i6ZFL6WFJKkiRJkqT6MAiYCHQJM7hjx47k5+dzySWXEI/HUxqssVi5ciU/+clPmDFjBkEQhJkSANOBHwP/Smk4KcUsKSVJkiRJ0t74N+CXQG6YwTk5OQwbNoxbbrmFNm1CLbbMOG+88QaFhYW8/vrrYadsAu4F7ga2pCyYlEJZUQeQJEmSJEmN0v4kz538L0KunszLy+Phhx/m+9//Ps2aNUtltkbtoIMO4uKLL+bwww/nb3/7W5gt4DlAX2A4sBZ4M8URpXrnSkpJkiRJklQbOcC1QBHQNsyErl27Mm7cOPr27ZvKXE3Spk2buP/++/nVr37Ftm3bwk6bC9yIZaUaEUtKSZIkSZIU1gCS5052CzO4ffv23HTTTVx++eVkZbmZc28sWbKEu+66i5kzZ4adkgB+T/K8ytUpCybVE0tKSZIkSZJUk2OAnwNnhxmck5PDRRddxK233kqHDh1SmyzDzJ8/nzFjxrBo0aKwUz4jeVblL4DQSzGldLOklCRJkiRJu7MvcAvwIyDUIZK5ubmMGzeOY489NqXBMtn27dv5wx/+wE9/+lPWrl0bdtp7QD7wLine truncated
+    }
+   },
+   "cell_type": "markdown",
+   "id": "d1ddc881",
+   "metadata": {},
+   "source": [
+    "<div>\n",
+    "<img src=\"attachment:g5958.png\" align=\"left\" width=\"300\"/>\n",
+    "</div>"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "09937668",
+   "id": "7efe441c",
    "metadata": {},
    "outputs": [],
    "source": [
-    "function rand_distance_table(n)\n",
-    "  threshold = 0.4\n",
-    "  mincost = 3\n",
-    "  maxcost = 10\n",
-    "  infinity = 10000*maxcost\n",
-    "  C = fill(infinity,n,n)\n",
-    "  for j in 1:n\n",
-    "    for i in 1:n\n",
-    "      if rand() > threshold\n",
-    "        C[i,j] = rand(mincost:maxcost)\n",
-    "      end\n",
+    "code2 = quote\n",
+    "    function distribute_input(C,comm)\n",
+    "        rank = MPI.Comm_rank(comm)\n",
+    "        P = MPI.Comm_size(comm)\n",
+    "        # Communicate problem size\n",
+    "        if rank == 0\n",
+    "            N = size(C,1)\n",
+    "            if mod(N,P) !=0\n",
+    "                println(\"N not multiple of P\")\n",
+    "                MPI.Abort(comm,-1)\n",
+    "            end\n",
+    "            Nref = Ref(N)\n",
+    "        else\n",
+    "            Nref = Ref(0)\n",
+    "        end\n",
+    "        MPI.Bcast!(Nref,comm;root=0)\n",
+    "        N = Nref[]\n",
+    "        # Distribute C row-wise\n",
+    "        L = div(N,P)\n",
+    "        myC = similar(C,L,N)\n",
+    "        if  rank == 0\n",
+    "            lb = L*rank+1\n",
+    "            ub = L*(rank+1)\n",
+    "            myC[:,:] = view(C,lb:ub,:)\n",
+    "            for dest in 1:(P-1)\n",
+    "                lb = L*dest+1\n",
+    "                ub = L*(dest+1)\n",
+    "                MPI.Send(view(C,lb:ub,:),comm;dest)\n",
+    "            end\n",
+    "        else\n",
+    "            source = 0\n",
+    "            MPI.Recv!(myC,comm;source)\n",
+    "        end\n",
+    "        return myC\n",
     "    end\n",
-    "    C[j,j] = 0\n",
-    "  end\n",
-    "  C\n",
-    "end"
+    "end;"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "72d82138",
+   "metadata": {},
+   "source": [
+    "### Running Floyd's updates in parallel\n",
+    "\n",
+    "As discussed above, we need to communicate row $k$ of matrix $C$ in order to perform iteration $k$ of the algorithm. The function below is similar to the sequential function `floyd!`, but there is a key difference. At the start of iteration $k$, the owner of row $k$ sends it to the other processors. Once row $k$ is available, all ranks can do the Floyd update locally on their portion of matrix $C$."
+   ]
+  },
+  {
+   "attachments": {
+    "fig-asp-efficiency-comm-2.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1AAAAMbCAYAAABDhHSeAAAACXBIWXMAAB7CAAAewgFu0HU+AAAAGXRFWHRTb2Z0d2FyZQB3d3cuaW5rc2NhcGUub3Jnm+48GgAAIABJREFUeJzs3Xt81dWV9/HPSggJIJcqNxEUW8TitYpYFFCsYlWIM9MiUYeZWqVFGwUqGrk8WquDIIpVaFrQUeplShEfnWnUto/ai0FEW7x3ULxFCYpEBCn3QNbzxz4hh3BOrifnknzfrxevnvM7+7d/C6EkK3vvtczdERERERGRzGRmxwP/BIwC+gPdAQO+AN4ESoEH3X1dA+aaD+TWM2wHsC0y/9vAq+7+ST3zTgaOibxd6u5/rC+WqHv7AD+JunSlJziJMbPpwG2Rt3nuvivuWCVQIiIiIiKZx8y+CswBxhISprrsAe4HbnD3L+uYcyvQqQnh/BUocvc/x5n3D8C5kbeT3X1+Qyc2s2OBt6IuZbt7VRNijDd/NvC/wMDIpToTqKxEPVhERERERJLDzM4EVgEXUZM8OWFF6C/As8A7QHWi0Q6YCLxqZke1QEhDgGfMbFwLzN3S5lKTPNWrXQsGIiIiIiIiCWZmI4HfU7PVbiswG/hV7a10ZtYD+AEwg7CydCTwFzP7hrtvqOdRPwLejfNZe+AwYARwMZBDyC0eMLMX3X1tY39fyWZmQ4AbgfzG3KcESkREREQkQ5hZd+ARapKnD4Bz3P3DWOPdvQK4zcz+m7AqdWjkVzFh9aouL7v7qnrG3GdmDwJPE5KqTsD1wKQG/HaSzsxmA6cBXwd6NWUObeETEREREckctxFWfgC+BL4VL3mK5u7/C3w/6tJ3zOxriQjI3Z8DHoi69N1EzNtCzoz8ik6enm/MBEqgREREREQygJn1BP4t6tIN7v5RQ+939z8AqyNvs2jk1rV6/N+o133M7JAEzp1IfwUeBxYCUwiVAb/dmAm0hU9EREREJDNcCuRFXm8EHmzCHAuB4ZHXOxIRVETtEumdCDGmFXefXPuameXFGhuPEigRERERkcwwIur1/7j7zsZOECkf3uAS4o3Qo9b7tEueEkVb+EREREREMsPpUa9fSlkUsV0Y9foDd9+WskhamBIoEREREZE0F2n2Gl344J1UxVJbpKx6YdSl/05RKEmhLXwiIiIiIunvK9Q0zIVQga+ldTazr8S43gnoAgwC/olwNis78tkm4PYkxJYySqBERERERNJfx1rvE1kAIp4/NXL8VuDCBjTozWjawiciIiIikv4213rfJSVRxLYXeAI4wd2XpzqYlqYVKBERERGR9PcPYA81378no8/Sy8CWGNerCFsI1wGvAc+6e+0y5rXtiXqdHXdUbNE5i0d+pYwSKBERERGRGMysH3ADcCqwEri9AYlCi3B3N7N3CeeOAE4Eft/Cj/2Ru69K0Fxbo153auS9B0W93unuSqBERERERNKFmR0CXA9MAjpELg8BJprZr4Cb3P2zFIS2nJoEalhTJjCzY4A+kbefuvvfExFYA0QnUI1dPYsen4ziGXXSGSgREREREcDMupjZT4EPCStPHWoNaQ/8EFhjZjeZWeckh/hc1OvzzOzQJsxxN/BM5NelCYmqYT6Ken1MI++NHl+egFiaRQmUiIiIiLRpZtbBzKYC7wM3AfUlRl2AnwLvm9kUM8tt6RgjHiecOwLIAW5szM1m1hUYHnXpbwmKqyHeino91MxqJ6d1OSvOPCmhBEpERERE2iQzyzGzHwJrgDuB7rXHdOjQgTFjxpCXlxdrih7AzwgrUleYWYsej3H3ykic1a40swsbMcVN1KyqbQJ+l6jYGuBlaoo/dAEua8hNZnYscHbUpb8kNqzGUwIlIiIiIm2KBRcRVjMWAX1rj8nJyWH8+PGsWLGCe++9l5dffpnCwkLat28fa8rDgf8E3jKzfzezlvweez41/ZkMWGZm3zczi3eDmWWZWRHw46jLt7v7zhaMcz/uXg78IerSHWZ2QV33mNlRwDJqqvZtBv5vy0TYcJbiIhYiIiIiIkljZucAtwMnx/o8KyuL0aNHM336dPr373/A5+Xl5cyfP58lS5awd+/eeI95C7jF3ZclKOz9mFkf4I/A0VGXXwR+BTxP2OZnwGGELXsTgcFRY/8IfNvdo0uLV8+9lZoqeacksAofZvZ1wkpU9RZJJ5zrehx4k1BoIhcYAIwCLo68r/ZDd78vUfFExZXH/o2J89x9V9zxSqBEREREpLUzs9OA24CR8caMGDGCm266iWOPPbbe+d555x3uuusunnzySer4fnoFMMPdE77tLFIp8AlgRCNv/S1wibtvjzNviyVQkfnPIcR9UH1ja5nl7v8nkbFUa2wCpS18IiIiItJqmdlxZvYoIZkZGWvMkCFDePzxx1m6dGmDkieAo48+mkWLFvHb3/6W4cOHxxt2OvBnM3vGzGKueDWVu28k/H7GA+814JY3CInTP8VLnpLB3Z8FTgGeJjTkrc8bwOiWSp6aQitQIiIiItLqmFl/YDowgTiLBt/4xjf48Y9/zKhRo5r9vNLSUmbNmsUbb7wRb4gDjwEz3f3dZj+wFjM7DjgHOIJQ3MKAL4C3gRfc/bUGznMWNWeOXnb3LYmONepZfQkFIo4jFPDoBPwD+DTy63l3f7Olnh8VRxbwrahLf3T3uMmdEigRERERaTUi35TfCFwOxKyKN2DAAK6//nrGjBlDHbUXGs3defbZZ5kzZw6rV6+ON6wSWEw4I7Uu3iBJX0qgRERERCTjRc4EXQ9M4sAGuAD06dOHKVOmcPHFF9OuXctVHK+qquKpp57itttu46OPPoo3bDeh6MNN7v5ZiwUjCacESkREREQylpkdBBQStut1jTXm4IMP5qqrrmLChAnk5iar5y1UVlaydOlS7rzzTjZs2BBv2FagGJjt7l8mLThpMiVQIiIiIpJxzKw9oRnrLUCvWGMOOuggvve97zFp0iQ6d+4ca0hSbN++ncWLF7NgwQK2bIl7pGgjcAdwTzL7M0njKYESERERkYxhZu2AS4GbgSNjjenQoQOXXnopkydPpnv37skMr06bN2+muLiY+++/n5074+ZIa4H/AB6I1adJUk8JlIiIiIikPQvVHsYSkouBscbk5ORQUFDA1KlT6dUr5qJUWli/fj3FxcU8/PDD7N69O96wdwh9qx6pqyKcJJ8SKBERERFJa5Hmq7cDMXspZWVlMXr0aKZPn07//v2TGltzlJeXM3/+fJYsWcLevXvjDXuLULFvWRJDkzoogRIRERGRtGRmpxFWYUbGGzNixAhuuummBjfATUfvvPMOd911F08++SR1fG++Apjh7n9JYmgSgxIoEREREUkrkaawNwEXxRszZMgQpk+fztChQ5MXWAtbtWoVc+bM4YUXXqhr2LNAkbu/mqSwpBYlUCIiIiKSFsysP6Ec+QQgK9aYb3zjG/z4xz9m1KhRSYwsuUpLS5k1axZvvPFGvCEOPAbMdPd3kxeZgBIoEREREUkxM+sL3AhcDsTscDtgwACuv/56xowZQ6gn0bq5O88++yxz5sxh9erV8YZVAosJZ6TWJS+6tk0JlIiIiIikhJkdAlwPTAI6xBrTp08fpkyZwsUXX0y7djFzq1atqqqKp556ittuu42PPvoo3rDdwK+AG909bsdeSQwlUCIiIiKSVGZ2EFBI2K7XNdaYgw8+mKuuuooJEyaQm5ub1PjSUWVlJUuXLuXOO+9kw4a4OdJWoBi4zd3jduyV5lECJSIiIiJJYWbtgcuAW4CYjZo6derEZZddxqRJk+jcuXMyw8sI27dvZ/HixSxYsIAtW+LmSJ8DdwL3uHvcjr3SNEqgRERERKRFmVkW8F1CL6cjY43p0KEDl156KZMnT6Z79+5JjS8Tbd68meLiYu6//3527oybI60lNB5+wN33JC+61k0JlIiIiIi0CAvVHsYSvokfGGtMTk4OBQUFTJ06lV69Yi5KSR3Wr19PcXExDz/8MLt374437B1CP61H3L0qedG1TkqgRERERCThzOwcworTybE+z8rKYvTo0UyfPp3+/fsnNbbWaO3atSxYsIAlS5awd+/eeMPeBG5192VJDK3VUQIlIiIiIgljZqcRVjtGxhszYsQIbrrpJo499tikxdVWvPPOO9x1112UlJTUNWwFMMPd/5KksFoVJVAiIiIi0mxmdhxwE3BRvDFDhgxh+vTpDB06NHmBtVGrVq1izpw5vPDCC3UNexYocvdXkxRWq6AESkRERESazMz6E8qRTwCyYo0ZNGgQU6ZMIT8/P4mRCUBpaSmzZs3ijTfeiDfEgceAme7+bvIiy1xKoERERESk0cysL3AjcDkQs8PtgAEDuP766xkzZgyhnoSkgrvz5JNPMnfuXN5///14wyqBxcAt7r4uedFlHiVQIiIiItJgZnYIcD0wCegQa0yfPn2YMmUKF198Me3axcytJAWqqqp46qmnmDVrFh9//HG8YbuBXwE3unvcjr1tmRIoEREREamXmR0EFBK263WNNebggw/mqquuYsKECeTm5iY1Pmm4yspKli5dyp133smGDXFzpK1AMXCbu8ft2NsWKYESERERkbjMrD1wGXALELNRU6dOnbjsssuYNGkSnTt3TmZ40gzbt29n8eLFLFiwgC1b4uZInwN3Ave4e9yOvW2JEigREREROYCZZQHfJfRyOjLWmA4dOnDppZcyefJkunfvntT4JHE2b95McXEx999/Pzt3xs2R1hIaIj/g7nuSF136UQIlIiIiIvtYqPYwlvDN8sBYY3JycigoKGDq1Kn06hVzUUoy0Pr16ykuLuahhx6isrIy3rC3CeXqH/M2mkgogRIRERERAMzsHMKK08mxPs/KymL06NFMnz6d/v37JzU2SZ61a9eyYMEClixZwt69e+MNexO41d2XJTG0tKAESkRERKSNM7PTgFnAWfHGjBgxghtvvJHjjjsueYFJSr3zzjvcddddlJSULine truncated
+    }
+   },
+   "cell_type": "markdown",
+   "id": "6abd8c34",
+   "metadata": {},
+   "source": [
+    "<div>\n",
+    "<img src=\"attachment:fig-asp-efficiency-comm-2.png\" align=\"left\" width=\"250\"/>\n",
+    "</div>"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "dd77ee3d",
+   "id": "52961b52",
    "metadata": {},
    "outputs": [],
    "source": [
-    "using Test\n",
-    "load = 10\n",
-    "n = nworkers()*load\n",
-    "C = rand_distance_table(n)\n",
-    "C_seq = floyd!(copy(C))\n",
-    "C_par = floyd_dist!(copy(C))\n",
-    "@test C_seq == C_par"
+    "code3 = quote\n",
+    "    function floyd_iterations!(myC,comm)\n",
+    "        L = size(myC,1)\n",
+    "        N = size(myC,2)\n",
+    "        rank = MPI.Comm_rank(comm)\n",
+    "        P = MPI.Comm_size(comm)\n",
+    "        lb = L*rank+1\n",
+    "        ub = L*(rank+1)\n",
+    "        C_k = similar(myC,N)\n",
+    "        for k in 1:N\n",
+    "            if (lb<=k) && (k<=ub)\n",
+    "                # Send row k to other workers if I have it\n",
+    "                myk = (k-lb)+1\n",
+    "                C_k[:] = view(myC,myk,:)\n",
+    "                for dest in 0:(P-1)\n",
+    "                    if rank == dest\n",
+    "                        continue\n",
+    "                    end\n",
+    "                    MPI.Send(C_k,comm;dest)\n",
+    "                end\n",
+    "            else\n",
+    "                # Wait until row k is received\n",
+    "                MPI.Recv!(C_k,comm,source=MPI.ANY_SOURCE)\n",
+    "            end\n",
+    "            # Now, we have the data dependencies and\n",
+    "            # we can do the updates locally\n",
+    "            for j in 1:N\n",
+    "                for i in 1:L\n",
+    "                    myC[i,j] = min(myC[i,j],myC[i,k]+C_k[j])\n",
+    "                end\n",
+    "            end\n",
+    "        end\n",
+    "        myC\n",
+    "    end\n",
+    "end;"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "54081b85",
+   "metadata": {},
+   "source": [
+    "### Collecting back the results\n",
+    "\n",
+    "At this point, we have solved the ASP problem, but the solution is cut in different pieces, each one stored on a different MPI rank. It is often useful to gather the solution into a single matrix, e.g., to compare it against the sequential algorithm.The following function collects all pieces and stores them in $C$ on rank 0. Again, we implement this with `MPI.Send` and `MPI.Recv!` as it is easier as we are working with a row partition. However, we could do it also with `MPI.Gather!`."
+   ]
+  },
+  {
+   "attachments": {
+    "g7043.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAABSkAAATJCAYAAAA7PJMlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N1peFXlofbx/95JmAXEAcQ6F5GqVYsTQphKsFXRVkF6PAoqHm1FwTYV8WhCTKRaa1pBqdUDrai1tUDPsah9a0BQUOtUFVSqVMUis6CgjCF7vR+2RgiBrITsvZLs/++68iFrP8/aN7b10rvPAJIkSZIkSZKidgQQVPnpFGmiNMqOOoAkSZIkSZKkJuFA4GTgJGCfKp/9FXhmdxMtKSVJkiRJkiTVVjvgeKD7Dj/dgNhuxn+CJaUkSZIkSZKkevIdkisj6028Pl8mSZIkSZIkqcmr94WPlpSSJEmSJEmS6ssa4NnaTnK7tyRJkiRJkqS6WA+8Cby6w8/bwEHAstq8yJJSkiRJkiRJUm28ABwOfFhfL7SklCRJkiRJklQba7/4qTeWlJIkSZIkSapvrYFmO/y+FdhUZcwRQH/gG8D+JO9OWQncDyzew7uzgW8CxwNHfzG3HfA58AmwFHgOeB2oqGP+fdi5N9sMbKkyphswAOgK7Ad8BqwmucpwDrv+edOhOdCqmuefpDuIJEmSJEmSFLWHgGCHn3t2+OxkYFaVz3f8+X417+sAXAE8AWzYw9wdfz4Gir6YW1vPVnnXmB0+O7Waz6v+rAduJFkahnVENe/pVIv5RwHvVplfTvKvW7p1Ztc/yw0R5JAkSZIkSVIG211J+RNgO3su+M6v5n0ba5izp5/lQI9a5t9dSXldiPw7/swG2oT8zr0pKU8DVlWZ+xlwVsj59a3WJWU8DaEkSZIkSZKkq4FSIKuGcdX1VTl7GL8eWEdyS3l1DiK5/bpnTQFrcDnwK2rOv6P+wG/38ntrci7wNHDgDs9WAn2BJ1P83ZIkSZIkSVKDVXUl5RySZzp++ftC4Kckt04fCnQhWej9Csir5n3b+GpV5K+BwcDXgViVcYcBw4G/s+tKvg+BfUPmr7qScsYO+RPAn4DvkSxAs0mewXkqMInkFuuq3312iO+sy0rKq9l1ZeeiL94VJbd7S5IkSZIkKXJVS8ovfxIkt07XZjUiJEvOQYTfFRwjuTW7osr33xxy/u7OnFwL9Klh7ll8Vap++fP/QnxnbUrKGHB7NePnk7zEJ2qWlJIkSZIkSbV0CMmVcKo/uyspf5rmHAVVvv8jdr61e3eqKym3krz0J4yfV5lbwc7bsasTtqRsDjxSzdjpQIuQ+VLNklKSJEmSJCmkwcA/+aqAug/oGGmipqO6kvJ50n8/SjNgRZUc3UPMq66kvKUW3/s1dl3FWdMlNmFKyvYkV5VWHXcXDevuGUtKSZIkSZKkGnQjuf22upV+nwNFJFerqe6qKymHRJTlgSo5rg4xp2pJuQXYv5bfu7DKO26sYXxNJeWhwJtVPq8geWN6Q+Pt3pIkSZIkSbvRAZhAsjw6czdjWgPjSJZBUZVqTdFW4C8RffcHVX4/tg7veBn4uJZz3qry+wF1+N4vnQi8wM7ZtwA/AH65F+9tMMLswZckSZIkSWrMckiunisiuV02jK+TvMF5NvBjksWm6u5NkkVlfWkDfBM4nuQKw9ZAy92MPaHK72Fv+N7Ra3WYs67K7+3q8A6AgSTPm9ynyru/B8yr4zsbHEtKSZIkSZLUlA0geV5fXVbPAXwb+AfwW5I3Q6+pp1yZpupqxrr6LnAl8B3qfklM2KJ6R2vrMGdTld93V6LuyWUkz8LM2eHZEpLnWy6qw/saLLd7S5IkSZKkpqgr8CRQRt0Lyi9lkyzGFpFckZm1l+/LROv3cv4BwBMk/zP9Hnt3i3VdzhvdXIc5QR3mVPUzdi4oXwfOoIkVlOBKSkmSJEmS1LS0BwqBa9i53Nmtk08+mXfffZcNGzbUNHQ/YBLwQ+A64Om9yJlptuzF3ANJ3mj9jd18vgL4iGQR+jlQXuXzriS3hjcF+wNtSf6ZJUmSJEmS1MDEgWHASqq/tXuXn8MPPzy47777guXLlwdvvfVWMGLEiCArKyvU3C9+ZgJHpe+P2KhUvd37nr141x/Z9a/9s8CFhLuMZkyVubNDzKl6u/eYWqeGX1R5xyM1jK/udu/xQKLKsxXAcXXIk061vt1bkiRJkiSpsetDchtsqHKxVatWQX5+fvDBBx8Ey5cv3+nnqaeeCk4//fTaFJXbSN4Y3jZ9f9xGob5KyhPZ9a/5z4BYLd5xa5X5jamk7ARcAVRUeb4G+FYdMqVLrUtKz6SUJEmSJEmN1deAB0luBa56g/MuYrEYgwcP5vnnnyc/P5/mzXc9mvC4447jz3/+M1OnTuXQQw8NkyEHGAX8k+S5lZ5XWb/Or/L7ApIXGNXmvMcD6y9OJCaTXCW8fYdn+5MsW0+PJFEKWFJKkiRJkqTGphVQBLwLXEKIVXUnnngif/nLX5g4cSIHHlhzZ5WXl8fcuXO56aabaNOmTZhMBwH3AS8CvcJMUCinVPl9Bsntz7Vxcj1lidLvgaEkV+5+qT3wFMmVxI2eJaUkSZIkSWosYsAQ4G1gHNCypgmdOnViwoQJPPHEE3Tv3r1WX9aiRQtGjhzJs88+y+DBg4nFQu0w7k5yq/CfgMNq9YWqTtVG+d+1nH8QTefSnD+TXFm64yVE+5C88XxgJInqkSWlJEmSJElqDLoD8whZ/u1YMA4ZMiRswVitTp06MXHixNoUnTuWqUWEKFO1W9lVfm9Ry/k/pGltwX8COAfYuMOzVsBfgEGRJKonlpSSJEmSJKkh+3Ib9UtAzzAT6rBVO5Qvt4xPmDAh1JZxkuXROJLb0odRu8telLSmyu+n1WLu14H8eszSUMwGvgNs2OFZc5Jb4YdEkqgeWFJKkiRJkqSGqBkwmq8upKmxw6jDpTe1FovFGDJkyB4v36nG14CpwAvUrmQTvFzl9x+QLB9rciDJ0q51vSdqGOYDA4B1OzzLAf5A8pzWRqfqkllJkiRJkqSoDQLuAo4MM3jfffflxz/+MZdddhlZWenZ2duqVSvy8/O54IILuO2225g5c2aYaacBz5O8BOV6YFUqMzYR/wuM3eH3FsBfSZ7NuHA3cwYAvwGO+uL3z4H6W1LbcLwM9Cd5ec6XS3uzgAdIHjFwf4q//1R2f/TCvtU8O549rPS0pJQkSZIkSQ3FMcCvSG5lrVFOTg7Dhg3j+uuvp23btqlNthuHH3449913H8OGDaOwsJBFixbVNCVOcqXb94FS4DZga4pjNmYvAY+TPIfxS18HXiNZVs4FVpBcMXk48F3gpB3Gvk9ydeFNqY8aiTeAvsAsoPMXz+IkS9qWwIQUfvdIkscYhPWfX/xUy+3ekiRJkiQpah1IlikLCVlQ5ubmUlZWRklJSWQF5Y569uxJWVkZEyZMYL/99gszpQ3J8yoX0ojPEUyTEcCSKs+ySBaXd5JcmXo/8N/sXFCuBM4C1qc+YqQWAb2BD3d4FiO5GnlstTMaIEtKSZIkSZIUlWyS502+A4wixI7PI488kgcffJBHH32Uo48+OtX5aiUejzNkyBDmz5/PyJEjycnJCTOtC8kby8uA41IasPFaDeQCz9VizhyS25HfSUmihuc9kkXlv6o8vw24Jf1xas9bpSRJkiRJUhS+TXKlV6hirm3btlx77bVceeWVYcu/yL333nsUFRUxe/bssFO2A78luTX545QFS4+h7Lyq8QXgsb18Zxw4j+TKyt7APjt8FgDLgWeAB0me0xh88Vlvkisqv/Qe8D81fNcPSW4f/9Jfv3h3bZz1xXd/6Q2SW893pwMwpsqz8cBntfjOg0gW/lU7v6kkV1zWpwuBb9XXyywpJUmSJElSOnUhWbyE2uIcj8c5//zzKSwsZP/9909tshSZN28eBQUFvPvuu2GnrAOKgXuAipQFa9yygANIXtCygeQlRNsjTaS9YkkpSZIkSZLSoQ3wU5Jn5DUPM6Fnz54UFxfTrVu3lAZLh/LycqZOncqdd97Jhg0bwk5bBPwY+FvqkkkNgyWlJEmSJElKpThwMXAH0DHMhM6dO3PDDTcwZEjTu0/mk08+4Ze//CUPPPAAFRWhF0k+DowmeVO11CRZUkqSJEmSpFQ5jeS5k6eHGdyqVSt+9KMfcc0119C8eajFlo3WwoULKSws5MUXXww7ZRvwG6CA5PZmqUmxpJQkSZIkSfXta8DPSK6grLF7iMVinHPOORQWFnLwwQenPFxDUlZWxs0338zSpUvDTllO8rbmyUAiZcGkNLOklCRJkiRJ9aUVcC1wM8kzKGt0wgknUFJSwsknn5zSYA3Zli1bmDJlCnfddRcbN24MO+0V4DrgudQlk9LHklKSJEmSJNWHQcDdwGFhBnfs2JH8/Hwuuugi4vF4apM1EitXruRnP/sZM2bMIAiCMFMCYDrJC4n+ndJwUopZUkqSJEmSpL3xLWAC0CvM4JycHIYNG8YNN9xAmzahFltmnNdee43CwkJeffXVsFM2Ab8Abge2pCyYlEKWlJIkSZIkqS72J3mJy0ggK8yEvLw8SkpKOPTQQ1MarCkIgoDp06czfvx4Vq9eHXbaUpJb7R8iucpSajQsKSVJkiRJUm3kAFcDxUDbMBO6dOnCLbfcQt++fVOZq0natGkT9957L3fffTfbtm0LO20uyfMq30hZMKmeWVJKkiRJkqSwBgG/Ao4KM7h9+/b85Cc/4bLLLiMrK9RiS+3GkiVLuO2225g5c2bYKQng9yTPqwy9FFOKiiWlJEmSJEmqSVeS5eR3wwzOyclh6NChjB07lg4dOqQ2WYaZP38+48aNY9GiRWGnfEryrMq7gK0pCybtJUtKSZIkSZK0O/sCRSS3d2eHmZCbm0txcTFdu3ZNZa6Mtn37dv74xz/y85//nLVr14ad9i6QDzyeumRS3VlSSpIkSZKkqrKBy4FbLine truncated
+    }
+   },
+   "cell_type": "markdown",
+   "id": "a0fb6974",
+   "metadata": {},
+   "source": [
+    "<div>\n",
+    "<img src=\"attachment:g7043.png\" align=\"left\" width=\"300\"/>\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8c725aa9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "code4 = quote\n",
+    "    function collect_result!(C,myC,comm)\n",
+    "        L = size(myC,1)\n",
+    "        rank = MPI.Comm_rank(comm)\n",
+    "        P = MPI.Comm_size(comm)\n",
+    "        if rank == 0\n",
+    "            lb = L*rank+1\n",
+    "            ub = L*(rank+1)\n",
+    "            C[lb:ub,:] = myC\n",
+    "            for source in 1:(P-1)\n",
+    "                lb = L*source+1\n",
+    "                ub = L*(source+1)\n",
+    "                MPI.Recv!(view(C,lb:ub,:),comm;source)\n",
+    "            end\n",
+    "        else\n",
+    "            dest = 0\n",
+    "            MPI.Send(myC,comm;dest)\n",
+    "        end\n",
+    "        C\n",
+    "    end\n",
+    "end;"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d573108e",
+   "metadata": {},
+   "source": [
+    "### Running and testing the code\n",
+    "\n",
+    "In the cell below, we run the parallel code and compare it against the sequential. Note that we can only compare both results on rank 0 since this is the only one that contains the result of the parallel code. We have also included a function that generates random distance tables of a given size $N$."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c8e9cea7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "code = quote\n",
+    "    using MPI\n",
+    "    MPI.Init()\n",
+    "    $code1\n",
+    "    $code2\n",
+    "    $code3\n",
+    "    $code4\n",
+    "    function input_distance_table(n)\n",
+    "        threshold = 0.1\n",
+    "        mincost = 3\n",
+    "        maxcost = 9\n",
+    "        inf = 10000\n",
+    "        C = fill(inf,n,n)\n",
+    "        for j in 1:n\n",
+    "            for i in 1:n\n",
+    "                if rand() > threshold\n",
+    "                    C[i,j] = rand(mincost:maxcost)\n",
+    "                end\n",
+    "            end\n",
+    "            C[j,j] = 0\n",
+    "        end\n",
+    "        C\n",
+    "    end\n",
+    "    function floyd!(C)\n",
+    "        n = size(C,1)\n",
+    "      @assert size(C,2) == n\n",
+    "      for k in 1:n\n",
+    "        for j in 1:n\n",
+    "          for i in 1:n\n",
+    "            @inbounds C[i,j] = min(C[i,j],C[i,k]+C[k,j])\n",
+    "          end\n",
+    "        end\n",
+    "      end\n",
+    "      C\n",
+    "    end\n",
+    "    comm = MPI.Comm_dup(MPI.COMM_WORLD)\n",
+    "    rank = MPI.Comm_rank(comm)\n",
+    "    if rank == 0\n",
+    "        N = 24\n",
+    "    else\n",
+    "        N = 0\n",
+    "    end\n",
+    "    C = input_distance_table(N)\n",
+    "    C_par = copy(C)\n",
+    "    floyd_mpi!(C_par,comm)\n",
+    "    if rank == 0\n",
+    "        C_seq = copy(C)\n",
+    "        floyd!(C_seq)\n",
+    "        if C_seq == C_par\n",
+    "            println(\"Test passed 🥳\")\n",
+    "        else\n",
+    "            println(\"Test failed\")\n",
+    "        end\n",
+    "    end\n",
+    "end\n",
+    "run(`$(mpiexec()) -np 3 julia --project=. -e $code`);"
    ]
   },
   {
@@ -772,10 +1022,12 @@
    "id": "91a772df",
    "metadata": {},
    "source": [
-    "### Is this implementation correct?\n",
+    "## Is this implementation correct?\n",
     "\n",
-    "Point-to-point messages are *non-overtaking* (i.e. FIFO order) between the specified sender and receiver according to section 3.5 of the MPI standard 4.0.\n",
-    "Unfortunately this is not enough in this case. The messages can still arrive in the wrong order if messages from different processes overtake each other."
+    "In the cell above, the result of the parallel code was probably the same as for the sequential code. However, is this sufficient to assert that the code is correct? Unfortunately, it is not. In fact, **the parallel code we implemented is not correct!** There is no guarantee that this code computes the correct result. This is why:\n",
+    "\n",
+    "In MPI, point-to-point messages are *non-overtaking* between a given sender and receiver. Say that process 1 sends several messages to process 3. All these will arrive in FIFO order. This is according to section 3.5 of the MPI standard 4.0.\n",
+    "Unfortunately, this is not enough in our case. The messages could arrive in the wrong order *from different senders*. If process 1 sends messages to process 3, and then process 2 sends other messages to process 3, it is not granted that process 3 will receive first the messages from process 1 and then from process 2 (see figure below)."
    ]
   },
   {
@@ -783,7 +1035,7 @@
    "id": "1277772d",
    "metadata": {},
    "source": [
-    "If we are lucky all messages will arrive in order and we will process all rows in the right order in all processors."
+    "If we are lucky all messages will arrive in order. In our parallel code, all processors would receive first row one, then row 2, then row 3, etc. The computed result will be correct."
    ]
   },
   {
@@ -806,7 +1058,7 @@
    "id": "df60e4e7",
    "metadata": {},
    "source": [
-    "However, FIFO ordering is not enough. In the next figure, communication between process 1 and process 3 is particularly slow. Note that process 3 receives messages from process 1 after it receives the messages from 2 even though FIFO ordering is satisfied between any two processors."
+    "However, FIFO ordering between pairs of processors is not enough to guarantee that rows arrive in consecutive order. The next figure shows a counter example. In this case, communication between process 1 and process 3 is particularly slow for some unknown reason. As result processor 3 receives first messages from processor 2, even though processor 1 sent the messages first. In our parallel code, the received rows would be first row 3, then row 4, then row 1, then row 2, which is not correct. Note however that process 3 received all messages from process 1 in the correct order (guaranteed by MPI). But this is not enough in our algorithm."
    ]
   },
   {
@@ -831,10 +1083,25 @@
    "source": [
     "### Possible solutions\n",
     "\n",
+    "There are several solution to this synchronization problem:\n",
+    "\n",
     "1. **Synchronous sends**: Use synchronous send MPI_SSEND. This is less efficient because we spend time waiting until each message is received. Note that the blocking send MPI_SEND used above does not guarantee that the message was received. \n",
     "2. **MPI.Barrier**: Use a barrier at the end of each iteration over $k$. This is easy to implement, but we get a  synchronization overhead.\n",
-    "3. **Order incoming messages**: The receiver orders the incoming messages, e.g. according to MPI.Status or the sender rank. This requires buffering and extra user code.\n",
-    "4. **MPI.Bcast!**: Communicate row k using `MPI.Bcast!`. One needs to know which are the rows owned by the other ranks."
+    "3. **Order incoming messages**: The receiver orders the incoming messages, e.g. using to MPI.Status to get the sender rank. This requires buffering and extra user code.\n",
+    "4. **MPI.Bcast!**: Communicate row k using `MPI.Bcast!`. One needs to know which are the rows owned by the other ranks since we cannot use `MPI.ANY_SOURCE` in `MPI.Bcast!`. This is trivial however if the number of rows is multiple of the number of ranks."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fb21edc4",
+   "metadata": {},
+   "source": [
+    "## Summary\n",
+    "\n",
+    "- We learned how to parallelize Floyd's algorithm.\n",
+    "- The considered strategy based on a row-wise data partition has little communication overhead if the problem size is large enough.\n",
+    "- One needs to be careful in which order the messages are received to have a correct algorithm.\n",
+    "- There are several strategies to solve this synchronization problem, each one with pros and cons.\n"
    ]
   },
   {
@@ -842,61 +1109,11 @@
    "id": "de96ad1b",
    "metadata": {},
    "source": [
-    "## Exercise \n",
-    "Rewrite the worker code of the parallel ASP algorithm so it runs correctly. Use the `MPI.Bcast!` to solve the problem of overtaking messages. Note: Only use `MPI.Bcast!`, do not use other MPI directives in addition. You can test your function with the following code cell. "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "31194529",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "function floyd_par!(C,N)\n",
-    "    comm = MPI.Comm_dup(MPI.COMM_WORLD)\n",
-    "    nranks = MPI.Comm_size(comm)\n",
-    "    rank = MPI.Comm_rank(comm)\n",
-    "    T = eltype(C)\n",
-    "    if rank == 0\n",
-    "        buffer_root = Vector{T}(undef,N*N)\n",
-    "        buffer_root[:] = transpose(C)[:]\n",
-    "    else\n",
-    "        buffer_root = Vector{T}(undef,0)\n",
-    "    end    \n",
-    "    Nw = div(N,nranks)\n",
-    "    buffer =  Vector{T}(undef,Nw*N)\n",
-    "    MPI.Scatter!(buffer_root,buffer,comm;root=0)\n",
-    "    Cw = Matrix{T}(undef,Nw,N)\n",
-    "    transpose(Cw)[:] = buffer\n",
-    "    MPI.Barrier(comm)\n",
-    "    floyd_worker_bcast!(Cw,comm)\n",
-    "    buffer[:] = transpose(Cw)[:]\n",
-    "    MPI.Gather!(buffer,buffer_root,comm;root=0)\n",
-    "    if rank == 0\n",
-    "        transpose(C)[:] = buffer_root[:]\n",
-    "    end\n",
-    "    C\n",
-    "end\n",
+    "## Exercise\n",
     "\n",
-    "@everywhere function floyd_worker_bcast!(Cw,comm)\n",
-    "   # Your implementation here\n",
-    "end\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "1b7eb4c2",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "load = 10\n",
-    "n = nworkers()*load\n",
-    "C = rand_distance_table(n)\n",
-    "C_seq = floyd!(copy(C))\n",
-    "C_par = floyd_par!(copy(C),n)\n",
-    "@test C_seq == C_par"
+    "### Exercise 1\n",
+    "\n",
+    "Modify the `floyd_iterations!` function so that it is guaranteed that the result is computed correctly. Use `MPI.Bcast!` to solve the synchronization problem. Note: only use `MPI.Bcast!`in `floyd_iterations!`, do not use other MPI directives. You can assume that the number of rows is a multiple of the number of processes."
    ]
   },
   {
@@ -914,15 +1131,15 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Julia 1.9.1",
+   "display_name": "Julia 1.11.6",
    "language": "julia",
-   "name": "julia-1.9"
+   "name": "julia-1.11"
   },
   "language_info": {
    "file_extension": ".jl",
    "mimetype": "application/julia",
    "name": "julia",
-   "version": "1.9.1"
+   "version": "1.11.6"
   }
  },
  "nbformat": 4,

notebooks/figures/fig_mpi.svg

@@ -365,6 +365,84 @@
          id="path8527-8"
          inkscape:connector-curvature="0" />
     </marker>
+    <marker
+       style="overflow:visible"
+       id="marker8529-3-3"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="Arrow2Mend"
+       inkscape:isstock="true"
+       inkscape:collect="always">
+      <path
+         transform="scale(-0.6)"
+         d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z"
+         style="fill:#0000ff;fill-opacity:1;fill-rule:evenodd;stroke:#0000ff;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1"
+         id="path8527-8-6"
+         inkscape:connector-curvature="0" />
+    </marker>
+    <marker
+       inkscape:isstock="true"
+       inkscape:stockid="Arrow2Mend"
+       orient="auto"
+       refY="0"
+       refX="0"
+       id="marker4031-1"
+       style="overflow:visible"
+       inkscape:collect="always">
+      <path
+         inkscape:connector-curvature="0"
+         id="path4029-0"
+         style="fill:#0000ff;fill-opacity:1;fill-rule:evenodd;stroke:#0000ff;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1"
+         d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z"
+         transform="scale(-0.6)" />
+    </marker>
+    <marker
+       style="overflow:visible"
+       id="marker4223-6"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="Arrow2Mend"
+       inkscape:isstock="true">
+      <path
+         transform="scale(-0.6)"
+         d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z"
+         style="fill:#0000ff;fill-opacity:1;fill-rule:evenodd;stroke:#0000ff;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1"
+         id="path4221-3"
+         inkscape:connector-curvature="0" />
+    </marker>
+    <marker
+       inkscape:isstock="true"
+       inkscape:stockid="Arrow2Mend"
+       orient="auto"
+       refY="0"
+       refX="0"
+       id="marker4031-5"
+       style="overflow:visible"
+       inkscape:collect="always">
+      <path
+         inkscape:connector-curvature="0"
+         id="path4029-3"
+         style="fill:#0000ff;fill-opacity:1;fill-rule:evenodd;stroke:#0000ff;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1"
+         d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z"
+         transform="scale(-0.6)" />
+    </marker>
+    <marker
+       style="overflow:visible"
+       id="marker4223-5"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="Arrow2Mend"
+       inkscape:isstock="true">
+      <path
+         transform="scale(-0.6)"
+         d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z"
+         style="fill:#0000ff;fill-opacity:1;fill-rule:evenodd;stroke:#0000ff;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1"
+         id="path4221-6"
+         inkscape:connector-curvature="0" />
+    </marker>
   </defs>
   <sodipodi:namedview
      id="base"
@@ -373,9 +451,9 @@
      borderopacity="1.0"
      inkscape:pageopacity="0.0"
      inkscape:pageshadow="2"
-     inkscape:zoom="0.70000001"
-     inkscape:cx="4005.3666"
-     inkscape:cy="58.38566"
+     inkscape:zoom="0.98994951"
+     inkscape:cx="6863.501"
+     inkscape:cy="332.85381"
      inkscape:document-units="mm"
      inkscape:current-layer="layer1"
      inkscape:document-rotation="0"
@@ -2678,5 +2756,439 @@
            y="194.96161"
            x="1153.2484">20</tspan></text>
     </g>
+    <g
+       transform="translate(192.3794,-112.09032)"
+       id="g5148-2"
+       inkscape:export-filename="/home/francesc/Downloads/g5148.png"
+       inkscape:export-xdpi="200"
+       inkscape:export-ydpi="200">
+      <path
+         style="fill:none;stroke:#000000;stroke-width:0.529167;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+         d="m 1359.9626,79.585912 h 20.9409"
+         id="path978-3-6-3-0"
+         sodipodi:nodetypes="cc"
+         inkscape:connector-curvature="0" />
+      <path
+         inkscape:connector-curvature="0"
+         sodipodi:nodetypes="cc"
+         id="path1168-6-1-6-6"
+         d="m 1314.5268,79.585912 h 20.9408"
+         style="fill:none;stroke:#000000;stroke-width:0.529167;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+      <g
+         transform="translate(1155.5468,183.31702)"
+         id="g1148-7-5-75-1">
+        <path
+           inkscape:connector-curvature="0"
+           id="path1063-5-5-3-5"
+           d="m 129.89807,-136.23112 12.99038,7.5 17.32051,-10 -12.99038,-7.5 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           sodipodi:nodetypes="cccc"
+           id="path1065-3-4-5-5"
+           d="m 142.88845,-128.73112 v 39.999996 l 17.32051,-10 v -39.999996"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           sodipodi:nodetypes="ccc"
+           id="path1067-56-7-62-4"
+           d="m 129.89807,-136.23112 v 39.999996 l 12.99038,7.5"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1069-2-6-9-7"
+           d="m 147.21857,-126.23112 v 15 l 4.33013,-2.5 v -15 l -4.33013,2.5"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1071-91-5-1-6"
+           d="m 145.48652,-96.231123 v 1.999999 l 12.99038,-7.499996 v -2 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1073-2-6-2-5"
+           d="m 145.48652,-100.23111 v 2.000003 l 12.99038,-7.500013 v -2 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1075-7-9-7-6"
+           d="m 145.48652,-104.23111 v 2 l 12.99038,-7.50001 v -2 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+      </g>
+      <g
+         transform="translate(1163.1327,183.31701)"
+         id="g1157-09-3-0-9">
+        <path
+           inkscape:connector-curvature="0"
+           id="path1086-3-7-93-3"
+           d="m 169.05642,-136.23112 12.99038,7.5 17.32051,-10 -12.99038,-7.5 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           sodipodi:nodetypes="cccc"
+           id="path1088-6-4-6-7"
+           d="m 182.0468,-128.73112 v 39.999996 l 17.32051,-10 v -39.999996"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           sodipodi:nodetypes="ccc"
+           id="path1090-0-5-0-4"
+           d="m 169.05642,-136.23112 v 39.999996 l 12.99038,7.5"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1092-6-2-6-5"
+           d="m 186.37692,-126.23112 v 15 l 4.33013,-2.5 v -15 l -4.33013,2.5"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1094-2-5-2-2"
+           d="m 184.64487,-96.231123 v 1.999999 l 12.99038,-7.499996 v -2 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1096-6-4-6-5"
+           d="m 184.64487,-100.23111 v 2.000003 l 12.99038,-7.500013 v -2 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1098-1-7-1-4"
+           d="m 184.64487,-104.23111 v 2 l 12.99038,-7.50001 v -2 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+      </g>
+      <text
+         y="15.986771"
+         x="1244.8425"
+         style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.93749809px;line-height:1.25;font-family:Hack;-inkscape-font-specification:Hack;letter-spacing:0px;word-spacing:0px;white-space:pre;shape-inside:url(#rect1130-2);stroke-width:1.87007797"
+         id="text1128-7-8-4-8-7"
+         xml:space="preserve"><tspan
+           style="stroke-width:1.87007797"
+           id="tspan78-79-4-7-4"
+           y="103.6842"
+           x="1289.8303">worker 1</tspan></text>
+      <text
+         y="15.986771"
+         x="1294.8483"
+         style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.93749809px;line-height:1.25;font-family:Hack;-inkscape-font-specification:Hack;letter-spacing:0px;word-spacing:0px;white-space:pre;shape-inside:url(#rect1184);stroke-width:1.87007797"
+         id="text1182-0-0-2-3"
+         xml:space="preserve"><tspan
+           style="stroke-width:1.87007797"
+           id="tspan83-2-7-0-0"
+           y="103.6842"
+           x="1339.8361">worker 2</tspan></text>
+      <g
+         id="g970-7-6-3-8"
+         transform="translate(1209.6376,183.31701)">
+        <path
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+           d="m 169.05642,-136.23112 12.99038,7.5 17.32051,-10 -12.99038,-7.5 z"
+           id="path956-5-8-7-6"
+           inkscape:connector-curvature="0" />
+        <path
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+           d="m 182.0468,-128.73112 v 39.999996 l 17.32051,-10 v -39.999996"
+           id="path958-92-8-5-8"
+           sodipodi:nodetypes="cccc"
+           inkscape:connector-curvature="0" />
+        <path
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+           d="m 169.05642,-136.23112 v 39.999996 l 12.99038,7.5"
+           id="path960-2-4-9-8"
+           sodipodi:nodetypes="ccc"
+           inkscape:connector-curvature="0" />
+        <path
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+           d="m 186.37692,-126.23112 v 15 l 4.33013,-2.5 v -15 l -4.33013,2.5"
+           id="path962-8-3-2-4"
+           inkscape:connector-curvature="0" />
+        <path
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+           d="m 184.64487,-96.231123 v 1.999999 l 12.99038,-7.499996 v -2 z"
+           id="path964-9-1-2-3"
+           inkscape:connector-curvature="0" />
+        <path
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+           d="m 184.64487,-100.23111 v 2.000003 l 12.99038,-7.500013 v -2 z"
+           id="path966-7-4-8-1"
+           inkscape:connector-curvature="0" />
+        <path
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+           d="m 184.64487,-104.23111 v 2 l 12.99038,-7.50001 v -2 z"
+           id="path968-3-9-9-4"
+           inkscape:connector-curvature="0" />
+      </g>
+      <text
+         xml:space="preserve"
+         id="text976-6-2-7-9"
+         style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.93749809px;line-height:1.25;font-family:Hack;-inkscape-font-specification:Hack;letter-spacing:0px;word-spacing:0px;white-space:pre;shape-inside:url(#rect1184);stroke-width:1.87007797"
+         x="1342.9568"
+         y="15.986771"><tspan
+           x="1387.9445"
+           y="103.6842"
+           id="tspan974-1-0-3-2"
+           style="stroke-width:1.87007797">worker 3</tspan></text>
+      <text
+         y="-54.098297"
+         x="1247.6248"
+         style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.93749809px;line-height:1.25;font-family:Hack;-inkscape-font-specification:Hack;letter-spacing:0px;word-spacing:0px;white-space:pre;shape-inside:url(#rect1130-2);fill:#0000ff;stroke-width:1.87007797"
+         id="text1128-7-8-7-3-7-0-06"
+         xml:space="preserve"><tspan
+           style="fill:#0000ff;stroke-width:1.87007797"
+           id="tspan78-79-8-2-1-3-8"
+           y="33.599152"
+           x="1292.6124">msg</tspan></text>
+      <path
+         sodipodi:nodetypes="cc"
+         inkscape:connector-curvature="0"
+         id="path1386-9"
+         d="m 1304.0316,27.773394 c 16.0706,-14.103005 27.5564,-13.085868 41.886,-0.615601"
+         style="fill:none;stroke:#0000ff;stroke-width:0.76499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#marker8529-3-3)" />
+      <text
+         y="-54.561871"
+         x="1290.4174"
+         style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.93749809px;line-height:1.25;font-family:Hack;-inkscape-font-specification:Hack;letter-spacing:0px;word-spacing:0px;white-space:pre;shape-inside:url(#rect1130-2);fill:#0000ff;stroke-width:1.87007797"
+         id="text1128-7-8-7-3-7-0-0-2"
+         xml:space="preserve"><tspan
+           style="fill:#0000ff;stroke-width:1.87007797"
+           id="tspan78-79-8-2-1-3-6-6"
+           y="33.135578"
+           x="1335.405">msg+=1</tspan></text>
+      <text
+         y="-53.871326"
+         x="1338.3951"
+         style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.93749809px;line-height:1.25;font-family:Hack;-inkscape-font-specification:Hack;letter-spacing:0px;word-spacing:0px;white-space:pre;shape-inside:url(#rect1130-2);fill:#0000ff;stroke-width:1.87007797"
+         id="text1128-7-8-7-3-7-0-0-3-6"
+         xml:space="preserve"><tspan
+           style="fill:#0000ff;stroke-width:1.87007797"
+           id="tspan78-79-8-2-1-3-6-2-4"
+           y="33.826126"
+           x="1383.3828">msg+=1</tspan></text>
+      <path
+         style="fill:none;stroke:#0000ff;stroke-width:0.76499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#marker4031-1)"
+         d="m 1354.3024,27.773394 c 16.0706,-14.103005 27.5564,-13.085868 41.886,-0.615601"
+         id="path4027-9"
+         inkscape:connector-curvature="0"
+         sodipodi:nodetypes="cc" />
+      <path
+         sodipodi:nodetypes="cc"
+         inkscape:connector-curvature="0"
+         id="path4219-5"
+         d="m 1402.8962,28.340358 c -14.7223,-30.9560949 -88.632,-34.6967981 -104.63,-1.371553"
+         style="fill:none;stroke:#0000ff;stroke-width:0.76499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#marker4223-6)" />
+    </g>
+    <g
+       id="g2038"
+       inkscape:export-xdpi="200"
+       inkscape:export-ydpi="200">
+      <path
+         style="fill:none;stroke:#000000;stroke-width:0.529167;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+         d="m 1861.8989,152.34405 h 20.9409"
+         id="path978-3-6-3-1"
+         sodipodi:nodetypes="cc"
+         inkscape:connector-curvature="0" />
+      <path
+         inkscape:connector-curvature="0"
+         sodipodi:nodetypes="cc"
+         id="path1168-6-1-6-2"
+         d="m 1816.4631,152.34405 h 20.9408"
+         style="fill:none;stroke:#000000;stroke-width:0.529167;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+      <g
+         transform="translate(1657.4831,256.07516)"
+         id="g1148-7-5-75-7">
+        <path
+           inkscape:connector-curvature="0"
+           id="path1063-5-5-3-0"
+           d="m 129.89807,-136.23112 12.99038,7.5 17.32051,-10 -12.99038,-7.5 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           sodipodi:nodetypes="cccc"
+           id="path1065-3-4-5-9"
+           d="m 142.88845,-128.73112 v 39.999996 l 17.32051,-10 v -39.999996"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           sodipodi:nodetypes="ccc"
+           id="path1067-56-7-62-3"
+           d="m 129.89807,-136.23112 v 39.999996 l 12.99038,7.5"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1069-2-6-9-6"
+           d="m 147.21857,-126.23112 v 15 l 4.33013,-2.5 v -15 l -4.33013,2.5"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1071-91-5-1-0"
+           d="m 145.48652,-96.231123 v 1.999999 l 12.99038,-7.499996 v -2 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1073-2-6-2-6"
+           d="m 145.48652,-100.23111 v 2.000003 l 12.99038,-7.500013 v -2 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1075-7-9-7-2"
+           d="m 145.48652,-104.23111 v 2 l 12.99038,-7.50001 v -2 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+      </g>
+      <g
+         transform="translate(1665.069,256.07515)"
+         id="g1157-09-3-0-6">
+        <path
+           inkscape:connector-curvature="0"
+           id="path1086-3-7-93-1"
+           d="m 169.05642,-136.23112 12.99038,7.5 17.32051,-10 -12.99038,-7.5 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           sodipodi:nodetypes="cccc"
+           id="path1088-6-4-6-8"
+           d="m 182.0468,-128.73112 v 39.999996 l 17.32051,-10 v -39.999996"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           sodipodi:nodetypes="ccc"
+           id="path1090-0-5-0-7"
+           d="m 169.05642,-136.23112 v 39.999996 l 12.99038,7.5"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1092-6-2-6-9"
+           d="m 186.37692,-126.23112 v 15 l 4.33013,-2.5 v -15 l -4.33013,2.5"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1094-2-5-2-20"
+           d="m 184.64487,-96.231123 v 1.999999 l 12.99038,-7.499996 v -2 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1096-6-4-6-2"
+           d="m 184.64487,-100.23111 v 2.000003 l 12.99038,-7.500013 v -2 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+        <path
+           inkscape:connector-curvature="0"
+           id="path1098-1-7-1-3"
+           d="m 184.64487,-104.23111 v 2 l 12.99038,-7.50001 v -2 z"
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+      </g>
+      <text
+         y="88.744911"
+         x="1746.7788"
+         style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.93749809px;line-height:1.25;font-family:Hack;-inkscape-font-specification:Hack;letter-spacing:0px;word-spacing:0px;white-space:pre;shape-inside:url(#rect1130-2);stroke-width:1.87007797"
+         id="text1128-7-8-4-8-75"
+         xml:space="preserve"><tspan
+           style="stroke-width:1.87007797"
+           id="tspan78-79-4-7-9"
+           y="176.44234"
+           x="1791.7666">rank 0<tspan
+   id="tspan76-2-3-9-2"
+   style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-family:Hack;-inkscape-font-specification:Hack;stroke-width:1.87007797" /></tspan></text>
+      <text
+         y="88.744911"
+         x="1796.7845"
+         style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.93749809px;line-height:1.25;font-family:Hack;-inkscape-font-specification:Hack;letter-spacing:0px;word-spacing:0px;white-space:pre;shape-inside:url(#rect1184);stroke-width:1.87007797"
+         id="text1182-0-0-2-2"
+         xml:space="preserve"><tspan
+           style="stroke-width:1.87007797"
+           id="tspan83-2-7-0-8"
+           y="176.44234"
+           x="1841.7723">rank 1<tspan
+   id="tspan81-3-8-2-9"
+   style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-family:Hack;-inkscape-font-specification:Hack;stroke-width:1.87007797" /></tspan></text>
+      <g
+         id="g970-7-6-3-7"
+         transform="translate(1711.5739,256.07515)">
+        <path
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+           d="m 169.05642,-136.23112 12.99038,7.5 17.32051,-10 -12.99038,-7.5 z"
+           id="path956-5-8-7-3"
+           inkscape:connector-curvature="0" />
+        <path
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+           d="m 182.0468,-128.73112 v 39.999996 l 17.32051,-10 v -39.999996"
+           id="path958-92-8-5-6"
+           sodipodi:nodetypes="cccc"
+           inkscape:connector-curvature="0" />
+        <path
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.79374999;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+           d="m 169.05642,-136.23112 v 39.999996 l 12.99038,7.5"
+           id="path960-2-4-9-1"
+           sodipodi:nodetypes="ccc"
+           inkscape:connector-curvature="0" />
+        <path
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+           d="m 186.37692,-126.23112 v 15 l 4.33013,-2.5 v -15 l -4.33013,2.5"
+           id="path962-8-3-2-2"
+           inkscape:connector-curvature="0" />
+        <path
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+           d="m 184.64487,-96.231123 v 1.999999 l 12.99038,-7.499996 v -2 z"
+           id="path964-9-1-2-9"
+           inkscape:connector-curvature="0" />
+        <path
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+           d="m 184.64487,-100.23111 v 2.000003 l 12.99038,-7.500013 v -2 z"
+           id="path966-7-4-8-3"
+           inkscape:connector-curvature="0" />
+        <path
+           style="fill:#e6e6e6;stroke:#000000;stroke-width:0.529167;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+           d="m 184.64487,-104.23111 v 2 l 12.99038,-7.50001 v -2 z"
+           id="path968-3-9-9-1"
+           inkscape:connector-curvature="0" />
+      </g>
+      <text
+         xml:space="preserve"
+         id="text976-6-2-7-94"
+         style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.93749809px;line-height:1.25;font-family:Hack;-inkscape-font-specification:Hack;letter-spacing:0px;word-spacing:0px;white-space:pre;shape-inside:url(#rect1184);stroke-width:1.87007797"
+         x="1844.8931"
+         y="88.744911"><tspan
+           x="1889.8807"
+           y="176.44234"
+           id="tspan974-1-0-3-7"
+           style="stroke-width:1.87007797">rank 2</tspan></text>
+      <text
+         y="-2.3611109"
+         x="1822.4244"
+         style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.93749809px;line-height:1.25;font-family:Hack;-inkscape-font-specification:Hack;letter-spacing:0px;word-spacing:0px;white-space:pre;shape-inside:url(#rect1130-2);fill:#0000ff;stroke-width:1.87007797"
+         id="text1128-7-8-7-3-7-0-0-3-61"
+         xml:space="preserve"><tspan
+           style="fill:#0000ff;stroke-width:1.87007797"
+           id="tspan78-79-8-2-1-3-6-2-0"
+           y="85.336334"
+           x="1867.4121">msg</tspan></text>
+      <path
+         style="fill:none;stroke:#0000ff;stroke-width:0.76499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#marker4031-5)"
+         d="m 1854.3678,99.061558 c 16.0706,-14.103008 27.5564,-13.085871 41.886,-0.615604"
+         id="path4027-6"
+         inkscape:connector-curvature="0"
+         sodipodi:nodetypes="cc" />
+      <text
+         y="18.408323"
+         x="1793.9163"
+         style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.93749809px;line-height:1.25;font-family:Hack;-inkscape-font-specification:Hack;letter-spacing:0px;word-spacing:0px;white-space:pre;shape-inside:url(#rect1130-2);fill:#0000ff;stroke-width:1.87007797"
+         id="text1128-7-8-7-3-7-0-0-3-61-2"
+         xml:space="preserve"><tspan
+           style="fill:#0000ff;stroke-width:1.87007797"
+           id="tspan78-79-8-2-1-3-6-2-0-0"
+           y="106.10577"
+           x="1838.9039">MPI_Send</tspan></text>
+      <text
+         xml:space="preserve"
+         id="text1992"
+         style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.93749809px;line-height:1.25;font-family:Hack;-inkscape-font-specification:Hack;letter-spacing:0px;word-spacing:0px;white-space:pre;shape-inside:url(#rect1130-2);fill:#0000ff;stroke-width:1.87007797"
+         x="1841.223"
+         y="18.408323"><tspan
+           x="1886.2107"
+           y="106.10577"
+           id="tspan1990"
+           style="fill:#0000ff;stroke-width:1.87007797">MPI_Recv</tspan><tspan
+           id="tspan1994"
+           x="1886.2107"
+           y="106.10577"
+           style="fill:#0000ff;stroke-width:1.87007797" /></text>
+    </g>
   </g>
 </svg>

notebooks/julia_async.ipynb

@@ -28,6 +28,56 @@
     "Understanding these concepts is important to learn distributed computing later."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "cde5ee75",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-info\">\n",
+    "<b>Note:</b> Do not forget to execute the next cell before starting this notebook! \n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0b0496c7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "function why_q1()\n",
+    "    msg = \"\"\"\n",
+    "    Evaluating compute_π(100_000_000) takes about 0.25 seconds on the teacher's laptop. Thus, the loop would take about 2.5 seconds since we are calling the function 10 times.\n",
+    "    \"\"\"\n",
+    "    println(msg)\n",
+    "end\n",
+    "function why_q2()\n",
+    "    msg = \"\"\"\n",
+    "    The time in doing the loop will be O(1) since the loop just schedules 10 tasks, which should be a (small) constant time independent of n.\n",
+    "    \"\"\"\n",
+    "    println(msg)\n",
+    "end\n",
+    "function why_q3()\n",
+    "    msg = \"\"\"\n",
+    "    It will take 2.5 seconds, like in question 1. The @sync macro forces to wait for all tasks we have generated with the @async macro. Since we have created 10 tasks and each of them takes about 0.25 seconds, the total time will be about 2.5 seconds.\n",
+    "    \"\"\"\n",
+    "    println(msg)\n",
+    "end\n",
+    "function why_q4()\n",
+    "    msg = \"\"\"\n",
+    "    It will take about 3 seconds. The channel has buffer size 4, thus the call to put!will not block. The call to take! will not block neither since there is a value stored in the channel. The taken value is 3 and therefore we will wait for 3 seconds.\n",
+    "    \"\"\"\n",
+    "    println(msg)\n",
+    "end\n",
+    "function why_q5()\n",
+    "    msg = \"\"\"\n",
+    "    The channel is not buffered and therefore the call to put! will block. The cell will run forever, since there is no other task that calls take! on this channel.\n",
+    "    \"\"\"\n",
+    "    println(msg)\n",
+    "end\n",
+    "println(\"🥳 Well done! \")"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "caf64254",
@@ -37,7 +87,7 @@
     "\n",
     "### Creating  a task\n",
     "\n",
-    "Technically, a task in Julia is a *symmetric co-routine*. More informally, a task is a piece of computation work that can be started (scheduled) at some point in the future, and that can be interrupted and resumed.  To create a task, we first need to create a function that represents the work to be done in the task. In next cell, we generate a task that generates and sums two matrices."
+    "Technically, a task in Julia is a *symmetric* [*co-routine*](https://en.wikipedia.org/wiki/Coroutine). More informally, a task is a piece of computational work that can be started (scheduled) at some point in the future, and that can be interrupted and resumed.  To create a task, we first need to create a function that represents the work to be done in the task. In the next cell, we generate a task that generates and sums two matrices."
    ]
   },
   {
@@ -233,7 +283,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "fun = () -> compute_π(4_000_000_000)\n",
+    "fun = () -> @show compute_π(4_000_000_000)\n",
     "t = Task(fun)"
    ]
   },
@@ -272,7 +322,7 @@
    "source": [
     "### `yield`\n",
     "\n",
-    "If tasks do not run in parallel, what is the purpose of tasks? Tasks are handy since they can be interrupted and to switch control to other tasks. This is achieved via function `yield`. When we call yield, we provide the opportunity to switch to another task. The function below is a variation of function `compute_π` in which we yield every 1000 iterations. At the call to yield we allow other tasks to take over. Without this call to yield, once we start function `compute_π` we cannot start any other tasks until this function finishes."
+    "If tasks do not run in parallel, what is the purpose of tasks? Tasks are handy since they can be interrupted and to switch control to other tasks. This is achieved via function `yield`. When we call `yield`, we provide the opportunity to switch to another task. The function below is a variation of function `compute_π` in which we `yield` every 1000 iterations. At the call to `yield` we allow other tasks to take over. Without this call to `yield`, once we start function `compute_π` we cannot start any other tasks until this function finishes."
    ]
   },
   {
@@ -299,7 +349,7 @@
    "id": "69fd4131",
    "metadata": {},
    "source": [
-    "You can check this behavior experimentally with the two following cells. The next one creates and schedules a task that computes pi with the function `compute_π_yield`. Note that you can run the 2nd cell bellow while this task is running since we call to yield often inside  `compute_π_yield`."
+    "You can check this behavior experimentally with the two following cells. The next one creates and schedules a task that computes pi with the function `compute_π_yield`. Note that you can run the 2nd cell bellow while this task is running since we call to `yield` often inside  `compute_π_yield`."
    ]
   },
   {
@@ -309,7 +359,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "fun = () -> compute_π_yield(3_000_000_000)\n",
+    "fun = () -> @show compute_π_yield(3_000_000_000)\n",
     "t = Task(fun)\n",
     "schedule(t)"
    ]
@@ -331,7 +381,7 @@
    "source": [
     "### Example: Implementing function sleep\n",
     "\n",
-    "Using yield, we can implement our own version of the sleep function as follows:"
+    "Using `yield`, we can implement our own version of the sleep function as follows:"
    ]
   },
   {
@@ -434,7 +484,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "fun = () -> a + b\n",
+    "fun = () -> add(a, b)\n",
     "t = Task(fun)\n",
     "schedule(t)"
    ]
@@ -679,6 +729,19 @@
     "take!(chnl)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "8a1ef849",
+   "metadata": {},
+   "source": [
+    "In summary:\n",
+    "\n",
+    "- `put!` will wait for a `take!` if there is not space left in the channel's buffer.\n",
+    "- `take!` will wait for a `put!` if there is no data to be consumed in the channel.\n",
+    "- `put!` will raise an error if the channel is closed.\n",
+    "- `take!` will raise an error if the channel is closed *and* empty."
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "9ddd66ca",
@@ -696,7 +759,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "t = @elapsed compute_π(100_000_000)"
+    "n = 140_000_000\n",
+    "t = @elapsed @show compute_π(n)"
    ]
   },
   {
@@ -711,7 +775,7 @@
     "    a) 10*t\n",
     "    b) t\n",
     "    c) 0.1*t\n",
-    "    d) near 0*t \n"
+    "    d) O(1), i.e. time independent from n \n"
    ]
   },
   {
@@ -721,11 +785,22 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "n = 140_000_000\n",
     "@time for i in 1:10\n",
-    "    compute_π(100_000_000)\n",
+    "    @show compute_π(n)\n",
     "end"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d6b8382e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "why_q1()"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "5f19d38c",
@@ -738,7 +813,7 @@
     "    a) 10*t\n",
     "    b) t\n",
     "    c) 0.1*t\n",
-    "    d) near 0*t \n",
+    "    d) O(1) \n",
     "\n"
    ]
   },
@@ -749,11 +824,22 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "n = 140_000_000\n",
     "@time for i in 1:10\n",
-    "    @async compute_π(100_000_000)\n",
+    "    @async @show compute_π(n)\n",
     "end"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "edff9747",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "why_q2()"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "5041c355",
@@ -766,7 +852,7 @@
     "    a) 10*t\n",
     "    b) t\n",
     "    c) 0.1*t\n",
-    "    d) near 0*t \n"
+    "    d) O(1) \n"
    ]
   },
   {
@@ -776,11 +862,22 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "n = 140_000_000\n",
     "@time @sync for i in 1:10\n",
-    "    @async compute_π(100_000_000)\n",
+    "    @async @show compute_π(n)\n",
     "end"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "87bc7c5c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "why_q3()"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "841b690e",
@@ -792,7 +889,7 @@
     "\n",
     "    a) infinity\n",
     "    b) 1 second\n",
-    "    c) near 0 seconds\n",
+    "    c) less than 1 seconds\n",
     "    d) 3 seconds"
    ]
   },
@@ -821,6 +918,16 @@
     "end"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a18a0a7d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "why_q4()"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "df663f11",
@@ -832,7 +939,7 @@
     "\n",
     "    a) infinity\n",
     "    b) 1 second\n",
-    "    c) near 0 seconds\n",
+    "    c) less than 1 seconds\n",
     "    d) 3 seconds"
    ]
   },
@@ -860,6 +967,42 @@
     "end"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d8923fae",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "why_q5()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0ee77abe",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-info\">\n",
+    "<b>Note:</b> If for some reason a cell keeps running forever, we can stop it with Kernel > Interrupt or Kernel > Restart (see tabs above).\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dfab0c90",
+   "metadata": {},
+   "source": [
+    "## Summary\n",
+    "\n",
+    "In order to start \"thinking in parallel\" you first need to be familiar with concepts of asynchronous programming, in particular tasks. In this notebook, we have seen the basics of working with tasks. Some key points to remember:\n",
+    "\n",
+    "- How to create, schedule, and fetch from a task.\n",
+    "- Tasks run asynchronously, but not in parallel. You can have a single core CPU and still be able to work with several tasks.\n",
+    "- Channels are used to communicate data between tasks.\n",
+    "- Adding data (`put!`) or taking data (`take!`) from a channel might wait depending on the channel state. Be careful to avoid dead locks.\n",
+    "\n"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "a5d3730b",
@@ -873,15 +1016,15 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Julia 1.9.0",
+   "display_name": "Julia 1.11.6",
    "language": "julia",
-   "name": "julia-1.9"
+   "name": "julia-1.11"
   },
   "language_info": {
    "file_extension": ".jl",
    "mimetype": "application/julia",
    "name": "julia",
-   "version": "1.9.0"
+   "version": "1.11.6"
   }
  },
  "nbformat": 4,

notebooks/julia_basics.ipynb

@@ -37,7 +37,7 @@
    "source": [
     "## Using Jupyter notebooks in Julia\n",
     "\n",
-    "We are going to use Jupyter notebooks in this and other lectures. You provably have worked with notebooks (in Python). If not, here are the basic concepts you need to know to follow the lessons.\n",
+    "We are going to use Jupyter notebooks in this and other lectures. You probably have worked with notebooks (in Python). If not, here are the basic concepts you need to know to follow the lessons.\n",
     "\n",
     "<div class=\"alert alert-block alert-info\">\n",
     "<b>Tip:</b> Did you know that Jupyter stands for Julia, Python and R?\n",
@@ -147,6 +147,44 @@
     "foo()"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "d18e679d",
+   "metadata": {},
+   "source": [
+    "### A very easy first exercise\n",
+    "\n",
+    "Run the following cell. It contains definitions used later in the notebook."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "81678b3d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "function why_q1()\n",
+    "    msg = \"\"\"\n",
+    "    In the first line, we assign a variable to a value. In the second line, we assign another variable to the same value. Thus, we have 2 variables associated with the same value. In line 3, we associate y to a new value (re-assignment). Thus, we have 2 variables associated with 2 different values. Variable x is still associated with its original value. Thus, the value at the final line is x=1.\n",
+    "    \"\"\"\n",
+    "    println(msg)\n",
+    "end\n",
+    "function why_q2()\n",
+    "    msg = \"\"\"\n",
+    "    It will be 1 for very similar reasons as in the previous questions: we are reassigning a local variable, not the global variable defined outside the function.\n",
+    "    \"\"\"\n",
+    "    println(msg)\n",
+    "end\n",
+    "function why_q3()\n",
+    "    msg = \"\"\"\n",
+    "    It will be 6. In the returned function f2, x is equal to 2. Thus, when calling f2(3) we compute 2*3.\n",
+    "    \"\"\"\n",
+    "    println(msg)\n",
+    "end\n",
+    "println(\"🥳 Well done! \")"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "92112bd1",
@@ -467,6 +505,24 @@
     "x"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "a2f94960",
+   "metadata": {},
+   "source": [
+    "Run next cell to get an explanation of this question."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fc562337",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "why_q1()"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "4d2cb752",
@@ -586,6 +642,24 @@
     "x"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "f69108c2",
+   "metadata": {},
+   "source": [
+    "Run next cell to get an explanation of this question."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "05c62aa3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "why_q2()"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "4fc5eb9b",
@@ -800,7 +874,7 @@
    "source": [
     "### Higher-order functions\n",
     "\n",
-    "Higher order functions are functions that take and/or return other functions. And example is the `count` function in Julia.\n",
+    "Higher order functions are functions that take and/or return other functions. An example is the `count` function in Julia.\n",
     "\n",
     "For instance, we can pass a user-defined function to count the number of even elements in an array."
    ]
@@ -1068,6 +1142,24 @@
     "x"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "062ff145",
+   "metadata": {},
+   "source": [
+    "Run next cell to get an explanation of this question."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6bf7818e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "why_q3()"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "bc8e9bcf",
@@ -1588,7 +1680,7 @@
    "source": [
     "### Exercise 3 (hard)\n",
     "\n",
-    "Function `mandel` estimates if a given point `(x,y)` in the complex plane belongs to the [Mandelbrot set](https://en.wikipedia.org/wiki/Mandelbrot_set)."
+    "Function `surprise(x,y)` takes the `x` and `y` coordinates of a point and returns a scalar value. Visualize the result of function `surprise(x,y)` for all points in a Cartesian grid of the box with range $(-1.7,0.7)$ in the horizontal direction and range $(-1.2,1.2)$ in vertical direction. Use at least `n=1000` points in each direction (1000$^2$ points in total). What do you see? Surprise!\n"
    ]
   },
   {
@@ -1598,7 +1690,11 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "function mandel(x,y,max_iters)\n",
+    "function surprise(x,y)\n",
+    "    # You don't need to understand\n",
+    "    # the body of this function.\n",
+    "    # Treat it as a black box.\n",
+    "    max_iters = 100\n",
     "    z = Complex(x,y)\n",
     "    c = z\n",
     "    threshold=2\n",
@@ -1608,22 +1704,38 @@
     "        end\n",
     "        z = z^2 +c\n",
     "    end\n",
-    "    max_iters\n",
+    "    return max_iters\n",
     "end"
    ]
   },
   {
+   "attachments": {
+    "Screenshot%20from%202025-09-05%2014-41-15.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAAA9EAAANbCAYAAABFGxzNAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AACAASURBVHic7N13fFRl3v7xa1JAQAyEJiGoIKIC+kukS9QlIAgEkIdQVpAiTQHdxfaoi66vfURZBSzAAxKDiItSRLoUKUuR3pSiS0JgEVjpxYCEDDm/P3bhMU4STpK555xJPu/Xa/7wzOS+v3Mte5IrmTnjsSzLEgAAAAAAuK4QpwcAAAAAACBYUKIBAAAAALCJEg0AAAAAgE2UaAAAAAAAbKJEAwAAAABgEyUaAAAAAACbKNEAAAAAANhEiQYAAAAAwCZKNAAAAAAANlGiAQAAAACwiRINAAAAAIBNlGgAAAAAAGyiRAMAAAAAYBMlGgAAAAAAmyjRAAAAAADYRIkGAAAAAMAmSjQAAAAAADZRogEAAAAAsIkSDQAAAACATZRoAAAAAABsokQDAAAAAGATJRoAAAAAAJso0QAAAAAA2ESJBgAAAADAJko0AAAAAAA2UaIBAAAAALCJEg0AAAAAgE1hTg8AOOHQoUM6ceKEz/HY2FiFhPjvd0vff/+9Ll68mO1YqVKlVKdOHb/tAQAAACBwKNEolr755hs99thjPseXLl2qVq1a+WWPU6dO6b777tOlS5eyHR88eLDGjx/vlz0AAAAABBYv50ax1LlzZ1WpUsXn+KRJk/y2xyeffOJToCWpf//+ftsDAAAAQGBRolEslShRQk888YTP8Xnz5uno0aN+2SM5OdnnWOPGjRUbG+uX9QEAAAAEHiUaxdaTTz6p0NDQbMe8Xq8++eSTQq+9Zs0a7d271+f4oEGDCr02AAAAAOdQolFs3XLLLWrTpo3P8aSkJGVlZRVq7ZxeFh4REaGuXbsWal0AAAAAzqJEo1h76qmnfI4dOHBAy5cvL/Cap06d0uzZs32OP/744ypTpkyB1wUAAADgPEo0irVHHnlENWrU8DmelJRU4DWnTp2a4wXF+vXrV+A1AQAAALgDJRrFWkhIiAYOHOhzfO7cuQW+wFhOFxS7//77FRMTU6D1AAAAALgHJRrFXv/+/VWyZMlsxwp6gbE1a9Zoz549Pse5oBgAAABQNFCiUexVrFhRnTt39jlekAuM5fQy8HLlyikxMbHA8wEAAABwD0o0oNwvMLZixQrba5w9e1Zffvmlz/HevXurdOnShZoPAAAAgDtQogFJcXFxuueee3yO5/RRVbmZMmWKLl686HN8wIABhZoNAAAAgHtQooH/ePLJJ32OzZ07V//6179sff1HH33kc+yBBx5Q3bp1Cz0bAAAAAHegRAP/8fjjj+umm27KdszuBcbWrl3LBcUAAACAYoASDfxH2bJl9dhjj/kct3OBsZxe9l2hQoUcL1gGAAAAIHhRooFfyekCY2lpaXleYCy3C4r16tVLN9xwg1/nAwAAAOAsSjTwK/fee6/uv/9+n+M5fXTVVbldUKx///5+nQ0AAACA8yjRwG/k9NfovC4wltMFxR566CHVqVPH77MBAAAAcBYlGviNLl26qHLlytmOZWZmaurUqT6PXbduHRcUAwAAAIoRSjTwGyVLllSfPn18jk+aNMnnAmO5XVCsU6dOpsYDAAAA4CBKNJCDQYMGKSQk+/890tLStHLlymv/ffbsWc2ePdvna/v27csFxQAAAIAiihIN5KBmzZpq1aqVz/FfX2Dsk08+8bmgmMfj0YABA4zPBwAAAMAZlGggFzldYGzOnDnXLjCW0wXFmjdvrtq1axufDQAAAIAzKNFALtq1a6dbb70127GrFxhbt26ddu/e7fM1XFAMAAAAKNoo0UAuQkNDc/ys50mTJunDDz/0OV6xYkV17NgxEKMBAAAAcIjHsizL6SEAt/rpp590yy23KDMzM9vxkJAQnyt1v/jii/rrX/8ayPEAAAAABBh/iQbycPPNN+f4cVW/LdAejyfHv1oDAAAAKFoo0cB15HSBsd9q2bKl7rjjjgBMAwAAAMBJlGjgOn73u9+pXr16eT5m4MCBAZoGAAAAgJMo0YANeZXkKlWqcEExAAAAoJigRAM29OrVS2XKlMnxvieeeELh4eEBnggAAACAEyjRgA0RERFq3769z/GQkBANGDDAgYkAAAAAOIESDdhgWZa2bNnic/zhhx9WjRo1HJgIAAAAgBMo0YANy5cv1/79+32ODxo0yIFpAAAAADiFEg3YMGHCBJ9jVatWzfEl3gAAAACKLko0cB2pqamaP3++z/EBAwYoLCzMgYkAAAAAOIUSDVzHe++9pytXrmQ7Fh4ezmdDAwAAAMUQJRrIw6FDhzRlyhSf4507d1a1atUCPxAAAAAAR1GigVxkZmbqiSee0IULF7IdDwkJ0SuvvOLQVAAAAACcRIkGfiUrK0uHDh3SvHnzFBcXpxUrVvg8plu3brrnnnscmA4AAACA0zyWZVlODwE4qV27dvrqq69sPTYiIkK7du1S9erVDU8FAAAAwI34SzRgk8fj0bhx4yjQAAAAQDFGiQZsCAkJ0TvvvKOePXs6PQoAAAAAB1GigTx4PB49/PDDWrt2rZ577jmnxwEAAADgMN4TjWJv1qxZSktLy3asbNmyqlatmpo0aaIqVao4NBkAAAAAt6FEAwAAAABgEy/nBgAAAADAJko0AAAAAAA2UaIBAAAAALCJEg0AAAAAgE2UaAAAAAAAbKJEAwAAAABgEyUaAAAAAACbKNEAAAAAANhEiQYAAAAAwCZKNAAAAAAANlGiAQAAAACwiRINAAAAAIBNlGgAAAAAAGyiRAMAAAAAYBMlGgAAAAAAmyjRAAAAAADYRIkGAAAAAMAmSjQAAAAAADZRogEAAAAAsIkSDQAAAACATZRoAAAAAABsokQDAAAAAGATJRoAAAAAAJso0YCfnT9/3ukRAAAAABhCiQb85NixY2ratKnKlSunZs2a6dChQ06PBAAAAMDPPJZlWU4PAQS7n376SQ8//LB2796tEiVK6PLly4qKitK6detUo0YNp8cDAAAA4Cf8JRoopIMHDyouLk67d+9WTEyMUlNT1bp1ax09elSPPPKITp065fSIAAAAAPyEEg0UwrFjx/Twww9r//79atasmVauXKnq1atr9uzZatSokfbt26f27dvr7NmzTo8KAAAAwA8o0UABZWZmql27dkpNTVVcXJyWLVum8uXLS5LKlCmjBQsWqFatWtqwYYMaN26sGTNmKCsry+GpAQAAABQGJRoooDFjxmjbtm26++67tXDhQrVv314ej0cej0ctWrRQ5cqVtXbtWtWvX1/79u1T9+7d1bRpU50+fdrp0QEAAAAUECUaKIB//vOf+stf/iJJeuqppxQREZHj426++WZt3rxZn376qWrVqqXNmzerUaNGGj9+vC5duhTIkQEAAAD4ASUaKIAFCxbo4sWLuvHGG9WpU6c8HxsSEqKePXtq06ZNatSokfbv36+hQ4cqLi6O90oDAAAAQYYSDeTT2bNnNWHCBEnSa6+9pujoaFtfFxkZqY0bN2rBggW6++67tW3bNrVv316XL182OS4AAAAAP6JEAzZdvHhRvXv3VmRkpPbu3avY2Fj98Y9/zNcaHo9HCQkJatq0qSRp3bp1+uqrr0yMCwAAAMAASjRgw6VLl9S8eXNNnTpVYWFhiomJ0YIFCxQeHl6g9e6//36FhPz7/37jx4/XhQsX/DkuAAAAAEMo0YANU6ZM0ebNm3XXXXcpJSVFO3bsULVq1Qq8Xr9+/TR//nyVK1dOy5cvV2xsrCZOnKiMjAw/Tg0AAADA3yjRwHWcOXNGo0ePliS9/fbbuvXWW/2ybrt27bRlyxbdd999SklJ0VNPPcXFxgAAAACXo0QDefj555/Vtm1bpaamqmHDhkpISPDr+rVq1dLWrVs1f/583X333dq6das6dOigX375xa/7AAAAAPAPSjSQi23btql+/frauHGj7rzzTi1cuFAej8fv+3g8HrVv314rV65U6dKltXbtWk2cONHv+wAAAAAoPEo08BunT5/WH/7wBzVt2lQpKSlq1KiRli9frsqVKxvd95dfftHFixclSTfddJPRvQAAAAAUDCUa+A+v16tx48apdu3a+uCDDyRJL7/8statW2f7s6ALo0aNGmrUqJEk6YMPPtDRo0eN7wkAAAAgfyjRwH+sXLlSTz/9tE6dOqV27dpp165devPNNwv8MVYFMWfOHN1777367rvv1L1794DtCwAAAMAeSjTwH61atdIf/vAHLVmyRAsXLtSdd94Z8BmioqK0cuVKSdLatWvVrVu3gM8AAAAAIHeUaOBX3nvvPbVu3drRGSpUqKDOnTtLkmbNmqUKFSo4Og8AAACA/0OJBlzoiy++0GOPPSZJysjI0IkTJxyeCAAAAIBEiQZca9q0aWrYsKEuXLigKlWqaPjw4U6PBAAAABR7lGjAxT7//HPFxcXJsiydOnXK6XEAAACAYi/M6QEA5K5mzZpau3atzp8/r1KlSjk9DgAAAFDsUaKBIHDTTTc5PQIAAAAA8XJuAAAAAABso0QDAAAAAGATL+d2gNfr1c8//5zt2A033OCa97xmZWVp7969+vbbb7V3714dP35c586d04033qhy5crptttuU5MmTRQbG6vw8HCnxwUAAACAgKFEG3blyhX98MMP2rZtm7Zu3apt27Zp586dunjxYrbHPfvssxo9erRDU0q//PKL5s+frwULFmjZsmW2Ppc4MjJS/fr109ChQ3XLLbcEYEoAAAAAcBYl2oB//etf+utf/6pt27Zpx44dunDhgtMj5SozM1ODBg3S7Nmzdf78+Xx97enTp/XOO+9owoQJGjVqlAYOHCiPx2NoUgAAAABwHiXagIMHD+r99993egxbMjIy9PHHHxdqjfT0dD355JPau3dv0DxvAAAAACgISjRydPvttysuLk5NmjRRLine truncated
+    }
+   },
    "cell_type": "markdown",
-   "id": "86ea2591",
+   "id": "de548741",
    "metadata": {},
    "source": [
-    "If the value of `mandel` is less than `max_iters`, the point is provably outside the Mandelbrot set. If `mandel` is equal to  `max_iters`, then the point is provably inside the set. The larger `max_iters`, the better the quality of the estimate (the nicer will be your plot).\n",
+    "The gid of points is shown in the next figure for `n=7` (but you should use at least `n=1000`).\n",
     "\n",
-    "Plot the value of function `mandel` for each pixel in a 2D grid of the box.\n",
+    "<div>\n",
+    "<img src=\"attachment:Screenshot%20from%202025-09-05%2014-41-15.png\", width=\"550\">\n",
+    "</div>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "95e81488",
+   "metadata": {},
+   "source": [
+    "### Hints\n",
     "\n",
-    "$$(-1.7,0.7)\\times(-1.2,1.2).$$\n",
-    "\n",
-    "Use a grid resolution of at least 1000 points in each direction and `max_iters` at least 10. You can increase these values to get nicer plots. To plot the values use function `heatmap` from the Julia package `GLMakie`. Use `LinRange` to divide the horizontal and vertical axes into pixels. See the documentation of these functions for help. `GLMakie` is a GPU-accelerated plotting back-end for Julia. It is a large package and it can take some time to install and to generate the first plot. Be patient."
+    "* Create a matrix `values` of the right size and fill it in so that `values[i,j]=surprise(x[i],y[j])` is the value for point `i` in the horizontal direction and `j` in the vertical direction (see the figure above).\n",
+    "* The vectors `x` and `y` can be created with function `LinRange`.\n",
+    "* Visualize the values in `values` with function `heatmap` from the Julia package `GLMakie`. `GLMakie` is a GPU-accelerated plotting back-end for Julia. It is a large package and it can take some time to install and to generate the first plot. Be patient.\n",
+    "*  See the documentation of these functions for help."
    ]
   },
   {
@@ -1649,15 +1761,15 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Julia 1.9.0",
+   "display_name": "Julia 1.11.6",
    "language": "julia",
-   "name": "julia-1.9"
+   "name": "julia-1.11"
   },
   "language_info": {
    "file_extension": ".jl",
    "mimetype": "application/julia",
    "name": "julia",
-   "version": "1.9.0"
+   "version": "1.11.6"
   }
  },
  "nbformat": 4,

notebooks/julia_distributed.ipynb

@@ -25,7 +25,7 @@
    "source": [
     "## Contents\n",
     "\n",
-    "In this notebook, we will learn the basics of distributed computing in Julia. In particular, we will focus on the Distributed module available in the Julia standard library. The main topics we are going to cover are:\n",
+    "In this notebook, we will learn the basics of distributed computing in Julia. In particular, we will focus on the [Distributed](https://docs.julialang.org/en/v1/manual/distributed-computing/) module available in the Julia standard library. The main topics we are going to cover are:\n",
     "\n",
     "- How to create Julia processes\n",
     "- How to execute code remotely\n",
@@ -60,7 +60,32 @@
     "    end |> println\n",
     "end\n",
     "q_1_check(answer) = answer_checker(answer,\"a\")\n",
-    "q_2_check(answer) = answer_checker(answer,\"b\")"
+    "q_2_check(answer) = answer_checker(answer,\"b\")\n",
+    "function why_q1()\n",
+    "    msg = \"\"\"\n",
+    "    We send the matrix (16 entries) and then we receive back the result (1 extra integer). Thus, the total number of transferred integers in 17.\n",
+    "    \"\"\"\n",
+    "    println(msg)\n",
+    "end\n",
+    "function why_q2()\n",
+    "    msg = \"\"\"\n",
+    "    Even though we only use a single entry of the matrix in the remote worker, the entire matrix is captured and sent to the worker. Thus, we will transfer 17 integers like in Question 1.\n",
+    "    \"\"\"\n",
+    "    println(msg)\n",
+    "end\n",
+    "function why_q3()\n",
+    "    msg = \"\"\"\n",
+    "    The value of x will still be zero since the worker receives a copy of the matrix and it modifies this copy, not the original one.\n",
+    "    \"\"\"\n",
+    "    println(msg)\n",
+    "end\n",
+    "function why_q4()\n",
+    "    msg = \"\"\"\n",
+    "    In this case, the code a[2]=2 is executed in the main process. Since the matrix is already in the main process, it is not needed to create and send a copy of it. Thus, the code modifies the original matrix and the value of x will be 2.\n",
+    "    \"\"\"\n",
+    "    println(msg)\n",
+    "end\n",
+    "println(\"🥳 Well done! \")"
    ]
   },
   {
@@ -112,24 +137,33 @@
     "\n",
     "\n",
     "<div class=\"alert alert-block alert-info\">\n",
-    "<b>Tip:</b> We can also start new processes when launching Julia from the command line by suing the `-p` command-line argument. E.g., `$ julia -p 3 ` would launch Julia with 3 extra processes.\n",
+    "<b>Tip:</b> We can also start new processes when launching Julia from the command line by using the `-p` command-line argument. E.g., `$ julia -p 3 ` would launch Julia with 3 extra processes.\n",
     "</div>\n"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d2e507dc",
+   "id": "a597f9be",
    "metadata": {},
    "source": [
     "### Each process runs a separated Julia instance\n",
     "\n",
-    "When adding the new processes, you can imagine that 3 new Julia REPLs have started under the hood (see figure below). The main point of the Distributed module is to provide a way of coordinating all these Julia processes to run code in parallel. It is important to note that each process runs in a separated Julia instance. This means that each process has its own memory space and therefore they do not share memory. This results in distributed-memory parallelism, and allows one to run processes in different machines.\n",
-    " \n",
-    " <div>\n",
-    "<img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAADE8AAAasCAYAAABTLlZXAAAACXBIWXMAABcRAAAXEQHKJvM/AAAAGXRFWHRTb2Z0d2FyZQB3d3cuaW5rc2NhcGUub3Jnm+48GgAAIABJREFUeJzs3Wl0XOd95/nvvbXvOwo7QGzcd4q7SJHaqN2bHDuZOEl3Okn38Tnd02dOz3ROpudNOt19ZubFxONJJz2ZyeIkdhLbkW3J1i5KpERS1M4VXAAQIHZUoRbUXnXnBWUPJVE7KVDw7/OSAG79ceu59z5F/H/PYyAiIiIiIiIiIiIiIrK0fAv45mIXISIiIiIiIiIiIiIiNw9zsQsQERERERERERERERERERERERERERERERG5kRSeEBERERERERERERERERERERERERERERGRJU3hCRERERERERERERERERERERERERERERERWdIUnhARERERERERERERERERERERERERERERkSVN4QkREREREREREREREREREREREREREREREVnSFJ4QEREREREREREREREREREREREREREREZElTeEJERERERERERERERERERERERERERERERFZ0hSeEBERERERERERERERERERERERERERERGRJU3hCRERERERERERERERERERERERERERERERWdIUnhARERERERERERERERERERERERERERERkSVN4QkREREREREREREREREREREREREREREREVnSFJ4QEREREREREREREREREREREREREREREZElTeEJERERERERERERERERERERERERERERERFZ0hSeEBERERERERERERERERERERERERERERGRJU3hCRERERERERERERERERERERERERERERERWdIUnhARERERERERERERERERERERERERERERkSVN4QkREREREREREREREREREREREREREREREVnSFJ4QEREREREREREREREREREREREREREREZElTeEJERERERERERERERERERERERERERERERFZ0hSeEBERERERERERERERERERERERERERERGRJU3hCRERERERERERERERERERERERERERERERWdIUnhARERERERERERERERERERERERERERERkSVN4QkREREREREREREREREREREREREREREREVnSFJ4QEREREREREREREREREREREREREREREZElTeEJERERERERERERERERERERERERERERERFZ0hSeEBERERERERERERERERERERERERERERGRJU3hCRERERERERERERERERERERERERERERERWdIUnhARERERERERERERERERERERERERERERkSVN4QkREREREREREREREREREREREREREREREVnSFJ4QEREREREREREREREREREREREREREREZElTeEJERERERERERERERERERERERERERERERFZ0hSeEBERERERERERERERERERERERERERERGRJU3hCRERERERERERERERERERERERERERERERWdIUnhARERERERERERERERERERERERERERERkSVN4QkREREREREREREREREREREREREREREREVnSFJ4QEREREREREREREREREREREREREREREZElzb7YBYiIiIiIiIiIiIiIiIiIyI1j2AzsHhPTdh1X17MsGlWLasmiYV2vg4qIiIiIiIiIiNw4Ck+IiIiIiIiIiIiIiIiIiCxZJja3jeAyJx6fgdO4Toet1inP10iPVClXoX6dDisiIiIiIiIiInKjKDwhIiIiIiIiIiIiIiIiIrJUmQ48zT5W/naUrmUO4tfpL8RWOs/UazmO/j9ZZtINFrT7hIiIiIiIiIiI3OQUnhARERERERERERERERERWaoME5vHQajXQ9MqFy3X6S/EjZkajZkibgeY1+eQIiIiIiIiIiIiN5TCEyIiIiIiIiIiIiIiIiIiS5aF1bColRpUFuqUr9fOE4UG1bJFQztOiIiIiIiIiIjI54TCEyIiIiIiIiIiIiIiIiIiS1WjQmE0w+v/W4nzARO3cX0Oa1WqFOeqzGQalBSgEBERERERERGRzwGFJ0REREREREREREREREREliqrQTVXZupImanFrkVERERERERERGQRmYtdgIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIyI2k8ISIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiCxpCk+IiIiIiIiIiIiIiIiIiIiIiIiIiIiIiMiSpvCEiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIgsaQpPiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIjIkmZf7AJERERERERERERERERERORjMg1sATsev4nXZ2BYFrVCndJMjXINao1FqstuYrptBKImTreBCVTnq5TzdQoFsBapLFmq7GB6iSSjhMMu3I082dkMqbk8ZWCxLgMRWapcuANBQs0JAvY6ZjlPdnqKbKlGobbYtf2ys+Pw+Ak0JQm5TVyNIguzE2QWymQri12byHVkd2N4YzRHvES8QCnNXCrPTKakec/NwuHH6Q3TnPDhtVcxylmmprPkFipUF6kelzdEsimA12nD9p5vqFFZyJKbniJTblCsf9oXtGPaPYSaEwT9Tvxmjdz0FNlsjkxlkT4PGjZwhgiFAzTF/dh596rzDaDGwtws2XSabAVq+uAqsqQpPCEiIiIiIiIiIiIiIiIi8jlj89oJbk+wYqePzbfYMetV5o5nGfzLGS7OWMwtLE5dZtSDb3mInQ/76Oy147bqTP90mguHcrx2sk5VzaVyPdmi4N7A/t/8Ivfe18fyhSM8+Z2f8Y9/e5jhOizSZSAiS5IJZiudm27n3v/+X7A7Oo9n+AgH/+RbPDk4x2tz6rJcVEaYeO8ObvuX3+SOAQ+9xTO8/Bf/Kz87foGnL6mlXJYOM7wMx7pf5RtfXs8X1xoYgz/gr3/wIn/52FkKwKfue5dPzUxsoHnDPfzb39zKhvgk5sjTfPvbz3LwlREmF+FRYUQ30LbxDv71797K2s4okXekBixoTHPp1ec5+F+/xZMX87yV/pRFGhHc4ZXc/i//Bfu2d7DdP83B//otnn7qME9eblBZjFuyw4/Zege7HtjH731jOzETvMZVX7eK0Jjm2Hf/kme//whPjjeYKy9CnSLymVF4QkRERERERERERERERETk88TlwJn00bXdT+8OL239JvXLVQpmnXyRRQ0oWOUGjXydmtuBs81Na8jCmwtQL9UYG1kglbMoqatLrhfDAbYwsfZe+tatYW12nLNNQXwGmAZLa6sT04F32Xb6+3u5dUUYu2Hw854vq1HHatSolosUF7LkMylmJy8zMTnN6MQ85YZWz5VPyLCDK0qiKUZLSwBXLUNudo7LI7MUgZsuD2fzgDNCe0eMeNiBqzbP9OVZpqeyFPm0u9EYYLjxBJO0rljH8uQsfsc454MO/Oq+ugk4cHojxHtW07fax+qFGjNxDyGX8eE/+nGYLnBGaG6NkYx7cNXnSU3OMXk5TZFP0bh+Q8eufCCbG1t0JctX9rJ9Yyue4ghT589z7NApZqsWhU/6/HQEsEVXsn5dDxtXRvEULzFyapBXjg6SqkPpEx/XixnqpL13JevXGtBI0hrxYgeu82iXT8hwBnFFOlm2YjVrk0EMxwniARcOrrxHn/mUzDAwTDsOpwuny4XLNAAH7kCQYCKG3xzHl73IOb8d3/V4nhkObI4Q8a5+elf3syE0xkhTkKjz7fn5ojDAMLHZHG+fA3AZNsBBIBEnGDTwG+OkXohxwm1gNz/0gCLyOafpu4iIiIiIiIiIiIiIiIjI54gZcuPrCTCwxU17jw2j3qB8foG50wWGZy3KixhOsDJlKkM5Lg0GCXW4aI2YhNb5acnW6DxUpFqtU9J2ACIfm2F3EVzzALu+/BD/6et9eGwGpmXRaDSwGjVq1SKl/DypyVEmLp3jzOtHOH78DQ4dO8NktkSmVKWq7lv5uGxu8C2ja+1a9uxaRrg0yPBrb/Ls6CzTjZswPOEIYASXs3LrOjau8BMuDvLyM69xbDpL1YLKp36BBlajTq1SolIuU65UqTYsGgon/fKwezECffRtWsv2jUnCxbOcfOkNDk+kmW58ivDEDR+7cm0GpiuEZ9kd7Pry/fz+720lPPEoR374A2ZePkOpXqfwid5UA5s3ga//fu787w7wrx7uJTz+GD/76+8z/cogxcanCE/ITc+yGlj1KtVymVK5glmpUW9Yi5fpLc1RmDrDm0cd5Ib8+E0TCNHU00fftihdrutdmYVl1alXK1TePgfVeoP6Yo75RhUKl5m6+DpHnisTNMFluMEI0rt1E739CTyuRaxPRD5zCk+IiIiIiIiIiIiIiIiIiHxumIQGPLTfFqApYcNva9AoVpg4UWLyVIVqY7FXJ25QL1eZfLNIU7uT8ko3TreLYIeX1XvdZA+WmDtb1wrKIp9Wo0y1XCa/UKZumWCz4/LGiXcFCTd307F8C5t2n+GBc0d59AePc+j4WU6l6tqBQj4e04Xp76ZrzS5uf2gLLblDvFpP89ZPXyNjwcLNNp7sbzegb9vHXXtjtGT9VIcvc+7YeVKfOlhogVVkITXK8KtHeC2SwXPpHEPpCtnq9ShePhdsb4cnNu7ljgd7ac0F8aamOfPcKeaNT9EQf0PHrrw/Fy5vglU7NrFyTQ/RRpGZ11/h3OtvcLbcIPOJJ6we/JFW1t62lYGBdoLFDFMvH2Hw5CnOVSCvifCSZpVTFGYGOf26H1tkBmNsjIlMkTKLsymatXCJ1PkMP/27V/C6bNgMB5gdrL79Hu5ZsZFg5Hq/YIV6Jc3k4JucCszh8c1yZizFdGkRw4b1Eo30SQaPjDJ/4TnsgGmGwezgLmeMO1sTtDpvtkmNiNxICk+IiIiIiIiIiIiIiIiIiHwe2Ezwuon2uene5Line truncated
-    "</div>\n",
-    " \n",
-    " "
+    "When adding the new processes, you can imagine that 3 additional Julia REPLs have started under the hood (see figure below). The main point of the Distributed module is to provide a way of coordinating all these Julia processes to run code in parallel. It is important to note that each process runs in a separated Julia instance. This means that each process has its own memory space and therefore they do not share any data. This results in distributed-memory parallelism, and allows one to run processes in different machines."
+   ]
+  },
+  {
+   "attachments": {
+    "image838.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAAByQAAAO7CAYAAAAV4zMQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13dBXV9gfw78zcnt47afQSqqCCNAEVARFURECxywN5lodi7x3R93zv/cSn2AsqIipIL9I7hADppLd7c3tv+/dHAiQhgSCJIO7PWmctF7lzzpmZMzOu2bPPEcAYY4wxxhhjjF14aQAKLnQnGGOMMcYYY4y1PfFCd4AxxhhjjDHGGGOMMcYYY4wxdunigCRjjDHGGGOMMcYYY4wxxhhjrN1wQJIxxhhjjDHGGGOMMcYYY4wx1m44IMkYY4wxxhhjjDHGGGOMMcYYazcckGSMMcYYY4wxxhhjjDHGGGOMtRsOSDLGGGOMMcYYY4wxxhhjjDHG2g0HJBljjDHGGGOMMcYYY4wxxhhj7YYDkowxxhhjjDHGGGOMMcYYY4yxdsMBScYYY4wxxhhjjDHGGGOMMcZYu+GAJGOMMcYYY4wxxhhjjDHGGGOs3XBAkjHGGGOMMcYYY4wxxhhjjDHWbjggyRhjjDHGGGOMMcYYY4wxxhhrNxyQZIwxxhhjjDHGGGOMMcYYY4y1Gw5IMsYYY4wxxhhjjDHGGGOMMcbaDQckGWOMMcYYY4wxxhhjjDHGGGPthgOSjDHGGGOMMcYYY4wxxhhjjLF2wwFJxhhjjDHGGGOMMcYYY4wxxli7kV3oDjDGGGOMMcYYY4z9lUlqQBQBQWjlBgR4bAC1a68YY4wxxhhrOxyQZIwxxhhjjDHGGLtQBCB6hISo8HPYxk8oWuKH2dduvWKMMcYYY6xNcUCSMcYYY4wxxhhj7EIRBIT1EZGcLLQ+Q9LjR833HJBkjDHGGGN/Hq39X13GGGOMMcYYY6w9pQEouNCdYOxCUCcIUCnPYQMiWIsAD8/ZyhhjjDHG/iQ4Q5IxxhhjjDHGGGPsAnKUExwXuhOMMcYYY4y1I/FCd4AxxhhjjDHGGGOMMcYYY4wxdunigCRjjDHGGGOMMcYYY4wxxhhjrN3wlK2MMcYYY4wxxhhj7U0AgnuKiE4AfFZC1U6Cw9sG9YpASDcREYmAaCOUbCe4/W1QL2uGgLBuI3DzmP4Qs77GovVl4GU82cVA1fkazJ3cD6a8tViydC+Ml/jADErsidHXXodU63b8b9lOmF2+C90ldhbB6VfghjGDEWvYjH99sweuC92hP0BAdDpGTpiMVMtOLFm5A9UWz/lXquqB2+ZOQLJ06p/81Tvx/uKNMLWyCnXPSXjw+hSUH1qD5auyYD3/XgHyJIyZMQH9Y4JP/hOZCvDrz2twuNQI/t8SdgIHJBljjDHGGGOMMcbamSxBRPr1EhJTBBg3eFDRVgEDAhAuIGWMiACJIDN6kZ3VRnWzJgREZIzFw0/OguzzHfhwfRnaIqb8e4ipV+PZubeiS2wgAILf64bVoEVJfhZ2bVqLjZkV4BBNW1Ii7apr0TvUhfz9m3G4vD1WfRUQGJOGvv17Q1NzAJsOFsHlbd2NQt3jBrz46iwU/WTDmh8u9YCkgJDkvrjt7y9gdMUr+G7V3nMLSAoa9BpxLVLESmTu34divfus7WkiOqDPwP4I1h/G1gMFsF5SX30I6HLjc3hyQih2LP0EX648CMtZdk8eloKxMx7ETT30+PStRVifrzvLxxkCQrsMwwPznka//Kew6C8RkBQQGNsFt8x9CaPL38Rv2/a1TUBSkY5rpt+BQQpAVAUjNiYKyqML8e3HG2Fq5XWv6TsNL744EjsW67BpTRasbTGcZTEYNO5m3NY9FqIkR0hUAiKM66E9vBdHOCDJGuCAJGOMMcYYY4wxxlh7kgMxA0VEJQJw+1G8kdBmCT0EmA/6UXOliNQ0EQljRFQX+mGwt1H97KIkhqXj6rE3YnCnCAAEEMHndcPpsMGkL8feb17E319YiuJL/63/H0MIxrAHF+KFDB0W/+NGZJU72iE7VobYbsPx95dfQOK2JzAhuxQ11taFvH32WhQdP47yGvMFC5L/aYjhmDj/n5gmrcdzjz3cioCkiIi0QZj14kJ0PfAKpjxRDGvt2bb5E5H3wF1PPoRxAbuw9wstHGeNHEmI6zQMd869D72PvoP3XO1xLVwKCD63HdryIpRWG+Bu5ccFZ2XdgPkTroNKEBDSZzLeeuUJDD3HKnzWGhQVl6Cq1oK26hacWfjvgzPwhUIGeVgypj6zGE/2baO62SWFA5KMMcYYY4wxxhhj7UiZJCKuuwClHHDt9KNc27b1k4VQepCQkgqokyQk9vHDuB38kvivwLsfD/Yeik8q1IhN641Rt9yPefePw4THv0Z44BTc9NAy1HBqShsQoQ6LQlQkIVAptlsrkkKN0MgoRASrIQmt3868+hl0TXum3fp1aZEQGB6NKCkEannrzqUkVyM0IgoRIRpI4jmcmIuegMiJT+CeDAHHvvkFa7aVnzWgLSgCkDHuToyMKcUnr61HXoXtD+npn5EuexMeua5L21bqt6KyqG6S1dBoLey/4+sm47JZ6LqsbbsFcqK2vBS1ABRmEbXWNsgGZZckDkgyxhhjjDHGGGOMtRcZEJouIDROAHyEsh3+dplK03zQD9P1EkLVQFgvEcEH/DC1x6yS7CJD8LocsJlsKDiwHgUHNmPltpex7pt56DNxPu5duRtvrGocZBAkBYLCIhEZFgy1Ugb4PLBbjajV6mB2tjw6BbkG4RERCAsJhFIugbxuOB1WGGtrYbC6TgbAFUGRSIyPhdpZiaPFtY0C47LgOKQnRkByViGvUAcPAFVYPDrEhcOtLYVDE4OIAAkusw6VWgsUYTGIjQiC4LZCV1UBncXdONAuiFAFhiEyMgyBaiVkAsHttMKg00JvdsDX4MeiTInoDh0RKZlRUq6DXx2GmMhQqOQCvE4r9DU1qLU44K/fRpCpERmfhNgQJSBGIiFQhCgpEJncDRm9wk5OQejWFSKnskFQRpBBHRSC0JBgBKiVUMhlEMgHt8MKY20NdGZXo32QQhLRrUMYBEGO1NR4BMoFKMIS0K1HL0Ta6s6c32lGRXk5DPYTZ1KAMjACCR3iENBgLTmXuQJFxbU4U/6eTB2CyKgIhASqIRcIHrcDZr0OWoMVXn/jzxjUEUlIiQ0FGYuQr/UjPCYG4UFqSPDCYdajslILx4WeG1hQIDQ2BlGhgVBIBLfdjJqqapgaLNIrqkKRmBiHELUMghSPKLUASQpGYsfuyLAZ684H+WGqKkCJzgkAkILj0aVDBCRRQkJ6IoIUAhQhcejSvSdU9VmVfpcVVRVlp4IvggwxqV0RKVlQUV0LUochKjwYCtEPl82MWq0WRpu72Y9FZKrg+vOigVwSQH4vnHYrzEY99CZ7o7HcZocusA8eeXwClPpCrFuyFHnOs28TGDsU06cNhGHPZ1i94yAMp51/EcqgMERHRSBIrQC8Tpj0OijPGGEXIA8IQ0xUOAI1SkiCHx6nHUZdDbSmM2RgChI0weGICA+t2w4+uF1OWE166A1mOJuk/4kyFYLDIxERGgiVQgaB/PC4HLCYDDAYzXB4zv/rDWV4EtLjQxt9UOC11OB4uRbOZuoPiO2ItCgVbNWFKKxpPLWBOroj0qMkGGvKUaG1nse0pyI0YdFIiI+C6kQMnvywGcpRUm5sPggtiFBqghEaGoJAjQoKuQySQHA7bfX3C1ub/L+MICkRFBqKkKBAqFVKyGVi3TPJYkStrhZmZ+PeiQo1YpPSECYYUVRWCzEoAjHhIVDKAa/DCl11NQxWVzPHSoI6LBKxkWHQKCX43XbotTWwiuHoEBsKmEuQU2o676llf8/zVREcjaT4aKidlThSZIA6LAYxkSHQKGUgrws2cy2qKvVwnRzOItShUYiJCkOAUga/2w6DtgYWMRRJsWEQLaXILuFpchljjDHGGGOMsTNJQ11CFxcul1SRwgTq8jc5jVukoHEvSRSuaae2RFD6Y3Iav0hB1z4no+TOF37fL70iUscpC+hYtY3yFgwh2QXsi6zffbQlR0d+9166P01sMhZi6a7lNeR1VNPKhdMpSX7qb4Kkoe6j7qQ3P19DWcXVZLJYyWyooqytP9LrcyZSj7hAEk9rTyBNVCcafft8+vjnbZRfYSCL1UL66lI6unsNffT8FEpWSSePUfrYR2hjgZOMP84mZZN6Yie+RYW1DtJvfpSSRRAgUd+7/kVHa5y057MF9N3OfNIZ9ZSz6VN66N4H6Z3vt1GZwUb6kv30/qPjKEkjnKpPkCg+41r6+xuf0uZDBVSjN5PFYqTynF309TuP0DV9EkktnWo/IKYbvbLBTLaD39Oc++6nFxavotxKI9mdNqop2ENfvHwnZcQGklD/e3lsL5q7aCuVVVRQRUU1GZ0+8vs8ZNFXU0VFRX0po50vXE5Sw3OTdBU98sZHtGrbISosrSa9yUIWk46KsrbS1wsfpmt7xZC8we/Dx79NRRUVVFFRSTW1JnJ6/eSxG6m6qvJkO/kbPqKbB8ac7Bsgp05X3U8/ZJVTRUUFVelt5Cei7OVzKVVsefwGxmfQTQ+/Tb/sOEZVBgtZLSaqOp5Jqz59jaYP70xKoeHvJRow9yuqMHuo7KPpNGrmC/T91qNUY7aRw1xDx377ih4Z342CWmpPDKDUPsNo8m3TaNrUqylF3lK/zrUIlDh4Bn1/2E6mta/RyMmz6T8/7aQirZkcdiOVZK6jd+dcQzGKU9eFptck+uDXQ1ReUUGVldVkcfvJ53aQQVd18hiXF+XQh/f0ODn+Q699kY4cL6WKikqq1hnI4fGTx26imgbnpWDrN3TX0IST50WmjqXnNzvJfHQtvfrU4/Tud1uoqNpAFquBjh9cR/+eP5V6xQc1ucYEUkd3pYlz3qAftx6miloTmcxmMugqKffAZvrmrTupY4C8xeOhiehBLine truncated
+    }
+   },
+   "cell_type": "markdown",
+   "id": "f7065cb8",
+   "metadata": {},
+   "source": [
+    "<div>\n",
+    "<img src=\"attachment:image838.png\"  align=\"left\" width=\"450\"/>\n",
+    "</div>"
    ]
   },
   {
@@ -217,7 +251,7 @@
    "source": [
     "### Creating workers in other machines\n",
     "\n",
-    "For large parallel computations, one typically needs to use different computers in parallel. Function `addprocs` also provides a low-level method to start workers in other machines. Next code example would create 3 workers in `server1` and 4 new workers in server `server2` (see figure below). Under the hood, Julia connects via ssh to the other machines and starts the new processes there. In order this to work, the local computer and the remote servers need to be properly configured (see the Julia manual for details). \n",
+    "For large parallel computations, one typically needs to use different computers in parallel. Function `addprocs` also provides a low-level method to start workers in other machines. Next code example would create 3 workers in `server1` and 4 new workers in `server2` (see figure below). Under the hood, Julia connects via ssh to the other machines and starts the new processes there. In order this to work, the local computer and the remote servers need to be properly configured (see the Julia manual for details). \n",
     "\n",
     "\n",
     "\n",
@@ -244,34 +278,6 @@
     "\n"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "6c09b2e5",
-   "metadata": {},
-   "source": [
-    "### Adding workers with ClusterManagers.jl\n",
-    "\n",
-    "Previous way of starting workers in other machines is very low-level. Happily, there is a Julia package called [ClusterManagers.jl](https://github.com/JuliaParallel/ClusterManagers.jl) that helps to create workers remotely in number of usual scenarios. For instance, when running the following code from the login node in a computer cluster, it will submit a job to the cluster queue allocating 128 threads. A worker will be generated for each one of these threads. If the compute node have 64 cores, 2 compute nodes will be used to create to contain the 128 workers (see below).\n",
-    "\n",
-    "\n",
-    "```julia\n",
-    "using Distributed\n",
-    "using ClusterManagers\n",
-    "addprocs(SlurmManager(128), partition=\"debug\", t=\"00:5:00\")\n",
-    "```\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "7c470ed9",
-   "metadata": {},
-   "source": [
-    "\n",
-    " <div>\n",
-    "<img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABkwAAAOBCAYAAACzvIyfAAAACXBIWXMAAB7CAAAewgFu0HU+AAAAGXRFWHRTb2Z0d2FyZQB3d3cuaW5rc2NhcGUub3Jnm+48GgAAIABJREFUeJzs3XmYXFWZ+PFvd7qTdEJ2IAkkkJCwhCVAwk4ia1SQqGgyqNOCmoA/Jwhij9sEARMVzICSKEqcxYksCoKELU4IhOyABBGRVRncAQVl30l+f7xdpFJdt+pWdVcvVd/P8/QDde8995zqVHVXn/ec962j1b333jv4rbfe+izwQWB3oBeSJEmSJEmSJEnV5S3gEeDaXr16fXP//fd/FqAOYOPGjfsAy4BRXTc+SZIkSZIkSZKkTvUn4PgDDjjg/rpf/epXQ15//fX7gNFdPSpJkiRJkiRJkqRO9sfevXvvW//6669/BoMlkiRJkiRJkiSpNo1+/fXXP1MPzOjqkUiSJEmSJEmSJHWhGfXAuK4ehSRJkiRJkiRJUhcaVw/06epRSJIkSZIkSZIkdaE+9V09AkmSJEmSJEmSpK5mwESSJEmSJEmSJNU8AyaSJEmSJEmSJKnmGTCRJEmSJEmSJEk1r6HE6+8CVlVgHJIkSZIkSZIkSR3pSODgtBeXFDCpq6tbPXny5C+WOiJJkiRJkiRJkqTOdM8993xj8+bNqQMmpuSSJEmSJEmSJEk1z4CJJEmSJEmSJEmqeQZMJEmSJEmSJElSzTNgIkmSJEmSJEmSap4BE0mSJEmSJEmSVPMMmEiSJEmSJEmSpJpnwESSJEmSJEmSJNU8AyaSJEmSJEmSJKnmGTCRJEmSJEmSJEk1z4CJJEmSJEmSJEmqeQZMJEmSJEmSJElSzTNgIkmSJEmSJEmSap4BE0mSJEmSJEmSVPMMmEiSJEmSJEmSpJpnwESSJEmSJEmSJNU8AyaSJEmSJEmSJKnmGTCRJEmSJEmSJEk1z4CJJEmSJEmSJEmqeQZMJEmSJEmSJElSzTNgIkmSJEmSJEmSap4BE0mSJEmSJEmSVPMMmEiSJEmSJEmSpJpnwESSJEmSJEmSJNU8AyaSJEmSJEmSJKnmNXT1ACRJkiRJ6kFGAk909SDUIw0H9kk49yvgr504FkmSlIcBE0mSJEmSCqsH3gOcAbwEvL9rh6Me6ijgRwnnZgDXduJYJElSHqbkkiRJkiQpv22A04AHgRuAY7t2OJIkSaokd5hIkiRJktTWhcBsYFBXD0SSJEmdw4CJJEmSJElttXT1ACRJktS5TMklSZIkSZIkSZJqngETSZIkSZIkSZJU8wyYSJIkSZIkSZKkmmfARJIkSZIkSZIk1TwDJpIkSZIkSZIkqeY1dPUAJEmSJKmTNABTgWmt/90B2A7oDzwHPAn8GrgDuA74XTv62gZ4J/AeYE9gJDAQeDarn+uB24BXU95zEjAu6/EfgTtb///dwOnAROAt4C7ga8D9WddPb71mT6AP8ARwI3AR8I8C/U4H+uY5vgx4qfX/64GTgBnAbsT39h/E9/A64MfAM0WfYZgI7J5w7mfAiwnnDgFGJ5y7FtiU5/g+wB4px5WxIzCzyDW/BH5T4n0nAscBR7X2sS3QG3i69esu4FZgJelfM13lCGD7PMfXAE+1/n8f4OPAicS/Q1/geeAB4KfAD4E32jGGfsAxwAnE93Y4MLi1jyeAh4j34Arg5Xb0AzAFOBk4gHivv0W89tcD3yHeqx2tL/F9Pq61322Jn2evE6+XPwGrgFuAezugv4HAB4DDgf2BYcCg1nOvED/bfk+87n8OrGt9LElSz7Jx48bNab/uueeeb3T1eCVJkiSpRPXAh4CHgc0lfK0iAh51JfTVGziLCA6k6eMp4FOkW8z23Zy2v2o9/tWEe78GvLf1mvkFxvB7kgMNEJPL+dod0Xp+J2Iyv9DzfJoIqKSxoMB9di3Q7scF2vVJaHN+kXGX+3VmyucKcBBwewn3/nPr/fMFsbqL28g/9k+0nt8beDThmszXr4GxZfTdQLynnipy/8zXM8R7tncZfY0Abipy/xeB97Ve/6EC130wZZ+9gNnE6yDta2YtETwqxzbAvxNBpVLfBw8QAStJkrrMPffc841SYiCm5JIkSZJUzbYhVpH/iORdC0mOICZD+6W8fjixO+WbwNCUbbYnAiHriNXhpdgb+CgwN+F8b2KV/geAswvcZyfgeyX2nel/GBFYOqjItcOIgEYpgYRacC6xS+jIEtrsAFwMrCYm7HuSvYFRxE6ZQsEvgL2I3TSDS7j/UOL1+F3y73BJavNN4G5iZ09aOxKBwvcUua4/scPpwyXcO0nm+f0H8TpIawrxPf8GpaVmHw3cA/wr0FRCu4w9id0okiT1GAZMJEmSJFWrocTK6s5Y4TyKSL8zqcz2B7e2L2UStA5YXOSaQUSwqJjjiOdQir2Jleel7AL4JnB8if1Uq+8C51HaDqZsBxET9qW8Zrra3sCFRHAxjTFEUCmN7YmUX4eXPiwg0nZtAHZJcW1v4GYi2JhGL+D7wPjyhgZEcGwtEfwo1+eBy1Je25dIV7ZbO/qTJKnHMWAiSZIkqRrVAT8A9uuEvnoBV7B1fZFy7EpMZpbyd1qaVd9pUg3VA+8ooV+A9wMfK7FNPbE7orHEdtXmk0TaqPbaiXjt9eqAe3WGg0ifeipjFsV3edUB/03sSmmPnYgAY7HX5xxg3xLvvQ2Fd3oVUk/8bNizzPbZPkKMv5jPUnxX3ltEraLX2jsoSZK6C4u+S5IkSapG/48t9TuS/J1YQf04MeE3gphwPYTS/lY6i8LBhr8CS4laIDsS9QyS0m8dTUxmfruE/le0ttmGSMG1d55r3iJqiKwHTgXm5blmX+DKEvrNTgf1KlFA/g9E4Oc4kieddwVOAf6zhL4q5c9EyqF8JiccfxZ4rMh9nypwbjdip02SB4jJ/18Tr8tRRNqnGeT/nh5JBF++U2RM3cGgrP9f3frVn0gbl7RTaQAwjUitl+RUCqfG+jtRSP7PxOv2vURx9nwOInZifC3hfBPw5QJ9ATxJpNmrI1L7bdt6PKmWTjFnAccmnNtE/Hy5jigu34cIrPwzUQw+nwuBGyhcjP6UhOMvAhcAVxMF3jMGEj9DpgDNdExwR5KkTmfARJIkSVK1aQS+VOD868C/EUGJ1/OcH0TsAPgcWyY6k/RpvS7Jj4gCzS9nHfssMSE+I6HN54BLgTeK9J0xly0TlxeTPxBxC1FHAaIA/CdpW69hTMr+cj1ETFY/nnVsX+BnJE9KfyRhnJ3tOyQHGjYnHF9N7K4p1xdJ3jHxPeDTRIAr2xVEkOUWYEiedmcSKb42tWNcnekLwIKsx18mamwclnD9JJIDJr2I72mS64laPy9kHTuLeI+dnNDmLOBbbP2+zTie/P8GGd9tbZ/52dJE/LsmBSCKaSL5+b1GBJuW5Ry/hfhZ8CXg63na9SWCbP+WcN/tSE7FdQaxey/X80TKsLXA+USg6KsJ95AkqdsyJZckSZKkavNPRLHifDYRE4wXkT9YAvAcMZk7gVi5XciHSS4uvYGYkM2ddH2BWIGdtLNhNMnBlFybgF9mPb474bq7ch7n67vUovMQE7YnsnWwBOA+4jkmmUoUgq81I4hgUT7rgdNpGyzJ2EhMxOczntiF0ROsZOtgCcArRD2XJHsUOPdeknen3Ad8iK2DJZn+ZhE1T/IZRnIwZWaBsdxK/Btm/2zJ9LW2QLtCTiY5cPtF2gZLsp1f4PypJC+iHZFwHODeAueyrSbe55emvF6SpG7BgIkkSZKkalNo9f93iGLNaTxNBAPyrTLPOLHAuc8Bbyace43Cu2CmFx7a215i650of0+47h85j5/Nc02x3TT5/AB4JOHcSuD2hHMNROqjWnMCyWmZvk7xHSJX0nbyP+PIMsfU2S5JOL6+QJvBBc4Veg9+iUgXl8+bROqtJEnvwUKv2y+Sf2fSW8SumnIk1Xx5hti5Usz3E45vC+yTcK7Q6/AzlDaX9LcSrpUkqcsZMJEkSZJUTeqIVc35vEXk7i9VUmqmOuDQhHOPETtMCrkV+EvCuaTnkOulnMcvJlyXezzfdX1T9pntR0XOX13gXC3WOEhKOfUa8Xoo5g223lGULem12N0k7ep4meTX74AC9zs84fhTwPIiY7mL5IDfYbSdMxlAcuq6h0neNQZwB8Vr3+TqRdRUymcF6YqtJ+06o8C9/0xy0OQUIrh1IuXXZJEkqduyhokkSZKkarIzMDzh3N0ULnJcqrEkp7FalaL95tbr8qVoGgXsQHJAJSN3B0vSjpY01/Uu0le+e95Z5JpCuwZ2L7G/anBwwvHngY+lvEdS8KBQ2qru4gVi51aSZ4Bt8hxPWuy5LbBLwrnVpKvpsor8r8XBrccfyjq2KxEozSdNyq3VwLgU12VMIPnfexBwWop7FJr3SXrNPAv8guSi8YcAPyUCXLe1fq0B7qfn1NGRJCkvAyaSJEmSqklSsAQKr/7u6L4eKnAu28MFzm1H8YBJRyo1YPIHkuvAZPy2wLlCaZaqVdJrZjtgcTvvXagQeXdRKL0dbJ1eLo1C78FC761shd6r2+WcH1Tg2kdT9JXmmmyFaokc1/rVHkMLnFsIXFak/TbA+1q/IIJhy4D/ITkdnyRJ3ZopuSRJkiRVk0KTxk91Yl9JtURyPVPgXKHJzEoo9e/D3Loo+bzS+pVPoTRL1aiOygaJepN/d0Z30tG7Dzr7PVjoNZumv0J95VPpoGKh798VwLUl3m9bokj9SiIF2b5ljkuSpC5jwESSJElSNSk0YZxb76OSfSUVms6VFEyA7h9QSFM/AZK/F/06aiA9RH+iJkUl1VpNiUq/B3Pv31TmfUq5JtvAEq8vVaHXy2bgw0Rh+aQ6ToUcQtRxelcZbSVJ6jIGTCRJkiRVkxcKnOvoAMTzBc4VmljNVihoUOj+3UF7n2Ox9EzV5mXgra4eRJXp7Pdgodds3xR9pbkm2Line truncated
-    "</div>"
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "8c8bc619",
@@ -413,7 +419,7 @@
    "source": [
     "### Useful macro: `@spawnat`\n",
     "\n",
-    "You have provably realized that in order to use `remotecall` we have written auxiliary anonymous functions. They are needed to wrap the code we want to execute remotely. Writing these functions can be tedious. Happily, the macro `@spawnat` generates an auxiliary function from the given block of code and calls `remotecall` for us. For instance, the two following cells are equivalent."
+    "You have probably realized that in order to use `remotecall` we have written auxiliary anonymous functions. They are needed to wrap the code we want to execute remotely. Writing these functions can be tedious. Happily, the macro `@spawnat` generates an auxiliary function from the given block of code and calls `remotecall` for us. For instance, the two following cells are equivalent."
    ]
   },
   {
@@ -444,7 +450,7 @@
    "source": [
     "###  `@async` vs `@spawnat`\n",
     "\n",
-    "The relation between `@async` and `@pawnat` is obvious. From the user perspective they work almost in the same way. However,  `@async` generates a task that runs asynchronously in the current process, whereas `@spawnat` executes a task in a remote process in parallel. In both cases, the result is obtained using `fetch`. "
+    "The relation between `@async` and `@spawnat` is obvious. From the user perspective they work almost in the same way. However,  `@async` generates a task that runs asynchronously in the current process, whereas `@spawnat` executes a task in a remote process in parallel. In both cases, the result is obtained using `fetch`. "
    ]
   },
   {
@@ -480,7 +486,7 @@
    "id": "10899cd4",
    "metadata": {},
    "source": [
-    "### Another usefull macro: `@fetchfrom`\n",
+    "### Another useful macro: `@fetchfrom`\n",
     "\n",
     "Macro `@fetchfrom` is the blocking version of `@spawnat`. It blocks and returns the corresponding result instead of a `Future` object. "
    ]
@@ -516,9 +522,9 @@
    "id": "6d6e04d9",
    "metadata": {},
    "source": [
-    "### Data movement in `remotecall` / `fetch`\n",
+    "### Explicit data movement in `remotecall` / `fetch`\n",
     "\n",
-    "When usig `remotecall` we send to the remote process a function and its arguments. In this example, we send function name `+` and matrices `a` and `b` to proc 4. When fetching the result we receive a copy of the matrix from proc 4."
+    "When using `remotecall` we send to the remote process a function and its arguments. In this example, we send function name `+` and matrices `a` and `b` to proc 4. When fetching the result we receive a copy of the matrix from proc 4."
    ]
   },
   {
@@ -531,8 +537,8 @@
     "proc = 4\n",
     "a = rand(10,10)\n",
     "b = rand(10,10)\n",
-    "ftr = remotecall(+,proc,a,b)\n",
-    "fetch(ftr);"
+    "ftr = remotecall(+,proc,a,b) # Send\n",
+    "fetch(ftr); # Receive"
    ]
   },
   {
@@ -556,8 +562,8 @@
     "a = rand(10,10)\n",
     "b = rand(10,10)\n",
     "fun = () -> a+b\n",
-    "ftr = remotecall(fun,proc)\n",
-    "fetch(ftr);"
+    "ftr = remotecall(fun,proc) # Send\n",
+    "fetch(ftr); # Receive"
    ]
   },
   {
@@ -567,7 +573,7 @@
    "source": [
     "### Data movement with remote channels\n",
     "\n",
-    "Another way of moving data between processes is to use remote channels. Their usage is very similar to conventional channels for moving data between tasks, but there are some important differences. In the next cell, we create a remote channel. Process 4 puts several values and closes the channel. Like for conventional channels, calls to `put!` are blocking, but next cell is not blocking the master process since the call to `put!` runs asynchronously on process 4."
+    "Another way of moving data between processes is to use remote channels. Their usage is very similar to conventional channels for moving data between tasks, but there are some important differences. In the next cell, we create a remote channel. Process 4 puts several values and closes the channel. Like for conventional channels, calls to `put!` might block, but next cell is not blocking the master process since the call to `put!` runs asynchronously on process 4."
    ]
   },
   {
@@ -621,8 +627,6 @@
    "source": [
     "### This will not work!\n",
     "\n",
-    "You really need remote channels to communicate different processes. Standard Channels would not work. For instance, the following code would block at the `take!`. Worker 4 will receive a different copy of the channel and will put values in it. The channel defined in the main process will remain empty and this will make the take! to block. \n",
-    "\n",
     "```julia\n",
     "chnl = Channel{Int}()\n",
     "@spawnat 4 begin\n",
@@ -632,7 +636,10 @@
     "    close(chnl)\n",
     "end\n",
     "take!(chnl)\n",
-    "```"
+    "```\n",
+    "\n",
+    "You really need remote channels to communicate different processes. Standard Channels would not work. For instance, the following code would block at the `take!`. Worker 4 will receive a different copy of the channel and will put values in it. The channel defined in the main process will remain empty and this will make the take! to block. \n",
+    "\n"
    ]
   },
   {
@@ -699,9 +706,14 @@
    "id": "b8b411f1",
    "metadata": {},
    "source": [
-    "### Remote channels are not iterable\n",
+    "### Remote channels are also iterable\n",
     "\n",
-    "One main difference with respect to conventional channels is that remote channels cannot be iterated. Let's repeat the example above."
+    "As with conventional channels, remote channels can be iterated. Let's repeat the example above.\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "<b>Note:</b> In old Julia versions, it was not possible to iterate over remote channels. If the cells below do not work, update Julia to a recent version.\n",
+    "</div>\n",
+    "\n"
    ]
   },
   {
@@ -735,7 +747,7 @@
    "id": "4bb35283",
    "metadata": {},
    "source": [
-    "Now, try to iterate over the channel in a for loop. It will result in an error since channels are not iterable."
+    "Now, try to iterate over the channel in a for loop."
    ]
   },
   {
@@ -755,7 +767,7 @@
    "id": "8eaea2fa",
    "metadata": {},
    "source": [
-    "If we want to take values form a remote channel and stop automatically when the channel is closed, we can combine a while loop and a try-catch statement. This works since `take!` raises an error if the channel is closed, which will execute the `catch` block and breaks the loop."
+    "As an alternative method (or if you are using an older version of Julia): if we want to take values form a remote channel and stop automatically when the channel is closed, we can combine a while loop and a try-catch statement. This works since `take!` raises an error if the channel is closed, which will execute the `catch` block and breaks the loop."
    ]
   },
   {
@@ -818,6 +830,16 @@
     "q_1_check(answer)"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9c4d4900",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "why_q1()"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "dbe373d1",
@@ -852,6 +874,16 @@
     "q_2_check(answer)"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e7c25fc4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "why_q2()"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "c561a73d",
@@ -877,6 +909,16 @@
     "x"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7b25a83f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "why_q3()"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "835080aa",
@@ -903,6 +945,16 @@
     "x"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "96b84cb5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "why_q4()"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "9e985c61",
@@ -1094,7 +1146,7 @@
    "id": "4d8bfa90",
    "metadata": {},
    "source": [
-    "We have activated the current folder. Now let's see which is the active project in another process, say process 2. You will see that process 2 is provably still using the global package environment."
+    "We have activated the current folder. Now let's see which is the active project in another process, say process 2. You will see that process 2 is probably still using the global package environment."
    ]
   },
   {
@@ -1317,6 +1369,41 @@
     "We have seen the basics of distributed computing in Julia. The programming model is essentially an extension of tasks and channels to parallel computations on multiple machines. The low-level functions are `remotecall` and `RemoteChannel`, but there are other functions and macros like `pmap` and `@distributed` that simplify the implementation of parallel algorithms."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "a75aa3bb",
+   "metadata": {},
+   "source": [
+    "## Exercises"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3153bd90",
+   "metadata": {},
+   "source": [
+    "### Exercise 1\n",
+    "\n",
+    "Implement this \"simple\" algorithm (the telephone game):\n",
+    "\n",
+    "Worker 1 generates a message (an integer). Worker 1 sends the message to worker 2. Worker 2 receives the message, increments the message by 1, and sends the result to worker 3. Worker 3 receives the message, increments the message by 1, and sends the result to worker 4. Etc. The last worker sends back the message to worker 1 closing the ring. See the next figure."
+   ]
+  },
+  {
+   "attachments": {
+    "g5148-2.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAAA/MAAAMYCAYAAACHf7fiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13uFxVvf/x9z7pIQFSCSWQ0EMILfSqgAUVLJRrA7wWREVQvNdy9V64Yi9XsWP9iWJBRQVEERCUSA+9SwuEQAiEVNLP5/fH2ieZzJmy95yZWVM+r+fZz8k5mb33d805s9dee631XQlmZmaWgzYBRqXb5sDogu83BTYDetJ/DwKGASPTncekX0emPx+Uvo70OIOBocAmGYPpO045LwGrCr5fDPSm/14HLCn4v5XAioLvlwBr0n1Wpcdalv7sxfTrsoJzLEp/tjQ9zkpIXsxYDjMzM8spiR2AmZlZc2kUMC7dxqdfxxb8bHM2bpiPKtjGlDigVbaoxLa4zM+L/s8PA8zMzMpxY97MzNqcJgJbAFunX/sa6H1fC7exVO7JttayDlgAPA88B8xPv19Q8P3z6ffPQrKkzHHMzMw6jhvzZmbWgtRDaJhPArZKv/Y11vu+bgNMJAxLN4Mw3L+vsf8soaH/NDAPeDL9OheYD0lvuYOYmZm1AzfmzcwsAo0Btku3Kem2HbAtofE+kTCf3KwR1hAa+08RGvtPp//ua+zPDf9OVkeL0MzMrAo35s3MrAE0kQ2N9VKN9k3L7WnWQvoa/I+n22MF/54DyZqIsZmZWZdzY97MzGqkrYGdgZ0Kvu4ITAVGRAys0XoJSdqWELK5LyNkcF9U8P3y9LV9CdyWA6uBtelrYUNm+b4s8kqP0WcJYc54HishSTPSa3M21PM9hGR+fYoz5hdnxe/LrD8KGEJI/Ne3zybpvzdPv44q2H+z9GejSxyz06wj9OA/TsnGfvJMxNjMzKwLuDFvZmYVaAyhod63FTbcR0UMbCDWAS8UbAtLfF/YMF+Wfr80/DtZUeKYVpJGEBr9m6VfS21jKvx/Oz8MWElo3D8KPFSwPQjJgpiBmZlZZ3Bj3sys6ykh9KbPAHYjNNR3ITTax0cMLKuVhOHQ89Kvz1C6gf4CsACSxZHitNw0ipDscCIwgfD3OCn9d982Kf35BMIognawEHgYeIDQwO/796Meum9mZlm5MW9m1lW0BaHRvnu69TXgW7GXfTkhMdn89Gthg72g4Z4sjBahtRiNYcPyhBMIKx9sSVj5YBtCcsVtCVMAWtFawjD9B9Mt7cnnXj+EMjOzYm7Mm5l1JG0KTGdDw306sAet09O+ltBAnwM8kW5z0u1pYC4ky2IFZ51OmxMa+pMJDfzJ6fd9jf2tgHHRwittDnBPut2dfn3YPflmZt3LjXkzs7anrYGZ6bYPofE+JWZEhGRvT7KhgV7caH8akrWxgjOrTiMIjfyp6bZ9wb+nAmPjxbbeauB++jXyk3lRozIzs6ZwY97MrK1s1HDv2yZFCmYVIbnXw8C/Cr4+Qhj+3hspLrMm0OZs3LgvbuzHTN63ELgLuJfQwJ8d/u1efDOzTuLGvJlZy9JWbGiw70uchnsvoSe9r7Het/0r/DzJu3SaWRdQQhiqvz0bkknumm5TCcv+NdtKQsP+NkLj/jbgfo+QMTNrX27Mm5m1BI0GDgAOYUPDfcsmBrCCMFz33vRrX+P9EUhWNTEOsw6nocAOhIb9zoTG/rT0380eur8CuJONG/gP+iGdmVl7cGPezCwKTSE03A9Kv84ABjXhxGsJw+DvITTc7yP01j3mG3iz2DSB0MjfhQ2N/Ok0NwfGcuAONjTwbyUk2lMTYzAzswzcmDczazgNJtyYHwIcChxGc27OnyE01u8v+Ho7JC814dxmVjfaFNiJ0LCfSVhOck/C8nvNsAS4BfgnoYH/Dy+VZ2YWnxvzZmZ1p3GEHveD2TBsvpHrWq8jrEXdN0z2drwutVkX0BTC6hUzCI373QkPDhs9J38toff+RuCGsCVPNficZmZWxI15M7MB02bA4cCR6TaDxl1f1wEPERrufY33OyFZ3qDzmVlb0VBCz/0MYA/Cw8R9gE0bfOK5hJ77vgb+HU6uZ2bWWG7Mm5nlppGEHve+xvtMGjPfvZcNDfe++at3QrKsAecys46lhJBgr3BljL2B0Q086UuE+fbXA9cBN3qKj5lZfbkxb2ZWlYYSMs33Nd4PBIY24ETzCb1aswjzU++EZGkDzmNmXU89hCH5fY37fYG9gE0adMLVwM3Atel2EyQrG3QuM7Ou4Ma8mVk/GkQYltrXeD+E+t/g9hIS0v2TDXNOH6nzOczMctAgQgb9vgb+foRr4ZAGnGwlcBMbGvc3Q7K6AecxM+tYbsybmQFp0rojgaOB44BJdT7BcsJ6zrNY34BPXqjzOczM6kxDCHPvDyU82DwCmNiAE60gJO+cBVwNXA/Jqgacx8ysY7gxb2ZdSj2E3qdjgNek/+6p4wmeIswVvZHQeL/b67ibWWfQzoQVOw4hrNoxjfpePyE8AJ0FXAX8BZL76nx8M7O258a8mXURjQWOIvS+vw7Yqo4HX0BI8vRPYBYks+t4bDOzFqbRhKXxDmFDD/6YOp9kPvAP4HLgMkherPPxzczajhvzZtbBlBDme/b1vu9P/bLOvwj8nTDX8xrgfkhUp2ObmbUxDSIMzT8ceHn6tZ6N+7WEZHp/Aa4EZkPSW8fjm5m1BTfmzazDaCih9/1NwLHAFnU68DLCkM+/pdudHjZvZpaFegiZ8l8OvIzQuK/nuvfPs344PldCMr+OxzYza1luzJtZB9BIQu/7m4DXApvV4aBrCFnmryE03m+BZE0djmtm1uU0iJAt/2WEBv6hwKh6HZyQbPRK4E+E9e394NXMzMysdWhz0ImgC0FLQarDNh90MeiUcHwzM2s8DQLNBH0MdBlocZ2u6QI9n9YTJ6Zz+83MzMys+bQF6DTQX0Cr63CTtwb0D9AnQHulc+zNzCwqDQUdAfocaDaot04N+5dAfwS9C9SI5fXMzMzMbANtmzbgL0sb3wO9mXuuoPe93tmWzcys7jQh7Vm/APR0nRr260C3gc4NowLMzMzMrA40Nm1sX1WHHpm+G7YvgA4lJGIyM7O2pemEIflXgVbVqXH/KOh80NGgwbFLaGZmZtZGNAr0NtCf6tADvwp0BejdoUfHzMw6kzYDHQ/6PmhOnRr2z4F+AHqFG/ZmZmZmJWkI6LWgi0DLBnjztRz0W9Bbw82dmZl1H+0G+gjoOtDaOjTsFxCG9x9FyMJvZmZm1s00Mx3O+OwAb7JeZMP893ota2RmZh1BY9mw6kk9MuS/kB7rWPfYm5mZWRfRboREQ/8a4M3U8wU3U0Njl8rMzNqBhhPmw58PmluHhn1hXeSGvZmZmXUabQ76AOj2Ad40zQV9DXQYTmBnZmYDoh7QAYSl7+6tQ8N+HugbOMmqmZmZtTclhPWBLySs6TuQIfQ/BL3cN0dmZtY42gF0NvWZZz8X9BXQjNilMjMzM8tIk0Bnge4ZwE3QSsJ68qeARsYukZmZdZv1S6NeBlo9wIb9fYQl9CbFLpWZmZlZEfUQ5iBePICbnnWgWemDgPGxS2RmZhZoXEHDfiBLpq4FXZUea5PYpTIzM7Oupm3S3obHB9hjcS5oSuzSmJmZVVa3hv0iwjS0o0FJ7FKZmZlZV9BgwhI/f01702u5iXkM9L+gXWKXxszMrDbaGnRmOqqsdwAN+0dA54C2j10iMzMz60iaAPov0FM13qysBP0y7YVwIjszM+sg2gb0YdCNA2jY94KuB70bNCp2iczMzKztaW/QBaDlNd6cPEAYij8hdknMzMwaT5PTeu9fA+itX5LWvfvELo2ZmZm1FfWAjiUk6qnlJuQlQjK8o2OXxMzMLB7NBJ0PWjCAhv1toNNAo2OXxszMzFqWJqS9CXMGeMPh4YFmZmbraVj6kHwgq7709dbPjF0aMzMzaxkDGkr/YrrvnrFLYWZm1vo0Nn3wPcu99WZmZlYD9YCOJyTbqeVG4ibQW0Nvg5mZmeWnGaAvg552b72ZmZlVoaGgd4EerOGmYRXo56D9Y5fCzMysc2gQ6BjQ76l9/frbQKf6IbuZmVnH0SjQWaAna7hBeBb0BdA2sUthZmbW2TSJkL/m0Rob9fNdZ5uZmXUEjQedC3qhxqf8p4GGxy6FmZlZd1EP6GhqT5q3Kt334NglMTMzs1y0HWE5nGU1Vv6HxC6BmZmZQR16628DnQIaErskZmZmVpZ2TBvxKz2U3szMrJMMuLd+Xjpab3zskpiZmdl6OhD0R1Bvzor9QUJCvKGxS2BmZmZZaUvQf4Eer6FRv5yQBX967FKYmZl1Me0L+nMNFfmthKXpemKXwMzMzGqlQaA3gq6t4V6gF/RX0FGxS2FmZtZFND0dZpe3J34W6FhQErsEZmZmVk/aM+1xX15Dw/4Owrz6QbFLYWZm1qG0K+hC0NocFfQ60GVhKL6ZmZl1Nk0gJMyrZTnaRwhL2XolGzMzs/rQdunT9jU5KuRVacN/WuzozczMrNk0FHQi6J81NOqfJSTL2yx2KczMzNqUJpM/O/3SdJ/JsaM3MzOzVqCZ6QP+vFnwF6f3FJNil8DMzKxNaEvQN3M24peAPg0aGzt6MzMza0WaDPo86IWcjfploK+5o8DMzKwsjQV9BfRSjgp2OehLeN1YMzMzy0SjQGeD5uZs1K8C/QS0S+wSmJmZtQgNAZ0Gei5nhXoBaKvY0ZuZmVk70lBCFvv7czbq1xFW1dk5dgnMzMwi0tGge3NUoKvTeW9TY0duZmZmnUA9hKVrb6qxUb9j7BKYmZk1kfYBXesK08zMzFqHDiUsaZunUd/X0bB97OjNzMwaSFulw+OzrhXf14j3/DQzMzNrEu2VNtCz3q8UTgHcOnb0ZmZmdaRNQOcQMsJmqRB7Qb8DTY8duZmZmXUrTSMkvcuzrN1LoP8DbRE7ejMzswFQD+hU8mWMLine truncated
+    }
+   },
+   "cell_type": "markdown",
+   "id": "6a485db2",
+   "metadata": {},
+   "source": [
+    "<div>\n",
+    "<img src=\"attachment:g5148-2.png\"  align=\"left\" width=\"320\"/>\n",
+    "</div>"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "9a49ad48",
@@ -1330,15 +1417,15 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Julia 1.9.1",
+   "display_name": "Julia 1.11.6",
    "language": "julia",
-   "name": "julia-1.9"
+   "name": "julia-1.11"
   },
   "language_info": {
    "file_extension": ".jl",
    "mimetype": "application/julia",
    "name": "julia",
-   "version": "1.9.1"
+   "version": "1.11.6"
   }
  },
  "nbformat": 4,

notebooks/matrix_matrix.ipynb

@@ -29,7 +29,8 @@
     "\n",
     "- Parallelize a simple algorithm\n",
     "- Study the performance of different parallelization strategies\n",
-    "- Implement them using Julia"
+    "- Learn the importance of \"grain size\" in a parallel algorithm\n",
+    "- Implement and measure the performance of parallel algorithms"
    ]
   },
   {
@@ -79,7 +80,8 @@
     "alg_2_complex_check(answer) = answer_checker(answer, \"b\")\n",
     "alg_2_deps_check(answer) = answer_checker(answer,\"d\")\n",
     "alg_3_deps_check(answer) = answer_checker(answer, \"c\")\n",
-    "alg_3_complex_check(answer) = answer_checker(answer, \"b\")"
+    "alg_3_complex_check(answer) = answer_checker(answer, \"b\")\n",
+    "println(\"🥳 Well done! \")"
    ]
   },
   {
@@ -89,6 +91,8 @@
    "source": [
     "## Problem Statement\n",
     "\n",
+    "We consider matrix-matrix multiplication as our first algorithm to parallelize. The problem we want so solve is defined as follows.\n",
+    "\n",
     "Given $A$ and $B$ two $N$-by-$N$ matrices, compute the matrix-matrix product $C=AB$. Compute it in parallel and efficiently."
    ]
   },
@@ -113,7 +117,7 @@
     "\n",
     "- compute the product in parallel using more than one process (distributed implementation)\n",
     "- study the performance of different parallelization alternatives\n",
-    "- implement the algorithms using Julia\n"
+    "- implement the algorithms using Julia's task-based programming model\n"
    ]
   },
   {
@@ -124,7 +128,7 @@
     "### Assumptions\n",
     "\n",
     "- All matrices `A`,`B`, and `C` are initially stored in the master process\n",
-    "- The result will be overwritten in `C`"
+    "- The result will be overwritten in `C` (in the master process)"
    ]
   },
   {
@@ -148,7 +152,8 @@
     "\n",
     "- Identify the parts of the sequential algorithm that can be parallelized\n",
     "- Consider different parallelization strategies\n",
-    "- Discuss the (theoretical) performance of these implementations\n"
+    "- Discuss the (theoretical) performance of these implementations\n",
+    "- Measure the actual performance of these implementations\n"
    ]
   },
   {
@@ -222,7 +227,7 @@
     "<b>Note:</b> The matrix-matrix multiplication naively implemented with 3 nested loops as above is known to be very inefficient (memory bound). Libraries such as BLAS provide much more efficient implementations, which are the ones used in practice (e.g., by the `*` operator in Julia). We consider our hand-written implementation as a simple way of expressing the algorithm we are interested in.\n",
     "</div>\n",
     "\n",
-    "Run the following cell to compare the performance of our hand-written function with respect to the built in function `mul!`.\n"
+    "Just to satisfy your curiosity, run the following cell to compare the performance of our hand-written function with respect to the built in function `mul!`.\n"
    ]
   },
   {
@@ -250,6 +255,19 @@
     "<b>Question:</b>  Which is the complexity (number of operations) of the serial algorithm? Assume that all matrices are $N$-by-$N$ matrices.    \n",
     "</div>\n",
     "\n",
+    "```julia\n",
+    "for j in 1:N\n",
+    "    for i in 1:N\n",
+    "        Cij = z\n",
+    "        for k in 1:N\n",
+    "            @inbounds Cij += A[i,k]*B[k,j]\n",
+    "        end\n",
+    "        C[i,j] = Cij\n",
+    "    end\n",
+    "end\n",
+    "```\n",
+    "\n",
+    "\n",
     "    a) O(1)\n",
     "    b) O(N)\n",
     "    c) O(N²)\n",
@@ -272,9 +290,14 @@
    "id": "0eedd28a",
    "metadata": {},
    "source": [
-    "### Where can we exploit parallelism?\n",
+    "## Where can we exploit parallelism?\n",
     "\n",
-    "Look at the three nested loops in the sequential implementation:\n",
+    "\n",
+    "The matrix-matrix multiplication is an example of [embarrassingly parallel algorithm](https://en.wikipedia.org/wiki/Embarrassingly_parallel). An embarrassingly parallel (also known as trivially parallel) algorithm is an algorithm that can be split in parallel tasks with no (or very few) dependencies between them. Such algorithms are typically easy to parallelize.\n",
+    "\n",
+    "Which parts of an algorithm are completely independent and thus trivially parallel? To answer this question, it is useful to inspect the for loops, which are potential sources of parallelism. If the iterations are independent of each other, then they are trivial to parallelize. An easy check to find out if the iterations are dependent or not is to change their order (for instance changing `for j in 1:n` by `for j in n:-1:1`, i.e. doing the loop in reverse). If the result changes, then the iterations are not independent.\n",
+    "\n",
+    "Look at the three nested loops in the sequential implementation of the matrix-matrix product:\n",
     "\n",
     "```julia\n",
     "for j in 1:n\n",
@@ -287,8 +310,12 @@
     "    end\n",
     "end\n",
     "```\n",
-    "- Loops over `i` and `j` are trivially parallelizable.\n",
-    "- The loop over `k` can be parallelized but it requires a reduction."
+    "\n",
+    "Note that:\n",
+    "\n",
+    "- Loops over `i` and `j` are trivially parallel.\n",
+    "- The loop over `k` is not trivially parallel. The accumulation into the reduction variable `Cij` introduces extra dependencies. In addition, remember that the addition of floating point numbers is not strictly associative due to rounding errors. Thus, the result of this loop may change with the loop order when using floating point numbers. In any case, this loop can also be parallelized, but it requires a parallel *fold* or a parallel *reduction*.\n",
+    "\n"
    ]
   },
   {
@@ -298,7 +325,7 @@
    "source": [
     "### Parallel algorithms\n",
     "\n",
-    "All the entries of matrix C can be potentially computed in parallel, but *is it the most efficient solution to solve all these entries in parallel in a distributed system?* To find this we will consider different parallelization strategies:\n",
+    "The loops over `i` and `j` are trivially parallel, implying that all the entries of matrix C can be potentially computed in parallel. However, *which is the most efficient solution to solve all these entries in parallel in a distributed system?* To find this we will consider different parallelization strategies:\n",
     "\n",
     "- Algorithm 1: each worker computes a single entry of C\n",
     "- Algorithm 2: each worker computes a single row of C\n",
@@ -330,7 +357,7 @@
    "source": [
     "### Data dependencies\n",
     "\n",
-    "Moving data through the network is expensive and reducing data movement is one of the key points in a distributed algorithm. To this end, we determine which is the minimum data needed by a worker to perform its computations.\n",
+    "Moving data through the network is expensive and reducing data movement is one of the key points to design efficient distributed algorithms. To this end, we need to determine which is the minimum data needed by a worker to perform its computations. These are called the *data dependencies*. This will give us later information about the performance of the parallel algorithm.\n",
     "\n",
     "In algorithm 1, each worker computes only an entry of the result matrix C."
    ]
@@ -380,8 +407,8 @@
     "\n",
     "Taking into account the data dependencies, the parallel algorithm 1 can be efficiently implemented following these steps from the worker perspective:\n",
     "\n",
-    "1. The worker receives the corresponding row A[i,:] and column B[:,j] from the master process\n",
-    "2. The worker computes the dot product of A[i,:] and B[:,j]\n",
+    "1. The worker receives the data dependencies, i.e., the corresponding row A[i,:] and column B[:,j] from the master process\n",
+    "2. The worker computes the dot product of A[i,:] and B[:,j] locally\n",
     "3. The worker sends back the result of C[i,j] to the master process"
    ]
   },
@@ -401,7 +428,7 @@
    "id": "9d22ccea",
    "metadata": {},
    "source": [
-    "A possible implementation of this algorithm in Julia is as follows:"
+    "A possible implementation of this algorithm in Julia is as follows. Try to understand why `@sync` and `@async` are needed here."
    ]
   },
   {
@@ -461,7 +488,8 @@
     "A = rand(N,N)\n",
     "B = rand(N,N)\n",
     "C = similar(A)\n",
-    "@test matmul_dist_1!(C,A,B) ≈ A*B"
+    "matmul_dist_1!(C,A,B)\n",
+    "@test  C ≈ A*B"
    ]
   },
   {
@@ -471,19 +499,19 @@
    "source": [
     "### Performance\n",
     "\n",
-    "Let us study the performance of this algorithm. To this end, we will analyze if algorithm 1 is able to achieve the optimal parallel *speedup*. The parallel speedup on $P$ processes is defined as  \n",
+    "We have a first parallel algorithm, but how efficient is this algorithm? Let us study its performance. To this end we need to consider a performance baseline as reference. In this case, we will use the so-called optimal parallel *speedup*. The parallel speedup on $P$ processes is defined as  \n",
     "\n",
     "$$\n",
     "S_P = \\frac{T_1}{T_P},\n",
     "$$\n",
     "\n",
-    "where $T_1$ denotes the runtime of the sequential algorithm on one node and $T_P$ denotes the runtime of the parallel algorithm on $P$ processes. If we run an optimal parallel algorithm with $P$ processes we expect it to run $p$ times faster than the sequential implementation. I.e., the *optimal* speedup of a parallel algorithm on $p$ processes is equal to $P$:\n",
+    "where $T_1$ denotes the runtime of the sequential algorithm and $T_P$ denotes the runtime of the parallel algorithm on $P$ processes. If we run an optimal parallel algorithm with $P$ processes we expect it to run $p$ times faster than the sequential implementation. I.e., the *optimal* speedup of a parallel algorithm on $p$ processes is equal to $P$:\n",
     "\n",
     "$$\n",
     "S^{*}_p = P.\n",
     "$$\n",
     "\n",
-    "The ratio of the actual speedup over the optimal one is called the parallel efficiency\n",
+    "The ratio of the actual speedup over the optimal one is called the parallel efficiency (the closer to one the better).\n",
     "\n",
     "$$\n",
     "E_p = \\frac{S_p}{S^{*}_p} = \\frac{T_1/T_P}{P}.\n",
@@ -523,7 +551,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dac6a50b",
+   "id": "501d4008",
    "metadata": {},
    "source": [
     "### Communication overhead\n",
@@ -538,8 +566,25 @@
     "\n",
     "<div class=\"alert alert-block alert-success\">\n",
     "<b>Question:</b>  How many scalars are communicated from and to a worker?  Assume that matrices A, B, and C are N by N matrices.\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "45f44171",
+   "metadata": {},
+   "source": [
+    " <div>\n",
+    "<img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABkwAAAOBCAYAAACzvIyfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15fFTV+cfxz50shFUIguxLXFr3CkittW6tVm21FhVxX1t3qlg1QEgmIQLuOy51a1Er2Fq1Vau2WutPVASsWK0Lsu9o2Lcsc39/nATmnnsnmZlkZgLzfb9eeb2Sc+eceQhZJvc553kc6n300Ued6+rqRgGnAd8BchAREREREREREREREdm11AFfAH/Oycm585BDDlkL4ADMnDnzQOAVoE/m4hMREREREREREREREUmrJcBJQ4YM+cSZM2dOl+rq6o+BvpmOSkREREREREREREREJM0W5+fnHxyqrq6+FiVLREREREREREREREQkO/Wtrq6+NgScnulIREREREREREREREREMuj0ELBnpqMQERERERERERERERHJoD1DQJtMRyEiIiIiIiIiIiIiIpJBbUKZjkBERERERERERERERCTTlDAREREREREREREREZGsp4SJiIiIiIiIiIiIiIhkPSVMREREREREREREREQk6+Um+PgPgH+lIA4REREREREREREREZGWdDTw/XgfnFDCxHGctwcPHlycaEQiIiIiIiIiIiIiIiLpNGvWrFtc1407YaKSXCIiIiIiIiIiIiIikvWUMBERERERERERERERkaynhImIiIiIiIiIiIiIiGQ9JUxERERERERERERERCTrKWEiIiIiIiIiIiIiIiJZTwkTERERERERERERERHJekqYiIiIiIiIiIiIiIhI1lPCREREREREREREREREsp4SJiIiIiIiIiIiIiIikvWUMBERERERERERERERkaynhImIiIiIiIiIiIiIiGQ9JUxERERERERERERERCTrKWEiIiIiIiIiIiIiIiJZTwkTERERERERERERERHJekqYiIiIiIiIiIiIiIhI1lPCREREREREREREREREsp4SJiIiIiIiIiIiIiIikvWUMBERERERERERERERkaynhImIiIiIiIiIiIiIiGQ9JUxERERERERERERERCTrKWEiIiIiIiIiIiIiIiJZTwkTERERERERERERERHJekqYiIiIiIiIiIiIiIhI1lPCREREREREREREREREsp4SJiIiIiIiIiIiIiIikvWUMBERERERERERERERkaynhImIiIiIiIiIiIiIiGQ9JUxERERERERERERERCTrKWEiIiIiIiIiIiIiIiJZTwkTERERERERERERERHJekqYiIiIiIiIiIiIiIhI1lPCREREREREREREREREsp4SJiIiIiIiIiIiIiIikvWUMBERERERERERERERkaynhImIiIiIiIiIiIiIiGQ9JUxERERERERERERERCTrKWEiIiIiIiIiIiIiIiJZTwkTERERERERERERERHJekqYiIiIiIiIiIiIiIhI1lPCREREREREREREREREsp4SJiIiIiIiIiIiIiIikvWUMBERERERERERERERkaynhImIiIiIiIiIiIiIiGQ9JUxERERERERERERERCTrKWEiIiIiIiIiIiIiIiJZTwkTERERERERERERERHJekqYiIiIiIiIiIiIiIhI1lPCREREREREREREREREsp4SJiIiIiIiIiIiIiIikvWUMBERERERERERERERkaynhImIiIiIiIiIiIiIiGQ9JUxERERERERERERERCTrKWEiIiIiIiIiIiIiIiJZLzfTAYiIyK5v5syZu7mue3goFOoB5GU6HpFm2gzM/frrrz8cPnx4XaaDEREREREREZGWoYRJ9hoBnN/MNf4C/K7hg+nTp7dt06bNMOAw13W7A04z14+b67rbgEU5OTl/GzRo0Hvpel7JWl2Ap5u5xhbgtBaIpaX9CBjdzDVmAGGAr776qs26desmAFc5jtPGdd1mLi3SehQVFS2aNWvWdYMHD34+avhPQLtmLn0uUNXMNRLRAZjWzDVqgVPswRkzZvRwHOcAx3G6NHP9RG12HOeLwYMHz03z80p2GgMc0cw17gD+2QKxiIiIiIhIMyhhkr32Ak5s5hpfNLwze/bsH0Qikamu6/Zt5ppJcRyTm4lEImNmzpz5Qm1t7QWHHXbY+kzEIlmhDc3//tnUEoGkQE+a/29zAd56663ctWvXvuA4zgnND0ukVernuu6fZs6c+eshQ4Y8Wj92PNCxmesWNHN+ovJo/vd9bfQHs2fP7h+JRO4HTiJDJWBd12XmzJmzI5HINUOHDp2eiRgkawym+d9DU1siEBERERERaR4lTORt4Mkk534KMGvWrH0jkchrNP8GUUs5NScn53nXdY93HCeS6WBkl7QOuCjJud2AW1swllT5ApiU5NzFAB07drwCULJEdnUOcO+MGTNeGzp06GLgciA/ybUeI/P95S4Fkikztv337QcffDAwEolMB3q0WFTJGxQKhd6aNWvWLwYPHvz3TAcju6z7gL8mOXcUcGALxiIiIqkxFDgm6uMlNL/qgIiItEJKmMiXJJ8wAcB13dtpPckSABzH+fHMmTOHA89mOhbZJW0h+e+bInaOhMkKmvmzAfh1C8QhsjNoGwqFLgAqgWeasc6jTT8k5f4A1DRngdzc3Cdc120NyZIG+a7r/uH999/fS6dPJUX+1Yy5Z6KEiYjIzuAovBvK/g8lTEREdkmZ3sUoO7mPPvqoM/DTTMcRxHGcMzMdg0i2+vTTT/OB/TMdh0i6OI7zvUzH0BrMmDFjf9d1j8p0HAG65eTknJHpIERERERERKR10wkTaZaampoBoVAoJ9b18R+PZ+a3Mz1jP+z+Q2484MaEn2vOmjmUflSKy46m0W1y2nD3oXfTq12voCl7J/wkQe6lDVW8DAz0jDvcSRkPJLxeORfhUmKNLqM9J3BDCvtahOkF/B1ob125jDD/SGK9W4DTrdH3CXMukLrO3mEOA57ClMFpsA34OWHmJbTWNHL4jD8BB3nGHf5AGeXNjDSr1dXVtcf7f+TR8/HH6frii56xLXvuybzbb8cNJZbLz1+2jL2vvRZn27Ydg47DgooKNh50UOyJzeW6DCwro/3HH3uG1x5zDEuuvTbh5TrOmkX/igrPWKSggLn33Ud19+7NCrUxoa1b2XvkSPJWrvSMr7joIr459dSE1+v25z+zxx/+4Bmr7tGDuffdRyQ/2UpVTctfsYK9R470fh0AC8rL2fi9xHMZfW+/nd3eeccztnHwYBaUlgY+3nXdTgk/yS7IcZzBsa7NmtWRior+nrGCggj33TeX7t2rE3qe2lqH66/fkwULvO1eTjttNeefvzJwjuM4QzBlz5rJvQywX8gsAk4CZ0uCa+0OvAZ0ti5cC06yJZ7ife67gFOswXfAuTCJtQ4G/oR3M1Yt8EtwPksywHieNxf4C7CfdeF34CRRVtI9HbjFGvwW+Ck4a5KJUEREREREdi46YSLNEgqF2jR2/ap9r2Jb3TaWbl66/W3agmm8svSVhJ5nfc16Sj4qYcnmJZ61zuh/RqxkCZjG3M03km2EGA30wZRTMm8ud9XfvI9fBfvjcr9nHRiAQzilyRKAMMtwmGg9dxHwNDfTM8G1fg7cYK1TSA4lpDJZYp77feBF67n3BaYSTrBvwGeMA0611qqlgDtbMmTxW3nuudS1b0+bpUu3v3X+97/p+bvfJbSOU1tLUUkJBfPmedZa96MfpTZZAuA4LL36anI2bfI89x5PPUXha68ltFTumjUMLCnxrNNm6VJWnXNOSpMlYJIyS665hvyVKz3P3ffWW2n35ZcJrdX2q6/oc+ednnXyV6xg2ZVXpjRZAiYps2rECN/ncOCYMeSuXZvQWl3eeIPuzz7rWSdn82aWXn55iqLfpcRMHA0evIHDDlvP0qVttr99/XVbbrqpiNramPnVQJMn9+Ldd3fzrNWuXYQzz1zd2LTdEnqS2H4HLMD7u+No4K7ElnEdTAJnkLXWzNQnSwCowPwtEP3cF4B7VWLLuO0xJVD3stZ6MrXJEgCnFijG9MuJfu6bwf1xYmu5RZiyeNHrDDRrKVkiIiIiIpItlDCRlCrML6RyUCUh60tt4pyJzN84P641XNel4uMKlm1e5hk/tsexnDEgTdU1SvkQh2JrNA94ljCFca1xG+2JMA1oZ10pp4x/tkCUTSvjj8DvrdHu1PAM04h5Usijkr6Y3hbeu1sOFzOO+P5Tm6snxcB0a3QI/l2hsVVwFPhO+mwlxHBuYkPzApSmRPLzmTdpEnXtvN8OPR97jE4ffBD3Or3vu4/2c+Z4xjbvuy9LrrmmReJsSnX37iwoLwfH++3Q7+abKVi0KL5FIhEGlpaSt9p7o3fN8cfzzSn25u/U2HTggSz7tbflTKi6mqLiYnI2b45rjdDmzRQVFxOyTngsu+oqNh58cIvF2pjVw4ZRdeKJnrH8VasYUFoKbny53DaLF9O/stI76DgsGDeOmhQnr7LB9dcv5jvf8X5NffJJex55JObmB5/p03fjqae8LVLatYswadI82rSJxJjVkpwIcA6m11O0y8A9J4GFbsB/wuNr4FfNCC4BzhpM/wz7eM8d4A5KYKEHge9aY2+Rtn5dzqfASGswBDwN8fbScfPM431JtTvBeTFggoiIiIiI7KKUMJGUG9J1CBfudaFnbEvdFopnFbMtsi14UpRn5j/Dv1b8yzPWo20PSg6273WnWBl3Ay9Yo/2Bh+Oav4kH8JeMeJv9uLn5wSWgPVcB9o7Po/kfY5qc+zB51PIs0NW6ci9l/KWFImzaZdSQywhMmYxov6GcXzY5P0x3IjwDviTRNZTycdAUaXnb+vVjUYn1fRyJMHDcOPK++abJ+bu9+y57POPtrx1p1475N9+Mm+LTDNHW/fCHrDz7bM9YzubNDLine truncated
     "</div>\n",
-    "\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dac6a50b",
+   "metadata": {},
+   "source": [
     "    a) O(1)\n",
     "    b) O(N)\n",
     "    c) O(N²)\n",
@@ -754,7 +799,9 @@
    "id": "f1f30faf",
    "metadata": {},
    "source": [
-    "### Experimental speedup"
+    "### Experimental speedup\n",
+    "\n",
+    "Measure the speedup with the following cell. Is this time better?"
    ]
   },
   {
@@ -779,7 +826,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e2c6f60a",
+   "id": "2b1e3b24",
    "metadata": {},
    "source": [
     "### Complexity\n",
@@ -792,8 +839,25 @@
     "\n",
     "<div class=\"alert alert-block alert-success\">\n",
     "<b>Question:</b>  Which is the complexity of the communication and computations done by a worker in algorithm 2?\n",
-    "</div>\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "77808863",
+   "metadata": {},
+   "source": [
     "\n",
+    "<div>\n",
+    "<img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABkwAAAOBCAYAAACzvIyfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15fFTV+cfxz51sJBAwAQIEEpIgLoggi6i4W7faShUrda9W6g7ihgnEEEIgca0IrqVWpe6/urZatWqVShVZBBSVJQkBAgQIkJCFLDO/Py6RnFmSmSyTYL7v14vXK3PuPWdOhiyT+9zneSwOWLFixWF1dXV3ApcARwIhiIiIiIiIiIiIiIiI/LzUAT8Cfw8JCXlkxIgRewAsgKVLlx4LvAcMaL/9iYiIiIiIiIiIiIiIBNVm4ILRo0evtlatWhVTXV29Ekho712JiIiIiIiIiIiIiIgE2abw8PDhjurq6ikoWCIiIiIiIiIiIiIiIp1TQnV19RQH8Nv23omIiIiIiIiIiIiIiEg7+q0DGNTeuxAREREREREREREREWlHgxxARHvvQkREREREREREREREpB1FONp7ByIiIiIiIiIiIiIiIu1NARMREREREREREREREen0FDAREREREREREREREZFOTwETERERERERERERERHp9EIDPP8r4D9tsA8REREREREREREREZHWdAZwgr8nBxQwsSzrs1GjRqUGuiMREREREREREREREZFgWrZs2f0ul8vvgIlKcomIiIiIiIiIiIiISKengImIiIiIiIiIiIiIiHR6CpiIiIiIiIiIiIiIiEinp4CJiIiIiIiIiIiIiIh0egqYiIiIiIiIiIiIiIhIp6eAiYiIiIiIiIiIiIiIdHoKmIiIiIiIiIiIiIiISKengImIiIiIiIiIiIiIiHR6CpiIiIiIiIiIiIiIiEinp4CJiIiIiIiIiIiIiIh0egqYiIiIiIiIiIiIiIhIp6eAiYiIiIiIiIiIiIiIdHoKmIiIiIiIiIiIiIiISKengImIiIiIiIiIiIiIiHR6CpiIiIiIiIiIiIiIiEinp4CJiIiIiIiIiIiIiIh0egqYiIiIiIiIiIiIiIhIp6eAiYiIiIiIiIiIiIiIdHoKmIiIiIiIiIiIiIiISKengImIiIiIiIiIiIiIiHR6CpiIiIiIiIiIiIiIiEinp4CJiIiIiIiIiIiIiIh0egqYiIiIiIiIiIiIiIhIp6eAiYiIiIiIiIiIiIiIdHoKmIiIiIiIiIiIiIiISKengImIiIiIiIiIiIiIiHR6CpiIiIiIiIiIiIiIiEinp4CJiIiIiIiIiIiIiIh0egqYiIiIiIiIiIiIiIhIp6eAiYiIiIiIiIiIiIiIdHoKmIiIiIiIiIiIiIiISKengImIiIiIiIiIiIiIiHR6CpiIiIiIiIiIiIiIiEinp4CJiIiIiIiIiIiIiIh0egqYiIiIiIiIiIiIiIhIp6eAiYiIiIiIiIiIiIiIdHoKmIiIiIiIiIiIiIiISKengImIiIiIiIiIiIiIiHR6CpiIiIiIiIiIiIiIiEinp4CJiIiIiIiIiIiIiIh0egqYiIiIiIiIiIiIiIhIp6eAiYiIiIiIiIiIiIiIdHoKmIiIiIiIiIiIiIiISKengImIiIiIiIiIiIiIiHR6CpiIiIiIiIiIiIiIiEinp4CJiIiIiIiIiIiIiIh0egqYiIiIiIiIiIiIiIhIp6eAiYiIiIiIiIiIiIiIdHoKmIiIiIiIiIiIiIiISKengImIiIiIiIiIiIiIiHR6CpiIiIiIiIiIiIiIiEinp4CJiIiIiIiIiIiIiIh0egqYiIiIiIiIiIiIiIhIpxfa3hsQEZGfv6VLl/ZwuVxjHQ5HXyCsvfcj0kIVwPoNGzZ8PWHChLr23oyIiIiIiIiItA4FTDqvy4BrWrjGm8Cf6x8sXrw4MiIiYjxwosvligOsFq4v0p4qgA1Op/ONMWPGfOd2LAZ4sYXrVwKXtHCNtnAqkNbCNZYAmQDr1q2L2Lt37xzgVsuyIlwuVwuXFuk4UlJSCpctW3bHqFGj3mgw/H9AVAuXvgooaeEagegGvNbCNWqBce6DS5Ys6WtZ1lDLsmJauL5Iu7Esq7Kuri7fy/uBetOAU1r4NA8DH7dwDRERERERaSEFTDqvw4FftnCNH+s/WL58+UlOp/NVl8uV0MI1RToUh8OR+fXXXz9pWdYdo0ePrjkwHEHLv3/KWzi/rfSj5Z+bC+DTTz8N3bNnz1uWZZ3f8m2JdEiJLpfr/5YuXXrD6NGjFxwYOxeIbuG6XVo4P1BhtPz7vrbhg+XLlw90Op3zgQtQCVg5xLlcLhwOB0uXLv3R5XLdefzxx7/ndsooWv499GoL54uIiIiISCtQwEQ+A55r5tzvAJYtW3a00+n8gJZfIBLpiByWZd1qWVYocNOBsb3Adc1crzfwQKvsrG39COQ2c+4mgOjo6JsBBUvk584CHluyZMkHY8aM2YT9cyK8mWv9hfYPLkwEmlNmzFn/wVdffZXsdDoXA31bbVciHcORlmW9u3Tp0j+MHj36+Qbj84B3m7nmncCxLd+aiIi0sTHAmQ0eb6blVQdERKQDUsBE1tL8gAkALpfrIRQskZ85l8t1w7Jly/48atSoZdjltJ5r5lIpHBoBk2208GcDcEMr7EPkUBDpcDh+D2QDL7VgnQVNn9LmXgBqmjyrEaGhoX91uVwKlsjPlQN48ssvv/z4xBNP3Hxg7D8tWO93KGAiInIoOB3zhrL/ooCJiMjPUnvfxSiHuBUrVhwGnNfe+xAJAguY0N6bOFR899134cAx7b0PkWCxLOu49t5DR7BkyZJjXC7X6e29D5E2FhkaGvqH9t6EiIiIiIi0PmWYSIvU1NQkORyOEF/HZ62cxdJdS42xk+NOZurQqQE/16rdq8hYkYGLg02jI0IiePT4R4mPig94PX9VO6uZsmQKWyq2GONXplzJhKTAr5+/s+kd/rLuL8ZY7y69mXfCPCJDIlu018bsqNrBpK8mUVlXaYxPHzadMb3GBLzeY98/xsdbzd6kx8Ycy6zjZmFZVov22pjVu1dz34r7jK+DcEc4j455lP5R/QNay4mTqUunsq50nTH+qwG/4oYjPJMjXC7X4ObtuvOpq6vrih1k8qrfs8/S8+23jbHKQYPIe+ghXI7AYvnhRUUMnjIFa//+g4OWRUFWFvuGDQtorYC4XCTNmEG3lSuN4T1nnsnmKVMCXi562TIGZmUZY84uXVg/bx7VcXEt2mpjHFVVDJ48mbDt243xbdddx86LLgp4vd5//zt9XnjBGKvu25f18+bhDG9upaqmhW/bxuDJk82vA6Bg5kz2HRd4LCPhoYfosWiRMbZv5EgKZszwer7L5eoe8JP8DFmWNcrXsWXLosnKGmiMdeniZN689cTFVQf0PLW1FnfdNYiCArPdyyWX7OCaa7b7mNU63nijN88/38cY69u3msceW09EhNPHLO/27AnlttsGU1ZmvpW6665NnHba3hbvtTGPPJLAZ5/1MMZGjNhHZmZBwGutXRvJvfcOwtng0w8JcfHQQxtISalq4U59q6uzuPvuQeTlmV8HF1+8k2uv3Rbweh9/HMNjj5nvJXr0qGX+/HV0725WqnO5XKMD37GIiIiIiHR0yjCRFnE4HBGNHb/16FvZX7efLRVbfvr3WsFrvLfFvVdm40prSklfkc7mis3GWpcOvLRNgyVgX4y/7ejbKK4qNp77ke8eYfXu1QGtlbcvjwe+fcBYZ2vlVm444oY2DZaAHZS5bvB1xnNvqdhC+op0du7fGdBai7YvYuGGhcY6pTWl3HLULW0aLAE7KHN639ON587fl0/a8jRqnIFVkVmwdgH/2fYfY60QK4SrBl3la0qjX+/iv+1XXUVd165EbNny07/DPv+cfn/+c0DrWLW1pKSn0yUvz1hr76mntm2wBMCyKLrtNkLKy43n7vO3vxH7wQcBLRW6ezfJ6enGOhFbtlB85ZVtGiwBOyizedIkwrdvN5474YEHiFq7NqC1ItetY8AjjxjrhG/bRtEtt7RpsATsoEzxZZd5vIbJ06YRumdPQGvFfPQRca+8YqwTUlZG0Y03ttHuf1Z8Bo5GjSrjxBNL2bIl4qd/GzZEcu+9KdTWBva744kn4vniix7GWlFRTn73ux0t/gSactFFO4iPrzaee9myaB5+eEBA67hckJU1kB9+iDLWGjKkos2DJQATJxbhclnGc//jHz157bXeAa1TWelg2rQUNm2KMNa68MJdbRosATsoM2nSFnbtCjOe+4kn+rNkSWDVYrdsiWDWrIHGOkVFEVx//TaPYMkBPbwNioiIiIjIoU0BE2lTseGxZI/MxuH2pZazKof8ffl+reFyuchamUVRRZExflbfs7g06dJW22tjhvQYwqSjJxljta5api2fRmlNqV9rVNZVkroslao68+LBHwf/sVkZHs1xXvx5/HrAr42xkv0lTF8+HSf+3RW7vWo7md9kGhkeABnDM4iPbNvgVb1JR09iWIx5MXzNnjXM+2Ge32ssL1nOgrVmu4BwRzi5o3KJColqlX2Kb87wcPJyc6mLMl/rfn/5C92/+srvdfrPm0fXVauMsYqjj2bzpEk+ZrSu6rg4CmbOBLdAYeLs2XQpLPRvEaeT5IwMwnaYF3p3n3suO8eNa62tNqr82GMpusHMqnJUV5OSmkpIRYVfazgqKkhJTcXhluFRdOut7Bs+vNX22pgd48dT8stfGmPhxcUkZWTYV6f9ELFpEwOzs81By2JjZibVfdWWo6XuumsTRx5pfk2tXt2VZ57x//fH4sU9+NvfzP+LqCgnubl5AWd4NIfDAdnZ+fTsaQbp33ijN++/H+v3OgsX9uXzzw8zxgYM2E96+sZW2WdTunevIycnj7Aw83vj0UcT+OEH/38P5uQkemT6jB5dxu9/37aZPvVSUiq5555NxpjTCenpyezaFebXGLine truncated
+    "</div>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e2c6f60a",
+   "metadata": {},
+   "source": [
     "    a) O(N) communication and O(N^2) computation\n",
     "    b) O(N^2) communication and O(N^2) computation\n",
     "    c) O(N^3) communication and O(N^3) computation\n",
@@ -964,7 +1028,9 @@
     "\n",
     "- Matrix-matrix multiplication is trivially parallelizable (all entries in the result matrix can be computed in parallel, at least in theory)\n",
     "- However, we cannot exploit all the potential parallelism in a distributed system due to communication overhead\n",
-    "- We need a sufficiently large grain size to obtain a near optimal speedup\n"
+    "- We need a sufficiently large grain size to obtain a near optimal speedup\n",
+    "- We measured the theoretical parallel performance by computing the complexity of communication over computation\n",
+    "- We measured the actual performance using the parallel speedup and parallel efficiency\n"
    ]
   },
   {
@@ -1001,8 +1067,6 @@
     "    @assert size(B,1) == l\n",
     "    @assert mod(m,nworkers()) == 0\n",
     "    # Implement here\n",
-    "    \n",
-    "    C\n",
     "end"
    ]
   },
@@ -1047,7 +1111,7 @@
    "outputs": [],
    "source": [
     "P = nworkers()\n",
-    "load = 100\n",
+    "load = 50 # try with 50, 100, and 200\n",
     "N = load*P\n",
     "A = rand(N,N)\n",
     "B = rand(N,N)\n",
@@ -1060,107 +1124,6 @@
     "println(\"Efficiency = \", 100*(T1/TP)/P, \"%\")"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "fa8d7f40",
-   "metadata": {},
-   "source": [
-    "### Exercise 2"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0e7c607e",
-   "metadata": {},
-   "source": [
-    "The implementation of algorithm 1 is very impractical. One needs as many processors as entries in the result matrix C. For 1000 times 1000 matrix one would need a supercomputer with one million processes! We can easily fix this problem by using less processors and spawning the computation of an entry in any of the available processes.\n",
-    "See the following code:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "023b20d1",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "function matmul_dist_1_v2!(C, A, B)\n",
-    "    m = size(C,1)\n",
-    "    n = size(C,2)\n",
-    "    l = size(A,2)\n",
-    "    @assert size(A,1) == m\n",
-    "    @assert size(B,2) == n\n",
-    "    @assert size(B,1) == l\n",
-    "    z = zero(eltype(C))\n",
-    "    @sync for j in 1:n\n",
-    "        for i in 1:m\n",
-    "            Ai = A[i,:]\n",
-    "            Bj = B[:,j]\n",
-    "            ftr = @spawnat :any begin\n",
-    "                Cij = z\n",
-    "                for k in 1:l\n",
-    "                    @inbounds Cij += Ai[k]*Bj[k]\n",
-    "                end\n",
-    "                Cij\n",
-    "            end\n",
-    "            @async C[i,j] = fetch(ftr)\n",
-    "        end\n",
-    "    end\n",
-    "    C\n",
-    "end"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "52005ca1",
-   "metadata": {},
-   "source": [
-    "With this new implementation, we can multiply matrices of arbitrary size with a fixed number of workers. Test it:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "c1d3595b",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "using Test\n",
-    "N = 50\n",
-    "A = rand(N,N)\n",
-    "B = rand(N,N)\n",
-    "C = similar(A)\n",
-    "@test matmul_dist_1_v2!(C,A,B) ≈ A*B"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ab609c18",
-   "metadata": {},
-   "source": [
-    "Run the next cell to check the performance of this implementation. Note that we are far away from the optimal speed up. Why? To answer this question compute the theoretical communication over computation ratio for this implementation and reason about the obtained result. Hint: the number of times a worker is spawned in this implementation is N^2/P on average."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "d7d31710",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "N = 100\n",
-    "A = rand(N,N)\n",
-    "B = rand(N,N)\n",
-    "C = similar(A)\n",
-    "P = nworkers()\n",
-    "T1 = @belapsed matmul_seq!(C,A,B)\n",
-    "C = similar(A)\n",
-    "TP = @belapsed matmul_dist_1_v2!(C,A,B)\n",
-    "println(\"Speedup = \", T1/TP)\n",
-    "println(\"Optimal speedup = \", P)\n",
-    "println(\"Efficiency = \", 100*(T1/TP)/P, \"%\")"
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "8e171362",
@@ -1175,15 +1138,15 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Julia 1.9.1",
+   "display_name": "Julia 1.11.6",
    "language": "julia",
-   "name": "julia-1.9"
+   "name": "julia-1.11"
   },
   "language_info": {
    "file_extension": ".jl",
    "mimetype": "application/julia",
    "name": "julia",
-   "version": "1.9.1"
+   "version": "1.11.6"
   }
  },
  "nbformat": 4,

notebooks/mpi_collectives.ipynb

@@ -0,0 +1,962 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "7606d30a",
+   "metadata": {},
+   "source": [
+    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/3/39/VU_logo.png/800px-VU_logo.png?20161029201021\" width=\"350\">\n",
+    "\n",
+    "### Programming large-scale parallel systems"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4ac1e5d9",
+   "metadata": {},
+   "source": [
+    "# Intro to MPI (collectives)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a341be2e",
+   "metadata": {},
+   "source": [
+    "## Contents\n",
+    "\n",
+    "\n",
+    "In this notebook, we will continue learning the basics of parallel computing using the Message Passing Interface (MPI) from Julia. In particular, we will learn:\n",
+    "\n",
+    "- How to use basic collective communication directives\n",
+    "- How to use MPI communicators\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c87b3c82",
+   "metadata": {},
+   "source": [
+    "## Collective communication\n",
+    "\n",
+    "MPI provides a set of routines for communication involving multiple processes. These are called *collective communication* operations. Some usual collective operations are:\n",
+    "\n",
+    "\n",
+    "- `MPI_Barrier`: Synchronize all processes\n",
+    "- `MPI_Bcast`: Send the same data from one to all processes\n",
+    "- `MPI_Gather`: Gather data from all processes to one\n",
+    "- `MPI_Scatter`: Scatter data from one to all all processes\n",
+    "- `MPI_Reduce`: Reduce data from all processes to a single one\n",
+    "- `MPI_Scan`: Scan (or prefix) reduction\n",
+    "- `MPI_Allgather`: Like a `MPI_Gather` but all processes receive the result\n",
+    "- `MPI_Allreduce`: Like `MPI_Reduce` but all processes receive the result\n",
+    "- `MPI_Alltoall`:  Exchange data from all to all processes\n",
+    "\n",
+    "\n",
+    "We will discuss some of them in greater detail in this notebook."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4ffa5e56",
+   "metadata": {},
+   "source": [
+    "## Why collective operations?\n",
+    "\n",
+    "Point-to-point communication functions provide all the building blocks needed in parallel programs and could be used to implement the collective functions described above. Then, why does MPI provide collective communication functions? There are several reasons:\n",
+    "\n",
+    "- Ease of use: It is handy for users to have these functions readily available instead of having to implement them.\n",
+    "- Performance: Library implementations typically use efficient algorithms (such as reduction trees).\n",
+    "- Hardware support: Hardware vendors can optimize the MPI library for their machine and even develop hardware specially design to perform MPI operations efficiently. For instance, [IMB Blue Gene](https://en.wikipedia.org/wiki/IBM_Blue_Gene) was equipped with  auxiliary networks for MPI collectives.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "53a0f5fa",
+   "metadata": {},
+   "source": [
+    "## Semantics of collective operations\n",
+    "\n",
+    "These are key properties of collective operations:\n",
+    "\n",
+    "\n",
+    "- Completeness: All the collective communication directives above are *complete* operations. Thus, it is safe to use and reset the buffers once the function returns.\n",
+    "- Standard mode: Collective directives are in standard mode only, like `MPI_Send`. Assuming that they block is erroneous, assuming that they do not block is also erroneous.\n",
+    "- Synchronization:  Completion of a call does not guarantee that other processes have completed the operation. A collective operation may or may not have the effect of synchronizing all processes, the only exception is `MPI_Barrier` of course.\n",
+    "\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "<b>Note:</b> Recent versions of the MPI standard also include non-blocking (i.e., incomplete) versions of collective operations (not covered in this notebook). A particularly funny one is the non-blocking barrier `MPI_Ibarrier`.\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e9e2a219",
+   "metadata": {},
+   "source": [
+    "## MPI_Barrier\n",
+    "\n",
+    "This function is used to synchronizes a group of processes. All processes block until all have reached the barrier. It is often invoked at the end of for loops to make sure all processes have finished the current loop iteration to move to the next one. We will see a practical example later in another notebook when studying the traveling sales person problem (TSP).\n",
+    "\n",
+    "In Julia:\n",
+    "```julia\n",
+    "MPI.Barrier(comm)\n",
+    "```\n",
+    "\n",
+    "In C:\n",
+    "```c\n",
+    "int MPI_Barrier(MPI_Comm comm)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d9753e4b",
+   "metadata": {},
+   "source": [
+    "### Example\n",
+    "\n",
+    "In this example the ranks sleep for a random amount of time and then they call barrier. It is guaranteed that the message \"Done!\" will be printed after all processes printed \"I woke up\" since we used a barrier. Try also to comment out the call to `MPI.Barrier`. You will see that the message can be printed in any order."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c6e78389",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "using MPI"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9a66c717",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "code = quote\n",
+    "    using MPI\n",
+    "    MPI.Init()\n",
+    "    comm = MPI.COMM_WORLD\n",
+    "    rank = MPI.Comm_rank(comm)\n",
+    "    sleep(rand(1:3))\n",
+    "    println(\"[rank $rank] I woke up\")\n",
+    "    MPI.Barrier(comm)\n",
+    "    println(\"[rank $rank] Done!\")\n",
+    "end\n",
+    "run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "df0223ff",
+   "metadata": {},
+   "source": [
+    "## MPI_Reduce\n",
+    "\n",
+    "This function combines values provided by different processors according to a given reduction operation. The result is received in a single process (called the root process).  The root process can be any process and it is rank 0 by default in Julia.\n",
+    "\n",
+    "In Julia:\n",
+    "```julia\n",
+    "MPI.Reduce!(sendbuf, recvbuf, op, comm; root)\n",
+    "```\n",
+    "\n",
+    "In C:\n",
+    "```\n",
+    "int MPI_Reduce(const void *sendbuf, void *recvbuf, int count,\n",
+    "               MPI_Datatype datatype, MPI_Op op, int root,\n",
+    "               MPI_Comm comm)\n",
+    "```\n",
+    "\n",
+    "\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "<b>Note:</b> `MPI_Reduce` (and related functions like `MPI_Allreduce`) use operations that are *commutative* and *associative*, such as addition, multiplication, maximum, minimum, logical and, logical or. Otherwise the result will be undefined as it will depend in which order the operations are done.\n",
+    "</div>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2fc1aa28",
+   "metadata": {},
+   "source": [
+    "### Example\n",
+    "Each process generates a random integer. Compute the sum of all these integers. Receive the result on rank 0."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "341b8438",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "code = quote\n",
+    "    using MPI\n",
+    "    MPI.Init()\n",
+    "    comm = MPI.COMM_WORLD\n",
+    "    rank = MPI.Comm_rank(comm)\n",
+    "    value = rand(1:4)\n",
+    "    root = 0\n",
+    "    sndbuf = [value]\n",
+    "    if rank == root\n",
+    "        rcvbuf = [0]\n",
+    "    else\n",
+    "        # This can be anything as only the root process\n",
+    "        # Will receive data\n",
+    "        rcvbuf = nothing\n",
+    "    end\n",
+    "    println(\"[rank $rank] I am contributing with $sndbuf\")\n",
+    "    MPI.Reduce!(sndbuf, rcvbuf, +, comm; root)\n",
+    "    println(\"[rank $rank] The sum is $rcvbuf\")\n",
+    "end\n",
+    "run(`$(mpiexec()) -np 3 julia --project=. -e $code`);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3ea77a8f",
+   "metadata": {},
+   "source": [
+    "## Reducing multiple values\n",
+    "\n",
+    "If more than one item is provided in the send buffers, `MPI_Reduce` does the reduction element by element.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b26fd0b2",
+   "metadata": {},
+   "source": [
+    "### Example\n",
+    "\n",
+    "Each process generates 2 random integers. Take a look to the result of reduce. Note that the values have ben reduced \"element by element\". "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1a145948",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "code = quote\n",
+    "    using MPI\n",
+    "    MPI.Init()\n",
+    "    comm = MPI.COMM_WORLD\n",
+    "    rank = MPI.Comm_rank(comm)\n",
+    "    sndbuf = rand(1:4,2)\n",
+    "    root = 0\n",
+    "    if rank == root\n",
+    "        rcvbuf = [0,0]\n",
+    "    else\n",
+    "        rcvbuf = nothing\n",
+    "    end\n",
+    "    println(\"[rank $rank] I am contributing with $sndbuf\")\n",
+    "    MPI.Reduce!(sndbuf, rcvbuf, +, comm; root)\n",
+    "    println(\"[rank $rank] The sum is $rcvbuf\")\n",
+    "end\n",
+    "run(`$(mpiexec()) -np 3 julia --project=. -e $code`);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "539df34f",
+   "metadata": {},
+   "source": [
+    "## MPI_Allreduce\n",
+    "\n",
+    "This is a variant of `MPI_Reduce` in which all processes get the result."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "be0381bb",
+   "metadata": {},
+   "source": [
+    "### Example\n",
+    "\n",
+    "This is the same example as above, but all processes get the result."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1595f4f5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "code = quote\n",
+    "    using MPI\n",
+    "    MPI.Init()\n",
+    "    comm = MPI.COMM_WORLD\n",
+    "    rank = MPI.Comm_rank(comm)\n",
+    "    sndbuf = rand(1:4,2)\n",
+    "    # All ranks allocate a receive buffer\n",
+    "    rcvbuf = [0,0]\n",
+    "    println(\"[rank $rank] I am contributing with $sndbuf\")\n",
+    "    MPI.Allreduce!(sndbuf, rcvbuf, +, comm)\n",
+    "    println(\"[rank $rank] The sum is $rcvbuf\")\n",
+    "end\n",
+    "run(`$(mpiexec()) -np 3 julia --project=. -e $code`);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "97dc2886",
+   "metadata": {},
+   "source": [
+    "## MPI_Gather\n",
+    "\n",
+    "Each rank sends a message to the root rank (the root rank also sends a message to itself). The root rank receives all these values in a buffer (e.g. a vector).\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "<b>Note:</b>  This function assumes that the amount of data sent from each rank is the same. See `MPI_Gatherv` below for more general cases.\n",
+    "</div>\n",
+    "\n",
+    "\n",
+    "In Julia:\n",
+    "```julia\n",
+    "MPI.Gather!(sendbuf, recvbuf, comm; root=0)\n",
+    "```\n",
+    "\n",
+    "In C:\n",
+    "```c\n",
+    "int MPI_Gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype,\n",
+    "    void *recvbuf, int recvcount, MPI_Datatype recvtype, int root,\n",
+    "    MPI_Comm comm)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3d0a1e9d",
+   "metadata": {},
+   "source": [
+    "### Example\n",
+    "\n",
+    "Each process sends an integer to rank 0.\n"
+   ]
+  },
+  {
+   "attachments": {
+    "g13884.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAABFEAAANWCAYAAAA2uxORAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13mGRlnf7/d01iYMg5BxUlKCiggiImDKiYV9c16xpXZY3o6q7u18SuusZVMcvq+jMHRFcRTBgQDICIiGSQnAcmz/3745xmTp+uTtXV/VR3v1/XVVdPV9c5dVd1z3PO+dQTOkiSJM24LAG2AbZufd0I2AxYCmxc/3sRsBWwGNi0vn9p/bPFwJaTfPKh/dwKrANWACuBNcDy+jE31V+X1/evrB+3DrgFuLl+zM317ZbGv2+GzqpJZpIkSbNAp3QASZI0F2QpsAuwM7A7sC0bCiNDRZJtG//erEzOGbOCYUUVbgauB64G/gZcA1wJXFt97dxWKKckSZoEiyiSJGkcWUZVGNmlvu1Wf921vu0MbF8sHqwC7pjkNquBJfW/N6XqnVLSHcBVVEWWdqHlKuBi4BLorC4VUJIkWUSRJEkAZBtg7/p298a/7wps0ecnuwm4ob7d2Pq6iqrXxhrgNjYMoxkaVnMTsLb+2QrorOxfrAwNIVoIbF7fuQWwANiEaqjRkvpnWzZuW7W+b96W0D/rqYorF49yuxI66/v4fJIkqcUiiiRJ80Y2Z0NxpF0s2XoKO14FXEF1gX9Z/fVquhdJbphfF/rZhA0Fle2oevBsT9V7Z0dgp/q2A9Vwp6lYDVzKhqLKRcB5wJ+oerGsm+L+JUma9yyiSJI052QRVYHkAODA+usBVENvJusOqgvzK+vb5fXXK7izcNK5tg+hRTaiKrDs0vi6A7AHsFd924Wqp8xkrQT+XN/OZUNx5a/QWTPl6JIkzRMWUSRJmtWyNXBvNhRKDgD2p1q9ZqJWAhfUt782/n0BdP7W17iaoiymmp9mr1Fuk52bZg3V7/pPjCiwWFyRJKnNIookSbNGtgUOrW8HURVMdpngxuuoCiR/YViRhAuAy6GTvsdVAVnGhoLK3YF9gf3qr5NZCno1cA7wu8bt7P7OQSNJ0uxjEUWSpIGUhcA+wMHAA4HDqS6EJ3LsvgX4I1Wvgj8BvwV+B53JrmCjOSVbUfVS2q/x9S71bSLWUhXhflvfzq2+dm7qf1ZJkgaTRRRJkgZCdqTqYXJY/fUQqhVhxrIWOJ+qx8BZ9e0c6FwxjUE152RbNvRW2Q+4J1VPp4n0XFlHVVgZ6q3yG+BMe6xIkuYqiyiSJBWRPYCHAw8FHkQ1eeiYG1BdrJ4O/Lr+ei50Vk1nSs1nuQtVMaV5224CG64GzgR+Cfyi+urkw5KkucEiiiRJMyLbAQ8BjqQamrPfOBssp+pZchrVheivoXPddCaUxpedqYaYDd2GhgSN5yo2/C2fBvx+fi11LUmaKyyiSJI0LbIF8GCq3iYPo5qDYrTj7nqquUtOB35F1dPkPC8yNTtkR6peKodQDUc7DNhinI1upPpb/0V9OwM6K6YzpSRJ/WARRZKkvsgiqglgH01VNDkYWDjag6km5TwVOAX4GXRunomU0vTLAqqi4eHAA6j+X+w1zkarqIoqp9S3M6CzdjpTSpLUC4sokiT1LMuoCiaPA54A7DDGgy9iw1CG7zn5q+aX7Ajclw0rTR0CbDTGBrdTFVV+VN9+5zLckqRBYBFFkqRJyd2Bo4HHUk0Iu2iUB15J1dOkvnUum5l80myQTdhQVHkAcASw2RgbXMWGnls/gs7l0x5RkiRJkjRZWQR5COS9kPMhGeW2EvIDyCsh9yidWppdsgjyAMi/Qn4CWTXG/7XU/xc/CnlKPf+QJEmSJKmMbAR5POQLkBvHuJC7BvLZ+kJurE/RJU1KNoE8CvKfkN9C1o3x/3A15BTIqyF7l04uSZIkSfNAFkIOh3wQct0YF2wX1o85sp5MVtK0y2b1/7njIGeO00vlQsjxkKMhS0onlyRJkqQ5IgshD6svuK4f5YJsBeT7kJdD9iidWBJAdoc8H/JlyM1jFFRuqh/zbMi2pVNLkiRJ0iyTBXWPkw9DrhpjfpNvQ/4BsmnpxJLGksV1MfR9kL+MUVBZCzkN8kbIvUqnliRJkqQBlgPri6zLx5hX4XuQ50K2LJ1WUq9yd8hrIKfW/6/HGvZzHOSQ0oklSZIkaQBkC8iL60+fu11Erat/dgxk+9JpJfVbltVzoxwP+dsYBZVL67mODod0SqeWJEmSpBmSTt21/wuQO0YpnPy0nuNkh9JpJc2ULIQ8EPJuyLljFFQuhrwHcj8LKpIkSZLmqOwG+de6i363C6M/Q46F7FI6qaRBkP0gb4WcM0ZB5RLIeyGHWlCRJEmSNMtlCeTvqFbPWdvlAug2yGeqLvqSNJrsC/k3yNnjDPn5r6qgIkmSJEmzRvaA/AfkulEudn4BeSFks9JJJc022QfyFshZYxRU/lIXXfYqnVaSJEmSRpGDISdA1nS5qLmhnjzygNIpJc0V2aueeHq0yakDObN+zDal00qSJEma97IU8gLIH7pcvKyFfAfyBMji0kklzWW5O+TNkPNGKabcAfkS5DGQRaXTSpIkSZpXsiPkbaMM2bmZajnSPUunlDQfZX/IcZBrRimoDPWMcz4mSZIkSdPpziE7q7tcmJxfd5tfVjqlJFEtm3xk3WbdPkpB5dy6ILxH6bSSJEmS5oQsgDytnlugfQGyDvItyMNKp5Sk0WUryEvria27FVPWQk6EPK4qvkiSJEnSpGQx5DmjzDFwS90dfp/SKSVpcrI75FjIBaMUVK6shwPZO0WSJEnSeLIx5J8gl3S5uPgT5GUO2ZE0+6UDeRDkk5Bbu7R3a+qedkdVPfIkSZIk6U7ZtJ7T5MouFxO/r3ul2M1d0hyUpZC/Y/Tlkq+oe6fsXjqpJEmSpKKyed21/YYuFw6nQY4unVCSZk4Orocr3jbK3Ckn1wUXi8qSJEnS/JEdIe/pcqGwHvJdyANKJ5SkcrJ5PRnt70fpnXIh5E2QHUonlSRJkjRtshXVkp7tOQDW1atTHFI6oSQNljF7p6yCfAVy/9IpJUmSJPVNNqmH7dzYugBYDTnBlXYkaTzZHPLiMXqnnOZQH0mSJGlWy5J6tZ2ruhRPPu5EiZLUixwG+Z+6J0q7mHIB5FWQzUqnlCRJkjQhWVB/InpBlzlPvgLZu3RCSZr9skM9RPK6LsWUWyEfhOxZOqUkSZKkUeXIUbqbnww5qHQ6SZp7shHVUvBnd2l7h+acOrJ0SkmSJEl3yoMhv+xyAv9zyOGl00nS3JcO5BGQk+qef+32+AzIM6uhlpIkSZIKyF6Qb3Q5Wf895DGl00nS/JR7QD4KWd6lfb4C8hrIpqVTSpIkSfNENqnH4q9onZxfXK8g4QoRklRcNoccA7mkSzHlFshxkK1Lp5QkSZLmqHQgz4Bc3joZvx7yMsji0gklSW1ZBHk65PRRJqF9D2Sn0iklSZKkOST3gfysdfK9BnI8ZNvS6SRJE5HD68lm28WUVZATIHcvnVCSJEmaxbJ1vVTm2tYJ96mQA0qnkyT1Iveuiybttn1oRZ+DSyeUJEmSZpEsquc3ua51gn15tZymJGn2y93qHoVrWm39+rqYcljphJIkSdKAy4Mg57ZOqJdD3gJZWjqdJKnfsle9ok97wvBAToEcWTqhJEmSNGCyRT10Z13rBPpEyB6l00mSplu2r1dfu6lLMeWXkIeXTihJkiQNgDwJcmXrhPn31SSEkqT5JVtB/rXLkM5AfuQwH0mSJM1T2Rny9dYJ8h2QN1TzokiS5q8sg7wGck2XYspJTkArSZKkeSKdaoLY3NA6Kf4p5B6l00mSBkk2gRwDubpLMeVkyH1KJ5QkSZKmSe5WTxTYPAm+qV6Np1M6nSRpUGVTyLFd5kxZD/mKRXhJkiTNIVlUn/y2V184EbJL6XSSpNkim9XHk5tbx5N1dTFl79IJJUmSpCnIAZCzWie7l0IeWzqZJGm2ynaQ99VzaTWPL6shx0N2K51QkiRJmoQsgLwWsrL1SeEHq08SJUmaquwM+XDrWJP6+/dBti6dUJIkSRpHdoOc2jqhPQ9yaOlkkqS5KLtDPln3RGkee26sC/oblU4oSZIkdZFntCb+Ww/5CGTj0skkSXNd7go5oe752CymXAT5eycxlyRJ0oDIFvWJa/Ok9WrI40onkyTNN9kP8t0uyyKfAXlw6XSSJEma1/IAyIWtE9WvQ7YtnUySNJ/lSMgfuhRTTobcs3Q6SZIkzStZDHlbq9v0rZAXl04mSVIlCyB/B7mkVUhZU6/ks2PphJIkSZrzsl+XpYtPg+xVOpkkSSNlGeTfILe1jl23QP4FsknphJIkSZqT8szWSejq+gR0YelkkiSNLTtCPl73RGkWUy6HPK/quSJJkiRNWTaCfKzL0sUHl04mSdLkZD/IiV3mSzkTcmjpdJIkSZrVsjvk160TzW9Uq/JIkjRb5TDIL1rHt/X1inM7lE4nSZKkWSePh9zUOLlcCXl56VSSJPVHOpBnQa5oFVNuhhwDWVQ6oSRJkgZeFnVZfedSuzlLkuambFIf91a2iil/hjyqdDpJkiQNrGwP+VHrJPJEyNalk0mSNL2yN+SkLvOlnAjZo3Q6SZIkDZQ8BPK3xknj2vqTOVcskCTNIzkaclGrkHJ7fUxcWjqdJEmSissb6qLJ0Mni3yBHlE4lSVIZ2bgumtzRKqb8BfKY0ukkSZJURDaCfLZ1gvgTyE6lk0mSVF52rVfs6TbE5y6l00mSJGnGZHvIaa2TwndDFpZOJknSYMmj6olmm8fMFZB/gywpnU6SJEnTKveCXNxavvi5pVNJkjS4srhe+viWLkN8Hlo6nSRJkqZFHg+5rXHydzXksNKpJEmaHbIL5H9bhZR1kI9BtiidTpIkSX2TY+oTvaGTvrMhe5ZOJUnS7JOHQM5vFVOugjyndDJJkiRNSdcJLine truncated
+    }
+   },
+   "cell_type": "markdown",
+   "id": "4bdf9c02",
+   "metadata": {},
+   "source": [
+    "<div>\n",
+    "<img src=\"attachment:g13884.png\"  align=\"left\" width=\"350\"/>\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1f8a70c6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "code = quote\n",
+    "    using MPI\n",
+    "    MPI.Init()\n",
+    "    comm = MPI.COMM_WORLD\n",
+    "    rank = MPI.Comm_rank(comm)\n",
+    "    msg = 10*(rank+2)\n",
+    "    root = 0\n",
+    "    sndbuf = [msg]\n",
+    "    if rank == root\n",
+    "        nranks = MPI.Comm_size(comm)\n",
+    "        rcvbuf = zeros(Int,nranks)\n",
+    "    else\n",
+    "        # This can be anything as only the root process\n",
+    "        # Will receive data\n",
+    "        rcvbuf = nothing\n",
+    "    end\n",
+    "    println(\"[rank $rank] I am sending $sndbuf\")\n",
+    "    MPI.Gather!(sndbuf, rcvbuf, comm; root)\n",
+    "    println(\"[rank $rank] I received $rcvbuf\")\n",
+    "end\n",
+    "run(`$(mpiexec()) -np 3 julia --project=. -e $code`);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0f85f08d",
+   "metadata": {},
+   "source": [
+    "## MPI_Allgather\n",
+    "\n",
+    "This is a variant of `MPI_Gather` in which all processes get the result."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4bf2ef34",
+   "metadata": {},
+   "source": [
+    "### Example"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "070e1fc1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "code = quote\n",
+    "    using MPI\n",
+    "    MPI.Init()\n",
+    "    comm = MPI.COMM_WORLD\n",
+    "    rank = MPI.Comm_rank(comm)\n",
+    "    msg = 10*(rank+2)\n",
+    "    sndbuf = [msg]\n",
+    "    nranks = MPI.Comm_size(comm)\n",
+    "    rcvbuf = zeros(Int,nranks)\n",
+    "    println(\"[rank $rank] I am sending $sndbuf\")\n",
+    "    MPI.Allgather!(sndbuf, rcvbuf, comm)\n",
+    "    # All ranks receive the result\n",
+    "    println(\"[rank $rank] I received $rcvbuf\")\n",
+    "end\n",
+    "run(`$(mpiexec()) -np 3 julia --project=. -e $code`);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fb6a9034",
+   "metadata": {},
+   "source": [
+    "## MPI_Gatherv\n",
+    "\n",
+    "This function is similar to `MPI_Gather`, but it is used when there is a different amount of data sent from each process. There is also `MPI_Allgatherv` for which all processes will receive the result."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e0d2998c",
+   "metadata": {},
+   "source": [
+    "### Example\n",
+    "\n",
+    "Each processes sends `rank+1` integers to rank 0."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "23814278",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "code = quote\n",
+    "    using MPI\n",
+    "    MPI.Init()\n",
+    "    comm = MPI.COMM_WORLD\n",
+    "    rank = MPI.Comm_rank(comm)\n",
+    "    sndbuf = fill(10*rank,rank+1)\n",
+    "    root = 0\n",
+    "    if rank == root\n",
+    "        nranks = MPI.Comm_size(comm)\n",
+    "        # Define the amount of data to be received\n",
+    "        # from each rank\n",
+    "        counts = [ rank+1 for rank in 0:(nranks-1)]\n",
+    "        # Total amount of receive data\n",
+    "        ndata = sum(counts)\n",
+    "        data = zeros(Int,ndata)\n",
+    "        rcvbuf = MPI.VBuffer(data,counts)\n",
+    "    else\n",
+    "        # This can be anything as only the root process\n",
+    "        # Will receive data\n",
+    "        rcvbuf = nothing\n",
+    "    end\n",
+    "    println(\"[rank $rank] I am sending $sndbuf\")\n",
+    "    MPI.Gatherv!(sndbuf, rcvbuf, comm; root)\n",
+    "    if rank == root\n",
+    "        println(\"[rank $rank] I received $data\")\n",
+    "    end\n",
+    "end\n",
+    "run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "25c2fb1e",
+   "metadata": {},
+   "source": [
+    "## How to get the amount of data to be received?\n",
+    "\n",
+    "Remember that `MPI_Probe` allows you to query the amount of data to be received with `MPI_Recv`. Unfortunately, this mechanism does not work for MPI collectives. A workaround is to first communicate the amount of data to be gathered, allocate the receive buffer accordingly, and then communicate the actual message.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "12f9e2ca",
+   "metadata": {},
+   "source": [
+    "### Example\n",
+    "\n",
+    "Each process sends a random amount of integers to rank 0. The root process will not know the amount of data to be gathered from each rank in advance. We need an auxiliary gather to inform about the message size."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b10430a9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "code = quote\n",
+    "    using MPI\n",
+    "    MPI.Init()\n",
+    "    comm = MPI.COMM_WORLD\n",
+    "    rank = MPI.Comm_rank(comm)\n",
+    "    # Get number of ints to send\n",
+    "    count = rand(1:4)\n",
+    "    # Generate data to send\n",
+    "    sndbuf = fill(rank,count)\n",
+    "    # Communicate counts\n",
+    "    root = 0\n",
+    "    if rank == root\n",
+    "        nranks = MPI.Comm_size(comm)\n",
+    "        counts = zeros(Int,nranks)\n",
+    "    else\n",
+    "        counts = nothing\n",
+    "    end\n",
+    "    MPI.Gather!([count], counts, comm; root)\n",
+    "    # Allocate receive buffer\n",
+    "    if rank == root\n",
+    "        ndata = sum(counts)\n",
+    "        data = zeros(Int,ndata)\n",
+    "        rcvbuf = MPI.VBuffer(data,counts)\n",
+    "    else\n",
+    "        rcvbuf = nothing\n",
+    "    end\n",
+    "    # Communicate data\n",
+    "    println(\"[rank $rank] I am sending $sndbuf\")\n",
+    "    MPI.Gatherv!(sndbuf, rcvbuf, comm; root)\n",
+    "    if rank == root\n",
+    "        println(\"[rank $rank] I received $data\")\n",
+    "    end\n",
+    "end\n",
+    "run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8b4254d1",
+   "metadata": {},
+   "source": [
+    "## MPI_Scatter\n",
+    "\n",
+    "This is the inverse operation of `MPI_Gather`. The root rank sends a distinct message to each rank.\n",
+    "\n",
+    "In Julia:\n",
+    "```julia\n",
+    "MPI.Scatter!(sendbuf, recvbuf, comm; root=0)\n",
+    "```\n",
+    "\n",
+    "In C:\n",
+    "```c\n",
+    "int MPI_Scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype,\n",
+    "    void *recvbuf, int recvcount, MPI_Datatype recvtype, int root,\n",
+    "    MPI_Comm comm)\n",
+    "```\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "<b>Note:</b> `MPI_Scatter` sends the same amount of data to each rank. To send a variable amount of data there is `MPI_Scatterv` (`MPI.Scatterv!` in julia), which works similar to `MPI_Gatherv`. \n",
+    "</div>\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e55aa353",
+   "metadata": {},
+   "source": [
+    "### Example\n",
+    "\n",
+    "Rank 0 sends a unique integer to each rank. "
+   ]
+  },
+  {
+   "attachments": {
+    "g13389.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAABFEAAANWCAYAAAA2uxORAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13nGV1ff/x190Ou5SlI10FKQpSBaLYsBB7xW6sUSMhxoJGjRiNIbGXRIk1qD8TuwI2FBsWBAsoICC91wUWtu++f3+cM+yZs3fq3pnvnZnX8/G4j9l7555z33Nn9nvO+dxv6SBJkjTpMg/YGtiq9XU+sBmwANik/vccYDEwF1hUP76g/t5cYMsxvvjAfu4C1gLLgRXAauDu+jlL6q9314+vqJ+3FrgTuKN+zh317c7Gv++AzsoxZpIkSVNAp3QASZI0HWQBsBNwH2BXYBvWF0YGiiTbNP69WZmck2Y5g4oq3AHcCtwIXA/cBFwH3Fx97SwtlFOSJI2BRRRJkjSCLKQqjOxU33apv+5c3+4DbFcsHqwElo1xm1XAvPrfi6h6p5S0DLiBqsjSLrTcAFwBXAmdVaUCSpIkiyiSJAmAbA3sWd/2avz7fsAWPX6xJcBt9e321teVVL02VgNLWT+MZmBYzRJgTf295dBZ0btYGRhCNBvYvH5wC2AWsCnVUKN59fe2bNwWt+43b/PonXVUxZUrhrhdB511PXw9SZLUYhFFkqQZI5uzvjjSLpZstRE7XglcS3WBf3X99Ua6F0lum1kX+tmU9QWVbal68GxH1XtnB2DH+rY91XCnjbEKuIr1RZXLgYuAC6l6sazdyP1LkjTjWUSRJGnayRyqAsn+wAH11/2pht6M1TKqC/Pr6ts19ddrubdw0rm5B6FF5lMVWHZqfN0e2A3Yo77tRNVTZqxWAH+ubxewvrjyF+is3ujokiTNEBZRJEma0rIV8GDWF0r2B/ajWr1mtFYAl9a3vzT+fSl0ru9pXG2kzKWan2aPIW5jnZtmNdXv+kI2KLBYXJEkqc0iiiRJU0a2AQ6vbwdRFUx2GuXGa6kKJJcwqEjCpcA10EnP46qALGR9QWUvYB9g3/rrWJaCXgX8Efhd43Z+b+egkSRp6rGIIklSX8psYG/gYOCvgIdSXQiP5th9J/Anql4FFwK/BX4HnbGuYKNpJYupeint2/h63/o2GmuoinC/rW8XVF87S3qfVZKk/mQRRZKkvpAdqHqYHFF/PYRqRZjhrAEupuoxcF59+yN0rp3AoJp2sg3re6vsCzyQqqfTaHqurKUqrAz0VvkNcK49ViRJ05VFFEmSishuwKOBRwIPo5o8dNgNqC5WzwZ+XX+9ADorJzKlZrLcl6qY0rxtO4oNVwHnAr8EflF9dfJhSdL0YBFFkqRJkW2BRwBHUw3N2XeEDe6m6llyFtWF6K+hc8tEJpRGlvtQDTEbuA0MCRrJDaz/Wz4L+P3MWupakjRdWESRJGlCZAvg4VS9TR5FNQfFUMfddVRzl5wN/Iqqp8lFXmRqasgOVL1UDqEajnYEsMUIG91O9bf+i/p2DnSWT2RKSZJ6wSKKJEk9kTlUE8A+nqpocjAwe6gnU03KeSbwI+Bn0LljMlJKEy+zqIqGDwWOpPp/sccIG62kKqr8qL6dA501E5lSkqTxsIgiSdK4ZSFVweSJwFOA7Yd58uWsH8rwHSd/1cySHYBDWb/S1CHA/GE2uIeqqPLD+vY7l+GWJPUDiyiSJI1J9gKeBDyBakLYOUM88Tqqnib1rXP15OSTpoJsyvqiypHAUcBmw2xwA+t7bv0QOtdMeERJkiRJ0lhlDuQRkPdBLoZkiNsKyPchx0EeUDq1NLVkDuRIyNshP4GsHOb/Wur/i/8FeUY9/5AkSZIkqYzMhzwZ8gXI7cNcyN0E+Wx9ITfcp+iSxiSbQh4H+Q/IbyFrh/l/uAryI8jrIHuWTi5JkiRJM0BmQx4K+TDklmEu2C6rn3N0PZmspAmXzer/cydBzh2hl8plkJMhT4LMK51ckiRJkqaJzIY8qr7gunWIC7LlkO9CXgPZrXRiSQDZFfISyP9B7himoLIE8r+QF0C2KZ1akiRJkqaYzKp7nHwUcsMw85t8C/I8yKLSiSUNJ3PrYuj7IZcMU1BZAzkL8mbIg0qnliRJkqQ+lgPqi6xrhplX4TuQF0O2LJ1W0nhlL8g/Qs6s/18PN+znJMjBpRNLkiRJUh/IFpBX1p8+d7uIWlt/73jIdqXTSuq1LKznRjkZcv0wBZWr6rmOHgrplE4tSZIkSZMknbpr/xcgy4YonPy0nuNk+9JpJU2WzIb8FeTfIBcMU1C5AvJeyGEWVCRJkiRNU9kF8va6i363C6M/Q06A7FQ6qaR+kH0h74D8cZiCypWQ90EeYkFFkiRJ0hSXeZBnUa2es6bLBdBSyGeqLvqSNJTsA/lnyPkjDPl5P+Tw0mklSZIkaQyyG+TfIbcMcbHzC8jLIJuVTippqsnekLdBzhumoHJJ3fNt99JpJUmSJGkIORhyCmR1l4ua2+rJI/cvnVLSdJE96omnh5qcOpBz6+dsXTqtJEmSpBkvCyAvhfyhy8XLGsi3IU+BzC2dVNJ0lr0gb4VcNEQxZRnkS5C/hswpnVaSJEnSjJIdICcOMWTnDqrlSHcvnVLSTJT9ICdBbhqioDLQM875mCRJkiRNpHuH7KzqcmFycd1tfmHplJJEtWzy0XWbdc8QBZUL6oLwbqXTSpIkSZoWMgvy7HpugfYFyFrINyGPKp1SkoaWxZBX1RNbdyumDAw/fGJVfJEkSZKkMclcyIuGmGPgzro7/N6lU0rS2GRXyAmQS4coqFxXDweyd4okSZKkkWQTyN9BruxycXEh5NUO2ZE09aUDeRjkk5C7urR3q+uedsdUPfIkSZIk6V5ZVM9pcl2Xi4nf171S7OYuaRrKAsizGHq55Gvr3im7lk4qSZIkqahsXndtv63LhcNZkCeVTihJkycH18MVlw4xd8oZdcHForIkSZI0c2QHyHu7XCisg5wGObJ0QkkqJ5vXk9H+fojeKZdB3gzZrnRSSZIkSRMmi6mW9GzPAbAWcirkkNIJJam/DNs7ZSXky5DDSqeUJEmS1DPZtB62c3vrAmAV5BRX2pGkkWRzyCuH6Z1ylkN9JEmSpCkt8+rVdm7oUjz5hBMlStJ45AjI5+ueKO1iyqWQ4yCblU4pSZIkaVQyq/5E9NIuc558GbJn6YSSNPVl+3qI5C1diil3QT4M2b10SkmSJElDytFDdDc/A3JQ6XSSNP1kPtVS8Od3aXsH5pw6unRKSZIkSffKwyG/7HIC/3PIQ0unk6TpLx3IYyCn1z3/2u3xOZDnV0MtJUmSJBWQPSBf73Ky/nvIX5dOJ0kzUx4A+S/I3V3a52shr4MsKp1SkiRJmiGyaT0Wf3nr5PyKegUJV4iQpOKyOeR4yJVdiim31u34VqVTSpIkSdNUOpDnQq7pcjL+asjc0gklSW2ZAzkWcvYQk9C+F7Jj6ZSSJEnSNJIDIT9rnXyvhpwM2aZ0OknSaOSh9WSz7WLKSsgpkL1KJ5QkSZKmsGxVL5W5pnXCfSZk/9LpJEnjkQfXRZN22z6woo8rqkmSJEmjlzn1/Ca3tE6wr6mW05QkTX25f92jcHWrrV9XF1OOKJ1QkiRJ6nN5GOSC1gn13ZC3QRaUTidJ6rXsUa/o054wPJAfQR5dOqEkSZLUZ7JFPXRnbesE+lTIbqXTSZImWrarV+1Z0qWY8kuLKZIkSRIAeRrkutYJ8++rSQglSTNLFkPe3mVIZyBnOMxHkiRJM1TuA/la6wR5GeRN1bwokqSZKwsh/wi5qUsx5TQnoJUkSdIMkU41QWxua50U/xTygNLpJEn9JJtCjofcOETPlANLJ5QkSZImSO5fTxTYPAleUq/G0ymdTpLUr7IIckKXOVPWQb5sEV6SJEnTSObUJ7/t1RdOhexUOp0kaarIZvXx5I7W8WRtXUzZs3RCSZIkaSNkf8h5rZPdqyBPKJ1MkjRVZVvI++u5tJrHl1WQkyG7lE4oSZIkjUFmQV4PWdH6pPDD1SeJkiRtrNwH8tHWsSb1/fdXq/1IkiRJfS27QM5sndBeBDm8dDJJ0nSUXSGfrHuiNI89t9Wr/MwvnVCSJEnqIs9tTfy3DvIxyCalk0mSprvcD3JK3fOxWUy5HHKsk5hLkiSpT2SL+sS1edJ6I+SJpZNJkmaa7As5rcuyyL+BPLx0OkmSJM1oORJyWetE9WuQbUonkyTNZDka8ocuxZQzIA8snU6SJEkzSuZCTmx1m74L8srSySRJqmQW5FmQK1uFlNX1Sj47lE4oSZKkaS/7dlm6+CzIHqWTSZK0oSyE/DNkaevYdSfkn5y7S5IkSRMkz2+dhK6qT0Bnl04mSdLwsgPkE3VPlGYx5RrIi6ueK5IkSdJGy3zIx7ssXXxw6WSSJI1N9oWc2mW+lHMgDymdTpIkSVNadoX8unWi+fVqVR5JkqaqHAH5Rev4tq5ecW770ukkSZI05eTJkCWNk8sVkNeUTiVJUm+kA3kB5NpWMWUJ5HjInNIJJUmS1Pcyp8vqO1dBDi+dTJKk3sum9XFvRauY8mfIY0unkyRJUt/KdpAftk4iT4VsVTqZJEkTK3tCTu8yX8qpkN1Kp5MkSVJfySMg1zdOGtfUn8y5YoEkaQbJkyCXtwop99THxPml00mSJKm4vKkumgycLF4POap0KkmSysgmddFkWauYcgnkmNLpJEmSVETmQz7bOkH8CWTH0skkSSovO9cr9nQb4nPf0ukkSZI0abId5KzWSeG/QWaXTiZJUn/J4+qJZpvHzOWQt0PmlU4nSZKkCZUHQa5oLV/84tKpJEnqX5lbL318Z5chPo8snU6SJEkTIk+GLG2c/N0IOaJ0KkmSpobsBPl/rULKWsh/QbYonU6SJEk9k+PrE72Bk77zIbuXTiVJ0tSTR0AubhVTboC8qHQySZIkbZSuE8ieDtm8dDJJkqauLine truncated
+    }
+   },
+   "cell_type": "markdown",
+   "id": "74d5b606",
+   "metadata": {},
+   "source": [
+    "<div>\n",
+    "<img src=\"attachment:g13389.png\"  align=\"left\" width=\"350\"/>\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5ea413fd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "code = quote\n",
+    "    using MPI\n",
+    "    MPI.Init()\n",
+    "    comm = MPI.COMM_WORLD\n",
+    "    nranks = MPI.Comm_size(comm)\n",
+    "    rank = MPI.Comm_rank(comm)\n",
+    "    root = 0\n",
+    "    rcv = Ref(0) \n",
+    "    if rank == root\n",
+    "        snd = [10*(i+1) for i in 1:nranks]\n",
+    "        println(\"[rank $rank] I am sending: $snd\")\n",
+    "    else\n",
+    "        # Only the root rank sends data\n",
+    "        # so this can be anything\n",
+    "        snd = nothing\n",
+    "    end    \n",
+    "    MPI.Scatter!(snd,rcv,comm;root)\n",
+    "    println(\"[rank $rank] I have received: $(rcv[])\")\n",
+    "end\n",
+    "run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ed8da7f9",
+   "metadata": {},
+   "source": [
+    "## MPI_Bcast\n",
+    "\n",
+    "Similar to `MPI_Scatter`, but we send the same message to all processes.\n",
+    "\n",
+    "In Julia:\n",
+    "```julia\n",
+    "MPI.Bcast!(buf, comm; root)\n",
+    "```\n",
+    "\n",
+    "In C:\n",
+    "```c\n",
+    "int MPI_Bcast(void *buffer, int count, MPI_Datatype datatype,\n",
+    "    int root, MPI_Comm comm)\n",
+    "```\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "97369713",
+   "metadata": {},
+   "source": [
+    "### Example\n",
+    "\n",
+    "Rank 0 sends the same integer to all processes."
+   ]
+  },
+  {
+   "attachments": {
+    "g1657.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAAA9MAAANWCAYAAAD9cN8uAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N13nC1lYf/xz7n9ApdeLyACgghIUbEhVowVOxqN0RhLjJpgDfYYuybGoPGn2BLRWIgVCyq2SBGlKCBNFOm9t9vv9/fHzMLs7Nm9Z8+W5+zu5/16zWvvPbsz892zu8/MM0/rIElSUekAWwJbNT5uBSwBNgcWAJvW/18KbAIsrD+3EFjW+Nyy+uvHYwtgBbASWAvcXr9+c/3xDmBN/fkVwDrgtvpzt9bbLaNvnTvHmUeSJM0AndIBJEmzTeYB2wM7ATsCOzCystysNG/J7L4erWFkhft64DrgGuAq4Nr643XQubZQTkmSNA6z+eZFkjTpsoiqcrwT91SWd64/7gjci6oiPd7W4cmynqriOh6hapFeCCwGNprsUOO0hqqifRVVZfvqersWuBK4BPgzdG4f7QCSJGnqWZmWJLVkCbBHl+0+VBXpybx23AncBNxYbzc0/n9H/fnVVBXkNVTdq1cBd3FP9+tbqo+TWbnMfKqu5QCbAfOoKtmLgUXAxlTvw+b15zfvYdt48vIB1Xv1Z+6uXA/bLoXOqkk+nyRJarAyLUlzUhYCuwJ71luz0rwz/V8f1lO1pl5B1bJ6GVWL6vXcU2G+6Z6PnZX9fw8zTYbGeW8ObANsR9Wav239cTuqhxXb1/+eP4GTrad6/4cq1xcDFwLnAxdY0ZYkaeKsTEvSrJcdgf3qbf/6430Zf1fstVTdjK9geGX5ynq7HLgaOmsnJ/dclnlUlex2hXs51UOQoW1xHwdfR1XBPpeqcn0+cB5VJfuOCUeXJGmOsDItSbNGFgP7cE+FeajyvNU4DrKeqoJ8UWP7Q/3xEuismcTAmpB0qFqyd2N4BXto25HxtW4HuBS4gKqiPfTx947PliRpJCvTkjQjZTHwAOChwEGMv7X5aqrK0kWt7U92AZ4tsohqQrhdgd2BvYH71duO4zjQeqrfjTMb22+hc/OYe0mSNMtZmZakGSHLgQcCBwOPqP+9pIcd11BVhM6gamU8DzgNOtdMUVDNCNmMakK5fagq2bvV/96LarK1XlxN9Xs1tPl7JUmaU6xMS9LAyUZUleWHAg+rP+7Qw45XAOcAZ9XbOcCFjmFW77IxVYX6ftxT0T6QalK6XlzOPa3XpwO/sgVbkjRbWZmWpOKyDHgk8FjgUVTjnDfUXfsG4NR6+zVwJnRumsqUmsuyDdWwgua2Ww87rqea4OzkejsFOn+cqpSSJE0nK9OSNO2ygKrCfGi9PZJq7eLRrKNa1ugM4CSqSsl50MkUB5XGkE2pxuo/sLH10k38WuA07vl9PmluLZEmSZotrExL0pTLAuBBVC3PjwUeDiwdY4eruafV+VTgdOjcNdUppYnLMqpu4Q+gGqJwMBue7GwFVZfwodbrX0HnxqlMKUnSZLAyLUlTIvcBngI8jqrr9qZjfPG1wM+Bn1Vb509Tn0+aLtmFqlL9cKrJ8/Zl7CW71gNnAz8BfgqcCJ07pzqlJEnjZWVakiZF5gMHAIcBT6Xq8jqaO6hanH9Sb2faZVtzRzah+lsZmpn+YGCLMXZYSzWh3tDfyy+hs3qqU0qStCFWpiWpb9kceAJVBfpJwJajfOFdVGND65ZnzoTOummJKA28zKcaez3Uev1oxp69/nbg/6harX8K/N6HUZIkSdLAy30gr4P8FLIaklG2cyAfgDwSMtbkYpJGyN6Qf4B8G3LLGH9ngVwD+R/I30J2Kp1ckiRJ0t2yP+T9kPPHuKFfBflxXQHYtXRiafbIfMhDIG+tH2KtGOPvcD3kTMh7IA+GbGhmcUmSJEmTK3tD3rWBCvQNkGMhL4JsVjqxNDdkKeTQ+u/zhA30ELnev1FJkiRpymX3uvXrrDFuzs+uW6kfZquXNAiyBeRZkE9BLh/jb3cl5Ed175HdSqeWJEmSZrjcC/JGyGlj3IT/DvIWb8ClmSAHQt4OORWyboy/63MhH4IcUnUllyRJkrQB2RLyGshJ9RjL0W603wm5b+m0kvqVbSEvgXwdctsGhmx8BvIXkAWlU0uSJEkDJPMgj4AcDblzlBvqSyBHVV8naXbJgroM+OAG5kK4CXIM5DCciV+SJElzV+5dT1R06RgV6A9DHlA4qKRplb0h/wQ5cYzu4DdCPgd5EmRh6cSSJEnSFMsSyPOpZvrtdpN8K+TTkIdDOqXTSiotO9RDP/5vjIr1TZDPQ55si7UkSZJmmRwI+Xh909tt/dlfQl4M2bh0UkmDKjtAXg35xQYq1v8NeYot1pIkSZqhsqSeYOiMUW56r4R8ALJn6aSSZppsD3kV5OeQtaOUMdfVcy08qHRaSZIkqQfZvh4LfV2Xm9u1dRfvw201kjQ5shXkRZDvQlaPUrE+ry6Xdi2dVpIkSWrJwyBfGeVm9jzIGyDblk4paTbLdpC/rycv67a83rr6gd6LIJuUTitJkqQ5K4vqVuaTx7hpPczJxCRNv+wMORLyh1Faq++CHFuXUa5hLUmSpOmQ7eqb1Mu73KDeRrVm9P1Kp5SkSh5Yj5++fow5HI6CHFA6qSRJkmal7A35AmRVl5vRC6hm2rXrpKQBlSV1b5rjxhhffSrV5IkblU4rSZKkGS/7Q47pMmvuertyS5qZsgXkFZCTRqlU31r3srG1WpIkSeOVQyDHj9KV+2O4rJWkWSH7QD5Yd/fuVrE+uZ60bGnppJIkSRpoeUTd4ty+obyBanmZLUsnlKTJl3mQQ+uJybqtX31L3Vq9X+mkkiRJGhjp1N21f93lBvLauhK9WemUkjQ9smM90eJlo7RWn153E3dstSRJ0tyUBZAXQs7tcrN4MeSVkMWlU0pSGVkIeQ7kx3Rfu/oGyEcc9iJJkjRn3N0SfXaXm8M/1i0urr0qSXfL7vXY6mu7lJtOyChJkjT75VDIaV1uBn9XT7Izv3RCSRpcWUS1xNYJo7RWXwg5wi7gkiRJs0YeCvlZlxu/30CeWDqdJM082bPu5n1Ll7L1esh7ITuUTilJkqS+5H717LTtFpTz69YVuyRK0oRkk3p4zHldKtWr6jL4oaVTSpIkqSfZuV7Gpb3Ey2WOiZakqXD38lrfHaUL+On1cBrLX0mSpMGTbSEfq1tDmjdx19Xj+JydW5KmXA6A/Bdk5SirJbwOsmnplJIkSaomxXkj5NbWTdutkHdClpVOKElzT7aDvGuUWcBvhXwIsn3plJIkSXNUDoX8vss4vaOrGzlJUll3zwL+61HGVR8D2aN0SkmSpDkie0C+17opW1/flO1UOp0kqZs8dpRx1Wvq8nvv0gklSZJmqWxcdxtsj8U7A3Jw6XSSpF5k37ryvKbLQ9HvOgO4JEnSpEmn7iZ4WevG64Z6crH5pRNKksYr94YcBbmrSxfwkyCHlU4oSZI0g+WBkJNbN1mr6xuwzUqnkyRNVLatex3d0qVSfWb9MLVTOqUkSdIMkS0hn4Osa91Y/QBy39LpJEmTLVvWleobu1Sqz4a8wJ5IkiRJY8phkCtaN1IXVa0TkqTZLRvXQ3gu71KpvhjyCivVkiRJw2TnevKZ5o3THZB/qpZXkSTNHVkM+TvIn0ZpqX6G3b8lSdIcl3mQf4Dc1qVL9y6l00mSSsoCyF9BLuhSqT4N8qTSCSVJkgrIvpBTWjdHN1Xd+CRJGpJ59WRkF3WpVP/K2b8lSdIckYWQIxm5ZvSxkG1Kp5MkDaoshLxolO7fJ0EeUzqhJEnSFMnBkHO7TCrzhNLJJEkzRRbVk5Fd2aVSfQLkoNIJJUmSJkmWQj4GWd+44VkL+TfIRqXTSZJmomwEeRPk+laFej3k25D9SieUJEmagDwAcl7rRue3kAeWTiZJmg2yDPJ2yM2ta806yP9A7l06oSRJ0jhkPuTNkFWNG5uV9WsLSqeTJM022Rzyni4rRKyEfLj6vCRJ0kDLvSG/bN3M/B6yf+lkkqTZLltD/hVyV+s6dAPktdWYa0mSpIGTw+slrppj1452bLQkaXplp/r6s7ZVqb60nhW8UzqhJEkSkM0gX2rdsFwNeXLpZJKkuSz7QL7fZebv30AeWTqdJEma0/I4yOWtm5T/hWxVOpkkSZUcCjmrS6X6u5A9SqeTJElzShZBPtpa8uoWyAtLJ5MkaaTMh7wcclWrQr0aclQ13lqSJGlK5V6QU1s3I7+A7FI6mSRJY8smkHdB7mhdx26GHAlZUjqhJEmalfKkelbUoZuPVZB/gswrnUySpN5lB8hnukxSdjHk6aXTSZKkWSPz6yf56xo3HJdDHl46mSRJ/ctekGO7jKf+CWTv0ukkSdKMlu0gP23dZBzv+DJJ0uyRQyHntK51qyAfqLqGS5IkjUsOgVzZWjv6g3brliTNPlkAOaKeULNZqb7S9aklSVKP0qlvKFY3biauh/xF6WSSJE2tbFXP8L2OkZNt7lc6nSRJGljZFPL11g3ELyHLSyeTJGn65EGQX7Wuh2sgR1cVbkmSpLtlX8ifWt26P1x1fZMkaa7JPMjftVayCOQayEvs+i1JkoA8sTVO7FbIc0qnkiSpvGxRd/1uL6V1OuShpdNJkqRi8obWDcJZkN1Kp5IkabDkQMjJrQr1Osj/g2xWOp0kSZo2WQT5XOum4AfeEEiSNJp0IIdDLmtdP6+uZv2WJEmzXLauJxZr3gh82GWvJEnqRTaF/Hs9KVnzWvptyI6l00mSpCmRPSEXNi78qyB/UzqVJEkzT/aH/LpVob4DciRkful0kiRp0uQJrYnGboA8unQqSZJmrsyDvAJyW6tSfQbkAaXTSZKkCcsrWt3RzoHsWjqVJEmzQ3aEfKtVoV4N+SBkSel0kiRp3LIA8unWxf27kGWlk0mSNPvk+ZBrW9fdC+0JJmmyuMi9NC2yLine truncated
+    }
+   },
+   "cell_type": "markdown",
+   "id": "26b56c81",
+   "metadata": {},
+   "source": [
+    "<div>\n",
+    "<img src=\"attachment:g1657.png\"  align=\"left\" width=\"350\"/>\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4de15781",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "code = quote\n",
+    "    using MPI\n",
+    "    MPI.Init()\n",
+    "    comm = MPI.COMM_WORLD\n",
+    "    nranks = MPI.Comm_size(comm)\n",
+    "    rank = MPI.Comm_rank(comm)\n",
+    "    root = 0\n",
+    "    buffer = Ref(0)\n",
+    "    if rank == root\n",
+    "        buffer[] = 20\n",
+    "        println(\"[rank $rank] I am sending: $(buffer[])\")\n",
+    "    end    \n",
+    "    MPI.Bcast!(buffer,comm;root)\n",
+    "    println(\"[rank $rank] I have received: $(buffer[])\")\n",
+    "end\n",
+    "run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e83be96d",
+   "metadata": {},
+   "source": [
+    "## Communicators\n",
+    "\n",
+    "A key component in MPI programming are so-called communicators. A communicator object has to main purposes:\n",
+    "\n",
+    "\n",
+    "1. To provide an isolated communication context\n",
+    "2. To define a group of processes\n",
+    "\n",
+    "\n",
+    "### Communication context\n",
+    "\n",
+    "When you write an MPI program it is very likely that you are going to use libraries that also use MPI to send messages. Ideally, these libraries should not interfere with application messages. Using tags to isolate the messages send by your application does not solve the problem. MPI communicators fix this problem as they provided an isolated communication context. For instance, `MPI_SEND` and `MPI_RECV` specify a communicator. `MPI_RECV` can only receive messages sent to same communicator. The same is also true for collective communication directives. If two libraries use different communicators, their message will never interfere. In particular it is recommended to never use the default communicator, `MPI_COMM_WORLD`, directly when working with other libraries. A new isolated communicator can be created with `MPI_Comm_dup`.\n",
+    "\n",
+    "\n",
+    "### Groups of processes\n",
+    "\n",
+    "On the other hand,  imagine that we want to use an MPI communication directive like `MPI_Gather`, but we only want a subset of the processes to participate in the operation. So far, we have used always the default communication `MPI_COMM_WORLD`, which represents all processes. Thus, by using this communicator, we are including all processes in the operation. We can create other communicators that contain only a subset of processes. To this end, we can use function `MPI_Comm_split`.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "556ca223",
+   "metadata": {},
+   "source": [
+    "## MPI_Comm_dup\n",
+    "\n",
+    "Duplicate a given communicator, creating a new one with an new communication context.\n",
+    "\n",
+    "\n",
+    "In Julia:\n",
+    "```julia\n",
+    "newcomm = MPI.Comm_dup(comm)\n",
+    "```\n",
+    "\n",
+    "In C:\n",
+    "```c\n",
+    "int MPI_Comm_dup(MPI_Comm comm, MPI_Comm *newcomm)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9acfb792",
+   "metadata": {},
+   "source": [
+    "### Example\n",
+    "\n",
+    "Send value 2 from rank 2 to rank 3. Then send value 1 also from rank 2 to rank 3. Note that we are using 2 different communicators for each send receive pair in such a way that we first receive value 1 and then value 2, even if this not was the other in which the data was sent. Try also to define `comm2 = comm` and see how the result will change."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e50d44f6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "code = quote\n",
+    "    using MPI\n",
+    "    MPI.Init()\n",
+    "    comm = MPI.COMM_WORLD\n",
+    "    # We create a new communicator\n",
+    "    comm2 = MPI.Comm_dup(comm)\n",
+    "    rank = MPI.Comm_rank(comm)\n",
+    "    if rank == 2\n",
+    "        sndbuf = [2]\n",
+    "        req1 = MPI.Isend(sndbuf, comm2; dest=3, tag=0)\n",
+    "        sndbuf = [1]\n",
+    "        req2 = MPI.Isend(sndbuf, comm; dest=3, tag=0)\n",
+    "        MPI.Wait(req2)\n",
+    "        MPI.Wait(req1)\n",
+    "    end\n",
+    "    if rank == 3\n",
+    "        rcvbuf = zeros(Int,1)\n",
+    "        MPI.Recv!(rcvbuf, comm; source=2, tag=0)\n",
+    "        @show rcvbuf\n",
+    "        MPI.Recv!(rcvbuf, comm2; source=2, tag=0)\n",
+    "        @show rcvbuf\n",
+    "    end\n",
+    "end\n",
+    "run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5e1b13eb",
+   "metadata": {},
+   "source": [
+    "## MPI_Comm_split\n",
+    "\n",
+    "Split a given communicator.\n",
+    "\n",
+    "\n",
+    "In Julia:\n",
+    "\n",
+    "```julia\n",
+    "newcomm = MPI.Comm_split(comm, color, key)\n",
+    "```\n",
+    "\n",
+    "In C:\n",
+    "```c\n",
+    "int MPI_Comm_split(MPI_Comm comm, int color, int key,\n",
+    "    MPI_Comm *newcomm)\n",
+    "```\n",
+    "\n",
+    "There are two key parameters:\n",
+    "\n",
+    "- `color`: all processes with the same color will be grouped in the same new communicator.\n",
+    "- `key`: The processes will be ranked in the new communicator according to key, breaking ties with the rank in the old communicator. \n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2b128840",
+   "metadata": {},
+   "source": [
+    "### Example\n",
+    "\n",
+    "Create two groups of processes, one for processes with rank id even, and another for the rest. Note that the rank id will be function of the given communicator."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b18cb74d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "code = quote\n",
+    "    using MPI\n",
+    "    MPI.Init()\n",
+    "    comm = MPI.COMM_WORLD\n",
+    "    rank = MPI.Comm_rank(comm)\n",
+    "    color = mod(rank,2)\n",
+    "    key = 0\n",
+    "    newcomm = MPI.Comm_split(comm, color, key)\n",
+    "    newrank = MPI.Comm_rank(newcomm)\n",
+    "    println(\"Hello, I am rank $rank in world and rank $newrank in group $color.\")\n",
+    "end\n",
+    "run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b22e4bf4",
+   "metadata": {},
+   "source": [
+    "### Example\n",
+    "\n",
+    "Processes with odd rank id generate a random integer. Sum all these integers and send them to the first process with odd rank id. Note that we are using a new communicator to perform the reduction."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4c0e504b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "code = quote\n",
+    "    using MPI\n",
+    "    MPI.Init()\n",
+    "    comm = MPI.COMM_WORLD\n",
+    "    rank = MPI.Comm_rank(comm)\n",
+    "    color = mod(rank,2)\n",
+    "    key = 0\n",
+    "    newcomm = MPI.Comm_split(comm, color, key)\n",
+    "    # Only odd rank ids will execute this\n",
+    "    if color == 1\n",
+    "        newrank = MPI.Comm_rank(newcomm)\n",
+    "        newroot = 0\n",
+    "        sndbuf = [rand(1:4)]\n",
+    "        if newrank == newroot\n",
+    "            rcvbuf = [0]\n",
+    "        else\n",
+    "            rcvbuf = nothing\n",
+    "        end\n",
+    "        println(\"[rank $rank, newrank $newrank] I am contributing with $sndbuf\")\n",
+    "        MPI.Reduce!(sndbuf, rcvbuf, +, newcomm;root=newroot)\n",
+    "        println(\"[rank $rank, newrank $newrank] The sum is $rcvbuf\")\n",
+    "    end\n",
+    "end\n",
+    "run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d465ebce",
+   "metadata": {},
+   "source": [
+    "Try to run the code without splitting the communicator. I.e., replace `newcomm = MPI.Comm_split(comm, color, key)` with `newcomm = comm`. Try to figure out what will happen before executing the code."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d334aea1",
+   "metadata": {},
+   "source": [
+    "## Conclusion\n",
+    "\n",
+    "- MPI also defines operations involving several processes called, collective operations.\n",
+    "- These are provided both for convenience and performance.\n",
+    "- The semantics are equivalent to \"standard mode\" `MPI_Send`, but there are also non-blocking versions (not discussed in this notebook).\n",
+    "- Discovering message sizes is often done by communicating the message size, instead of using `MPI_Probe`.\n",
+    "- Finally, we discussed MPI communicators. They provide two key features: isolated communication context and creating groups of processes. They are useful, for instance, to combine different libraries using MPI in the same application, and to use collective operations in a subset of the processes.\n",
+    "\n",
+    "After learning this material and the previous MPI notebook, you have a solid basis to start implementing sophisticated parallel algorithms using MPI."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "843b40cd",
+   "metadata": {},
+   "source": [
+    "## Exercises"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5c2045d9",
+   "metadata": {},
+   "source": [
+    "### Exercise 1\n",
+    "\n",
+    "Implement the parallel matrix-matrix multiplication (Algorithm 3) using MPI collectives instead of point-to-point communication. I.e., this is the same exercise as in previous notebook, but using different functions for communication."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5e8f6e6a",
+   "metadata": {},
+   "source": [
+    "# License\n",
+    "\n",
+    "This notebook is part of the course [Programming Large Scale Parallel Systems](https://www.francescverdugo.com/XM_40017) at Vrije Universiteit Amsterdam and may be used under a [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) license."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Julia 1.11.6",
+   "language": "julia",
+   "name": "julia-1.11"
+  },
+  "language_info": {
+   "file_extension": ".jl",
+   "mimetype": "application/julia",
+   "name": "julia",
+   "version": "1.11.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

notebooks/mpi_tutorial.ipynb

@@ -1036,7 +1036,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Julia 1.9.1",
+   "display_name": "Julia 1.9.0",
    "language": "julia",
    "name": "julia-1.9"
   },
@@ -1044,7 +1044,7 @@
    "file_extension": ".jl",
    "mimetype": "application/julia",
    "name": "julia",
-   "version": "1.9.1"
+   "version": "1.9.0"
   }
  },
  "nbformat": 4,

notebooks/pdes.ipynb

@@ -1534,7 +1534,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Julia 1.9.1",
+   "display_name": "Julia 1.9.0",
    "language": "julia",
    "name": "julia-1.9"
   },
@@ -1542,7 +1542,7 @@
    "file_extension": ".jl",
    "mimetype": "application/julia",
    "name": "julia",
-   "version": "1.9.1"
+   "version": "1.9.0"
   }
  },
  "nbformat": 4,

notebooks/tsp.ipynb

@@ -22,8 +22,8 @@
     "In this notebook, we will learn\n",
     "\n",
     "- How to parallelize the solution of the traveling sales person problem\n",
-    "- How to fix dynamic load imbalance\n",
-    "- The concept of search overhead\n"
+    "- The concept of search overhead\n",
+    "- A dynamic load balancing method\n"
    ]
   },
   {
@@ -51,9 +51,17 @@
     "        \"It's not correct. Keep trying! 💪\"\n",
     "    end |> println\n",
     "end\n",
-    "tsp_check_2(answer) = answer_checker(answer, 2)\n",
+    "tsp_check_2(answer) = answer_checker(answer, 4)\n",
     "tsp_check_3(answer) = answer_checker(answer, \"d\")\n",
-    "tsp_check_4(answer) = answer_checker(answer, \"a\")"
+    "tsp_check_4(answer) = answer_checker(answer, \"a\")\n",
+    "function q_superlinear_answer(bool)\n",
+    "    bool || return\n",
+    "    msg = \"\"\"\n",
+    "    Negative search overhead can explain the superlinear speedup in this algorithm. The optimal speedup (speedup equal to the numer of processors) assumes that the work done in the sequental and parallel algorithm is the same. If the parallel code does less work, it is possible to go beyond the optimal speedup. Cache effects are not likely to have a positive impact here. Even large search spaces can be represented with rather small distance matrices. Moreover, we are not partitioning the distance matrix.\n",
+    "    \"\"\"\n",
+    "    println(msg)\n",
+    "end\n",
+    "println(\"🥳 Well done!\")"
    ]
   },
   {
@@ -64,9 +72,16 @@
     "## The traveling sales person (TSP) problem\n",
     "\n",
     "\n",
+    "In this notebook, we will study another algorithm that works with graphs, the [traveling sales person (TSP) problem](https://en.wikipedia.org/wiki/Travelling_salesman_problem). The classical formulation of this problem is as follows (quoted from Wikipedia) \"Given a list of cities and the distances between each pair of cities, what is the shortest possible route that visits each city exactly once?\" This problem has applications in combinatorial optimization, theoretical computer science, and operations research. It is a very expensive problem to solve (NP-hard problem), which makes parallel computing often required.\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "<b>Note:</b> There are two key variations of this problem. One in which the sales person returns to the initial city, and another in which the sales person does not return to the initial city. We will consider the second variant for simplicity.\n",
+    "</div>\n",
+    "\n",
+    "\n",
     "### Problem statement\n",
     "\n",
-    "Given a graph $G$ with a distance table $C$ and an initial node (i.e. a city) in the graph, compute the shortest route that visits all cities exactly once, without returning to the initial city."
+    "Our version of the TSP problem can be formalized as follows. Given a graph $G$ with a distance table $C$ and an initial node  in the graph, compute the shortest route that visits all nodes exactly once, without returning to the initial node. The nodes on the graph can be interpreted as the \"cities\", and the solution is the optimal route for the traveling salesperson to visit all cities. The following figure shows a simple TSP problem and its solution."
    ]
   },
   {
@@ -86,14 +101,12 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c303dddf",
+   "id": "fd4f87fc",
    "metadata": {},
    "source": [
     "### Sequential algorithm (branch and bound)\n",
     "\n",
-    "The sequential algorithm finds a shortest path by traversing the paths tree of the problem. The root of this tree is the initial city. The children of each node in the graph are the neighbour cities that have not been visited on the path so far. When all neighbour cities are already visited, the city becomes a leaf node in the tree.\n",
-    "\n",
-    "The possile solutions of the problem are the paths from the root of the tree to a leaf node. Note that we assume the children are sorted using the **nearest city first heuristic**. This allows to quickly find a minimum bound for the distance which will be used to prune the remaining paths (see next section). "
+    "A well known method to solve this problem is based on a [branch and bound](https://en.wikipedia.org/wiki/Branch_and_bound) strategy. It consists in organizing all possible routes in a tree-like structure (this is the \"branch\" part from the branch and bound strategy). The root of this tree is the initial city. The children of each node in the graph are the neighbor cities that have not been visited in the path so far. When all neighbor cities are already visited, the city becomes a leaf node in the tree. See figure below for the tree associated with our TSP problem example. The TSP problem consists now in finding which is the \"shortest\" branch in this tree. The tree data structure is just a convenient way of organizing all possible routes in order to search for the shortest one checking one by one, from left to right, using a [depth-first search](https://en.wikipedia.org/wiki/Depth-first_search). We refer to it as the *search tree* or the *search space*."
    ]
   },
   {
@@ -113,10 +126,12 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9da5f5ae",
+   "id": "c303dddf",
    "metadata": {},
    "source": [
-    "Of course, visiting all paths in the tree is impractical for moderate and large numbers of cities. The number of possible paths might be up to $O(N!)$. Therefore, an essential part of the algorithm is to bound the search by remembering the current minimum distance. "
+    "### Nearest city first heuristic\n",
+    "\n",
+    "When building the search tree we are free to choose any order when defining the children of a node. A clever order is using the *nearest city first heuristic*. I.e., we sort the children according to how far they are from the current node, in ascending order. This heuristic increases the chances that the the optimal route is located at the left of the tree, which will allow us to speed-up the search using pruning (next section). The figure above used the nearest city first heuristic. In blue you can see the distance between cities. The first child is always the one with the shortest distance."
    ]
   },
   {
@@ -126,9 +141,8 @@
    "source": [
     "### Pruning the search tree\n",
     "\n",
-    "The algorithm keeps track of the best solution of all paths visited so far. This allows to skip searching paths that already exceed this value. \n",
-    "\n",
-    "For example, in the following graph only 3 out of 6 possible routes need to be visted when we cut off the search after the minimum distance is exceeded. (The grey nodes are the ones we don't visit because the minimum distance had been exceeded at the previous node already.)"
+    "The basic idea of the algorithm is to loop over all possible routes (all branches in the search tree) and find find the one with the shortest distance. One can optimize this process by \"pruning\" the search tree. We visit all possible routes in the tree from left to right. In this process, we keep track of the shortest route visited so far. While visiting a new node in a route, we check if the distance traveled in this route is already larger than the shortest route traversed so far. If this is the case, we can skip all routes that continue from this city as we now for sure that a route will not be in this part of the tree. This is called \"pruning\" since we are cutting branches of the tree, and is the \"bound\" part of the branch and bound strategy.  \n",
+    "For example, in the following graph, only 3 out of 6 possible routes need to be fully traversed to find the shortest route. In particular, we do not need to fully traverse the second branch/route (figure below left). When visiting the third city in this branch, the current distance is already equal to the full previous route. It means that the solution will not be in this part of the tree.  In figure below (right), the gray nodes are the ones we do not visit because the minimum distance had been exceeded before completing the route."
    ]
   },
   {
@@ -148,37 +162,14 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f3c78a1d",
+   "id": "9da5f5ae",
    "metadata": {},
    "source": [
-    "Note that it is not necessary that the graph be fully connected. Variations of this algorithm work for sparse graphs or directed graphs as well. \n",
+    "###  Computation complexity\n",
     "\n",
-    "In the previous example, the shortest route was also the leftmost path in the graph. Although it is more likely that the shortest route be found in the left part of the graph when using the nearest city first heuristic, the solution can be anywhere in the search tree."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "85d771de",
-   "metadata": {},
-   "source": [
-    "<div class=\"alert alert-block alert-success\">\n",
-    "<b> Example:</b> Look at the following graph and its corresponding search tree. If $x\\leq 15$, the shortest route is the leftmost branch of the search tree. If $x > 16$, the route is situated on the right side of the search tree. \n",
-    "</div>"
-   ]
-  },
-  {
-   "attachments": {
-    "fig-tsp-question-1.png": {
-     "image/png": "iVBORw0KGgoAAAANSUhEUgAACQQAAAJCCAYAAACcIlijAAAACXBIWXMAAB7CAAAewgFu0HU+AAAAGXRFWHRTb2Z0d2FyZQB3d3cuaW5rc2NhcGUub3Jnm+48GgAAIABJREFUeJzs3XeYnVW1+PHvOjNJCL0mEyJSFFBEUbGRhBJIQtGr/iwI0rwWmvWqeMWLF5Frv16v/doRUARUVARJIYmkAYK9ASoqJQUIEBJSZuas3x9n0DPvnJlMOTNnyvfzPPPIu/Z+914H+eM9511778hMJEmSJEmSJEkaziI4Hri4TsPdm8nL6zSWJEnSmBLBVOCVwJHAwcBuwK7Aw8Ba4A7gp8C1mfyxDvPtApwCHNcx3yRgAvAQ8BdgCfCdTG4f6FzSaBIWBEmSJEmSJEmShrsI3g78b52G+3MmT63TWJIkSWNCBFOADwOvBcb34pYy8CPgvZnc0Y/5moB3AxcA2/filuuBN2fy177OJY1GpUYnIEmSJEmSJElSL+ze6AQkSZLGqghmAb8EXkfvioGgUo/wcuDWiL7tzhjBROBa4KP0rhgI4ATgtgim92UuabRqbnQCkiRJkiRJkiT1ggVBkiRJDRDB0cCPqRzTVfR34BfAY8BOwCHAkwt9dgSujuAlmcztxXwBfAc4vkbzPcCvgfXAvsDz6LwRym7AdRFMz+R3W5tLGs0sCJIkSZIkSZIkjQTFgqDFwBX9HGvdwFKRJEkaGyLYB7iGrsVAtwLvzmRJjXteAHwCOKIq3AxcGcEzMrlvK9OeC7y0EFsLnA18L5Ny1Vz7A5+mc/HQTsC3Izg0k7atzCWNWpGZjc5BkiRJkiRJkqQeRbAYOLIq9F+ZvL9B6UiSJI0JEXwXeGUh/B3gjEy29HBfM3A58JpC01cyObOH+7YF7gYmVYUfBqZl8sdu7mnqmOukQtPrMvlmd3NJo11p610kSZIkSZIkSWq4PQrXDzUkC0mSpDEigmfQtRjoN2ylGAigY2ee10OX3YBeF8FOPdx6Fp2LgQD+rbtioI652qnsHnRPoel9HcVC0phkQZAkSZIkSZIkaSQoHhn2YEOykCRJGjtOrhH7wNaKgZ6QyePAFwvhccCsHm57W+H6TuDSXsz1KJVjyqodABy3tXul0cqCIEmSJEmSJEnSsBZBCdi1ELYgSJIkaXAVi2nWA9f3cYxFNWLPrtUxgoOAfQrhr2WSvZzrW8CmQuyEXt4rjToWBEmSJEmSJEmShrtdgOZCzIIgSZKkQRJBAAcWwr/L7FJwszX31ogVjwR7wvE1Yr0uQMpkLXBLL8aUxoTiFyhJkiRJkiRJ0ggSwW7AcmCPQtN1mZzWyzH+E3hHIbwFmJnJH6r67UelOGew/LybFeC71Yg9NIh5SJIkbVUEh1D7SKrvZvLnXo7xUuDphXA78NlMNg8wxYEYB7y+ELu/H+PU2qRkXDd9jy5cr4d/Pov20q3AkVXX+0awXyZ/6eM40ohnQZAkSZIkSZIkjWCZPBTBO4DrgKhqOjWCqzK5tqf7IzgAOB/YptD0nupioA4fAU4caM49aKbyAqyo1iryBwYxD0mSpN64C7iCrgU9x0cwc2tHXXU8h11J1+ewi4vFQBF8D9hxgPl259eZvKs6kMkW4Oo6jN1SI7a2m75PK1z/KrPms2FPft7NuBYEacyxIEiSJEmSJEmSRrhMfhLBV4AzC02fiWBhJhtq3ddxFMRX6PoSaiHwyfpn2m/FHYI2ZbIeIIJ9gBOA5wNTgO2ovGS6D1gEzMvk0aFLVZIkjRWZPB7BKcDNwPiqpiOBfwW+3t29Hc9hX6Drc9htwMU1bjmS2rsm1kN3O/bUw2E1Yn8sBiIYD+xdCP+9H/PdUyO2fz/GkUa8WttzSZIkSZIkSZJGnncCdxRi+wAf6OGe1wNHFGJrgTMyKdcts4HbvXD9YAT7RfAD4M/A54HXAccCM4CXAucAVwErI/h4BLsOYb6SJGmMyOQXwPtrNH0iouYuh084HTimEFsPnJxJa73yGwZeUSO2vEbsKUBTIdafI8ruqxGzIEhjkgVBkiRJkiRJkjQKdOwCdCp0eYH0jgieXezf8YLq4zWGOiuTe7uZZgPw8CD+dWePwvXOwK+Bl7H137knAucBv4tgxlb6SpIk9cd/U9mZsNqudLPjYgS7d9xT9LZM/lTn3BomgsOgy/PXLzL5fY3uT6oRW9mPaWvds1c/xpFGPI8MkyRJkiRJkqRRIpPbIrgQ+HBVuBn4cgSHZdJeFf8f6LJrztcz+W4P47+eyq5CQ624Q9D2/RijBbgxgtMyuaoOOUmSJAGQSTmCM4BfAbtUNZ0awaWZzC/c8t90fb75bibf6GGad9L1eLF66U/hTY86jgD7vxpNn+7mlu1qxNb3dd5MNkfQSudj0Prz7CiNeBYESZIkSZIkSdLo8nHgODofBfZ84FzgswARzAFOKdz3J+DtQ5FgPxR3CHrCw8B3geuBvwOPdfR9PnAS8KJC//HANyO4N7PmURWSJEn9ksk9EbwJuhRXfzmCgzt2cySCI6kcF1btPuDsrYx/ad2SHWQRBPAl4FmFptuAy7q5rVZB0KZ+prCZzgVB2/ZzHGlE88gwSZIkSZIkSRpFOnYBOh14tND0XxE8KYKJwBcKbW3AqZl9X4U9RIor6KHykumpmZyZyQ8y+Xkmd2WyPJNPZ3IY8CpgXeG+bYArIgZthb0kSRqjMvkedCnc2Qd4H0DH88eXgahqLwOnZ/LQUOQ4RD4MvK4QWwucnEm5m3tqFe1s6ef8xUKiWsVG0qhnQZAkSZIkSZIkjTKZ/I3KjkDVdgQ+A7wfeEqh7aJMbhmK3PqpuiCoDJybydmZrO3ppo6XckdBZUV+lSezlVX4kiRJ/fRW4O5C7LwIDqZSGHRAoe2TmSwcksyGQAQfAd5bCG8BXpPJn3q4dWKNWH8Lgor31RpbGvUiMxudgyRJkiRJkiRpEETwLeC1hXA70FR1vRQ4qmNnoWEpgh35Z87tmV12/dna/W+ishq/2p8y2b8e+UmSJFWLYDrwUzo/c/0GOJDKEaZP+AXwosx+F74MGxGMo7IL5RsLTW3AiZlcs5X7zwL+rxA+OZPv9COXNXQ+cvZ3mRzc13Gkkc4dgiRJkiRJkiRp9Hoz8LdCrPrF1KPAacO5GAggk3WZPNzx16dioA7fAP5aiD01gv0Gnp0kSVJnmSyjcmxWtWfSuRjoceCUUVIMtBNwHV2LgbZQ+Yw9FgN1KO7oCDChnykVj4atNbY06jU3OgFJkiRJkiRJ0uDI5JEITgcW0rkQ6AlvzuxSKNOtCJ4JTK5TerXcmEndt7XPpC2Ca4B/KzRNB/5S7/kkSZKADwKzgRd10/6uTP4whPkMigimAj8Gnl1oWg+8OpMbejnU4zVi9SoIWt/PcaQRzYIgSZIkSZIkSRrFMrkpgmuBlxea/pDJt/o43AXAifXJrKZmGLTdim6uEWsZpLkkSdIY11GQfBqVY8G2LzT/GPhSX8aL4HZg5zqlV3RrJif39aYIngdcS9dnqpXACZn8sg/D1drFZ7t+5DQOGFcI1yo2kkY9C4IkSZIkSZIkaRSL4EXAS2o0PT2CN2by1aHOqUFW1YjtMeRZSJKksWQCtd/J70mlaKUvx4XtDexWj6RquKevN0QwB/gusEOh6U/A8Zn8qY9D3lcj1p/i7T1rxO7txzjSiFdqdAKSJEmSJEmSpMERwQ7A5XS/OPRTEew/hCk10qYasc1DnoUkSRoTIpgAfJuux1cBPBf4wJAmVEcRvAG4jq7FQLcA0/pRDATwZ6BciE3txzi1CoLu6sc40ojnDkGSJEmSJEmSNHp9BnhKIdYONHX88/bA5RHMyKS1F+P9Bbi9jvkV5SCOXWtF/YODOJ8kSRrbPgE8q4f2f49gbiY/HaqEBiqCAD4MvLdG8/eBUzPZ2J+xM9kYwb3Ak6vCe/VjqFpFRBYEaUyyIEiSJEmSJEmSRqEIXg28rhD+PvBH4H1VsRcAFwIXbG3MTM4Hzq9TilsVwUTgtEL4L5ks6MdwT6sRq3U0hSRJ0oBEcDzwlkJ4MZWdgXbsuC4Bl0ZwSCaP9GLYV1I5ZmwwPLy1DhGMB74BvLZG8yeB92R22eGnr/5I54KgQyIo9XHc59SI3TGwtKSRKTIHc8GFJEmSJEmSJGmoRbAX8Ctgl6rwo8BBwNqOtgOq2tqBozJZOmRJ9kIEJWAdsF1V+PZMntePsX4AvKwqlMCUTFYPLEtJkqR/imAS8GtgclX4MeAZwEuBzxVu+U4mJw9Rev0SQRPwHeBVhaYy8NZMvlCned4LfKQQPiiTP/RhjAXAMVWhe4EnZw7qTpTSsFRqdAKSJEmSJEmSpPrpKKK5lM7FQAD/nsn9mWwCzqLz8VxNwGUR7DREafZKx2rw3xbCz4pg976ME8FU4IRC+LcWA0mSpHrqOFLra3QuBgJ4Xyb3AF8EVhTaTorg1KHIbwD+j67FQO3AGfUqBurwkxqx2b29OYIdgBcVx7QYSGOVBUGSJEmSJEmSNLqcBxxViC0BvvzERSaLqbysqrYP8PlBzKu/ritcj6PrUWhb80G6HrHx5VodJUmSBuAc4CWF2AqoFM10FDu/CdhS6PP5CPYZ9Oz6IYJzgDcWwmXglEwur/N0v6brka7FuXtyEp13lgS4fkAZSSOYR4ZJkiRJkiRJ0igRwaHAcmB8VXgz8OxM/ljouwvwe6ClMMxrM7liUBPtgwieDNxN5wWujwDPyeSvvbj/RODKQng1sG8mG+uVpyRJGtsiOAi4DZhYFd4CHJrZecfDCD4IvL8wxFIqR7i2D2qifRDBAVSKdCYUmt6TyScGac7zgQ8Xwq/K5HtbuW874BfA/lXhvwIHZNJa1ySlEcIdgiRJkiRJkiRpFOh4CfItOhcDAXyoWAwEkMnDwNtqDPWFjiKcYSGTv1M5pqLazsCCCJ7V070RvB5qrlx/m8VAkiSpXiKYQOU5bGKh6ePFYqAOH4IuLine truncated
-    }
-   },
-   "cell_type": "markdown",
-   "id": "d1df7878",
-   "metadata": {},
-   "source": [
-    "<div>\n",
-    "<img src=\"attachment:fig-tsp-question-1.png\"  align=\"left\" width=\"800\"/>\n",
-    "</div>"
+    "The total number of routes is $O(N!)$, where $N$ is the number of cities. This comes from the fact that the number of possible routes is equal to the number of possible permutations of $N$ cities. Thus the cost of the algorithm is $O(N!)$, which becomes expensive very quickly when $N$ grows.\n",
+    "\n",
+    "In practice, however, we will not need to traverse all $O(N!)$ possible routes to find the shortest one since we consider pruning. The nearest city first heuristic also makes more likely that the shortest route is among the first routes to be traversed (left part of the tree), thus increasing the chance to prune routes afterwards. However, the solution can be anywhere in the search tree, and the number of routes to be traversed is $O(N!)$ in the worse case scenario (the optimal route is the right-most one)."
    ]
   },
   {
@@ -188,7 +179,12 @@
    "source": [
     "## Serial implementation\n",
     "\n",
-    "Let's implement the serial algorithm. First, we sort the neighbours according to their distance. "
+    "Let's implement the serial algorithm.\n",
+    "\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "<b>Note:</b> The implementation of this algorithm is rather challenging. Try to understand the key ideas (the explanations) instead of all code details. Having the complete functional implementation is useful to analyze the actual performance of our parallel implementation at the end of the notebook, and to check if it is consistent with the theory.\n",
+    "</div>"
    ]
   },
   {
@@ -196,7 +192,9 @@
    "id": "f2b70f85",
    "metadata": {},
    "source": [
-    "### Nearest-city first heuristic"
+    "### Nearest-city first heuristic\n",
+    "\n",
+    "The first step is preprocessing the distance table to create a new data structure that takes into account the nearest city first heuristic. This is done in the following function."
    ]
   },
   {
@@ -217,27 +215,12 @@
     "end"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "2eeecdd6",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "C = [\n",
-    "    0 2 3 2\n",
-    "    2 0 4 1\n",
-    "    3 4 0 3\n",
-    "    2 1 3 0    \n",
-    "]"
-   ]
-  },
   {
    "cell_type": "markdown",
-   "id": "f769627c",
+   "id": "9fc1a398",
    "metadata": {},
    "source": [
-    "The data structure we will use for the connections table is a matrix of tuples of the form (destination, distance). The tuples are sorted by their distance in ascending order (per start city). "
+    "Execute the next cell to understand the output of `sort_neighbors`."
    ]
   },
   {
@@ -247,15 +230,21 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "C = [\n",
+    "    0 2 3 2\n",
+    "    2 0 4 1\n",
+    "    3 4 0 3\n",
+    "    2 1 3 0    \n",
+    "]\n",
     "C_sorted = sort_neighbors(C)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "51ed8312",
+   "id": "a63dc266",
    "metadata": {},
    "source": [
-    "The connections matrix can be indexed by a city. This returns a `Vector{Tuple}}` of all the destinations and their corresponding distances.  "
+    "The output is a vector of vector of tuples. The outer vector is indexed by a city id, for instance:"
    ]
   },
   {
@@ -271,6 +260,29 @@
     "C_sorted[city]"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "f769627c",
+   "metadata": {},
+   "source": [
+    " This returns a vector of tuples that contains information about the connections to this city of the form (destination, distance). In this case, city 3 is connected to city 3 at distance 0 (itself), then with city 1 at distance 3, then with city 4 at distance 3, and finally with city 2 at distance 4. Note that the connections  are sorted by their distance in ascending order (here is where the nearest city first heuristic is used). Remember the graph with the distance between cities:"
+   ]
+  },
+  {
+   "attachments": {
+    "g27477.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAF9CAYAAADoebhRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7d13vBxV/f/x100lCZDQOwECoTdRASlSlSK9ioAoKiI/CzYUpQkCKqKoIB1FvwKKShUFpClFEOldegs1QCAk4Sbn98fnXjP3zNm9d3dn5szMvp+Pxzxg9+6d+ezNzHtnZ07pQaTeJgEbAasAk/uW+RPLe8A7wJvAK8Ajfcv9wD+AqcWXLCIiQzUc2BY4F3gKcB0svcC/geOB1Qt8DyIiMohlgR8BL9BZ0Ddb/gMcAowp6D2JiIhnJeAcYBb5hb2/vAQcBsxXwPsTyVRP7AJE2jQOOBI4FBg5yGvnAI8CD/X990XmXscfjl3bHw8sh13zXw1Ycgg1vAB8Dbiw5epFRGTItgGepvnZ+CvAacAuwIJtbGNF4HPApQz+LeJa7PKSiIhkaARwAnbm3iiArwZ2AkZluN2FgS8BjzfZ7mvADhluU0Skqy0I3ETj0L0C+GDONYwA9sUuEYVqmIO19NFlUxGRDiwB3EM4aJ8Fdi+4npHAl4FpDWo6n8HvM4iISMDyNL5+fw4wNl5pTAbuDtTlgEtQ8IuItGRRwpdSpmOXWcpgHuBMGp/x61KPiMgQzAvcSTpIXwc2jlhXI4cTvsH8w5hFiYhUxW8JN8NcM2ZRgziIcPDvGbMoEZGy+wzp4HwH2DBmUUN0GOnapwErxyxKRKSsJmEBnwzNXuCjMYtq0c9JB/+twLCYRYmAbjJJ+VwBbO89dwxwdAbrHoaN1fN+YGnvZ9dgA6plYSRwI+lvJgdhN31FRATYlfQZ8g3Y+DjtWBLrJXs0cDnwamD9/csh7ZcdtBw2to/fa3fhjLcjIlJJw7CJS5IhOROb/KRViwNv0Djgiwh9sA5c/nZ+kMN2REQqZ3fSAXlcm+taJrCuGKE/ArjL285btDcAnIhIrfht8l+j/fHqm4X+dGw2rCJCH2wWL39bR+W0LRGRSng/6WA8soP19Yf+LOABrGfsl7FOXaOBBQLbyyv0Ae7wtvUc7d+nEBGpvFMYGIrv0tklkHmAtbDLKyFFh/5ege1tmeP2RERKawTwMgMD8aKct1l06M8DTPW2d16O2xNpSJ1FJLb1gEW8586PUUiOZgB/8J77KOonIxEo9CW2LbzH07HpB+vmUu/xErTXHFWkIwp9iW0z7/HNWPv8urkJG04iyf/AE8mdQl9iW9t7fEOMIgowDWuWmuS/d5HcKfQlpvHAYt5z98UopCD3e48nR6lCuppCX2IKDTf8SOFVFMd/bxpuWQqn0JeYlvAeO+DJGIUU5HHv8WKok5YUTKEvMfnDLEwH3otRSEHe9B73AONiFCLdS6EvMfmh/3aUKoozLfDc/IVXIV1NoS8xjfIez4pSRXFCTVFHF16FdDWFvsT0jve47pc65g08Fzr7F8mNQl9i8gOv3aGUqyL0/hT6UiiFvsTk39gcCUyIUUhBFvUez8JGFBUpjEJfYnoi8FydOyz57fJD718kVwp9iekJ0k006zwImf+B9miUKqSrKfQlpl7SHZY+EKOQgnzQe1zn3sdSUgp9ie027/HmUarI30rAst5zt8QoRLqbQl9iu857vBqwZIxCcuZPjzgbuDFGIdLdFPoSmx/6PdicsnWzt/f4bmwKRZFCKfQltueBf3nP7RejkBxNBDbxnvtjjEJEFPpSBv6cuOsC68coJCefY+CxNgf4v0i1iIhEtxA2Lo1LLP6csllawNuWAw7JaVsTgDe8bfmXtEQK0xO7AJE+5wCfTjx2wHrAXW2u75PAFxv8bDiwjvfcM8ArDV7/GPDxNus4Cjjae25n8v1QExEpvUlYR63kGfE/af/E5DDSZ/PtLne3WcMy2HDRyXXdjy6rSkTa+aQsHgd+7z23EXbGXlWnkB459Bjsmr6ISNdbBht1MnlmPBVYvo11xT7T/3hgPTehS6oSmXZAKZtvAj/wnrsD2JjWJlmZBxiTUU2zgbdaeP1k4N8MHEq5F7tHcW9GNYmI1MJI4E7SZ8nnUo2TlIWBh0nXf2zMokREyuwD2JmxH5wnxCxqCMYCN5Ou+1bsw0xERDwLYJdGGl1fP45ynvFPwMbS8eudDiwdsS4RkdJaCGuXP9iN1TOBEZFqDFkSu1bfqN6fxCtNRKScJmA3bP3A9Nvu9y83AktFqXSgzYEXGPyD6uRYBYqIlM144HbSQfkONiTxaYGfOeBlYNcI9YK1DDoea9kz1A8qnfGLSNcbj42yGboWnhyD/sjAa/qXK7EevUX5GNaZLFTLG8Bm2AeCzvhFRBLGYzNn+cE4A9gu8Pr9SA9r0L/Mwpp15jmx+naEW+f0Lw8BayRer+AXEekzP9aU0Q/EmcD2TX5vVeC+wO/1L7OBq4B9sCaUnVoa6937QJNtOmx46HkDv39Cg9f/OIPaREQqYRzh5o0zsUsngxmDdXaaEViHf0/gauBb2CWXxQZZ73DsMtEOwEnAfwhfs08uzwK7DbJeBb+IdK1xwA2EA3+HFtc1GbueP1jLmeQyFbsMcwdwDXA91i/gcdJj+jdbpmPDRYTO7kNObLCek1p8zyIilTEWC9lQ4O/YwXrXxi6vhHrxZr1Mw0bPbGfydgW/iHSNsdiMUaEbsDtltI2VsKGLnwhsp5NlDjZC5mex/gTt6sE+MELb+FEH6xURKZWxwN8JB/7OOWyvBxu/51vYNf1GLX6aLS9g89geCCybcW0KfhGprTHAtaQDrhfYu6AaeoCJwNbYtIzJOmYD38da6BwAbIgNB5F3PT8jHPw/zHnbIiK5GU34Rmsv7c8526m9vVpaGaM/Swp+EamVUcAVhAN/n4h1lSX0wYL/54SD359IRkSktEYBlxMO/H0j1gXlCn1Q8ItIxY0CLqOcgQ/lC32w4P8F4eA/MWJdIiJNjQIuJR1cs4H9I9aVVMbQh+bBf1TEukREgkYClxBu4/65iHX5yhr6YMF/Kl0X/G5hcJPBrWT/L1IqbmVwXwJ3IbgHwb0Bbg6418DdD+7X4PYGNyp2pQUaCfyZcOAfFLGukDKHPjQP/iMj1pUhtya4w8Bd3XfcOG95Fdxl4PYAp7mFJQY3Hty3+0Le30EbLS+AG2wgrjoYDlxAOPAPjlhXI2UPfaht8Lt9wd3bwjHk+k6k1otduXQdt0mLO2py+V7s6nM0HPgd4cD/QsS6mqlC6IMFf6MZw46IWFcH3BVNjpNpfd+aQz97B9xmsauXrpIK/bvBHQ9uW3Crg1sa3GrgDgR3Z2Cn3S/2O8jBcOC3hAP/kIh1DaYqoQ+1C/4Bof8AuO+A2wjcPInXLATuU+Ce8I6h18EtEq926TJuE3DvgTsP3PsGee1wcD/2dtgp4MYUU2shhgO/IRz4/y9iXUNRpdAHC/5fEg7+70asqw3uCnBXgvvQEF47Htxt3nGk+QekKG4pcCu28PoecH/1dthWx4ovq+HYMMahwP9ixLqGqmqhD7UJftficNRuIrh3E8fQ0/nUJZIJ91Ev9I+LXVEGhgG/Jhw+h0WsqxVVDH2w4D+d8N/+OxHrypm73DuOOhnaWiRPbnFvZz0rdkUd6gHOIBw634pYV6uqGvrQlcHvTvKOoxa+cYsUyk2s0fXIZpcXvh2xrnZUOfTBvm39iq4JfneOdxwtELsikQbczt7OWtYmjINp1oKkiiFT9dCH5sF/eLyy8uAeThxDz9v9MpFSGnAtcg645WJX1IZmnYQqdANxgDqEPnRF8LstvROnX8auSKQBt423s14cu6I2NBsArIJtxP+nLqEPzW+sV+2ym8eNwtry9x9DM8FNil2VSIBbDNyziZ11OrjJsatqUbMx3is8DABQr9AHa0Jbw+B3p3snTsfHrkgkwI0Ed4O3s34+dlUtajZxdx0m9ahb6EPjvhNVa1nVxx3sHUP/QgOvSfm44eB+7+2sv4ldVRtOJBwedZm3tY6hD7UJfrcnuN7EMfQcuImxqxLxuB5wZ3uBf2kFz05OIBwaJ8UsKmN1DX1oPDyGoxKd59w24GYkjqFXwK0WuyoRj+uxVgUDAv/vDBhEqhK+Tzgsqty/IKTOoQ+VDX63NQOHXHgDDaks5eR+5gX+zeDmjV1Vi44jHBInxywqJ3UPfWg+IN6XItbVgNsca/DQfwy9BW7D2FWJBLgfBG44zR+7qhYdSzjwfxKzqBx1Q+hD86GvSxT8biNsLP3+Y+htcJvGrkokwB3rBf5d4BaMXVWLvkc48H+KteKpo24JfWge/CUYEdVtAO7NxDE0HdwWsasSCXBHe4F/D7iFYlfVoqMJB/6Z1DfwobtCHyz4/4/SBb9bl4Hz484Et328ekQacl/3Av9hcIvHrqpF36Q7Ax+6L/ShefBHmPTGrRMI/LrMNyH14g71Av8RcEvErqpFXycc+Gdj3frrrhtDH5rPZ1xg8Lu1wL2aOIZmgdupuO2LDJn7ihf4j9HyTEDRfY1w4J9DdwQ+dG/oQ/PgL2BeY7cKuBcTx1AvuL3z364k1f2rfEbcDsClDPx7/Q54toWVPAM9p2VaVmu+SrjN/XnAZ7ADvxvsDVyQePweMCpSLTH039z1w9Zh1/hPzW/T7kFg1cQTzwK3tbiSH0HPHdnVJBLkDvHO8ttZbo34Br5C+Az/PLrnDL9fN5/p9xuOffCFzvhznPfBPZPBcbRzfvV1h2474LvRVwi3ub+Q7jrDl7lmA/sDf/ae7x9Ou6oT/sgQjIhdQEU8gLVs6cSTWRTSoi8R7lX7e2A/7OCX7vQesBe2LyTPnvuDH2zGtCx9C+i0t/rdWRQiUkefxc7i/a/wv6e7P+x1eWegkdgZf+hSz8ER6xKRFvRftvEP5D/Q3YEPCv0QBb9IhR2IXbbxD+CLUeCDQr+RUcAlhIO/ahMBiXSNLine truncated
+    }
+   },
+   "cell_type": "markdown",
+   "id": "6758ba5d",
+   "metadata": {},
+   "source": [
+    "<div>\n",
+    "<img src=\"attachment:g27477.png\"  align=\"left\" width=\"125\"/>\n",
+    "</div>"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "44025bd5",
@@ -284,7 +296,7 @@
    "id": "6c91a99f",
    "metadata": {},
    "source": [
-    "Next, we write an algorithm that traverses the whole search tree and prints all the possible paths. The tree is traversed in depth-first order. Before we go to a neighbouring city, we also have to verify that it has not been visited on this path yet. If we reach a leaf node, we print the complete path and continue searching. "
+    "Next, we write an algorithm that traverses the whole search tree and prints all the possible paths. To this end, the tree is traversed in [depth-first order](https://en.wikipedia.org/wiki/Depth-first_search) using a recursive function call. Before we go to a neighbouring city, we also have to verify that it has not been visited on this path yet. If we reach a leaf node, we print the complete path and continue searching."
    ]
   },
   {
@@ -333,7 +345,7 @@
     "        end\n",
     "        return nothing\n",
     "    else\n",
-    "        println(path)\n",
+    "        println(\"I just completed route $path\")\n",
     "        return nothing\n",
     "    end\n",
     "end"
@@ -357,7 +369,7 @@
    "source": [
     "### Serial implementation without pruning\n",
     "\n",
-    "Now, we add the computation of the minimum distance. At each leaf node, we update the minimum distance. Furthermore, as we add another node to our path, we update the distance of the current path. That makes it necessary to include two more parameters in our recursive algorithm: `distance`, the distance of the current path, and `min_distance`, the best minimum distance found so far. "
+    "Now, we know how to traverse all possible routes. We just need a minor modification of the code below to solve the TSP problem (without pruning). We add a new variable called `min_distance` that keeps track of the distance  of the shortest route so-far. This variable is updated at the end of each route, i.e., when a leaf node is visited. After traversing all routes, `min_distance` will contain the distance of the shortest route (the solution of the ASP problem)."
    ]
   },
   {
@@ -375,6 +387,16 @@
     "</div>"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "9ac4875c",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-info\">\n",
+    "<b>Note:</b> We could further modify the function so that we also return a vector containing the cities in the shortest route. However, in this notebook, we will only return the distance of the shortest route (a single value) for simplicity.\n",
+    "</div>"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -382,6 +404,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "verbose::Bool = true\n",
     "function tsp_serial_no_prune(C_sorted,city)\n",
     "    num_cities = length(C_sorted)\n",
     "    path=zeros(Int,num_cities)\n",
@@ -411,7 +434,7 @@
     "    else\n",
     "        # Set new minimum distance in leaf nodes\n",
     "        min_distance = min(distance,min_distance)\n",
-    "        #@show path, distance, min_distance\n",
+    "        verbose && println(\"I just completed route $path. Min distance so far is $min_distance\")\n",
     "        return min_distance\n",
     "    end\n",
     "end"
@@ -425,6 +448,7 @@
    "outputs": [],
    "source": [
     "city = 1\n",
+    "verbose = true\n",
     "min_distance = tsp_serial_no_prune(C_sorted,city)"
    ]
   },
@@ -435,7 +459,7 @@
    "source": [
     "### Final serial implementation\n",
     "\n",
-    "Finally, we add the pruning to our algorithm. Anytime the current distance exceeds the minimum distance, the search in this path is aborted and continued with another path. "
+    "Finally, we add the pruning to our algorithm. Anytime the current distance exceeds the minimum distance, the search in this path is aborted and continued with another path. By running the function below, you will see that only three routes will be traversed thanks to pruning as shown in next figure."
    ]
   },
   {
@@ -472,6 +496,7 @@
     "function tsp_serial_recursive!(C_sorted,hops,path,distance,min_distance)\n",
     "    # Prune this path if its distance is too high already\n",
     "    if distance >= min_distance\n",
+    "        verbose && println(\"I am pruning at $(view(path,1:hops))\")\n",
     "        return min_distance\n",
     "    end\n",
     "    num_cities = length(C_sorted)\n",
@@ -493,7 +518,7 @@
     "    else\n",
     "        # Set new minimum distance in leaf nodes\n",
     "        min_distance = min(distance,min_distance)\n",
-    "        #@show path, distance, min_distance\n",
+    "        verbose && println(\"I just completed route $path. Min distance so far is $min_distance\")\n",
     "        return min_distance\n",
     "    end\n",
     "end"
@@ -507,6 +532,7 @@
    "outputs": [],
    "source": [
     "city = 1\n",
+    "verbose = true\n",
     "min_distance = tsp_serial(C_sorted,city)"
    ]
   },
@@ -526,13 +552,14 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "n = 12 # It is safe to test up to n=12\n",
+    "n = 11 # It is safe to test up to n=11 on a laptop\n",
     "using Random\n",
     "using Test\n",
     "Random.seed!(1)\n",
     "C = rand(1:10,n,n)\n",
     "C_sorted = sort_neighbors(C)\n",
     "city = 1\n",
+    "verbose = false\n",
     "@time min_no_prune = tsp_serial_no_prune(C_sorted,city)\n",
     "@time min_prune = tsp_serial(C_sorted,city)\n",
     "@test min_no_prune == min_prune"
@@ -543,7 +570,7 @@
    "id": "6088ddc9",
    "metadata": {},
    "source": [
-    "You can observe that, especially for larger numbers of cities (n=11 or n=12), the performance of the algorithm with pruning is much better than the performance of the algorithm without pruning. "
+    "You can observe that, especially for larger numbers of cities (n=11), the performance of the algorithm with pruning is much better than the performance of the algorithm without pruning. "
    ]
   },
   {
@@ -556,11 +583,13 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c6375465",
+   "id": "732a3ffb",
    "metadata": {},
    "source": [
     "### Where can we extract parallelism ?\n",
-    "Unlike the previous algorithms we studied, in this problem we don't know beforehand how much work is performed since we don't know where the pruning cuts off part of the search tree. Still, we want to divide the workload among multiple processes to enhance the performance. "
+    "\n",
+    "All branches of the search tree can be traversed in parallel. Let us discuss how we can distribute these branches over several processes.\n",
+    "\n"
    ]
   },
   {
@@ -585,7 +614,7 @@
    "source": [
     "### Option 1\n",
     "\n",
-    "The first idea how to parallelize the TSP algorithm is to assign a branch of our search tree to each process. However, as mentioned in an earlier section, the number of branches in the search tree can be up to $O(N!)$. This would require an unfeasibly large amount of proecesses which each do only very little work. "
+    "We can (at least in theory)  assign a branch of our search tree to each process. However, as mentioned in an earlier section, the number of branches in the search tree can be up to $O(N!)$. This would require an unfeasibly large amount of processors which each do only very little work. Thus, we skip this option as it is impractical."
    ]
   },
   {
@@ -609,7 +638,8 @@
    "metadata": {},
    "source": [
     "### Option 2\n",
-    "Instead of assigning one branch per worker, we can assign a fixed number of branches to each worker. This way, each worker can perform the pruning within their own subtree and less workers are needed. \n"
+    "\n",
+    "Instead of assigning one branch per worker, we can assign a fixed number of branches to each worker. This would be a good strategy if we do not consider pruning. However, it is not efficient if we include pruning (which is essential in this algorithm). "
    ]
   },
   {
@@ -632,13 +662,14 @@
    "id": "e3af7def",
    "metadata": {},
    "source": [
-    "### Performance issues\n",
+    "### Performance issues:  Load balance\n",
     "\n",
-    "#### Load balancing\n",
-    "However, this approach has a problem with load balancing. Since we don't know beforehand how much pruning can be done in each subtree, some workers might end up doing less work than others. This uneven distribution of workload leads to some workers being idle, which impairs the speedup. \n",
+    "Pruning is essential in this algorithm but makes challenging to evenly distribute the work over available processors. Image that we assign the same number of branches per worker and that the workers use pruning locally to speed up the solution process. It is not possible to know in advance how many branches will be fully traversed by each worker since pruning depends on the values in the input distance matrix (runtime values). It might happen that a worker can prune many branches and finishes fast, whereas other workers are not able to prune so many branches and they need more time to finish. This is a clear example of bad load balance. We will explain a strategy to fix it later in the notebook.\n",
     "\n",
-    "#### Search overhead \n",
-    "Another disadvantage of this kind of parallel search is that the pruning is now less effective. The workers each run their own version of the search algorithm and keep track of their local minimum distances. This means that less nodes will be pruned in the parallel version than in the serial version. This is called **search overhead**."
+    "\n",
+    "### Performance issues: Search overhead \n",
+    "\n",
+    "Another disadvantage of this kind of parallel search is that the pruning is now less effective. The workers each run their own version of the search algorithm and keep track of their local minimum distances, which are  different from the global minimum found so far. This means that less nodes will be pruned in the parallel version than in the serial version. The parallel code might search more routes than the sequential ones. This is called *search overhead*."
    ]
   },
   {
@@ -647,7 +678,7 @@
    "metadata": {},
    "source": [
     "<div class=\"alert alert-block alert-success\">\n",
-    "<b>Question:</b> How many nodes are pruned in total when we assign two branches to each worker? Look at the illustration below.\n",
+    "<b>Question:</b> How many routes are fully traversed in total when we assign two branches to each worker? Look at the illustration below. Assume that each worker does pruning locally and independently of the other workers.\n",
     "</div>"
    ]
   },
@@ -682,33 +713,42 @@
    "id": "d0bc4fdd",
    "metadata": {},
    "source": [
-    "In this example, the parallel algorithm prunes less nodes than the serial version because not all workers are able to use the global minimum distance as a pruning bound."
+    "In this example, the parallel algorithm traverses more routes (1 more) then the serial version because not all workers are able to use the global minimum distance as a pruning bound. Remember that the sequential code only traverses 3 routes completely. See figure:"
+   ]
+  },
+  {
+   "attachments": {
+    "g26375.png": {
+     "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1oAAAHpCAYAAACbY01sAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15fFxl9cfxz7mTZtKWRQrIItAmadkqyCaI7IsisikKgsiOFOVHwQLNTFrgKs3MpC1Vyr6oQBWxZRGogFAoImURQUCRpSRpWcsiFeiSSTNzfn/cDEymycwkmZk7k5z369WX5s6duac0mcy5z/N8H8EYY4wpnBpgD2BXYOuuP1sB6wAju/73f8BKYAXQCrza9edp4AUgWfKqjTHGGGOMMabMbAZMAhYAqwAdwJ//AncAJ+I1ZsYYY4wxxhgzZAhwFHAf0MnAmqve/nwK3Iw3OmaMMcYYY4wxg1YAOAH4F8Vprnr78wCwTwn+fsYYY0xBiN8FGGOMqRhfBa4hvxGmj4F/4629agU+xJtW2I43JXAksDHeGq5tgO3x1nflcjvwM+CtPtZujDHGGGOMMWVlBHAVkCD7qNMTwIXAbngjX30xHDgIaAJez3GdT4FzsJuFxhhjjDHGmAq1HdmnCS4HongjU4UiwF7ATUBHlmv/CdiggNc1xhhjjDHGmKI7Gi+CvbcGKwysV+QaRgNXA2t6qaMNGF/kGowxxhhjjDGmIE6h9+ZmLrBJievZEXi8l3o+AvYucT3GGGOMMcYY0ycX0nND8z5wqI91CXA+PU8nXAl8w7/SPhd5PbJXZGlke7/rMMYY4y9bSGyMMSbdGcD1rP37YRHwA+Dtkle0tq8B84AtMo6vAA4Enil5RV2u+8d1wz4a9dFLQD1wRyARCE8eN7nFr3qMMcb4xxotY4wxKUcBd7B2YuDdwHF40ezlYkvgL3hhHek+xAvSeK3kFQGRtshEUbk87dAa4LcBCVwyuXbyMj9qMsYY4w9rtIwxxgCMBZ5l7XCLOcBpQGfJK8ptQ+BBYJeM4/8C9gBWl7KYWEtsfRVdjLc/WKYViv5KAjIrPDq8vJR1GWOM8YfjdwHGGGN8FwT+yNpN1nzKt8kC+C9wCN6myOl2AC5f+/TikmEyUpC/4q0Zy7SOIFNJsCTSGmma+erMjUpdnzHGmNKyES1jjDHT8QIw0j0NHECJR4X6qRb4O5DZvHwHb9pjScVaYjuo6EXAMVlOWwn82qlymhu2aninRKUZY4wpIWu0jDFmaBsP/BMYlnZsOd50vCUFeP31u15rR6Am7XgH8MsCvH7KoXgjcOkzNd7EW8O1soDXyVvTkqYDnKQTxZvG2JtVKDc4SWdmw7iGt0pVmzHGmOKzRssYY4YuAf4K7JNx/LvAn/rxeusAOwG7pv3Zlp6nqa8A1u3HNbKZAVyQcSyGt7GyL1RVokuiR4mKC3wly6lrgNsEuSxUF3qhNNUZY4wpJmu0jDFm6DoMbxQo3e1kn/LWm58DU8l/7W8xGq3hwEt4UwlT2oE64N0CX6vPoi3Rg1W0SZDdc5y6SJDmhtqG+SLS03ovY4wxFcAaLWOMKQPNS5q304Tur6L7iSM3h8aE7i/BZRcBX0/7egWwPd6Uu766HJjYh/OL0WgBHMna67JmAJOLcK1+iS2JHapJvQjYM9t5gjwP/HJEYsQfJ46bGC9NdcYYYwrFGi1jjPFBdGm0jiQHo+wN7I+3L1TK7HBd+Nwil7Af8GjGsYFMs8vWaC3BS+JLH2kqVqMF8DjeXlrp19oKb+1Z2Yi2RA/GIYxyYI5TP1D0hmQyee3UsVP70wQbY4zxgTVaxhhTAtHXo2MlIAep6gF4jdUmWU5/IVwX3qnIJf0OOCHt64FOsUs1Wu/i7ceV+vM08D4QBUJp5xez0ToUuC/j2NnA1UW63oA0tTXt5KgzCW9T6GFZTk0Cjwgy26YVGmNM+bNGyxhjiiCyOLKxBGR/4GBBvqFoba7npEkGhwU3nrTlpI+KVN66wDJgRNqx64CzBvCadcAnwIe9PF7KRgvgBbykw5SnyDFVz2/TWqeNrtKq81T0dHL9t1H+o47ewDDmNG7R+N/SVGiMMaYvrNEyxpgCcF9y16mpqdlPHT0I5WDgy/T9PbZT0eccdR4NaOCyC8de+H4RSgU4Gbgp49ieeM1IsZS60ZoEXJZxbGtgcRGvWRBum/uFIMEJKD/Fm/KYTRzhThLcGKoPLbRRLmOMKR/WaBljTD+4C92q4JbBrxDg4K7Gal+guo8vk0B4HmUR8Hhc4g+5te7/Cl/tWv6AN00t5VW8GPZiKnWjtSleqEdV2rFzgCuLeM2CmqtzAy1LW46UpJyt6IHk+J2toi2SlF87w5ybbRNkY4zxnzVaxhiTB1WVyJLIVxx1DkI5CGFfYGQfXyYBPIfyiATkr9Iujzds2/BpEcrNRoB38BqRlFKk8pW60YK1UxXvBL5X5GsWRaQlsrUgp4nIBEW/kOP0JPAkwi3xVfFb3fHuilLUaIwxpjtrtIwxphfNbzRvnuxM7gUcjHIYwpf68TKtwAJgQXBY8OEirrvK1/Z4e02l+zZQ7Dh5PxqtS/H29kpZDmyE14hUJHexu17QCf4I4Qxg5zyesgK4I6nJOWvq1ix0xa3Yv7sxxlQaa7SMMaZLZHFkY6mSA0XlIODgPgZYpLwLPIzwsNPpLGgY1/BWgcscqOOBW9O+TgBfwPtAXkx+NFoH4TW56cYAS4t83ZKIvh7dFeEMhB8C6+XxlLcQ5klCbm+ob3jS1nMZY0xxWaNljBmy5urcQEtLy05d66yOwAuEcPr4MivxpmktIMGCUH3ouTL/AOsCl6R93QKMLcF1/Wi0NsFLV0z3TeChIl+3pGYsmzFyzco1x4rI6XTfPyybt1DuEOT29rr2J2ykyxhjCs8aLWPMkJK2UfDBeB+61+/jS3QivEDSmw4Yb48/5o53OwpfadHcijeqlXIfcFgJrutHowXedMH0NU0VFYjRV7GW2Fbq6PEoZ5B/A/0hcL8g89qXtt/vHuB2FrFEY4wZMqzRMsYMajOWzRjZubJzT+Bg4AiE7fvxMp+ts4on4n9xx7mfFLTI0lqIt2FyylXA/5Xgun41Ws8Cu6R9HQGmlOC6vou+Ht0Vh5OAH+KtTcuHNV3GGFMg1mgZYwYVV10n2BLceYCx6581VlqtjwyyDWH/Aeya9nUT3QMjisWvRiuzsbwCmFiC65YNt82tCSaDhyEcgzd6uU6eT31PRe8UlT+NTIz868RxE+NFLNMYYwYda7SMMRVvetv0TTuTnYeKyCF4I1cb9uX5gvxP0YcRHgQeDNeGlxSjzjLxKt7GvSlhIFaC6/rVaN0NHJn29U3AqSW4blly29yaGq35hqLHAEeRX4gGwGpgkYjMJ8ldofrQG8Wr0hhjBgdrtIwxFSdj1Ko/IRZJhH9W8DqrgWgF0tMUzwdmleC6fjVac4Fj0r7+A95UuiEvo+k6kr6tV2wF5qPcO2r5qL9O2G3CmuJUaYwxlcsaLWNMRZj56syN1gxbcwDeiNWRdN9wNx/vAY8JMr96WPX8MtjPyi//Ar6c9vUlwC9KcF2/Gq37gW+lfX0DcGYJrltRZi+eHVwVWPXN/jRdin4kyMMIC5KSvHfKmCnvFq9SY4ypHNZoGWPKUg9rrfYHqvrwEquBRRUUu14qTwJfS/t6BjC5BNf1q9F6nO6R55cBF5TguhVr9uLZwRXOim+IyPfx1nTlG6QB3r5sfxfkPhFZsHrJ6n9YoIYxZqiyRssYUzYib0U2lLh8E+HbwCHAxn18iX8L8oAgD66W1X9za932IpRZ6TJHeG4BTi7Bdf1qtF4DxqV9fTFwaQmuOyj0ME336/Tts0Ol7TNnjDEFY42WMcY3qiqxltgu4sihin4b2B0I9OElVgAPi8j9JLnfFujn5Srgp2lfP4W3xq3Y/Gi0qvE+6KePhJ6At5eY6YfI4sjGTsA5VNHD6d8+dB8AjyIs6JTOhy4ac1Fb4as0xpjyYI2WMaakZr05a3i8I74XwhHA0cAWfXyJVmCBIPNHJEY8aJHTfTYRuDzt6+XAqBJc149Ga1vg5YxjX8WLuDcDNFfnBlpbW/dU9HAcDka7bRuQr3eBxxEWJKoSf5665dS3C12nMcb4xRotY0zRzXh9xhcTTuJbXXfBDyX/fXwgba2Vonc31ja+Upwqh4xDgAcyjm0NLC7ydf1otE7Gi3NPUbwRmE+LfN0hafri6fWdVZ2HicqhwD7AyD6+hAL/VvQxR5zHJSCPNWzV8E7hKzXGmNKwRssYUxSxxbHxWqWH93Ndx2cbBsdXx+93x7srilLk0PQF4EO6T9E8C7iuyNf1o9G6GTgp7esXga8U+ZoGcBe6VcEtg19JC7PZBwj246XeBR4XkUWa0MdtjZcxppJYo2WMKYgBTglsp2v6kDp6b+Poxv8Up0rT5Rlgt7Svb6f7XlPFUOpGS4A3gS+lHbscOK+I1zS9cN9xRwRXB78OHNw1zXBn+rb3Xcr7wN8RHpekLNpg+QZP2x5exphyZY2WMabfmhc3b6FVepiqHgEcCAzvw9PfAeYLMr+9pv1hd3N3VXGqND1opnuk+ypgM+CTIl6z1I3WfsCjGceOAu4p4jVNnlL74qnqQSJyEDC2ny/1CcoihEVJJ/nEmpVrnrERcGNMubBGyxjTJwOaEqj8B7hXkPkNdQ2LbAqQb3YHns44djrwmyJes9SN1o14f6eUT/CaSWvoy1CsJbZV0kkeIMi+KHvjrRvsj07gX4o+4eA85SScJyePm9xSwFKNMSZv1mgZY7JyX3Kra0bUHJBMJr8rIkcAm/fh6auBhxHudQLOfFvYXjaCQBte45Hyd2CPIl6zlI3WKGBJxuv/HvhRka5nCqyptWmTAIHdVXUvhL3xbg4M6+fLvQc8g/Is8Hh8ePwJG0E3xpSCNVrGmLXMenPW8I41HQcreowgRyj6hT48/QO8VLt7nQ7ngYZtGyzhrXx8EW8Prf8DLine truncated
+    }
+   },
+   "cell_type": "markdown",
+   "id": "f6329f5d",
+   "metadata": {},
+   "source": [
+    "<div>\n",
+    "<img src=\"attachment:g26375.png\"  align=\"left\" width=\"300\"/>\n",
+    "</div>"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "95643cd8",
+   "id": "1c8e5eb7",
    "metadata": {},
    "source": [
-    "<div class=\"alert alert-block alert-success\">\n",
-    "<b>Question:</b> The previous example described positive search overhead. There is also negative search overhead, resulting in superlinear speedups. Can there be negative search overhead in this parallel TSP algorithm? (Provided the workers communicate the minimum distance with each other)\n",
-    "</div>\n",
-    "\n",
-    "    a) No, because we use the nearest city first heuristic. \n",
-    "    b) No, because each worker has to search the whole subtree before the algorithm completes. \n",
-    "    c) Yes, because the parallel algorithm does not need to search the whole search tree.\n",
-    "    d) Yes, because the global minimum distance can be found more quickly, enabling the parallel version to do more pruning."
+    "In order to minimize search overhead, workers need to collaboratively keep track of a global minimum distance. However, this needs to be done carefully to avoid race conditions. We show how to do this later in the notebook."
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "fff58498",
+   "cell_type": "markdown",
+   "id": "5e4cee1a",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "answer = \"x\" # Replace x with a,b,c or d\n",
-    "tsp_check_3(answer)"
+    "### Negative search overhead\n",
+    "\n",
+    "The parallel algorithm might search more branches than the sequential one when we parallelize the pruning process. However, it is also possible that parallel algorithm searches less branches that the sequential one for particular cases. Imagine that the optimal route is on the right side of the tree (or the last route in the tree in the limit case). The parallel algorithm will need less work than the sequential one in this case. The last workers might find the optimal route very quickly and inform the other workers about the optimal minimum, which can then prune branches very effectively. Whereas the sequential algorithm will need to traverse many branches in order to reach the optimal one. If the parallel code does less searches than the sequential one, we say that the search overhead is negative. \n",
+    "\n",
+    "Negative search overhead is very good for parallel speedups, but it depends on the input values. We cannot rely on it to speed up the parallel execution of the algorithm."
    ]
   },
   {
@@ -717,7 +757,16 @@
    "metadata": {},
    "source": [
     "### Option 3: Dynamic load balancing with replicated workers model\n",
-    "In our parallel implementation, we will use a coordinator process and several worker processes. The coordinator process (or _master_) searches the tree up to a certain maximum depth _maxhops_. When _maxhops_ is reached, the coordinator creates a job and delegates it to a worker. The workers repeatedly get work from the master and execute it. This is an example of **dynamic load balancing**: the load is distributed among the workers during runtime."
+    "\n",
+    "In this third option, we explain a strategy to improve load balance based using the [*replicated workers model*](https://en.wikipedia.org/wiki/Thread_pool) also known as *worker pool* or *thread pool*. In this model, the main processes (aka master or coordinator process) sends jobs to a job queue. Then, workers take one available job from the queue, run it, and take a new job when they are done. In this process, workers never wait for other workers thus fixing the load balance problem. It does not matter if there are some jobs that are larger than others as long as there are enough jobs to keep the workers busy. The main limiting factor of this model is the number of jobs and speed in which the main process is able to generate jobs and send them to the queue. This is an example of **dynamic load balancing**: the load is distributed among the workers at runtime.\n",
+    "\n",
+    "\n",
+    "In our parallel implementation, we will use a coordinator process and several worker processes. The coordinator process will search the tree up to a certain maximum depth given by a number of hops/levels _maxhops_. When _maxhops_ is reached, the coordinator will stop searching the tree and will let any available worker to continue searching in the subtree. In the figure below, the master process will only visit the nodes in the top green box. The worker processes will search in parallel the subtrees below.\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n"
    ]
   },
   {
@@ -745,28 +794,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "81219a27",
+   "id": "9e345393",
    "metadata": {},
    "source": [
-    "<div class=\"alert alert-block alert-success\">\n",
-    "    <b>Question:</b> To find the right <i> maxhops </i> level is a tradeoff between...\n",
-    "</div>\n",
-    " \n",
-    "    a) Communication overhead (large maxhops) and load imbalance (small maxhops). \n",
-    "    b) Search overhead (large maxhops) and load imbalance (small maxhops). \n",
-    "    c) the number of workers (large maxhops) and the job size (small maxhops). \n",
-    "    d) buffer for the job queue (large maxhops) and idle time of the coordinator process (small maxhops)."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "0cf1ec88",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "answer=\"x\" #Replace x with a,b,c, or d\n",
-    "tsp_check_4(answer)"
+    "### Performance impact of  maxhops\n",
+    "\n",
+    "We introduced a new parameter `maxhops`.  Which is then the optimal value for it? When choosing `maxhops`, there is a trade off between load balance and communication overhead.\n",
+    "\n",
+    "- A small `maxhops` will reduce the number of jobs communicated to the workers (less communication), but reducing the number of jobs is bad for load balance. In the limit, we might generate even less jobs than the number of workers.\n",
+    "\n",
+    "- A large `maxhops` will increase the number of parallel jobs, improving dynamic load balance, but it will lead to more communication.\n",
+    "\n",
+    "\n",
+    "The optimal value of `maxhops` will depend on the given system, the number of workers, problem size, and also the particular input values. It is not possible to determine it in advance."
    ]
   },
   {
@@ -776,6 +816,8 @@
    "source": [
     "## Implementation of the parallel algorithm \n",
     "\n",
+    "We will implement this algorithm using the task-based programming model provided by Distributed.jl as it is convenient to implement the replicated workers model.\n",
+    "\n",
     "First, let's add our worker processes. "
    ]
   },
@@ -937,7 +979,7 @@
     "\n",
     "### Simplified example\n",
     "\n",
-    "We will demonstrate how the workers communicate the minimum distance with each other with a short example. Each worker generates a random value and updates a globally shared minimum. The variable for the global minimum is stored in a `RemoteChannel`. The buffer size of the channel is 1, such that only one channel can take and put new values to the channel at a time."
+    "We will demonstrate how the workers communicate the minimum distance with each other with a short example. Each worker generates a random value and updates a globally shared minimum. The variable for the global minimum is stored in a `RemoteChannel`. The buffer size of the channel is 1, such that only one worker can take and put new values to the channel at a time, thus solving the race condition problem."
    ]
   },
   {
@@ -1087,7 +1129,7 @@
    "metadata": {},
    "source": [
     "## Testing the parallel implementation\n",
-    "Next, we will test the correctness and performance of our parallel implementation by comparing the results of the parallel algorithm to the results of the serial algorithm for multiple problem instances. "
+    "Next, we will test the correctness and performance of our parallel implementation by comparing the results of the parallel algorithm to the results of the serial algorithm for multiple problem instances. Run it for different values of `n` and `max_hops`. Try to explain the impact of these values on the parallel efficiency."
    ]
   },
   {
@@ -1097,12 +1139,13 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "n = 18 # Safe to run up to 18\n",
+    "n = 18 # Safe to run up to 18 on a laptop\n",
     "using Random\n",
     "Random.seed!(1)\n",
     "C = rand(1:10,n,n)\n",
     "C_sorted = sort_neighbors(C)\n",
     "city = 1\n",
+    "verbose = false\n",
     "T1 = @elapsed min_serial = tsp_serial(C_sorted,city)\n",
     "max_hops = 2\n",
     "P = nworkers()\n",
@@ -1115,6 +1158,50 @@
     "@test min_serial == min_dist"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "92e68978",
+   "metadata": {},
+   "source": [
+    "### Super-linear speedup\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9a724509",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-success\">\n",
+    "<b>Question:</b> For some values of `n` and `max_hops` the parallel efficiency can be above 100% (super-linear speedup). For example with `n=18` and `max_hops=2`, I get super-linear speedup on my laptop for some runs. Explain a possible cause for super-linear speedup in this algorithm."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "eebe7e9a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "uncover = false\n",
+    "q_superlinear_answer(uncover)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "19835531",
+   "metadata": {},
+   "source": [
+    "## Summary\n",
+    "\n",
+    "- We studied the solution of the TSP problem using a branch and bound strategy\n",
+    "- The problem is $O(N!)$ complex in the worse case scenario, where $N$ is the number of cities.\n",
+    "- Luckily, the compute time can be drastically reduced in practice using the nearest city first heuristic and branch pruning.\n",
+    "- Pruning, however, introduces load imbalance in the parallel code. To fix this, one needs a dynamic load balancing strategy as the actual work per worker depends on the input matrix (runtime values).\n",
+    "- A replicated workers model is useful to distribute work dynamically. However, it introduces a trade-off between load balance and communication depending on the value of `maxhops`.\n",
+    "- The parallel code might suffer from positive search overhead (if the optimal route is on the left of the tree) or it can benefit from negative search overhead (if the optimal route is on the right of the tree).\n",
+    "- In some cases, it is possible to observe super-linear speedup thanks to negative search overhead.\n"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "c789dc7a",
@@ -1130,15 +1217,15 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Julia 1.9.1",
+   "display_name": "Julia 1.11.6",
    "language": "julia",
-   "name": "julia-1.9"
+   "name": "julia-1.11"
   },
   "language_info": {
    "file_extension": ".jl",
    "mimetype": "application/julia",
    "name": "julia",
-   "version": "1.9.1"
+   "version": "1.11.6"
   }
  },
  "nbformat": 4,