diff --git a/notebooks/julia_mpi.ipynb b/notebooks/julia_mpi.ipynb index 6fe4bef..391f731 100644 --- a/notebooks/julia_mpi.ipynb +++ b/notebooks/julia_mpi.ipynb @@ -528,7 +528,7 @@ "metadata": {}, "source": [ "
\n", - "Note: Function `mpiexec` provided by `MPI.jl` is a convenient way of accessing the `mpiexec` program that matches the MPI installation used my Julia.\n", + "Note: Function `mpiexec` provided by `MPI.jl` is a convenient way of accessing the `mpiexec` program that matches the MPI installation used by Julia.\n", "
" ] }, @@ -974,7 +974,7 @@ "\n", "\n", "\n", - "`MPI_Send` is also often called a blocking send, but this is very misleading. `MPI_Send` might or not wait for a matching `MPI_Recv`. Look into the following example (which in fact is an incorrect MPI program)." + "`MPI_Send` is also often called a blocking send, but this is very misleading. `MPI_Send` might or not wait for a matching `MPI_Recv`. Assuming that `MPI_Send` will block waiting for a matching receive is erroneous. I.e., we cannot assume that `MPI_Send` has synchronization side effects with the receiver process. However, assuming that `MPI_Send` will not block is also erroneous. Look into the following example (which in fact is an incorrect MPI program). In contrast, `MPI_Send` guarantees that the send buffer can be reused when function returns (complete operation)." ] }, { @@ -1047,7 +1047,7 @@ "source": [ "## Fixing cyclic dependencies\n", "\n", - "Returning to the incorrect program seen before. Remember that on my laptop it worked for `n=1`, but it failed for `n=10000`. For `n=1` the MPI implementation provably decided that the best trade-off was to buffer the small message and return from the `MPI_Send` as soon as possible. For the large message (`n=10000`) it provably decided to wait for a matching receive in ordeer to avoid copying data. In this case, both ranks were blocked on the call to `MPI_Send` and the receives were never posted leading to a dead lock.\n", + "Returning to the incorrect program seen before. Remember that on my laptop it worked for `n=1`, but it failed for `n=10000`. For `n=1` the MPI implementation provably decided that the best trade-off was to buffer the small message and return from the `MPI_Send` as soon as possible. For the large message (`n=10000`) it provably decided to wait for a matching receive in order to avoid copying data. In this case, both ranks were blocked on the call to `MPI_Send` and the receives were never posted leading to a dead lock.\n", "\n", "We can fix these cyclic dependences by smartly ordering the sends and the receives:" ] @@ -1149,10 +1149,11 @@ "source": [ "### Standard mode\n", "\n", - "* Function `MPI_Send` \n", + "* Function `MPI_Send`.\n", "* Programmer cannot make any assumptions whether the message is buffered or not. This is up to the system.\n", - "* Assuming buffering might lead to incorrect programs (as the example above).\n", - "* Assuming that a matching receive started is also incorrect." + "* Assuming that `MPI_Send` does not block because it copies the message into a buffer is erroneous (as the example above).\n", + "* Assuming that `MPI_Send` will block waiting for a matching receive is also incorrect. \n", + "* Assuming that a matching receive started when `MPI_Send`returns is also incorrect (no synchronization guarantee)." ] }, { @@ -1160,9 +1161,9 @@ "id": "014ef71f", "metadata": {}, "source": [ - "### Buffered\n", + "### Buffered mode\n", "\n", - "* Function `MPI_Bsend`\n", + "* Function `MPI_Bsend`.\n", "* Programmer provides additional internal buffer space with function `MPI_Buffer_attach`.\n", "* `MPI_Bsend` completes when message is copied into a local buffer.\n", "* Erroneous if buffer space is insufficient.\n", @@ -1174,12 +1175,12 @@ "id": "bf456a3e", "metadata": {}, "source": [ - "### Synchronous\n", + "### Synchronous mode\n", "\n", - "* Function `MPI_Ssend`\n", - "* It waits for a matching receive and it is guaranteed that the receive started.\n", - "* No extra data copy\n", - "* Easy to get deadlocks\n", + "* Function `MPI_Ssend`.\n", + "* It waits for a matching receive and it is guaranteed that the receive started (synchronization guarantee).\n", + "* No extra data copy.\n", + "* Easy to get deadlocks.\n", "* It can be started whether or not a matching receive was posted." ] }, @@ -1188,9 +1189,9 @@ "id": "a1bd72c3", "metadata": {}, "source": [ - "### Ready\n", + "### Ready mode\n", "\n", - "* Function `MPI_Rsend`\n", + "* Function `MPI_Rsend`.\n", "* It may be started only if the matching receive is already posted. This allows the underlying implementation to skip some operations, such as a [handshake](https://en.wikipedia.org/wiki/Handshake_(computing)).\n", "* Erroneous if there is no matching receive yet.\n", "* Otherwise, the same as the synchronous `MPI_Ssend`." @@ -1242,7 +1243,7 @@ "source": [ "## Incomplete operations\n", "\n", - "Functions `MPI_Isend` and `MPI_Irecv` are *incomplete* operations, hence the `I` in `MPI_Isend` and `MPI_Irecv`. This means that, when these functions return, there is no guarantee that the underlying operation has finished. Function `MPI_Wait` should be used to wait for completion of the send and/or receive.\n", + "Functions `MPI_Isend` and `MPI_Irecv` are *incomplete* operations, hence the `I` in `MPI_Isend` and `MPI_Irecv`. This means that, when these functions return, there is no guarantee that the underlying operation has finished. Function `MPI_Wait` should be used to wait for completion of the send and/or receive.\n", "\n", "In particular:\n", "\n", @@ -1354,7 +1355,7 @@ "\n", "Remember that we used `MPI_Probe` to query for the size of an incoming message. However, if we use `MPI_Probe` we miss the opportunity to do local work before a matching send started, i.e. we cannot do latency hiding.\n", "\n", - "This can be fixed using an `MPI_Iprobe`, i.e., an incomplete probe. It allows us to check for incoming messages without blocking.\n", + "This can be fixed using an `MPI_Iprobe`, i.e., an incomplete probe (aka a non-blocking probe). It allows us to check for incoming messages without blocking.\n", "\n", "In Julia:\n", "```julia\n", @@ -1363,8 +1364,11 @@ "\n", "In C:\n", "```C\n", - "int MPI_Iprobe(int source, int tag, MPI_Comm comm, int *flag,\n", - " MPI_Status *status)\n" + "int MPI_Iprobe(int source, int tag, MPI_Comm comm, int *ismsg,\n", + " MPI_Status *status)\n", + "```\n", + "\n", + "Function `MPI_Iprobe` returns immediately without waiting for a matching send. The value of `ismsg` tells if a send was posted or not. If there was a send, the `status` object can be used to get information about the incoming message as we saw before." ] }, { @@ -1464,7 +1468,7 @@ "source": [ "## Summary\n", "\n", - "We have seen different ways of sending and receiving messages with MPI, each one with its pros and cons. You as user should decide which is the best option for a given problem. We also learned how to get information about an incoming messages with `MPI_Status`, `MPI_Probe`, and `MPI_Iprobe`. In addition, we saw how easy is to write incorrect programs if you do not follow the semantics of the MPI operations properly.\n" + "We have seen different ways of sending and receiving messages with MPI, each one with its pros and cons. You as user should decide which is the best option for a given problem. We also learned how to get information about an incoming messages with `MPI_Status`, `MPI_Probe`, and `MPI_Iprobe`. In addition, we saw how easy is to write incorrect programs with if you do not follow the semantics of the MPI operations properly.\n" ] }, {