\n",
@@ -742,10 +608,10 @@
},
{
"cell_type": "markdown",
- "id": "001f022b",
+ "id": "ead20fc0",
"metadata": {},
"source": [
- "### Blocking send/receive\n",
+ "## Blocking send and receive\n",
"\n",
"In Julia:\n",
"\n",
@@ -772,7 +638,7 @@
},
{
"cell_type": "markdown",
- "id": "19b6c4e1",
+ "id": "be4fae1c",
"metadata": {},
"source": [
"### Example"
@@ -780,7 +646,7 @@
},
{
"cell_type": "markdown",
- "id": "334be457",
+ "id": "c2301992",
"metadata": {},
"source": [
"Send 5 integers from rank 2 to rank 3."
@@ -788,18 +654,10 @@
},
{
"cell_type": "code",
- "execution_count": 54,
- "id": "4fffb94a",
+ "execution_count": null,
+ "id": "61fe5db3",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "rcvbuf = [1, 2, 3, 5, 8]\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
@@ -821,7 +679,7 @@
},
{
"cell_type": "markdown",
- "id": "29340e48",
+ "id": "ee680aa5",
"metadata": {},
"source": [
"### Any source, any tag\n",
@@ -841,7 +699,7 @@
},
{
"cell_type": "markdown",
- "id": "6c8af148",
+ "id": "1f0cb6c8",
"metadata": {},
"source": [
"### Example"
@@ -849,7 +707,7 @@
},
{
"cell_type": "markdown",
- "id": "f8806fb6",
+ "id": "2add5065",
"metadata": {},
"source": [
"Send 5 integers from rank 2 to rank 3."
@@ -857,18 +715,10 @@
},
{
"cell_type": "code",
- "execution_count": 61,
- "id": "96c00b8a",
+ "execution_count": null,
+ "id": "ccdf660a",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "rcvbuf = [1, 2, 3, 5, 8]\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
@@ -892,7 +742,7 @@
},
{
"cell_type": "markdown",
- "id": "b4f838cc",
+ "id": "7833da72",
"metadata": {},
"source": [
"### Who was the sender? Which was the tag?\n",
@@ -911,20 +761,10 @@
},
{
"cell_type": "code",
- "execution_count": 67,
- "id": "0faa96df",
+ "execution_count": null,
+ "id": "2f037032",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "rcvbuf = [1, 2, 3, 5, 8]\n",
- "status.source = 2\n",
- "status.tag = 0\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
@@ -950,7 +790,7 @@
},
{
"cell_type": "markdown",
- "id": "d519b6f0",
+ "id": "590bc407",
"metadata": {},
"source": [
"### Which is the incoming message size?\n",
@@ -959,26 +799,19 @@
"\n",
"```julia\n",
"status = MPI.Probe(comm,MPI.Status; source, tag)\n",
- "```\n",
- "```C\n",
- "int MPI_Probe(int source, int tag, MPI_Comm comm, MPI_Status *status)\n",
- "```\n",
- "\n",
- "We can get the message size from the status object using function `MPI_Get_count`. We can also get the source and tag from the status object as shown before.\n",
- "\n",
- "```julia\n",
"count = MPI.Get_count(status, T)\n",
"```\n",
"```C\n",
+ "int MPI_Probe(int source, int tag, MPI_Comm comm, MPI_Status *status)\n",
"int MPI_Get_count(const MPI_Status *status, MPI_Datatype datatype, int *count)\n",
"```\n",
"\n",
- "Note that one needs to specify the data type."
+ "We can get the message size from the status object using function `MPI_Get_count`. We can also get the source and tag from the status object as shown before."
]
},
{
"cell_type": "markdown",
- "id": "56896b06",
+ "id": "517d052e",
"metadata": {},
"source": [
"### Example"
@@ -986,19 +819,10 @@
},
{
"cell_type": "code",
- "execution_count": 73,
- "id": "b20f565a",
+ "execution_count": null,
+ "id": "1b54af36",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "I am about to receive 5 integers.\n",
- "rcvbuf = [1, 2, 3, 5, 8]\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
@@ -1025,7 +849,7 @@
},
{
"cell_type": "markdown",
- "id": "2d6938ad",
+ "id": "04bdb01f",
"metadata": {},
"source": [
"### Blocking semantics\n",
@@ -1040,18 +864,10 @@
},
{
"cell_type": "code",
- "execution_count": 76,
- "id": "497fe2df",
+ "execution_count": null,
+ "id": "2299bf78",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "rcvbuf = [1, 2, 3, 5, 8]\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
@@ -1075,7 +891,7 @@
},
{
"cell_type": "markdown",
- "id": "5619caba",
+ "id": "245893cf",
"metadata": {},
"source": [
"However:\n",
@@ -1086,31 +902,20 @@
},
{
"cell_type": "markdown",
- "id": "5f739652",
+ "id": "59fcac63",
"metadata": {},
"source": [
"### Incorrect program\n",
"\n",
- "The following program will or will not work depending whether the underlying implementation uses buffering. In my case, it works with `n=1` below, but leads to a dead lock when `n=10000`. The MPI implementation might decide to buffer or not depending on the message size."
+ "The following program will or will not work depending whether the underlying implementation uses buffering. On my laptop, it works with `n=1`, but leads to a dead lock when `n=10000`. The MPI implementation decided to buffer or not depending on the message size."
]
},
{
"cell_type": "code",
- "execution_count": 93,
- "id": "e8bedc57",
+ "execution_count": null,
+ "id": "42fb8089",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "(rcvbuf[1], rank) = (0, 0)\n",
- "(rcvbuf[1], rank) = (0, 1)\n",
- "(rcvbuf[1], rank) = (3, 2)\n",
- "(rcvbuf[1], rank) = (2, 3)\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
@@ -1135,7 +940,7 @@
},
{
"cell_type": "markdown",
- "id": "a6f47642",
+ "id": "8b7eeec2",
"metadata": {},
"source": [
"### Correct program\n",
@@ -1145,21 +950,10 @@
},
{
"cell_type": "code",
- "execution_count": 92,
- "id": "276e40c5",
+ "execution_count": null,
+ "id": "94c8d5e9",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "(rcvbuf[1], rank) = (0, 0)\n",
- "(rcvbuf[1], rank) = (0, 1)\n",
- "(rcvbuf[1], rank) = (3, 2)\n",
- "(rcvbuf[1], rank) = (2, 3)\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
@@ -1184,7 +978,7 @@
},
{
"cell_type": "markdown",
- "id": "0890f627",
+ "id": "2b37ed85",
"metadata": {},
"source": [
"Another solution is to use `MPI_Sendrecv`."
@@ -1192,21 +986,10 @@
},
{
"cell_type": "code",
- "execution_count": 94,
- "id": "d648e1cf",
+ "execution_count": null,
+ "id": "9c0c84f0",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "(rcvbuf[1], rank) = (0, 1)\n",
- "(rcvbuf[1], rank) = (0, 0)\n",
- "(rcvbuf[1], rank) = (3, 2)\n",
- "(rcvbuf[1], rank) = (2, 3)\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"code = quote\n",
" using MPI\n",
@@ -1227,6 +1010,270 @@
"run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
]
},
+ {
+ "cell_type": "markdown",
+ "id": "4fca501c",
+ "metadata": {},
+ "source": [
+ "## Communication modes\n",
+ "\n",
+ "In all cases, it is safe to reuse the send buffer once the corresponding send returns. I.e., all the following sends are *complete* MPI operations. However, there are some important differences.\n",
+ "\n",
+ "### Standard\n",
+ "\n",
+ "* `MPI_Send` \n",
+ "* Programmer cannot make any assumptions whether the message is buffered or synchronous.\n",
+ "* This is up to the system.\n",
+ "\n",
+ "### Buffered\n",
+ "\n",
+ "* `MPI_Bsend`\n",
+ "* Programmer provides additional internal buffer space with function `MPI_Buffer_attach`.\n",
+ "* `MPI_Bsend` completes when message is copied into a local buffer.\n",
+ "* Erroneous if buffer space is insufficient.\n",
+ "\n",
+ "### Synchronous\n",
+ "\n",
+ "* `MPI_Ssend`\n",
+ "* The send will only return if a matching receive was posted.\n",
+ "* No buffering → easy to get deadlocks\n",
+ "* It can be started whether or not a matching receive was posted.\n",
+ "\n",
+ "### Ready\n",
+ "\n",
+ "* `MPI_Rsend`\n",
+ "* It may be started only if the matching receive is already posted.\n",
+ "* Erroneous if there is no matching receive yet.\n",
+ "* Otherwise, same as an `MPI_Ssend`.\n",
+ "\n",
+ "\n",
+ " All these send types are matched with a `MPI_Recv`. I.e., there is no `MPI_Brecv`, `MPI_Srecv`, `MPI_Rrecv`. For further information about the communication modes, refer to [this section](https://www.mpi-forum.org/docs/mpi-2.2/mpi22-report/node53.htm) of the MPI standard.\n",
+ " \n",
+ " \n",
+ "
\n",
+ "Note: `MPI_Bsend`, `MPI_Ssend`, and `MPI_Rsend` are not exposed in the Julia bindings via a high-level interface like for `MPI.Send`, but they can be accessed using the low-level bindings in the submodule `MPI.API` (not shown in this notebook).\n",
+ "
\n",
+ " \n",
+ " \n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7a935b14",
+ "metadata": {},
+ "source": [
+ "## Non-blocking send and receive\n",
+ "\n",
+ "- They return immediately.\n",
+ "- The send and receive is not completed when the function returns. I.e. they are *incomplete* operations.\n",
+ "- `MPI_Wait` used to wait for completion of the send and/or receive.\n",
+ "\n",
+ "- Used to overlap communication and computation (\"latency-hiding\").\n",
+ "\n",
+ "\n",
+ "```julia\n",
+ "request = MPI.Isend(sndbuf, comm; dest, tag)\n",
+ "request = MPI.Irecv!(rcvbuf, comm; source, tag)\n",
+ "```\n",
+ "\n",
+ "```C\n",
+ "int MPI_Isend(const void *sndbuf, int count, MPI_Datatype datatype, int dest,\n",
+ " int tag, MPI_Comm comm, MPI_Request *request)\n",
+ "int MPI_Irecv(void *rcvbuf, int count, MPI_Datatype datatype,\n",
+ " int source, int tag, MPI_Comm comm, MPI_Request *request)\n",
+ "\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "85903e61",
+ "metadata": {},
+ "source": [
+ "### Example\n",
+ "\n",
+ "Send 5 integers from rank 2 to rank 3. Both ranks do some work while messages are being communicated."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "42284c69",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "code = quote\n",
+ " using MPI\n",
+ " MPI.Init()\n",
+ " work() = sum(rand(1000))\n",
+ " comm = MPI.COMM_WORLD\n",
+ " rank = MPI.Comm_rank(comm)\n",
+ " if rank == 2\n",
+ " sndbuf = [1,2,3,5,8]\n",
+ " request = MPI.Isend(sndbuf, comm; dest=3, tag=0)\n",
+ " work()\n",
+ " MPI.Wait(request)\n",
+ " end\n",
+ " if rank == 3\n",
+ " rcvbuf = zeros(Int,5)\n",
+ " request = MPI.Irecv!(rcvbuf, comm; source=2, tag=0)\n",
+ " work()\n",
+ " MPI.Wait(request)\n",
+ " @show rcvbuf\n",
+ " end\n",
+ "end\n",
+ "run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0180c576",
+ "metadata": {},
+ "source": [
+ "### Incorrect program\n",
+ "\n",
+ "This program in incorrect both on the send and the receive side.\n",
+ "\n",
+ "- One needs to wait for completion before reseting the send buffer\n",
+ "- One needs to wait for completion before using the receive buffer\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "52991807",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "code = quote\n",
+ " using MPI\n",
+ " MPI.Init()\n",
+ " comm = MPI.COMM_WORLD\n",
+ " rank = MPI.Comm_rank(comm)\n",
+ " if rank == 2\n",
+ " sndbuf = [1,2,3,5,8]\n",
+ " request = MPI.Isend(sndbuf, comm; dest=3, tag=0)\n",
+ " sndbuf .= 10 # We cannot set the sndbuf before MPI.Wait.\n",
+ " MPI.Wait(request)\n",
+ " end\n",
+ " if rank == 3\n",
+ " rcvbuf = zeros(Int,5)\n",
+ " request = MPI.Irecv!(rcvbuf, comm; source=2, tag=0)\n",
+ " @show rcvbuf # Not guaranteed to have the correct value.\n",
+ " MPI.Wait(request)\n",
+ " end\n",
+ "end\n",
+ "run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4cc264f1",
+ "metadata": {},
+ "source": [
+ "### Which is the incoming message size?\n",
+ "\n",
+ "If we use `MPI_Probe` we miss the opportunity to do local work before a send is started."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "03b11966",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "code = quote\n",
+ " using MPI\n",
+ " MPI.Init()\n",
+ " work() = sum(rand(1000))\n",
+ " comm = MPI.COMM_WORLD\n",
+ " rank = MPI.Comm_rank(comm)\n",
+ " if rank == 2\n",
+ " sleep(5) # Sleep 5 seconds\n",
+ " sndbuf = [1,2,3,5,8]\n",
+ " request = MPI.Isend(sndbuf, comm; dest=3, tag=0)\n",
+ " MPI.Wait(request)\n",
+ " end\n",
+ " if rank == 3\n",
+ " # We are going to wait here for about 5 seconds\n",
+ " # Missing the opportunity to do some useful work\n",
+ " status = MPI.Probe(comm,MPI.Status; source=2, tag=0)\n",
+ " count = MPI.Get_count(status,Int)\n",
+ " rcvbuf = zeros(Int,count)\n",
+ " request = MPI.Irecv!(rcvbuf, comm; source=2, tag=0)\n",
+ " work()\n",
+ " MPI.Wait(request)\n",
+ " @show rcvbuf\n",
+ " end\n",
+ "end\n",
+ "run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6761b4fa",
+ "metadata": {},
+ "source": [
+ "We can fix this using an `MPI_Iprobe`. It allows us to check for incoming messages without blocking.\n",
+ "\n",
+ "```julia\n",
+ "ismsg, status = MPI.Iprobe(comm, MPI.Status; source, tag)\n",
+ "```\n",
+ "```C\n",
+ "int MPI_Iprobe(int source, int tag, MPI_Comm comm, int *flag,\n",
+ " MPI_Status *status)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "75fdb2b1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "code = quote\n",
+ " using MPI\n",
+ " MPI.Init()\n",
+ " work() = sum(rand(1000))\n",
+ " comm = MPI.COMM_WORLD\n",
+ " rank = MPI.Comm_rank(comm)\n",
+ " if rank == 2\n",
+ " sleep(5) # Sleep 5 seconds\n",
+ " sndbuf = [1,2,3,5,8]\n",
+ " request = MPI.Isend(sndbuf, comm; dest=3, tag=0)\n",
+ " MPI.Wait(request)\n",
+ " end\n",
+ " if rank == 3\n",
+ " while true\n",
+ " ismsg, status = MPI.Iprobe(comm, MPI.Status; source=2, tag=0)\n",
+ " if ismsg\n",
+ " count = MPI.Get_count(status,Int)\n",
+ " rcvbuf = zeros(Int,count)\n",
+ " reqrcv = MPI.Irecv!(rcvbuf, comm; source=2, tag=0)\n",
+ " work()\n",
+ " MPI.Wait(reqrcv)\n",
+ " @show rcvbuf\n",
+ " break\n",
+ " end\n",
+ " work() # Do work while waiting for an incoming message.\n",
+ " end\n",
+ " end\n",
+ "end\n",
+ "run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c168ca8e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "? MPI.Iprobe"
+ ]
+ },
{
"cell_type": "markdown",
"id": "6caa8d74",
@@ -1285,21 +1332,10 @@
},
{
"cell_type": "code",
- "execution_count": 13,
+ "execution_count": null,
"id": "1f8a70c6",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "I am sending 20\n",
- "I am sending 30\n",
- "I am sending 40\n",
- "I have received: [20, 30, 40]\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"code = quote\n",
" using MPI; MPI.Init()\n",
@@ -1344,21 +1380,10 @@
},
{
"cell_type": "code",
- "execution_count": 14,
+ "execution_count": null,
"id": "5ea413fd",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "I am sending: [20, 30, 40]\n",
- "I have received: 20\n",
- "I have received: 30I have received: 40\n",
- "\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"code = quote\n",
" using MPI; MPI.Init()\n",
@@ -1406,21 +1431,10 @@
},
{
"cell_type": "code",
- "execution_count": 15,
+ "execution_count": null,
"id": "4de15781",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "I am sending: 20\n",
- "I have received: 20\n",
- "I have received: 20\n",
- "I have received: 20\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"code = quote\n",
" using MPI; MPI.Init()\n",
@@ -1449,20 +1463,10 @@
},
{
"cell_type": "code",
- "execution_count": 17,
+ "execution_count": null,
"id": "ff61f64a",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Done!\n",
- "Done!\n",
- "Done!\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"code = quote\n",
" using MPI; MPI.Init()\n",
@@ -1828,15 +1832,15 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Julia 1.9.0",
+ "display_name": "Julia 1.10.0",
"language": "julia",
- "name": "julia-1.9"
+ "name": "julia-1.10"
},
"language_info": {
"file_extension": ".jl",
"mimetype": "application/julia",
"name": "julia",
- "version": "1.9.0"
+ "version": "1.10.0"
}
},
"nbformat": 4,