\n",
- "Question: Which of the two loops in the Gauss-Seidel method are trivially parallelizable?\n",
- "
\n",
- "\n",
- "```julia\n",
- "for t in 1:niters\n",
- " for i in 2:(n+1)\n",
- " u[i] = 0.5*(u[i-1]+u[i+1])\n",
- " end\n",
- "end\n",
- "```\n",
- "\n",
- " a) Both of them\n",
- " b) The outer, but not the inner\n",
- " c) None of them\n",
- " d) The inner, but not the outer\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "4edad93f",
- "metadata": {
- "scrolled": true
- },
- "outputs": [],
- "source": [
- "answer = \"x\" # replace x with a, b, c or d\n",
- "gauss_seidel_1_check(answer)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9df06442",
- "metadata": {},
- "source": [
- "## Parallelization of the Jacobi method\n",
- "\n",
- "Now, let us parallelize the Jacobi method."
- ]
- },
{
"cell_type": "markdown",
"id": "97a2d7d5",
@@ -521,203 +466,16 @@
},
{
"cell_type": "markdown",
- "id": "8ed4129c",
- "metadata": {},
- "source": [
- "## MPI implementation\n",
- "\n",
- "We consider the implementation using MPI. The programming model of MPI is generally better suited for data-parallel algorithms like this one than the task-based model provided by Distributed.jl. In any case, one can also implement it using Distributed.jl, but it requires some extra effort to setup the remote channels right for the communication between neighbor processes.\n",
- " \n",
- "Take a look at the implementation below and try to understand it.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "e15082fb",
- "metadata": {},
- "outputs": [],
- "source": [
- "] add MPI"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "a66cbf9a",
- "metadata": {},
- "outputs": [],
- "source": [
- "using MPI"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "d39c7bb2",
- "metadata": {
- "code_folding": []
- },
- "outputs": [],
- "source": [
- "code = quote\n",
- " using MPI\n",
- " MPI.Init()\n",
- " function jacobi_mpi(n,niters)\n",
- " comm = MPI.COMM_WORLD\n",
- " nranks = MPI.Comm_size(comm)\n",
- " rank = MPI.Comm_rank(comm)\n",
- " if mod(n,nranks) != 0\n",
- " println(\"n must be a multiple of nranks\")\n",
- " MPI.Abort(comm,1)\n",
- " end\n",
- " load = div(n,nranks)\n",
- " u = zeros(load+2)\n",
- " u[1] = -1\n",
- " u[end] = 1\n",
- " u_new = copy(u)\n",
- " for t in 1:niters\n",
- " # Communication\n",
- " if rank != 0\n",
- " neig_rank = rank-1\n",
- " s = 2\n",
- " r = 1\n",
- " MPI.Sendrecv!(view(u,s:s),view(u,r:r),comm;dest=neig_rank,source=neig_rank)\n",
- " end\n",
- " if rank != (nranks-1)\n",
- " neig_rank = rank+1\n",
- " s = load+1\n",
- " r = load+2\n",
- " MPI.Sendrecv!(view(u,s:s),view(u,r:r),comm;dest=neig_rank,source=neig_rank)\n",
- " end\n",
- " # Local computation\n",
- " for i in 2:(load+1)\n",
- " u_new[i] = 0.5*(u[i-1]+u[i+1])\n",
- " end\n",
- " u, u_new = u_new, u\n",
- " end\n",
- " # Gather the results\n",
- " if rank !=0\n",
- " lb = 2\n",
- " ub = load+1\n",
- " MPI.Send(view(u,lb:ub),comm,dest=0)\n",
- " u_all = zeros(0) # This will nevel be used\n",
- " else\n",
- " u_all = zeros(n+2)\n",
- " # Set boundary\n",
- " u_all[1] = -1\n",
- " u_all[end] = 1\n",
- " # Set data for rank 0\n",
- " lb = 2\n",
- " ub = load+1\n",
- " u_all[lb:ub] = view(u,lb:ub)\n",
- " # Set data for other ranks\n",
- " for other_rank in 1:(nranks-1)\n",
- " lb += load\n",
- " ub += load\n",
- " MPI.Recv!(view(u_all,lb:ub),comm;source=other_rank)\n",
- " end\n",
- " end\n",
- " return u_all\n",
- " end\n",
- " function jacobi(n,niters)\n",
- " u = zeros(n+2)\n",
- " u[1] = -1\n",
- " u[end] = 1\n",
- " u_new = copy(u)\n",
- " for t in 1:niters\n",
- " for i in 2:(n+1)\n",
- " u_new[i] = 0.5*(u[i-1]+u[i+1])\n",
- " end\n",
- " u, u_new = u_new, u\n",
- " end\n",
- " u\n",
- " end\n",
- " function testit(load)\n",
- " comm = MPI.COMM_WORLD\n",
- " nranks = MPI.Comm_size(comm)\n",
- " rank = MPI.Comm_rank(comm)\n",
- " n = load*nranks\n",
- " niters = 100\n",
- " u_par = jacobi_mpi(n,niters)\n",
- " if rank == 0\n",
- " # Compare agains serial\n",
- " u_seq = jacobi(n,niters)\n",
- " if u_par ≈ u_seq\n",
- " println(\"Test passed 🥳\")\n",
- " else\n",
- " println(\"Test failed\")\n",
- " end\n",
- " end\n",
- " end\n",
- " testit(3)\n",
- "end\n",
- "run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "eff25246",
- "metadata": {},
- "source": [
- "\n",
- "Question: In function jacobi_mpi, how many messages per iteration are sent from a process away from the boundary (excluding the messages to collect the data on rank 0)?\n",
- "
\n",
- "\n",
- " a) 1\n",
- " b) 2\n",
- " c) 3\n",
- " d) 4\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "98bd9b5e",
- "metadata": {},
- "outputs": [],
- "source": [
- "answer = \"x\" # replace x with a, b, c or d\n",
- "jacobi_2_check(answer)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "c9aa2901",
- "metadata": {},
- "source": [
- "### Latency hiding\n",
- "\n",
- "Can our implementation above be improved? Note that we only need communications to update the values at the boundary of the portion owned by each process. The other values (the one in green in the figure below) can be updated without communications. This provides the opportunity of overlapping the computation of the interior values (green cells in the figure) with the communication of the ghost values. This technique is called latency hiding, since we are hiding communication latency by overlapping it with computation that we need to do anyway.\n",
- "\n",
- "The modification of the implementation above to include latency hiding is leaved as an exercise (see below).\n"
- ]
- },
- {
- "attachments": {
- "fig16.png": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6gAAADyCAYAAABAvOgkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7d17nFxlYf/xz9lssrmQm4EQkhACRi7h1kD4YSIqEGuwEi4tSNEakWpqBWuLtEKrgtW2WkDbn1VJ0Cra3w8lKMKGCIJc1BBCxCgghGsSJIEk5ELut93pH89s9uxkZnez7M7z7O7n/XqdV84zc2bmm9nNZL97Lk82e/bsApIkSZIkRVYTO4AkSZIkSWBBlSRJkiQlorZkPD3LsuUxgkiSJEmSepdCoTAeuKdp3KKgZlm2fNasWc9WO5QkSZIkqfeZM2cOhULzZZE8xFeSJEmSlAQLqiRJkiQpCRZUSZIkSVISLKiSJEmSpCRYUCVJkiRJSbCgSpIkSZKSYEGVJEmSJCXBgipJkiRJSoIFVZIkSZKUBAuqJEmSJCkJFlRJkiRJUhIsqJIkSZKkJFhQJUmSJElJsKBKkiRJkpJgQZUkSZIkJcGCKkmSJElKggVVkiRJkpQEC6okSZIkKQkWVEmSJElSEiyokiRJkqQkWFAlSZIkSUmwoEqSJEmSkmBBlSRJkiQlwYIqSZIkSUqCBVWSJEmSlAQLqiRJkiQpCRZUSZIkSVISLKiSJEmSpCTUxg4gSZKSMxB4b3F9PrA1YpZq619cdtO7/t6SlAQLqiRJKjUCuLW4/mbgxQ4+Tw3wFmAyMAkYXLz9CtItf/8IfBa4H5gWOYsk9ToWVEmS1BW+AXwAGFLmvqtJt6DG8F7gHOA54PrIWSQpKguqJEkqtRG4qri+voPPMZHmcrqy+JzHvsFc1fAH4BHg91V8zZOAWcAvqF5BPRQ4ObccCWTAbcCnq5RBkvZhQZUkSaU2A19+g89xE3Ad8BjwKvAh4Ltv8Dmr4abi0pNNBhZXuO+gagaRpFIWVEmS1BX+X+wA3ci3CBej2rKfjzsSOJFwjvBjHXjdDcCvi4/9IDCmA88hSZ3KaWYkSVKpQ4FCcTkicpZquwR4ALihiq/5CqEkPrOfj5tBuJjVrP183LOEi1+9CXg34Zzgdfv5HJLUJdyDKkmS1OwI4HSgsYqvOZFwfu5a4MEqvN6m4iJJyXEPqiRJUlx/RtgT+vnYQSQpNguqJEmSJCkJFlRJkiRJUhI8B1WSJLXXUGBChfuew/MaJUlvkAVVkiS11zuAOyvcdxZwTxWz9BZHUPkqvacU/zwV+FKFbW4HFnV2KEnqKhZUSZLUXhupPN+me0+7xjjg021sc2JxKWcZFlRJ3YgFVZIktdcvgcmxQ/QyLwFfrnDfKcCZwO+Auyts89uuCCVJXcWCKkmSlK4Xgasq3PcpQkFd1Mo2ktSteBVfSZIkSVIS3IMqSZK6womECyc1OSm3/klgW278dWBLNUJJktJmQZUkSV2htSvLfq5k/D0sqJIkLKiSJGlfW4A5xfWOXp336dxztGVb25tUzRZgLbAhdpAu9h1gbG58RPHP6cC9udsfAP61WqEkyYIqSZJKbQD+6g0+xy+LS3fz78Wlp5sCHFXm9tHFpcna6sSRpMCCKkmSFNePgKXsfxmsJ0xD82IHXvNvgcHt2O6lDjy3JHWYBVWSJCmup4rL/nq2uHREpXlTJSkqp5mRJEmSJCXBgipJkiRJSoIFVZIkSZKUBAuqJEmSJCkJFlRJkiRJUhIsqJIkSZKkJDjNjJJXKBSy+vr6R2PnaM0NN9xw2LJlywYCbN68+bUNGzakNrH5gcBBxfXtwPJ4UcoaAIzPjZ8BGuNEKW/cuHFHUfyl3iWXXPLy5MmTN0eOVFGWZbfOmDHjumq8Vn19/XcLhcKx1Xitjpg7d+7Ihx56aATAzp07t65evTq1OR0HAeNy46djBWnFMbn1l4CtsYJUMI7wPgKsA9ZEzLKPYcOGjRgyZMhIgDFjxuy4+uqrl8XO1Jq6urrTp0+f3uVf47vuuusdDQ0NN3T163TUjh07aq688sqjmsarV69evnPnzu0xM5UxnvD/J4Q5dF+LF6Wsgwg/fwBsA1ZEzLKP/v37Dxw5cuRhTeOvfe1rT2dZFjNSWz5xzjnnPBI7RDVYUJW8z3/+89nJJ588OXaO1qxfv56XXtr7c+8g4LBWNo9tEM3/YaTqpNgBSuW+vjQ0NByV+H9ii6r1QoVC4dgsy5L997l9+/bSf5sjI8Zpj2Tfy6Jj2t4kqtLCH93GjRvZuHEjAP369RuUZdmIyJFaVVNTU5WfDRsaGoan/NmRZVmLz30g2V/EFQ2i5S96UzOI5l+UJ2HHjh2lX+PJKf/fnmXZ0NgZqsVDfCVJkiRJSXAPqrqjm7IseyF2iLzdu3dfQ/EwmzFjxqxYuXLlNyNHKvWXwFuK67uAz0XMUs5k4ILc+IvAlkhZyqqtrf3Snj17AFi/fv0DWZbdEzlSqZmFQmFizACFQuHumpqaB2NmKLV58+aPU9yjNmLEiO3r1q37fORIpd4B/ElufFWsIK34UvPqmIfh4CXxopTz249AY11x8BLwjZhpSo0dO/bSl19++UiAhobCnhdfHDY7dqa8gQN3jRw1atuFkWPszrLss5EztLBz584DgM80jUeNGnXbq6+++uuIkcr5Z6Bfcf154FsRs5TzMZr36u4Aro2WpIzRo0eftmrVqrObxnv27Pmnurq6hpiZShUKhX8D0t2t20UsqOp2CoXCD2bMmHF/7Bx5EyZMuIpiQR03btzKlStXfjlypFLvormg7gZSy/cRWhbU/wJWR8pSVp8+ffYW1GXLli06++yzk3oP582bNwWIWlCzLPtFau/L1KlTz6NYUIcPH75j3bp1SeUjnGudL6j/DhQiZakkV1An/gZ+9v14Ucrp+8FcQV1FYp9vRxxxxJnNBbVhzyc/edr3YmfKu/TSpyaef/6L0Qtqap8dF1988cHkCuopp5xyT319fWoF8DM0F9QVJPa9D8ygZUFNKt/JJ5+8K19Qf/CDH1w/d+7cXTEzlaqvr/9XemFB9RBfSZIkSVISLKiSJEmSpCRYUCVJkiRJSbCgSpIkSZKSYEGVJEmSJCXBgipJkiRJSoIFVZIkSZKUBAuqJEmSJCkJFlRJkiRJUhIsqJIkSZKkJFhQJUmSJElJsKBKkiRJkpJgQZUkSZIkJcGCKkmSJElKggVVkiRJkpQEC6okSZIkKQkWVEmSJElSEmpjB1CXGQBMzI2XAI1tPKYGmJQbPwVs7+RckiRJklSWBbXnOgL4dW48kLbLZr+SxxwPPNnJuSRJkiSpLA/xlSRJkiQlwYIqSZIkSUqCBVWSJEmSlAQLqiRJkiQpCRZUSZIkSVISLKiSJEmSpCRYUCVJkiRJSbCgSpIkSZKSYEGVJEmSJCXBgipJkiRJSoIFtecqlIyzdjzmgK4IIkmSJEntYUHtubaUjNtTPg/riiCSJEmS1B4W1J5rPS33ok5ox2Pe1UVZJEmSJKlNFtSeawuwLDc+q43tBwOXdV0cSZIkSWqdBbVnuy+3fjkwpsJ2dcDNwKFdnkiSJEmSKrCg9mxzgMbi+nDgl8AFNJ+PeiBwEfAocD7w22oHlCRJkqQmtbEDqEs9BnwN+GRxfDgwt7jeSMtfUKwilNfnq5au4y6rr68/N3aIvH/4h38Y0LT+/PPPHw78Z8Q45RyZW+9HevmOLRl/EdgWI0gle/bs2bs+YcKEd9XX1w+MGGcfhULh+NgZgLPr6+tHxQ6R981vfnN80/ratWsHkN73/kkl4/+IkqLdfvt2mHBQ7BQtNfbPDcaT2Nd46dKlRzet9+lT0/emmx64ImaeUn377nlT7AxAv/r6+qS+bps2bRp4yy237B0vWLDgz4EUPmfz6nLrR5LY9z7h584myX3+Lly4cFJ+/P73v/+GmTNnNlbaPpL2zMLR41hQe75PFf/8BC0LaX79V8CfA+uqFeqNyLLsT2NnKFVb2/xPae3atYcAfxMvTZv6knY+gI/EDlCqoaFh7/rQoUMnA5PjpUnW1OKSjIEDm3+P8Prrr/cn/e/9xPOtPTEsyRpFYu/hmjVr9q736VPTZ9SorRdFjJOqWhL7uvXr16/FeP369dOAaXHStMuhJPYelqgjsXyvvfZai3FNTc3lkaKohAW152sA/hb4NqGEngQcRLjK7wuEPaoPEvao9gHel3vsH6oZVJIkSVLvZkHtPZ4oLq1poPkQ4GRcc801hXnz5t0ZO0drRo8efXRDQ8MQgA0bNqx65ZVXXo6dqcRoYGxxfQvwVMQs5RwATMyNf03z+dNJOProo0+uqanpA9C3b9/ngA2RI1WUZdnjVXythwinCCRpyJAhYydOnDgaYPv27ZuWLVu2NHamEkOAo3PjR2MFacX/ya0vBTbFClLBUcDQ4vorJPbL1YMPPviQESNGHFpc3wr8PnKkVu3YsWN3NV4ny7JXgJT/b6+ZOHHi3iNlVqxY8dTWrVtL55iPbSLN1xV5mfQ+i8cSfv6A8LmR1OfvAQccMHjcuHHHNI0LhcJiWk7RmJSGhoY1bW/VM2SzZ8/e+4XIsuyoWbNmPRszkCRJkiSpd5gzZ86RhULhmaaxV/GVJEmSJCXBgipJkpS+fm1vIkndnwVVkiQpfZ8lXHX/wNhBJKkrWVAlSZLS9yvgZOBx4MzIWSSpy1hQJUmS0rcQeB04BLiLMH1c36iJJKkLWFAlSZLStwnYU1zvD7wfWAIcES2ROqp/7ABSyiyokiRJ3cP9ufX+hHkwFwJ/ESeOOqA/4XDt2XjhK6ksC6okSVL3cCewNTfOgJHAfxXvGxgjlPbLjYRzic8DDoqcRUqSBVWSJKl7eAjYUub2ocB04GnghKom0v74OPAhoIGw13tl3DhSmiyokiRJ3cNaYFeF+/oB4wiHAV9VtURqrynAV4vrnwbujZhFSpoFVZIkqft4rI373wR8Abi9ClnUPgcDtxJ+iXA78JW4caS01cYOIEmSpHa7A3gPUFdy+zbgeeA24DfA+irnUnm1wA+BscAzwCVAIWYgKXUWVEmSpO7jQWAjYa8chEN++xGuDnslHjqamn8H3glsBs4nTBckqRUe4itJktR9LCdcZAfCFX1/Cnyb8DPd94FD4sRSGRcBf0fYY/phwkWsJLXBgipJktS9PEUoPU8DfwZ8AniCsFf1Zvz5LgWTCb84APg34EcRs0jdih9gkiRJ3cudhL2oFxT/3E7YW7cV+GPgc/GiCRgP1AODgPn49ZD2iwVVkiSpe5kP/F9gRe62p4GPFdc/B5xX7VACYDjh6zOKcLGqi2g+JFtSO1hQJUmSupcXgE+Vuf1/CHNtZoTzUY+rZijRD5gLHAOsBM4FtkRNJHVDFlRJkqSe4++Bu4EDgB8Dw+LG6TUy4FvANMKVev8EeDlqIqmbsqBKkiT1HA3AXwDLgLcA/x9/3quGLwIfJEz7cz7weNw4UvflB5YkSVLPso5wDupW4D3A1+PG6fH+EvhHwpWVZwH3x40jdW8WVEmSpJ7nceASoJFw8aQroqbpud4L3Fhcv4YwzY+kN8CCKkmS1DPdRjgnFeA64MKIWXqiMwnvcS3w38AX4saRegYLqiRJUs/1FeA/CT/zfQ84LW6cHmMKcAfQH/gJ8Fdx40g9hwVVkiSpZ7uCcEXfpjLl9DNvzCnATwlXSp5PmOt0T9REUg9iQZUkSerZGglX9n0YGAHcCxwVNVH3dSrwM2Ao4WJIFxCu3Cupk1hQJUmSer7thAv6/AYYBfwceHPURN3PaYRyOgz4BXAu4X2V1IksqJIkSb3DRmAasAQYAzwAjI8ZqBt5B+Fw3iHAQ4SyvyVqIqmHsqBKkiT1HhsJc6MuBQ4F7sOS2pZzgLuBwcU/34PlVOoyFlRJkqTeZTXwLuB5wmG+vwSOiZooXR8lXGBqAOGqvefhYb1Sl7KgSpIk9T4rgbcDvwPGAguAt0ZNlJ5PA3OAPsB3CRdE2hkzkNQbWFAlSZJ6p1eBM4FFwHDCBYDOiJooDXXAd4AvFcf/DHwYp5KRqsKCKkmS1HutJxzuex/N51heGjVRXAcTpo+5hFBIPwZcEzOQ1NtYUCVJknq3LcDZwA+BfsC3gRsIh7b2JpOAR4GphOL+HmB21ERSL2RBlSRJ0k7gYuAqoBG4AriLMOdnbzAT+BUwDngOeBthr7KkKrOgSpIkCaAAfBl4H7AVmA4sBibHDNXFhgFzgZuBgcCdwCmEaXgkRWBBlSRJUt6PCFf4XQ5MIFzh91NAFjFTVzgd+A3h6ry7gL8jTCPzesRMUq9nQZUkSWr2l4R5Qb8aO0hkSwjnZN5KOC/1emA+cEjMUJ3kAOC/CBdDOhx4gXBI738Q9iJLisiCKkmS1Oww4DTghNhBErARuAj4KLANOAt4Gvg43fdnyLOAx4HLiuMbCUX819ESSWqhu364SJIkqTq+RTgP9RFgKPB1wgWFjo8Zaj+9hXB+6U8Je02XEabX+Wtgc8RckkpYUCVJkpp9CTgIOD92kMQ8TTgM9jLCOZpTCOdv3gSMjZirLQcRDk9+EphBONf0esIe8vsj5pJUgQVVkiSp2TbgNWBT7CAJagS+AUwkXPm2FvgIYVqW64ED40Xbx0jgOsKe0k8RzqO9i7DX9+8Jc79KSpAFVZIkqdkUwhygF8QOkrBVhKlopgAPAP0JJXAFMIe4h/5OKmZYBlwJDAIWEabMORt4Nl40Se1hQZUkSWr2HuAGwrmJat0jwJnAuwklcCDhgkqPE4rrTMI8o13toOLrPkI47PijxSyPAH8CvBX4WRVySOoEtbEDSJIkqVu7t7hMBf4G+FPCHKOnE875vBf4MfAQYUqXN6oGOBY4g3Cu8NuBPsX7dhLmcb2RMF2QpG7GgipJkqTO8HBxGU2YT/YCwsWI3ltcAF4FFhD2dL5QXFYA6wjnuOb1J1w1+AjgSMKVeCcRinB+z2wBWEyYs/VmYG3n/rUkVZMFVZIkSZ1pFfCF4nIUoaieRZiqZhTwZ8Wl1G6aL140mNZ/Tt0MLCRMG/Nj4KXOCC4pPguqJEmSusozwL8UlzpCSZ0KHAO8mbB3dAyQAX2B4SWPbySUz+cJVwt+irAH9nGgoevjS6o2C6okSZKqYSehXC4oub0PMAQYQDisNyNM97MVp/uReh0LqiRJkmJqADYUF0m9nNPMSJIkSZKSYEGVJEmSJCXBgipJkiRJSoIFVZIkSZKUBAuqJEmSJCkJFlRJkiRJUhKcZkaSJKnZbOAu4PXYQSSpN7KgSpIkNVtZXCRJEXiIryRJkiQpCRZUSZIkSVISLKiSJEmSpCRYUCVJkiRJSbCgSpIkSZKSYEGVJEmSJCXBaWbULdx+++3jY2doTX19/YEbNmzoD7Bq1apNixYt2hQ7U4nBwNDi+i5gTcQs5fQDRubGK4FCpCxlnXvuuaNrampqAE466aR1xx133PbYmSoZMGDAprPOOmt9NV7r7rvvPmT79u111XitjliwYMGQF154YQjAli1bdtx7772vxc5Uog44KDd+OVaQVozNra8FdsYKUsGBQP/i+mYSm790ypQpg0eNGjUUYPDgwbvOP//81D5/WzjvvPNWZFnW5Z+/t95664C+ffse3NWv01G7d+/ObrnlljFN40cffXTNypUrd8XMVMZIwv+fEL7vN0fMUs6Q4gLhc2NtxCz7OPzww+v+6I/+aO/n78yZM1P8/N1r2LBhr55xxhk7YueoBguqknfttdfW1NbWLoudozWLFy/miSeeiB1DXeiOO+7Yuz5x4kRqa9P9+Ny9e/fXgcur8Vq7du26s7a2dnI1Xqsjli9fzu233x47hnqxhQsX7l2fMGECF154YcQ0bbvvvvuGUYWS379//3dnWfaTrn6djtqzZ4+fHT3csmXLWLas+cfLSy65hOLvoZO0devWs4B7YueohnS/CpIkSZKkXsWCKkmSJElKQrrHqEmV/UWWZQvb3qx6duzYsYTieRaTJk16bMmSJe+LHKnUzcBpxfVtwPERs5TzPuDfcuNTgaTOFayrq3th585w6t3SpUtvnDJlynWRI5W6sVAo/HHkDNdlWXZj5AwtrFmz5jZgEsD48eNfX758+UmRI5X6KHDV3tFX+Xi8KBX8Hd/Yu/5Obuc87o2YZl9/z/XsYWAYjH4W/vGq1h9QXUceecdnn3323kkAu3fv3p5l2XGxM+U1NjYen8Chtsm9L3/4wx8OBBY1jU899dSrFy1adGvESOU8AU3f+ywAZkbMUs4PgaZTQDZR/CxOxdSpUy99+OGH/6lpvGnTpqOHDx++O2amUoVC4Tl64Q5FC6q6nUKh8MqMGTNejJ0jb8KECY1N6/37998JJJUPyJ9UXyC9fKVldAWwOkaQ9ti6devGs88+O6n3cN68edtiZwA2pPa+TJ06de8FfWpraxtJ73u/5cWszmQNNWldIKyFYWzhXYn928xobB7U7oLLVsYLs6+6uvv3fg8WCoXG1P6N3HnnnSNiZwAKqb0vF1988db8eOTIka+R3udH7nufHaSXL39BteQ+f0eMGLEuP77zzjuXzZ07N6kLYdXX18eOEEWva+SSJEmSpDRZUCVJkiRJSbCgSpIkSZKSYEGVJEmSJCXBgipJkiRJSoIFVZIkSZKUBAuqJEmSJCkJFlRJkiRJUhIsqJIkSZKkJNTGDtDDZcCbgTFAf2AV8BTQ8AafdwIwChgMbAReBFa/weeUJEmSpKhS34M6Hnghtwxux2NqSh5zXFeFK/p47rX+J5fhE8BS4DngQeBu4HFCkfwCMHA/X+dQ4OvAy8Xn/CUwH3gYeAX4DfBhWv+aXp3L+p9tvN4/0fJ9/HAb29+W2/ZDbWwrSZIkSftIfQ9qX+CI3Li9hTr/mLrOi1PW8NzrLSeU6NuAd1fYfgTwGeBdwHRgUzte4zLgesJe2HIyYBLw34QieR6wvsx2T+ayXgj8LVCo8Jzn0/J9PAf4ToVtDwBmAP2K499W2E6SJEmSKkq9oHY3NYS9qO8mHMb7EPAYsBk4DDgXOLC47VsJpXNWG895LXBNbrwV+CmwBHgdGEkouqcW7387cC/wNmBHyXM9BOwhfN0PASYCvy/zmm8iFN6804E+lD88+e00l9O1hD3FkiRJkrRfLKid6zTCe/p74CL2LX9XAj8CziyOLwW+CLxU4flmAJ/LjX9I2Ju6rmS7awh7RL9LOHT4JOBfgE+VbLcJWAxMKY6nlckIcAbNe6t3EvZCDwNOBh4ts/203Pr9VN4rK0mSJEkVpX4OandTC6wkFLxyxW8j8H5gS3Hch1Asy+lLOOc0K45vBS5m33LaZC7wkdz444S9q6V+nlufVub+0ttv3M/t76+wjSRJkiS1yoLa+T5DOMy1ktXAvNx4coXtLiRcGAlCob2ctvdM3kLYQwrhfNWLy2yTL5DvpPxe9KbCuRn4MtBYcnvegcAJufHPy2wjSZIkSW2yoHau3YQ9nW15LLc+vsI25+TW76D10pv349z628vc/zCwvbg+FDil5P5DgSOL678gXCH4d8Xx24ABJdufSfP30XLCVXwlSZIkab9ZUDvXM8C2dmy3Jrc+tMI2+XK5cD8yLM2tH1Pm/p3Ar3Lj0r2i+fF9xT+b9or2B6a2sr17TyVJkiR1mBdJ6lzlpnYpZ1duvV+Z++uA0bnxFbQ9DymEKW7yc8W+qcJ2Pwf+uLg+jXChJnLj/HZNf16Zuz9fRM8ss70kSZIk7TcLaufa00nPM7xkfETZrdo2qMLt+fNQpxCu/Nu057epcL5KmDcV4JeEUt2PlgX2MGBCcb0APNDBnJIkSZJkQU1Un5Lx/VS+em9rtle4/TfABkIRriNMj/MzwryoTXtu89PFbAUeAd5BmGpmePHx+bL6e0KplSRJkqQO6YkFtdwhs91NaRn9GvCTTnz+BuBB4PzieBqhoLZ2uO7PCQW1D3A6cDuefypJkiSpE6V+kaQdJeP+7XhMubk/u5sdtNwbeUKlDd+AcvOhlrtAUqXtMzz/VJIkSVInSr2gbioZH9KOx5zaFUEiyJ/PeU7FrTouXygnEYr96cXxc8BLJds/SpgXFUJBPRYYVRzvAR7qgoySJEmSepHUC+rrwOrc+G3teMxHuyhLtf0ot34y8J5Ofv6lwMrieg3hKr3DiuNye0N3E+ZFBTgamJm779fs+8sESZIkSdovqRdUaDkH6Mdo/bzZD9M8fUp3dzvhwkNN/hs4fD8efxDNhbOS/NV8L8+tlx7e2yRfXC+vcLskSZIkdUh3KKjfz60fB9wMDCnZZiDwOeAmYEuVcnW1RkLhbjoPdxSwGLiUyheC6gOcAcwBVgBvbuM18sVyQO51K00XU2770tslSZIkqUO6w1V8fwIsoPnw3vcDZxdve52wp/CthDk/GwmHnv64+jG7xGLC3+d7hAtEjQC+DXyFsGf5ZUKBHUbYu3oicMB+PH+5YrkEWF9h+yeANbS8ENV2Wu7lliRJkqQO6Q4FtRG4CLgXOKZ42xD2PSdzG/DXwB3Vi1YVc4HlwHcJ85QCDAXOauNxrwAb29jmZeAZ4KjcbZUO74UwL+r9wJ/nblvAvldbliRJkqT91h0KKoSL+ZwCXAF8gJaFagOhxP0H8DRh+pO5Jfd3padzr/dkOx/zUu4xq1vbsGgxcDzwp8AFhMN4S6fT2UK48NEvgHsIe0cb2vHcXwPemRv/qNKGRd8nHErc3u0lSZIkqV26S0EF2Ap8obgMJhzau5F9D0ctAO+rYq4fs/+HFD9cXPZHI3BbcYFwSPMIwtew3PvQXl8vLu01v7hIkiRJUqfqTgU1bzPNc3L2VluLiyRJkiT1CN3hKr6SJEmSpF7AgipJkiRJSoIFVZIkSZKUhO56DmpHfRAY0AnP8yzwYCc8jzogy7Jb6+vrd8XOkXf11VcPaVpfvHjxZGBVxDjlvCm3PpD08pX+u/wd4cJgydi1q/lbbtKkSZdfdtllH4oYZx+FQmF47AzA1fX19Z+IHSLvO9/5zoim9WXLlg0lve/9QS1Gp3JTpBztM58LuIezY8doYU/+PVx5NPS7K16YfT31VDa0ab1fv34D6+vrNL98BAAAAwFJREFUU/se7Bs7ADAgtfdl27ZtNbfccsve8fz5868D/jleorLynx+nkd7n24jcenKfv/Pnz2/x+fuBD3xg+cyZM2PFqaRX7kzsbQX1OuDgTniem7GgxjSi7U2qK8uyvet79uzpBxwSL02bMtLOB53z77RTFQqFves1NTUHAAfES5OswcUlGTU1zf+3NzQ01JD69/4OUvhFQ2W7GcDuTvlFbxdpqIWGA2OnyGtoOeFbd/j8jSG59yX/2QHQ0NAwDBgWJ0271JHYe1giua9xQ+k/zixLKl9v1itbuSRJkiQpPb1tD+pRdE4pT+rw0p7u2muvbZw3b94HY+dozQknnHDs6NGjhwGsXLnyD08++eRLsTOVGAscVlzfDDweMUs5g4ETcuNHgIYK20Yxbdq0t9bW1vYBGDRo0NNZlnV07uEu19jY+Ey1XqtPnz6fLRQKSe2xyhs9evRh06dPHwuwadOmjQsXLvx97EwlhgLH5cYLYgVpxdty608Cr8cKUsGxNO/ZehlYETHLPo455pgx48aNGw8wbNiwzVmWpfb528Lw4cO3VeN1amtrH2tsbEz2//Ysy/pMnz79rU3jJUuWPL5mzZrUpjg8Hmg6xWgF4fs/JeOAQ4vrrxM+P5IxatSooSeeeOLez98syx7OsqzQ2mNiqq2tTfqzozNls2fP3vuFyLLsqFmzZj0bM5AkSZIkqXeYM2fOkYVCYe8v1z3EV5IkSZKUBAuqJEmSJCkJFlRJkiRJUhIsqJIkSZKkJFhQJUmSJElJsKBKkiRJkpJgQZUkSZIkJcGCKkmSJElKggVVkiRJkpQEC6okSZIkKQkWVEmSJElSEiyokiRJkqQkWFAlSZIkSUmwoEqSJEmSkmBBlSRJkiQlwYIqSZIkSUqCBVWSJEmSlAQLqiRJkiQpCRZUSZIkSVISLKiSJEmSpCRYUCVJkiRJSajNDwqFwvg5c+bEyiJJkiRJ6kUKhcL4/Li25P57CoVC9dJIkiRJklTkIb6SJEmSpCRYUCVJkiRJSfhf0SSRmgN3HHEAAAAASUVORK5CYII="
- }
- },
- "cell_type": "markdown",
- "id": "7d66b1a2",
- "metadata": {},
- "source": [
- "\n",
- "![](\"attachment:fig16.png\")
\n",
- "
\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9d4de5a9",
+ "id": "75f735a2",
"metadata": {},
"source": [
"## Extension to 2D\n",
"\n",
"\n",
- "Now, let us study the method for a 2D example."
+ "The Jacobi method studied so far was for a one dimensional Laplace equation. In real-world applications however, one solve equations in multiple dimensions. Typically 2D and 3D. The 2D and 3D cases are conceptually equivalent, but we will discuss the 2D case here for simplicity.\n",
+ "\n",
+ "Now the goal is to find the interior points of a 2D grid given the values at the boundary.\n",
+ "\n"
]
},
{
@@ -727,7 +485,7 @@
}
},
"cell_type": "markdown",
- "id": "f14142ea",
+ "id": "18659ed7",
"metadata": {},
"source": [
"\n",
- "\n",
@@ -737,7 +495,7 @@
},
{
"cell_type": "markdown",
- "id": "b1e1a12b",
+ "id": "ca411d89",
"metadata": {},
"source": [
"For the Laplace equation in 2D, the interior values in the computational grid (represented by a matrix $u$) are computed with this iterative scheme. The entry $(i,j)$ of matrix $u$ at iteration $t+1$ is computed as:\n",
@@ -750,7 +508,7 @@
},
{
"cell_type": "markdown",
- "id": "6f5d2895",
+ "id": "8c916627",
"metadata": {},
"source": [
"### Serial implementation\n",
@@ -761,7 +519,7 @@
{
"cell_type": "code",
"execution_count": null,
- "id": "4ab59b2f",
+ "id": "1e843565",
"metadata": {},
"outputs": [],
"source": [
@@ -785,19 +543,49 @@
"end"
]
},
+ {
+ "cell_type": "markdown",
+ "id": "f4dc50b1",
+ "metadata": {},
+ "source": [
+ "If you run the function for zero iterations you will see the initial condition."
+ ]
+ },
{
"cell_type": "code",
"execution_count": null,
- "id": "6da0aa54",
+ "id": "4acf219e",
"metadata": {},
"outputs": [],
"source": [
- "u = jacobi_2d(10,0)"
+ "n = 10\n",
+ "niters = 0\n",
+ "u = jacobi_2d(n,niters)"
]
},
{
"cell_type": "markdown",
- "id": "45d786dd",
+ "id": "efd4df6a",
+ "metadata": {},
+ "source": [
+ "If you run the problem for enough iterations, you should converge to the exact solution. In this case, the exact solution is a matrix with all values equal to one."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5c120b8b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "n = 10\n",
+ "niters = 300\n",
+ "u = jacobi_2d(n,niters)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "85895681",
"metadata": {},
"source": [
"### Where can we exploit parallelism?\n",
@@ -823,7 +611,7 @@
},
{
"cell_type": "markdown",
- "id": "f93e2024",
+ "id": "65e4f745",
"metadata": {},
"source": [
"### Parallelization strategies\n",
@@ -844,7 +632,7 @@
}
},
"cell_type": "markdown",
- "id": "267ecd2a",
+ "id": "11e834b5",
"metadata": {},
"source": [
"
\n",
@@ -854,7 +642,7 @@
},
{
"cell_type": "markdown",
- "id": "e0cee28b",
+ "id": "7384ddfe",
"metadata": {},
"source": [
"Which of the thee alternatives is more efficient? To answer this question we need to quantify how much data is processed and communicated in each case. The following analysis assumes that the grid is of $N$ by $N$ cells and that the number of processes is $P$."
@@ -862,7 +650,7 @@
},
{
"cell_type": "markdown",
- "id": "f6ea16f5",
+ "id": "f733d88f",
"metadata": {},
"source": [
"### 1D block partition\n",
@@ -877,7 +665,7 @@
}
},
"cell_type": "markdown",
- "id": "0cfeca62",
+ "id": "6b983ffd",
"metadata": {},
"source": [
"
\n",
@@ -887,7 +675,7 @@
},
{
"cell_type": "markdown",
- "id": "4600c94a",
+ "id": "4f1e0942",
"metadata": {},
"source": [
"- We update $N^2/P$ items per iteration\n",
@@ -898,7 +686,7 @@
},
{
"cell_type": "markdown",
- "id": "969c42ee",
+ "id": "a9bed431",
"metadata": {},
"source": [
"### 2D block partition"
@@ -911,7 +699,7 @@
}
},
"cell_type": "markdown",
- "id": "e5142fa1",
+ "id": "6f7b5b36",
"metadata": {},
"source": [
"
\n",
@@ -921,7 +709,7 @@
},
{
"cell_type": "markdown",
- "id": "091b0310",
+ "id": "abb6520c",
"metadata": {},
"source": [
"- We update $N^2/P$ items per iteration\n",
@@ -932,7 +720,7 @@
},
{
"cell_type": "markdown",
- "id": "a8fb49a3",
+ "id": "9aaf1ca6",
"metadata": {},
"source": [
"### 2D cyclic partition"
@@ -945,7 +733,7 @@
}
},
"cell_type": "markdown",
- "id": "1bc30b74",
+ "id": "77701278",
"metadata": {},
"source": [
"
\n",
@@ -955,7 +743,7 @@
},
{
"cell_type": "markdown",
- "id": "c45f4e44",
+ "id": "9cd32923",
"metadata": {},
"source": [
"- We update $N^2/P$ items\n",
@@ -966,7 +754,7 @@
},
{
"cell_type": "markdown",
- "id": "3d0693a7",
+ "id": "191fa03b",
"metadata": {},
"source": [
"### Summary\n",
@@ -980,7 +768,7 @@
},
{
"cell_type": "markdown",
- "id": "850b1848",
+ "id": "c0c6d216",
"metadata": {},
"source": [
"### Which partition is the best one?\n",
@@ -995,6 +783,935 @@
"\n"
]
},
+ {
+ "cell_type": "markdown",
+ "id": "8caf9b30",
+ "metadata": {},
+ "source": [
+ "## The Gauss-Seidel method \n",
+ "\n",
+ "Now let us study a slightly more challenging method. The implementation of Jacobi used an auxiliary array `u_new`. The usage of `u_new` seems a bit unnecessary at first sight, right? If we remove it, we get another method called Gauss-Seidel.\n",
+ "\n",
+ "\n",
+ "The following cell contains the sequential implementation of Gauss-Seidel. It is obtained taking the sequential implementation of Jacobi, removing `u_new`, and only using `u`. This method is more efficient in terms of memory consumed (only one vector required instead of two). However it is harder to parallelize."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "efd792c3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "function gauss_seidel(n,niters)\n",
+ " u = zeros(n+2)\n",
+ " u[1] = -1\n",
+ " u[end] = 1\n",
+ " for t in 1:niters\n",
+ " for i in 2:(n+1)\n",
+ " u[i] = 0.5*(u[i-1]+u[i+1])\n",
+ " end\n",
+ " end\n",
+ " u\n",
+ "end"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5a38cef6",
+ "metadata": {},
+ "source": [
+ "Note that the final solution is nearly the same as for Jacobi for a large enough number of iterations."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c7efa254",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "n = 5\n",
+ "niters = 1000\n",
+ "gauss_seidel(n,niters)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "37b7c288",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "jacobi(n,niters)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "97d9ad59",
+ "metadata": {},
+ "source": [
+ "
\n",
+ "Question: Which of the two loops in the Gauss-Seidel method are trivially parallelizable?\n",
+ "
\n",
+ "\n",
+ "```julia\n",
+ "for t in 1:niters\n",
+ " for i in 2:(n+1)\n",
+ " u[i] = 0.5*(u[i-1]+u[i+1])\n",
+ " end\n",
+ "end\n",
+ "```\n",
+ "\n",
+ " a) Both of them\n",
+ " b) The outer, but not the inner\n",
+ " c) None of them\n",
+ " d) The inner, but not the outer\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "0b383598",
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "answer = \"x\" # replace x with a, b, c or d\n",
+ "gauss_seidel_1_check(answer)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6b06f709",
+ "metadata": {},
+ "source": [
+ "If you look into the algorithm closely, you will realize that the steps in loop over `i` are not independent. To update `u[i]` we first need to update `u[i-1]`, which requires to update `u[i-2]` etc. This happens for Gauss-Seidel but not for Jacobi. For Jacobi the updates are written into the temporary array `u_new` instead of updating `u` directly. In other words, in Jacobi, updating `u_new[i]` does not require any entry updated entry of `u_new`. It only requires entries in vector `u`, which is not modified in the loop over `i`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "099634d9",
+ "metadata": {},
+ "source": [
+ "### Backwards Gauss-Seidel\n",
+ "\n",
+ "In addition, the the result of the Gauss-Seidel method depends on the order of the steps in the loop over `i`. This is another symptom that tells you that this loop is hard to parallelize. For instance, if you do the iterations over `i` by reversing the loop order, you get another method called *backward* Gauss-Seidel."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ff03c246",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "function backward_gauss_seidel(n,niters)\n",
+ " u = zeros(n+2)\n",
+ " u[1] = -1\n",
+ " u[end] = 1\n",
+ " for t in 1:niters\n",
+ " for i in (n+1):-1:2\n",
+ " u[i] = 0.5*(u[i-1]+u[i+1])\n",
+ " end\n",
+ " end\n",
+ " u\n",
+ "end"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "63c4ce1f",
+ "metadata": {},
+ "source": [
+ "Both Jacobi and *forward* and *backward* Gauss-Seidel converge to the same result, but they lead to slightly different values during the iterations. Check it with the following cells. First, run it with one `niters=1` and then with `niters=100`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "abe31423",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "n = 5\n",
+ "niters = 1\n",
+ "u_forward = gauss_seidel(n,niters)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "23430f70",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "u_backward = backward_gauss_seidel(n,niters)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "0b2b2437",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "u_jacobi = jacobi(n,niters)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0deb5549",
+ "metadata": {},
+ "source": [
+ "### Red-black Gauss-Seidel\n",
+ "\n",
+ "There is another version called *red-black* Gauss-Seidel. This uses a very clever order for the steps in the loop over `i`. It does this loop in two phases. First, one updates the entries with even index, and then the entries with odd index."
+ ]
+ },
+ {
+ "attachments": {
+ "g28304.png": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkUAAAJhCAYAAABPQ8ThAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15mFxVnf/xd1V1eklI0lkgkARIAgQIIGERBJVFBcddEYHZHEfGCMjMiI6D4PgbZxydkRl1xj2Ny4zjNqKCuCMii+wgCRIggZgg2SBbZ+u9q35/nFup6qZ6r657q/r9ep566lbVrVvfrqru++lzzj03tXz58hySJEkTXDruAiRJkpLAUCRJkgTU9bv96lQqtT6OQiRJkiopl8stAH6Rv90nFKVSqfXLli1bU+miJEmSKq2lpYVcrjC02u4zSZIkDEWSJEmAoUiSJAkwFEmSJAGGIkmSJMBQJEmSBBiKJEmSAEORJEkSYCiSJEkCDEWSJEmAoUiSJAkwFEmSJAGGIkmSJMBQJEmSBBiKJEmSAEORJEkSYCiSJEkCDEWSJEmAoUiSJAkwFEmSJAGGIkmSJMBQJEmSBBiKJEmSAEORJEkSYCiSJEkCDEWSJEmAoUiSJAmAurgLkDQu5gJ/Hi3/AlgRYy2jNQv4q2j5DuC+GGuRNAEYiiQ4AbgyWr4W2B4t/wcwFfgx8KMY6hqLQ4F/i5ZbeWEomgP8c7T8n8AT0fLfA0cADwBfGecah3IghZ/hWgxFksaZoUiCw4Fl0fLHKYSitxN2zJupvlA0lOkUfubvUQhFbwLOBKYQfyiSpIpyTJEkSRKGIkmSJMBQJEmSBDimSBqtOuCcaPlZYDWQIozHOZ8w0DkH/CvwdL/nNkTPfTnhKLEUsIFwhNVtQHaYNbwceC0wjzCY+gngO8DOkf84ZXU8cDDh5/9VdN8RwNuAhUA9sAb4AeF9G4k08ArgVYSfezfwW+CGaHkohwPnRtfzgUnAVuBB4GfAnmFsIxPV8NJoG42EcWjbgMcJn+FQn8Ek4KzoMj/a5ibCd+BXQM8w6pBUZoYiaXQmA7+Mlj8H/DvwLcKOsti36RuK3hate3iJbf4D8DvgHYQd/UAOAv6XEL76uw64HHhq0OrH1zXAnxB27E2EYPh+Qvgr9i/AZ4G/Y3ghYCHwf8CLSzz2ceAiQqgo5ZXApwlHGg5kJ/BBoGWQdU4BvgEcM8g6vYRB+t8a4PE3RLUcUeKxa4AngUuBewZ5DUnjwFAkjd1UQkBaTGgx+C3QRdgBF3dRf5Cw804RWoPuAVYRdqKnAKdHz7mdsBN/cJDXelF0+zlCC8cO4GhCUPo6IYgkwccIoacV+CnhSL4FhBauJuBvgWZCEBzMgcCvCWFyNfAIoSXqjGh7BxGOEDwBeKbE85dEj/USpidYRWgVmkX43E4CZgDLCS15ny2xjUMJrTjTo9urCJ/Rtuj1D41eY3bROv1dCfwX4XuRA+4nBOEu4ERCqD4GuBV4DQOHPEnjwFAkwcOEVgYIXSl5f0XYQT4+xPP/lLCTu5rQAtAd3Z+Kng/wOgqB6FHgj0ts948ILSHTCN1gS4DOfuv8K4VAdAPwl8C+osdfRAhJHxyi5s0UfuZHi+7/MCEo/GGI5w9HBvgAIeS9jRAe8hYCPwGOBf4iWr5hkG39NdAGvAW4qej+OuCTwN8QAuPVwBUlnr+JEMC+1a+OvFMJUxMcTmht+z/g+X7r/C0h7OQILUHfKLGdDKF7rqPEY+cQ5oRKE4LdJbxw/qhzgO8DM6Naj2F4XXqSysCB1lIICDdEl7ai+2+O7ls1xPPrCC0i11EIRBB2nh2E37NPEQLRc8B5lA5aPyd0fQEsInRBFZtDYW6hJ4A/o28gghBwLuKFXVX97aHwMz9XdP9t0X33D/H84UgRQuYFvDCIrAPeTGghAfh/Q2yrjhCebup3fw+hJSof4t48wPO/D3ymRB15DwEXRsuNhKDb3ynR9eOUDkQQWqJuBX5T4rFPEkJTK2FMVKlZxm+n0Go2lxB6JVWIoUgauzbC7NcDOZfQRQOhJal/C0Sx7xAGXUOYSLHYhYQBuhACWBel3U0IN0nweQYedLyG0CIDYXD2iwZYD0LY6x+I8ropDOg+hNB9NRoPURiL1X9sGBRafyYTws1InAacHC1/nsJnXMqPKAxA7/8dkDSODEXS2N3P4Ec+nVO0/LMhtpWlMJbolH6PvWQE2/npEI9XykjqPGOQ9YYaW1Pc3TdriHVnEo4e+xNCy1vxJd/CdliJ5/0uus4P+D5uiNcpdk7R8lDvCRRa6vp/BySNI8cUSWO3fojHjy5a/gx9u9hKWRJd92/xWBBdt9K3y6uUJ4d4vFKGqqP48VJH5OUN1roGfVujGgdY50zC+d7OZeh/CEsNlP4CYZzZDOCt0WUdYQD4bRQGvJdS/B34BNA+xOvn159OmMJgoFZBSWVkKJLGrm2Ix2cULZ89gu02Enbe+XmLpkXXw5mPZ9cIXme85IC9Q6xTXGfzIOv1jrGWPwX+h0K311ZgLSFMFdd4DuFIt1LdY+sJXWCfJxwZliK0Gi0E3hnV+GvCYPX+J68t/g6U6pobTBOGIqkiDEXS2OWGeLx4h5afv2c02863LjSUWrGfgVpLKilFaOXofwRdseI6h2o9Ga2DgC8Sgs6zhMHLvxpg3XsIoWgg6wlHEs4D3kjohjubQpB6FSFYXUIY3J1X/B14J0OHxWJDhW5JZWIoksZfcVdX8WDekcofOTWL0HowWIg4dJSvUW7zCS0yAymuc6gustF6M+FwfQiH6w8UiCAM1B6OjYSg9UVCa95JwFWEFqk6QmvSzRS6Sou/AysI8yxJShgHWkvjr3hm4lePYTv5Wa7rCEczDebMMbxOOQ1VR/Hjg83iPRZHFS2XOlQ+bz6FcVsjkSXMdfVnwI3RfXMIR9TlFX8H/mgUryGpAgxF0vj7MYVWnfcSDukejeIjtd41yHozCJMlJsFfDfJYA2HuIQiDx+8epxqK52warGvsb8rwWg8XLRd/zr+gMBbsSgrjwyQliKFIGn9bCEcuQTjf1beBA4Z4zsvoewg+hMG7+cP1/5jSExWmgS8x+KDlSjqLEAJKuY5Cy8xXGL+xM8VHuL17gHVeT+j+GswyBj59B4Q5pF4TLfcS5mHKayVM4AlhUsbvDbEtCK2BZw2xjqQyckyRVBkfIszDcyZhgO6TwJcJrSM7CDvUQwmnm3gj4fQOy3jhUUyXE7pi6gkzT3+BMKB3e/ScqwhHN91POJdanPLnGfsM4SSuXycExAWEsT2vjdZ7hnCo/Hj5AWFyzenA+whdW98gTKA4jzAD+DsIY7Za6Xv4fLF/Jky++WPCIfiPEo5eayZMzvluCt2B36DvKWMgzHr+MsJg7PMIEzReT+jS20b4ezyfMDfRGwjdb+8H7hzdjy1ppAxFUmV0Ek7W+iXC2JN5wD8Osv5Ah7M/DFxMOC9WE6HLp3+3z63AR4n/ZKI5QuC4lXCusLeXWOcPhPdlONMMjNYOQjfd/xG67P4suhR7nnBetX9l4FAEoUvsIgrnjSvlp5RuHeshhJ3PApcSwtk/DFG75z2TKshQJI1OF9ASLQ93LMw+4M8JZ0n/K8IkgodROCz9WcJ51n4M/JCBTwVxE+GM6tcSumvmEMYsPQb8N+FM74cV1ffEMOsbD78ntHx8gHCakgWELr71hJByHQPPqdRK4WcY6mit3xWtu73E4z8ktJxdQ2ipmUUIqs9Ej32K0IqVP8VG/1YeCIfenxtdjiccqZaff2gjoXXufwhHnQ2kgzAe7LOElsBXEN6TpujxTYTP8WeEz3n9INuSVGap5cuX758HJZVKHb1s2bI1gz1BUtk1MPhcPkOpY2RzH423b1KYj2lSv8dShFA01skYx2oSQ88sPhzlmm16rN8BSaPQ0tKyOJfL5c816EBrKQHGujNMUiAaSo74AxGUJxBB+WaaNhBJCWAokiRJwlAkSZIEGIokSZIAjz6TVH73EwYyJ2HskCQNm6FIUrl9JrpIUlWx+0ySJAlDkSRJEmAokiRJAgxFkiRJgKFIkiQJMBRJkiQBhiJJkiTAUCRJkgQYiiRJkgBDkSRJEmAokiRJAgxFkiRJgKFIkiQJMBRJkiQBhiJJkiTAUCRJkgQYiiRJkgBDkSRJEgB1cRcg1Yh6YH60/AzQG2MtkqRRMBRJ5XE08Gi0PBvYXsHXTgOLgWMJv9O9wA8q+PqSVBMMRSrlt8B04HLgljJsrwGYHC3vBbrLsE3Bh4FXAicDU4vub8NQJEkjZihSKQuAGcCUUT7/SOA84FTgFOA4Ct+184Bbx1hfEu2l8HNVKvT9JbCw6HYH0Fih15akmmMo0nh4D/DeuIuosHWEwDdSc4GbgV2EVp+R+DmwGXg4urwL+OgoapAkYShSaTPH+Pws8DiFnfUe4CtjLapG1RNa00YzBumKMtciSROaoUjj4e+A9xfdPiGuQiroSOA70fIrCS0/kqQqYihSKf8INAH/C6waxfNz5S2nKjQRWnzA3ytJqkr+8VYpf0sYaH0/owtFkiRVHWe0liRJwlAkSZIEGIqkStlOGGvV/7IuenzWAI/nKM8EmpKkIRiKJEmScKC1hu8g4JsDPPZl4P8qWEs1OgJIlbj/MGAFsINwWH8pnhZFkirAUKThagReNcBjtXjajnJrHeD+6dF1DthZoVokSSUYijRczzPwaSyermQhkiSNB0ORhqsDW4QkSTXMUCRVr7nAIf1uQziA4pR+6z5ckYokqYoZiqTqdTnwDyXubwQeKrqdBTIVqUiSqpihSOPhZPp2tRXvkG+k79FUfwz8ohJFjbMu4PfRcu8IntcTPW+ggdiD2Vn0moPJjmLbkjThGIo0HuoI504r5YB+tyeNcy2Vsppw2P1IbRjl8wA+FV0kSWVgKFIpJxPGpTw3yuevZPg7+i2jfA1JksrKUKRS1o/x+Z0Mr1tHkqTE8DQfkiRJGIokSZIAQ5EkSRJgKJIkSQKqeKD1F7/4xYPS6fRVcdcxmK9//eunt7W1NQJs3rx59ZYtWzbHXVM/RwHzouVtwKoYayllJnBCtJwF7oyxloGcU7S8ktHNNzSejgdmRcsbSN556uYRvocA7cADMdZSSiNwetHt+wl1JsmLgSnR8tPAxhhrKWURcGi0vAN4NMZaSmkGlhbdvpNwgubEOPHEE1+eTqczAOecc87vjjnmmO1x1zSQVCr1wLJly26Mu47RqtpQlMlkDszlch+Mu47BPPnkk2zfvv+7e0actQzTG+IuYAgvjbuAIVTDZ5x0r4i7gCEk/TNOen0Ar4u7gCEk7j1cuXLl/uUzzjjjJTGWMqRsNns9YZLeqmT3mSRJElXcUlTCb3K5XFfcRRTL5XLnAimAQ5uauqfX1XXEXFIfT+3bN6Uzm00DNDc3s2jRokSdDmLz5s2pzZs3pyC8icdNnbon5pJe4LE9e6bml4866qjc1KlTE9Xs/uijj6Z7enoAmDVpUu8hjY1tMZfUx7Pt7U27enrqABobG1myZEmivoN79+5NrVmzJpW/ffTkyW2TMpmRnMZl3K3as2f/l27+/PkcdNBBiXoPV69end63bx8AUzKZ3MLJk/fGXFIf27u66jd3djbkb5900knZVCo12FMqbsWKFelsNnysHR0d63O5XKLmgUulUscDB8VdRznUUii66LLLLkvUmJ3m5uZuovf4wvnz91w0Y8YzMZfUx0UrVx77bGdnI8AJJ5yQ+8QnPtETd03Frr/++szXvva1DEAmleL6Y49N2ngYznjggZPyy+95z3uyp512WqJ2mK997Wsntba2pgBOnz6988OLFiXqPfy7NWsW3N3aOgPgwAMP5HOf+1yivoMrVqxIX3HFFfv/Tl67aNGmxZMnJyqcv/zBB5f25HIpgAsuuCB7ySWXJOo9vPTSSyc98cQTKYBFTU29LQn7Pf7apk0HtmzYMD9/+zOf+UxPJpOs8ye/4hWvqO/oCP9TP/XUU9/+6le/em3MJfWxfPny7wJvi7uOcrD7TJIkCUORJEkSYCiSJEkCDEWSJEmAoUiSJAkwFEmSJAGGIkmSJMBQJEmSBNTW5I2j0QCcCJwSXU4FDose+xjwyZjqkiSpHOqA4+i7nzuScKKAbwB/E19pyTPRQ9EFwLcGeKypkoVIkjQOTgHuG+CxyZUspBrYfQa9wOPA/wLvBTrjLUeSpLJbC3wXuBp4NuZaEmuitxT9FGgGik9Q+PGYapEkqdweA2YCO4vue3tMtSTeRA9Fu+IuQJKkcbQvumgY7D6TJEnCUCRJkgQYiiRJkgBDkSRJEmAokiRJAmr36LMPAVNL3L+KMB+RJElSH7Uait4DHFLi/hsxFI2Lm2++Of3LX/4y/f73v793wYIFubjrqUbbt29PffjDH86cffbZuYsuuqg3lUrFXZIkTSi1Goo+BhxQ4v41lS5korjtttvSDz/8cPr222/PveMd7+iNu55qdMcdd6RWrFiR7urqyl188cW+h5JUYbUaij4fdwETzTnnnJN94IEH0nfccUfaUDQ6v/71r9MAZ511VjbuWiRpInKgtcrirLPOyqXTaVavXp3auHGj/T4jtGfPHlauXGkokqQYGYpUFjNnzsydcMIJOYA77rjD79UI3XHHHemenh4WLVqUc0yWJMXDnRc8ADxUdGmM7r+83/2Xx1JdFTn77LOzYCgajdtvvz0NhfdQksroJvruzxZF97+p3/0fjaW6BKnVMUUjcQqlw+Hc6JL3s8qUU73OPffc7Gc/+9nMqlWrUlu3bk0deOCBtngMQ1tbGw899FAawtisuOuRVHOOB44ocf/s6JL3ZGXKSS5DEVwMDGcMzBPjXUi1mzNnTu7oo4/OPfnkk6m77rordcEFFxiKhuHuu+9Od3V1MW/evNxRRx3leyap3P6a0kdk9/eH8S4k6QxF8L24C6glZ599dvbJJ5/M3HHHHekLLrjAVo9huPPOO+06kzSe7OkYJsd+qKzOPffcLMAjjzyS3rVrV9zlJF53dzf33XefXWeSlACGIpXVYYcdllu4cGGup6eH3/zmN36/hnDvvfem9+3bx+zZs3PHHXecXWeSFCN3Wio7j0IbvuKuM0/rIUnxcqelsst3Az3wwAPpffv2xV1OYvX29nL33Xfnu85sJZKkmBmKVHaLFy/OzZ07N9fV1bV/vIxe6OGHH07v2rWL6dOns3TpUscTSVLM3GFpXORbi+xCG1h+wsazzjorm8lk4i5HkiY8d1gaF/lQlJ+DR31ls1nuuuuuFHiuM0lKilqap+jclpaWnXEXUezqq6/eP3J23b59k34N0+Ksp7/2bHZ/KN6xY0fqnnvuKVtIzuVyNDc309rayte//vXMkiVLRjxmpvjEsjng1zt3Jur96+/xxx9P9fT0DOs9XLduXWr79u2phoYGent7Ked7X6ynp2f/e/h8V1cmae/hjq6uSfnl9vb2cXsfRmv9+vV9bv929+7JGzs7EzUivvgXa/369WX9PS6HvXv37l/e09ubStp3cF17e1Px7XvvvTedTifqLaS3t3f/8qxZsxa1tLS8JsZyXiCbzR5cKweKpJYvX77/dyqVSh29bNmyNXEWNFwtLS3H5XK5x+KuYzDXXnst27dvj7sMSVKNuOKKKzjxxBPjLmNAuVzu+ssuu2xZ3HUMV0tLy+JcLrc6fztZcViSJCkmVdt9ls1mc6lUqifuOgaTCVIA2Wy2N5fLJW3sSIZCMM4BvYOsO1r571gvfVv6hyNFqDGvuywVldekouXh/ozFP9d4f4czFM7tl2V8PuOxSFN4L8brOzhWxX8nexj593i81dH3M07a35k0ff/OJO3vdooXfsaJkk6ni+sbzd/Sikn6fnkoVRuKLrvsssfpu0NSMn0ZuBRYDrwn5lqS4qPAPwA3AhfEXIukhMtmCzn3C1/4QoyV1D67zzTeboyuL8DvW95bo+sfxFqFJKkPd1Iab7cCu4GDgZfEXEsSLAaOJXQF/iTmWiRJRQxFGm+dFHb+b4mzkIR4W3T9KyBRU0hI0kRnKFIl5LuJLqQwIHSiygdDu84kKWEMRaqEnwJtwAJgabylxGoBcDLh6KAfxVuKJKk/Q5EqoQ24JVqeyF1oFxBayu4CtsRciySpH0ORKiXfXTSRD0HPB8IbB11LkhQLQ5Eq5UdAF3AccEzMtcRhDnAGYdI1Q5EkJZChSJXSCvw6Wp6IrUVvIczc/CDwh5hrkSSVYChSJeW70CbiuKJ8ELSVSJISylCkSrqJcN6eU4GFMddSSc3A2dGyoUiSEspQpEp6Hrg7Wn5znIVU2JuAeuB3wOqYa5EkDcBQpErLt5RMpC60fNeZEzZKUoIZilRp3yMcgfVS4JCYa6mEA4DzomW7ziQpwQxFqrQNwEOE794bY66lEl4LNAHrgJUx1yJJGoShSHHIt5hMhEPz892EN8RahSRpSIYixeF70fW5wMw4CxlnDYSWIrDrTJISz1CkODwFrAImAa+PuZbxdD4wDdgI3B9zLZKkIRiKFJfvR9e13IVWfK6zXJyFSJKGZihSXPLdSa8GpsZZyDipA94QLXsoviRVAUOR4rICWAs0An8Ucy3j4RxgNrAduCveUiRJw2EoUpxqeSLH/M90E9ATZyGSpOExFClO+W6l1xNajGpFmsJpTOw6k6QqYShSnO4jTOY4FXhVzLWU00uAucAe4LaYa5EkDZOhSHHKAT+MlmupCy1/RN2PgI44C5EkDZ+hSHHLdy+9iXDEVi0oPhRfklQlDEWK2x3AVmAWcHbMtZTDScAioB34Wcy1SJJGwFCkuPUSupmgNrrQ8l1nPwf2xVmIJGlkDEVKgnwX2lup/u9kPhTZdSZJVabad0CqDbcCu4GDgdNjrmUsFgNLgG7gxzHXIkkaIUORkqAT+Em0XM3nQrswur4N2BlnIZKkkTMUKSny3U0XDrpWsuUDnRM2SlIVMhQpKX4CtAELgKXxljIqhwMnA1ng5phrkSSNgqFISdEG3BItV2MX2gVACvgNsCXmWiRJo2AoUpLku9CqMRTlpxOw60ySqpShSElyM9AFHAccE3MtIzEHOJNw2hIPxZekKmUoUpK0ArdHy9U0keNbgAzwEPCHmGuRJI2SoUhJk+9+qqYuNM91Jkk1wFCkpLmJcOqPU4GFZdzuhcA1wCvKuE2AZuCcaNlQJElVzFCkpHkOuCdaflMZt/sXwMeB15VxmwBvBOqBx4Any7xtSVIFGYqURNXUheaEjZJUIwxFSqLvEY7keilwSJm2+WngHcC3y7Q9gMnAedGyXWeSVOUMRUqiDcDDhO/nG8u0zduA/yEcIVYuryMEo3XAijJuV5IUA0ORkirfHVWuQ/PnAUcDB5Zpe1DoOvteGbcpSYqJoUhJlQ8arwBmlmF7XyIMhP5gGbYF0AC8Nlq260ySaoChSEn1FLAKmAS8PuZaSjkPmAZsBO6PuRZJUhkYipRk5e5CK6fiCRuzcRYiSSoPQ5GSLB+K/giYGmch/WSAN0TLdp1JUo0wFCnJVgBrgUbg1THXUuwcwoDt7cCd8ZYiSSoXQ5GSLt8Sk6SJHPNdZz8EeuIsRJJUPoYiJV0+FL2e0GIUtxSF0484i7Uk1RBDkZLuXsJkjlOBV8ZcC8AZwHxgD/CrmGuRJJWRoUhJlyN0U0EyutDyXWc/BjriLESSVF6GIlWDfBfam4C6OAuhEIrsOpOkGmMoUjW4HdgKzALOirGOpcARhBaiX8RYhyRpHBiKVA16gR9Fy3F2oeVf++eEMUWSpBpiKFK1yHehvYX4vrf5UOSEjZJUg+IenyEN1y+B3cBc4HTCUWkj8R3CZJB3j/L1jwKOA7oJg6wlSTXGUKRq0Qn8FLiE0Fo00lD0zTG+/oXR9W3AjjFuS5KUQHafqZrkj/i6cNC1xoddZ5JU4wxFqiY/AdqAhYQjwSplPnAKkAVuruDrSpIqyFCkatJGGFsEhfmCKuFCwuk97gY2V/B1JUkVZChStcl3oVXy0Pz8azlhoyTVsKodaP2Rj3wkPXPmzAPirmMwd9111+TOzs40wNq1aztWrVrVHXdN/TQCk6LlHpJ32ooM0BQt54C9hIHOXcDxhC6tNeNcw0HAmdHyrYRzsBUrvt1OmFMpSZoI7yOE960zxlpKmUThRL9ZQmtgkqSAKUW39xK+i0kymcJn3Ek4QjJJGuj7d6Y9xlpKyRDew7y9cRUykDe+8Y0HZLPZFMBRRx3VvnDhwp64axrI1KlTu/7yL/8yafuSYavaUDR37txjc7ncY3HXMZjbbruN7du3x11GLXuowq/3uwq/niRx882FoYxXXHEFDQ0NMVYzuM7OzuuBZXHXMVp2n0mSJGEokiRJAqq4+6y/VCr12t7e3kT1VWWz2XuI+vovOfTQ3WcfcECijlz6yNq1C57r6moAWLp0KVdeeWWixiL88Ic/zPzoRz9KA2RSKT53zDGr8499Yt26w9Z3dDS96aCDnv+jWbN2jsfr37Vz57RvbdlyyJz6+s6PHHHE+lLrXP7EE0fnl9/3vvdllyxZkqgxRVddddWkPXvCadrOnD6948/nzn0m5pL6+MKGDXN/t2fPVIA5c+bwsY99LFHfwaeeeir1iU98Yv/fyX9YtOjZeQ0NiRr3dOWTTy7uzeVSABdffHHuvPPOS9R4k4997GN169atSwEc0dTU+3cLFjwdd03Ffr5t24wfbt16UP52S0tLdzqdrPaCK664YlJXVxcAzz777H+fcMIJX4y5pD7S6fTHgVfGXUc51EwoyuVyKy6//PJEhY7m5ub9AzIPaWzsWTp1aqL+mNanUvvrmzp1au64445L1ADSe+65Z389KaD4/Ttv1qwd12/cOG9NW9vkDy5YsHE8Xv9rmzYdDHDOzJk7h/PZHXbYYYl7DzOZTI7w9tE8aVI2ad/BaZnM/h14fX09SXv/uru7U8W3j5o8uXPx5MmJeg+LCzz44IMT9x1sbGzcvzw5k8kl7Tv4yJ49xQPpWbJkSS6TyQy0eiyKQ1pra+vmyy+//IEYy3mB5cuX18ws/8mKw9IwvXr27NYU8OTevQc819U1acgnjFB7b296RdSC8apZs1rLvX1JUvIYilSVBBlQNwAAIABJREFU5jU0dC1samrLAb/avn168WM7enrq8t0Jw9Xa09PnX8Pbdu6c1pXNpmfX13cdP2VK0g4hliSNA0ORqtbLZsxoBbirtXV/KHq+u7vuTx999Jjf7d3bNPAz+3q+u7vurStXHvdMR0d9/r7bd+xoBnhZc7OtRJI0QRiKVLXOnzmzFeB3e/dObe3pyaxpa2u89LHHjm7PZjNNmUx2uNtpTKdzvblc6vInnli8cs+eyV25XOrh3bunAbwieg1JUu2rmYHWmhjae3vT+cBzxOTJnfMbGjo2dHY2vn/NmkXPtrc37untrZtWV9czOZ0ediianE5nG9Lp7M7u7kkfWLPmiEObmjras9nM9Lq67lOnTduXX68jl0s3pFLZEfXLSZKqhqFIVWN3b2/mTY88clxjOp2dXlfXk4XUtu7uSQCP7927/5QvKciNpKWoLpXKpaJTN+zp7a3Lb6sjm81ctHLlsXXpdG53T0/d7p6eum+ccMLjhzc2dpX7Z5Mkxc/uM1WNaZlM7wcXLnymJ5dLPdPR0fRsR0djRzb7gmNnc5AaSUsR9D2sOa8zm01v6OxsXN/e3tSRzabfPnfuJgORJNUuW4oKjiCcYHRhdPsu4J74ylEpr541a9cje/bs+MW2bbNKBaK8kbQUAaRSA3eK1aVSuaOnTNn3rnnznh/JNiUpYeYT9nOLCY0ijwI/i7WihJnooejFwMcJX5IZ/R77MIaiRLp6wYKNT+zdO+Xp9vbJ2RKH3qdgZMfjA+lBznw+a9Kkrk8eddS6ERcqSfFbCHwWOBWY0++xr2Ao6mOid58dCbyKQiB6DhhRC4MqLwX81zHHrG2uqyt5SojBWn0GkhngOdPq6no+ffTRa0fa8iRJCXEQ8DoKgWgn0BFfOck20UPReuCjwJuBQ4GD8ctSFZrr6no/sXjx7w8oOk1E3mi+1JmiU57kHZDJ9Lzv8MP/sLCpqXNURUpS/HYA1wEXExoCZgJrY60owSZ699m90UVV6PgpU9ovnT9/0/UbNsxr6+3dP74oNUhX2EDqU6k+LUGN6XT2lTNn7nj1rFm7ylGrJMXkKeDquIuoFhO9pUhV7pI5c7afNHXq7rqilp6BusIG01B0tFoKcvMaGjo+sGDBpjKVKUmqAoYiVb2PH3XUM3Pq68fUxVU80Lp50qSe/zr22LWlutQkSbXLUKSqV59K5f7rmGPWTslkegGmlBhnNJTpkyb1ADRlMr3/euSRv59VVzfibUiSqpuhSDVhXkND14cXLVrflE73No7iSLHJmUx2UiqVe+fcuZtOnDq1bTxqlCQlm6FINePsGTN2v/7AA7c2j6alqK6u57Tp01v/7JBDto1HbZKk5KvVo8+mUzrwdQN7K1yLKuiqww/f3NbbO+Kw/9eHHrq5Pp12DJEkTWC1GoqeAA4pcf+NwAUVrkUVlAKmjLL7bBzKkSRVkVoNRbuAxhL376t0IZIkqTrUaig6Nu4CJElSdXGgtSRJEoYiSZIkoHa7z0biQsL43Lz8ObSOA95WdP8TwGOVKkqSpDJ5DXBA0e1p0fVC+u7n/gDcX6mikshQBP9H6RazS6JL3r9gKJIkVZ/PAkeUuP8V0SXvmxiKJryHGV43oicHlSRVo8eA1mGst268C0k6QxGcFncBkiSNozfHXUC1cKC1JEkShiJJkiTAUCRJkgQYiiRJkgBDkSRJEmAokiRJAgxFkiRJgKFIkiQJgNTy5ctz+2+kUkcvW7ZsTZwFDVdLS8txuVyu+LQbq4CeuOop5ZprrnnRjh07UgCz6+t7p6TTXXHXVGxTZ2dDdy6XBpgyZQqHHHJIbqjnVNKOHTvIv38p4LDGxvaYS3qBZzo6mvLL8+bNyzU1NQ22esWtW7cu1dvbC8DUurrszLq6zphL6mNrd3d9W29vBmDSpEkcfvjhifoOtre3s3Hjxv3nRpzf0NCZSaWycdbU3x86Opryb9qBBx7I9OnTE/UebtiwIdXR0QFAYzqdm1Nf3xFzSX3s7ump29nTMyl/+8gjj0zU+wfw+9//PpXNhq/du971rudOPfXULTGX1N/hQDNALpe7/rLLLlsWcz3D1tLSsjiXy63O366lGa2Pi7uA/lKpwnlmt3V1ZbZBsvaYRfbt28fTTz+dGnrNeOToG0CSqHjnmUR7enrSe3p6Evsednd3J/o7CLChs7Mh7hoGs3XrVrZu3ZrY97Ajm00l/fc46d/BSZMmzQHmxF1HrbL7TJIkiSpuKerq6tpSX1//gbjrGMwJJ5xweltbWyPAxo0bV2/cuHFz3DX1cxQwL1reRuiCTJKZwAnRci9wV4y1DOScouWVDO+ki5V0PDArWn4WWBtjLaXMBRZHy23AgzHWUkojcHrR7fuBpHXjvhiYEi0/DWyMsZZSFgKHRcs7gEdjrKWUZmBp0e07CY3TifHiF7/45alUKg3Q0NDwWCqV2h53TQPJ5XIr465hLKo2FF155ZXbgf+Iuw5NeJ+MuwBJte3BBx/cv6974IEH4iyl5tl9JkmShKFIkiQJMBRJkiQBhiJJkiTAUCRJkgQYiiRJkgBDkSRJEmAokiRJAgxFkiRJgKFIkiQJMBRJkiQBhiJJkiTAUCRJkgQYiiRJkgBDkSRJEmAokiRJAgxFkiRJgKFIkiQJMBRJkiQBhiJJkiTAUCRJkgQYiiRJkgBDkSRJEmAokiRJAgxFkiRJgKFIkiQJMBRJkiQBhiJJkiTAUCRJkgQYiiRJkgBDkSRJEmAokiRJAgxFkiRJgKFIkiQJMBRJkiQBhiJJkiTAUCRJkgQYiiRJkgBDkSRJEmAokiRJAgxFkiRJgKFIkiQJMBRJkiQBhiJJkiTAUCRJkgQYiiRJkgBDkSRJEmAokiRJAgxFkiRJgKFIkiQJMBRJkiQBhiJJkiTAUCRJkgQYiiRJkgBDkSRJEmAokiRJAgxFkiRJgKFIkiQJMBRJkiQBhiJJkiTAUCRJkgQYiiRJkgBDkSRJEmAokiRJAgxFkiRJgKFIkiQJMBRJkiQBhiJJkiTAUCRJkgQYiiRJkgBDkSRJEmAokiRJAgxFkiRJANTFXYBUpTLA0mj5eeDZMWzrYGBetLwK6BjDtsbT4cDsaPm3QC7GWsbTYcCB0fIjQDbGWiRVkC1FUtjRvyq6NBbdf1Z036ISz5kCPBRd/n6Mr/8XRds6fIzbGk8folBn/Si3cTThPT2j6L5pFN7/6WMpsEyupvBzTo65FkkVZCiS4CXAL6PLnKL7vxfd9/Y4iqpRVxLe068U3beYwvt/TBxFSRIYiiRJkgBDkSRJEmAokiRJAjz6TCq3ycDJhLFJe4GHgW1l3P4BhMHYh0XLO4HfAc+NYltp4ETCkW8N0TaeBZ4pS6XJMIXC57Gb8HlsH+W2DgOOi7a5kTAQu3uYz80AC4C5hKMN24H1hKMNR3IU3xzgBGAG0EX4WZ4Dnh7BduYQfo5ZwD5gNbB2BDVINctQJJXHJOCfgSsIR1Pl9QDfAd4HbB3ltqcB7wZeB7yU0r+39wEfBm4dxvZmE46w+kvCjrG/Jwk1/9Moam0EvgG8Nbr9NULtww0PozWbwvv7KeAa4F+Ay4CpRet1A98C3s/ww9GRwBeBVwKpovu3A9cCLYM89wLgQuDVwMwSjz8H/Bfw74TvykBeAlwHvKxfDXlbgF8A7xhkG+cD/0g48q//NlYB/wDcNMjzpZpn95k0dmngB8AHCTvgfGtLlhBg/gy4GzhklNs/irBDPDvaXjvwB/ru1F9C2Cm+a4htLSXMvfN3FALRLuD30TWEI8DeN4o6ZxFCWT4Q/RNwKeMfiPrLADcDHyC06vwhumQJ4fUvgN8ABw1jW0sJrUuvim7vpBBeZgHLgb8e5Pn/BvwxIRD1EMLLBqA3enwO8HHgxwz8T+r5wB3AywlhpjPaxrpoGULr058M8PxU9Bq/AM6MbndEz98ZrXMccCMhNEkTlqFICgGmJbrsLbr/69F9Dw/x/EuA1wM/AY4gdLMsiC7fj9Y5CvhfSv+XPxyPAe8lzPOT70KbTZjX51JCF10a+Gz0WqXMBX4OzI9u/x/wIqA5qruZMCfT/2Pkk1EuIgS/lxJ2/pcCH+GFXTp3E97T7xbdt5XC+//8CF+3lD8ntMz8MKrr8OiyMLoPQvD772Fs67uEAPEOQovdzOj6ckL3FYTgU6oVCELQ/DQh0EwjBONDCd2sLyW8H0T1vr/E81PAFwjzQu0jTA/RHG1jEaFlbn50/x0D1HAVoeUM4FHgjwjhfREh2L0UWBk9/hEKoVaaeJYvX57LX1paWhbHXY9UJaYRdvj5y52EVoj+MoT/0PPrvb7EOlcXPX50icfrGTpMLSXspHPAZwZY59tFr/NvQ2yv1D9MLUXPbyi6/8WEFpAcsIew06202fT9PH5F6ZaXuuix/Hrnl1jn80WP72bguZOuKVpvoBa6hgHuL378gWgbmwjfl2JHFr3GNYzcYYTWpBxhDNSUAdZrJrQW5oCnStQh1aSWlpbFxTnIliKpPN5P6W6iXsJ/6nmXjmLb+bAzmBXAz6LlV5Z4fB7wtmj5ccL4kcEM99QWbwB+TegG2kzo4vv5MJ87nt5H6TE6PfRtkRnq8/hPwhirUv6naPm0AdbpHOD+4sc/HS0fQujGKlYcYjYPsa1SrqQw+/gVhNamUloJY7AgBLEzR/FaUtUzFEljtw54cJDHHyd0W0A4dchou9CKTSWMI1lUdMl3PR3NC3+3X0nhv/+vMPig3uG6jDAOZQrwBGFH+tsybHesVlPoDiplBYWgc/YQ2/rhII9tJozvguGPF2sgdLUVf25tRY8f22/9pykE1PcTugBHIt9q93tCi9RgbilaNhRpQvLoM2nsVgxznRcRdogHM/L/+tPARcCfAqdTOGFpKRnCWKOdRfe9qGj53hG+dikfpzAY+y7gzcCOMmy3HIb7eRxDaOGaxcBHog02PUGOcPTYAsI4r4GcSAiQ+fPoDfbP6Ix+t/cRjgi7ADgeWENombuF0GX7MIVB2/3VU2h5ageW9Xt8Gn27yYrrmjtIjVLNMhRJYzeceYiKBxDPZGShaDphx3hOicfaKHTRNFE4oW3/E7YWH3o/mjmN+ssHop3Am+gbwOI2ms9joFDUMcR2BgokedcAH+WFY3R6COOviB7LT+NQagzSXxE+29cQ/mafF10g/Kw/IbT+3dXveTMpBJ3jCEfKDde0oVeRao+hSKqM4i6zkUzWB/AlCoHoJsKcOY8SgkjxmJVPAH8/wDZG+ppD+V/CUV4zCF1ob6Cwk68GY/k8huuNhBY1CN1gHyccIbaVvu/VSQze7bgTeC2hhfAthMH6Swg/w2zCFAN/AXwO+BsKP0/x3/fVDHx0WinlaE2Uqo6hSBq72cNYp7i7ayStKocQus0gHN5/4SDr9u96KVbcEnIwYYzJWCwjHG5+JWFczi2ElozWMW63HMbz8xiJ90bXrYTD3geabmCwz63Y/dHlg4T6X0Fhcsg04bN4gBBYoe9nvpYwiaakQTjQWhq7k4axzsnR9XbC4evDdQqF39NvDbHu0kEeKx5nU45BtDnCpIXXRbdfQjjUfTiBZLwN5/PIr7OF0Z/2YyinRte/ZPD5lwb73AaylTDP1MWEVrq8S4qW2wljkCB8Pv4TLA3BUCSN3QJC18ZAjiMMkoXQhTGS7priAbyDPW8JhZ1wKbdRmDLgnZRvB3k1YcI/CMHv14SWqDgtZvBgdBKF+aBuH6caUgw8J1D/9f5sjK/1Uwqhq/8A6Z9G1zMpTMkgaQCGIqk8/oOBJ2/8dNHtL49wuxuKlgeaFLGBMIh2sEP9NxMmb4Rw2PfHhnjdkfxt+CfCWKYcIfzdQZhxOU6fZODJGz9ZdHukn8dw5SdjhNB1NtDRae9l8AA3hzD79WBmUZhRe1O/x/6Twszb/0mYg2goCyjPtBFS1TEUSWO3jXCizpvpu9NZAHyPwpFCt1CYYHG4HqBwpNq7CEczNRU9fhqh2+plDN0t9wEKO82/B24gtO4U7wCPIJzm47ER1vnvhO60HKGl5g5GPqdOuWwDziUMAF9UdP/C6L5zo9s/Jrx34yU/x9FcwntdXMscQpD+JIN/bucQztv2KcJn3Njv8dOBH1EIgN/p9/gzFCarPIjwfXovL2zNm0cYu/Y9nNFaE5h9zNLYfZdw/qk3EnYomwjhYC6FwPEk4fxUI9VFOKLouxRO7PlRYCOh9SHfQvBdwiSSVw+yrecJ59j6CeH0DxdGl32EcTUzKbRo7B5FrZ8njGNpIQSQOwlz86wexbbG4huEYPZ64HWEUNn/83iM0I04nj5KmK5gPqGVby1hLFBnVEua8F15NyHYDGQWYVb0qwiH8j9H+F70b0X6JuFn7+9z0XofJwzq/nR02UE4198MwmSg0oRnS5E0OjnCUUs7CaHircC/Eo7Imkv4zztF2Hl9jdCFMtD8QB1F2yo17833CDvXtdHtDCHUzCS0MlxNOBN7W9F2BjpNx2OE7pp/pzCfz5Roe/lAtIKwQ++vePsD+SrhUP3t0XZvIrQ+VVKWMJnkdYRD34s/j07gesIJWrcO8Pzin3Oo8V+7ovVKTUfwHOFz/0nRdg4khKRuQpA9GVhf9Hr950V6EPgvwhQMWcI/svMIoTMfiB4nzGX09kHqvY7QqvgDCrNwzyR87vlA9BzhvXktQ8+/JNWk1PLly/f/EqVSqaOXLVu2ZrAnSBpUIyF0HEw4FHsF5TvkO00Ys7OQsNPaSNhZjnYHliYMAj+UMB5qC6G7ZSRHxyXFbAoh51MUuoyaCJ/HHMLn8QjxTBtwCGFm68mE8LFqFHU0Ez6ruYTJFXcQji57doTbaSTMcH4Q4f3ZTghm6xn+Oe+kmtDS0rI4l8vtb822+0wqrw7Gb+K7LCEEPTrUiiPY3u+iS61qB+6JuwhCF95oTuharDW6jPXz6mDo86BJE5LdZ5IkSRiKJEmSAEORJEkSYCiSJEkCHGgtqTZsw1mYJY2RLUWSJEkYiiRJkgBDkSRJEmAokiRJAgxFkiRJgKFIkiQJMBRJkiQBhiJJkiTAUCRJkgQYiiRJkgBDkSRJEmAokiRJAgxFkiRJgKFIkiQJMBRJkiQBhiJJkiQA6uIuQFJFNQJNQA+wJ+ZaJClRbCmSJpb3ATuAn8Xw2hngeOCtwNuAl8RQgyQNyJYiVaulwJej5dOAbIy1aGCzgQ8DpwAnAZOLHvsOcF8cRUlSKYYiVasDCDtagNQot/FiQmvFKcCpwDGE1ox2+u68a8k64FZgVYVe72Dgb/rd10t4nyUpUQxFmsi+DRwRdxEV9u3oMlJvBt4D3AL8+wie1w58H3i46PIT4PRR1CBJ48pQpGp1HzAzWu4d5TZ6gJUUdtanAO8ce2k16VDgVcCGET5vLXBh+cuRpPIzFKla9QA7x7iNE4DuotsfGuP2qsEfE4LfKuC9MdciSYliKFK1mg/8abR8HZAbxTa6h16l5iwktPg0xV2IJCWNoUjVagHwb9HyfzD6LjRJkgDnKZIkSQIMRZIkSYChSFLBSwljs0pdPhOt845B1uk/H5EkVRVDkSRJEg60Vu15NXDpAI+9j5HPszORPMzAk1m+HfhHwkSMfz/AOtvHoyhJqhRDkWrNkYSTjZbykQrWUY06gN8P8Fg+8OwZZB1JqmqGItWaXwAXDfCYrUSSpAEZilRrno4ukiSNiKFI0nhbQt8ZtKdE1zMJ55vLayWcK02SYmEokjTevgWcWOL+86NL3k3AWypSkSSVYCjSRHYVcGXR7eboupG+LRY5wgDuWtBOOJHunhE+ryN6XtsoXnM3wzt5775RbFuSysZQpIlsBrCoxP2pfveP5mSzSfXp6DJS10eX0ThrlM+TpIoyFKlarQBOjZazo9zGl4AflqccSVK1MxSpWu0lTDY4FpuiiyRJnuZDkiQJDEWSJEmAoUiSJAkwFEmSJAFVPNB6+fLls3O53GVx1zGYb37zm6e3t7c3AmzatGn1pk2bNsddUz9HAfOi5W3AqhhrKWUmcEK0nAXujLGWgZxTtLySMCtzkhwPzIqWN5C8U6DMBRZHy23AgzHWUkojcHrR7fsJcz0lyYspzBL+NLAxxlpKWQQcGi3vJPyeJEkzsLTo9p0kbBqOk08++eXpdDoD8LKXveyxY445ZlvcNQ3it5dddtlP4y5itKo2FKVSqTnAR+OuYzCPP/4427fnTy7Oq+KspUacP/QqsfIzHrs3xl3AEJL+GSe9PoC3xV3AEBL3Hv72t7/dv3zaaae9MpVKxVjN4HK53PVA1YYiu88kSZKo4paiEn4LdMddRLFcLncaYXZk5jQ09EzLZDpjLqmPZzo6mrqy2TTA1KlTOfTQQxPVZLx161a2bt2agvAmHjl5cuJOA/FUW1u+24LDDz88N2XKlMFWr7jVq1enent7AWiuq8seWF+fqK6fzZ2djXt7ezMA9fX1HHnkkYn6Dra1tbF+/fr9/5Yf3tjYUZ9O98ZZU39Pt7VNyb9pBx98MDNnzkzUe7hu3bpUe3v42jVlMrn5DQ2jOVXMuNnZ3T1pW3d3ff72kiVLEvX+ATz55JOpbDbMUdvV1bUJeDbeil7gKMJwh6pXS6Ho9e9+97sTNWanubm5m+g9/pPDDtt90YwZz8RcUh8XrVx57LOdnY0AS5cuzV133XWJCpXXX3995qtf/WoGIJNK8fXjj18Td039nfHAAyfll6+66qrs6aefnqgd5mte85pJra2tKYAzm5s7PrxoUaLew79bs2bB3a2tMwDmzJnDV77ylUR9B1esWJG+/PLL9/+d/Ocjj9ywePLkkZ43bly9/MEHl/bkcimAiy++OHvJJZf0xF1TsXe+852TnnjiiRTAkU1NvS1LliTqO/i1TZsObNmwYX7+dktLS3cmk4mzpBc499xz6zs6OgBYvXr1/3z5y1++NuaS+li+fPl3SX636LDYfSZJkoShSJIkCTAUSZIkAYYiSZIkwFAkSZIEGIokSZIAQ5EkSRJgKJIkSQJqa/LG0cgAxwCnRJdTgYOjxz4FfD6muiRJKocUcASFfdwpwOHRYzcAH4yprkSa6KHoIuBbAzw2o5KFSJI0Dk4D7hvgsdmVLKQa2H0WPAP8APgQ0BVzLZIkldsW4CfAPwObYq4lsSZ6S9EtwIHAtqL7PgTUl15dkqSqsgqYD2wsuu+twNx4ykm2iR6KtsddgCRJ42hvdNEw2H0mSZKEoUiSJAkwFEmSJAGGIkmSJMBQJEmSBNTu0WcfAqaWuH8V8L8VrkWSJFWBWg1F7wEOKXH/jRiKJqxdPT2Z72zZMvuptram/1i8eH3c9VSrn2zb1nxva+u01x944I6XTJ/uob6SakathqLPM3BLkSaoFPCNzZsP6cnlUqvb2rYcPXlyR9w1VaOfb9s286Hdu6cfWF/fbSiSVEtqNRR9LO4ClDzT6up6T5g6dc8ju3dPu3XHjuajJ0/eEndN1WZfb2965Z49UwHOnzWrNe56JKmcHGitCeWsGTNaAe7euXN63LVUo1/t2DG9O5dLH1xf33nslCntcdcjSeVkKNKEct6sWbsyqVRuXXv75Gfa2xvirqfa3BGFyTOjcClJtaRWu89G4hbCcJO8/I7yHcDZRfd/E/jvypSk8TKrrq7n6ClT9j2+d+8Bt+zYMf1d8+Y9H3dN1aIrl0v9dvfuaQCvmDlzV9z1SBq2/6HvCWAXRNevAX5ZdP9twL9WqKZEMhTBKyndYnZEdMm7rzLlaLy9vLm59fG9ew+4e+fOZkPR8N2+c+e0jmw2M2PSpO6Tp07dF3c9kobtpfTdn+XNpW9Yeq4y5SSXoQgupm9L0UCeGO9CVBmvnjWrtWXDhvlr2tqmbOnqmnRwfX133DVVgzt27GgGOLO5uXU4vzCSEuOvgQOGsd4fxruQpDMUwffiLkCVdUhDQ/eRTU37nmpvn/LL7dun//khh2yLu6ak68nlUg/u2jUN4JUzZzqeSKouP4u7gGrhQGtNSC+dMWMXwF2trc1x11IN7m5tPWBPb2/d1Eym58XTptl1JqkmGYo0IZ0XzbHz+N69B2zv6bHFdAi379zZDHD69Om76lKpXNz1SNJ4MBRpQlrU1NR5WGNje28ul7pt+/ZpcdeTZDngvtbW6QDn2nUmqYYZijRhndncvAvgzqgVRKU9tHv3lNaenklNmUzvy2bM2BN3PZI0XgxFmrDyXWgr9uyZtru3NxN3PUn1q+ios1OnTdtdb9eZpBpmKNKEtWTKlPa5DQ2dPblc6vYdO+xCG8C9UdfZOc5iLanGGYo0ob2kubkVCqevUF8r9+yZ/HxXV0N9Op09Z8aM3XHXI0njqZaOujn9S1/60o64iyh2zTXX7J/jbkN7e9196fRwJs+qmM5cbn99u3btSj3yyCOJmpNvy5Yt++vJAfft2lX2929ufX0XwEO7d0+/c+fOqfXp9Ki7h9auXZuqr69P1HvY09Ozv54d3d2Zkb6HNz3//CyAIyZPbnt0797J5a5vV2/v/r9BnZ2dJO07uHbt2j63V+3b17ijuztRXYjFxWzYsCFxv8dtbW37l/f29qbG4/d4LDZ2dvY5B+IjjzySzmQyifqMs9ns/uUZM2Yc+qUvfemsGMspZXbcBZRLavny5fs//FQqdfSyZcvWxFnQcLW0tByXy+Uei7uOwVx77bVs37497jIkSTXiiiuu4MQTT4y7jAHlcrnrL7vssmVx1zFcLS0ti3O53Or8bbvPJEmSMBRJkiQBVTym6F3vetfjN9xwQ8PQa8bnrW99a117e3sK4N577+15+umns0M9p8Iy0QUgC/TGWEspKQrf0RwwXiduzQDPAAcBrwVuHcFz64uWe+g7xCMJMhT++ellZJ/x/cBJwOXAV8pcV16avp9xzzi9zlhMKlruiq2KgdXR9zNO2t+ZNH3/ziTtM07R9zNO3Amizz///EnTp09PARxzzDHdM2bMSNpQxwu4AAARf0lEQVTfmWJJ24+MSNWGolSYLyWJf6CKJb2+xP3yl9BZgdfoBm4G/gp4AyM7eWI1vIejsQBYStiJ3cT4/pyV+IzHwt/j2pfoz/iWW27ZX98NN9wQZyk1z+4zKfhBdH0Bhf9qJ7ILCP9B3wVsibkWSaoIQ5EU3ArsBOYAZ8RcSxJcEF3/YNC1JKmGGIqkoBv4abT8ljgLSYB8MMwRus4kaUIwFEkFxV1oiZoAr8IuIPxteBD4Q8y1SFLFGIqkgp8D+wiDjE+Kt5RY5VvK7DqTNKEYiqSCNuAX0fIFg61Yw5qBs6Nlu84kTSiGIqmvG6Prt8VaRXzeTJh76XfA6iHWlaSaYiiS+voRYc6SxcCxMdcSB7vOJE1YhiKpr13Ar6LlidaFdgBwXrRsKJI04RiKpBfKd6FNtFD0WqAJWAc8GnMtklRxhiLphW4knJ/pZGBRzLVUUj4Eeh4BSROSoUh6oW3A3dHym+MspIIagNdEyzcOtqIk1SpDkVRa8USOE8H5wDRgI3B/zLVIUiwMRVJp3yec5uIMYG7MtVRC8bnOcnEWIklxMRRJpW0knOYiDbwp5lrGWx3w+mjZrjNJE5ahSBpYvgut1k8Qew4wG9gO3BVvKZIUH0ORNLDvR9fnEkJDrcqHvpsIR91J0oRkKJIG9jThdBfF3Uu1Jk3hCDsnbJQ0oRmKpMHVehdafiD5HuC2mGuRpFgZiqTB5UPRqwmHrNeafNj7EdARZyGSFDdDkTS4R4E19J3csJZ4AlhJihiKpKHdFF3XWhfaSYTTmLQDP4+5FkmKnaFIGlq+FeV1QGOchZRZfsLGnwP74ixEkpLAUCQN7QHgWeAA4LyYaymnfChywkZJwlAkDUeOQhdarZwLbTGwBOgGfhxzLZKUCIYiaXjyrSlvBibFWUiZXBhd/wrYGWchkpQUhiJpeO4EngeagbNjrqUc7DqTpH4MRdLw9AI3R8vV3oV2OHAykKXwM0nShGcokoYv36pyAdX9u3MBkAJ+A2yJuRbp/7d3t7FxVfkdx793ZvwU25PEBCUkPO8mKaSBjXgoYUVIRLdturQhu2SD1CK1QFAQSYUqpKLICazEm6LQGqFWSUpV+qIVJd3CtstGS5UIUliUdhOw4ogmLFg8GJc8b9ZxHjwzty9mxh47Tsg4Y9879vcjXfmMM77+6+aO/fM5Z86RYqOaf7BLY+0/gePAdPLbY1SrYk+XCzZKUglDkXTx+oCfFtrVupBjMdCFOJ9IkgYxFEnlKfaufJ/8EFS1WQ4kgV8An0VciyTFiqFIKs828qs/X0t+m4xq415nknQehiKpPL3Am4V2tQ2hTQEWF9oOnUnSEIYiqXzFXpZqe2v+MqAW6AD2R1yLJMWOoUgq338AZ8lvk3FDxLWUw6EzSboAQ5FUvl8BOwrtauktagJ+p9A2FEnSMAxF0sgUg0Wl5xX9WeGYVeHzLgUagE6gvcLnlqRxwVAkjcxrQAa4Bbi+gud9oXB8s4LnhIEerX+t8HkladwwFEkjcxh4t9C+L8pCLkId8PuFtkNnknQehiJp5Ipva6/kENrjheOjCp7zO0Aa6AJ2VfC8kjSuGIqkkfsR+e0y7gSuqNA5/7ZwfFmh88HA0Nlr5OuVJA3DUCSN3BfA/5B/HS2r0DmvLBy1FTpfEviDQtsFGyXpAgxF0qUpBo1KvTX/88KxsELnWwxMA44AOyt0TkkalwxF0qX5UeHjYqAlwjrOpzjf6XXy75aTJJ2HoUi6NB+R3zajhoFhqrgIGBjWc+hMkr6GoUi6dMXeoritbr2Q/PykXwPbI65FkmLPUCRdumIvzO8CzVEWMkRx6OwnwOkoC5GkamAoki5dO3CA/CKJSyOupZQbwEpSGQxFUmX8uPCx0nuhjdS3gG8Ap4BtEdciSVXBUCRVRrE35l6gPspCCorzm34GnIyyEEmqFoYiqTJ2kV9fqIn8thpRK4Yih84k6SIZiqTKCInPENocYB7QR36StSTpIhiKpMop9sr8IZCKsI77Cx93AMcirEOSqoqhSKqcncAh4DLg7gjrKPZUuWCjJJXBUCRVThb490I7qoUcrwRuAXIltUiSLoKhSKqs4hDa9xjZ6+uvC8fnI/z+95Pf3uMdoHuE55CkCSnKeQ/SeLQd+BUwA7gD+HmZX//nl/j9iz1UDp1JUpnsKZIq6wzwRqE91kNo04E7C+0fX+iJkqRzGYqkyiv20nyf/FDWWLkPSAK/ADrH8PtK0rhgKJIq76fkV5G+lvx2G2PFBRsl6RIYiqTK6wXeLLTHaghtCrC40HY+kSSNQFVPtH7mmWdiH+r27dsXAGzdujUXdS3DCBgY3gmjLOQCSuuLY42l92Bpfa+TXy/oe8DTY1DHMqAW2AccGFJXtVzDONYGg4dAfR2PTLXcgxDD+lasWBEcOnQoAFi8eHEc78F+Tz/9dBgEQeyu4cWq2lC0ZcuWeWEYdkRdx4WsW7eOI0eORF2GonUj+fWLxsq8Mf5+kkbZ1q1b+9s33ngjN998c4TVXNjmzZv/Dng06jpGKvY9LZIkSWPBUCRJkkQVD58NFYbhimQyeTTqOkrlcrk3yb9FmvtnzjxxR3PzVxGXNMhznZ3XHDx7thbgpptu4pFHHslEXVOpbdu2JbZt25YASAYBfzlnzkdR1zTUk/v3zy62H3/88dzcuXP7x/t7enpobW1N5XI5NmzYkJ02bdqojLO//PLLyT179gT33HNPbtmyZYPmG6xbty7V09MDwO3p9JkfXHHFZ6NRw0j9Q1fXFft6epoALr/8ctavXx+re/CTTz6hra2t/+fkk9dc0zWjvr43ypqG+osDB76ZDcMAYPny5eGSJUtiNXy6cePG5GeffRYAXNfQkH386qs/ibqmUtuPHp267dChacXHbW1tmUQiXv0FTz75ZOrs2bMAdHV1/fOCBQv+PuKSBsnlchuIdr/Hihk3oSgIgndXrVoVq20NpkyZ0v9L8KrGxsy3J0/uibKeoeqCoP8X6OTJk8PbbrstVhP4Pvjgg/4JrgEQt+s31OzZs8+5hvPnzw/b29uDY8eOsXTp0opf376+Pvbv358CWLlyZXb+/PmDglcqlQopTHKdVlubjds1fO2rr/qK7fr6euJ2D9bU1Az67Ti/ufnUnEmTYnUNS2eBX3311bF7HTc2NiaL7aZkMozbPXjg5MmG0se33nprLplMnu/pkSgNaUePHv101apVOyIs5xybN29eHXUNlRKvOCyNM8V3irz99tuj8lrbtWtX4uTJk7S0tITz5s2r2nd8SFIcGIqkUbRkyZJcEAR0dHQEhw8fHrS6dXFY62L19fXR19c36HPFsLV48eJc3Lr8Jana+FNUGkXTp08P586dG+ZyOXbu3Nkfinbv3h088MADNeWc64033kisXbs2lcvlR0dyuRzvvPNOAuDuu++2l0iSLpGhSBplxSG0t956KwGwffv2RGtra00Ylpdj6urq2LdvX2Lt2rWp3t5edu/enTh+/DjpdJpbbrklVvNIJKkajZuJ1lJcZLNZgiDonxy5ZMmS3KZNm5J79uxJrF69OtXZ2Zk4ceIEM2bMKOu89fX1pFIp9u7dm1i5cmXNrFmzQoBFixYNmhiayWRIpXxpS1K5/MkpVdi7776b2LBhQ2rmzJlhOp2mt7eXZDJJNpulvb29v3e23DlA9fX1JJNJTp8+zeHDh4MjR44EAO+9915i1apVNdlslkOHDtHY2Mgrr7zS93XnkyQNZiiSKmzRokW522+/Pbdr165EZ2fneZ9XW1tb1nkbGhrC0h6h4vDbkSNHKAakdDrN888/byCSpBFwTtGAGcB3ye/Z8iiwINpyVM2effbZzPTp0y84aai+vr6sc9bX11+wd6m5uZnVq1dn5syZ46RrScOZCnyHgd9z42LBxUqa6KFoAfndzD8HuoGfAJsLx3cjrEtVrra2lra2tszUqVPPG1BGEoqCIBj232pra1m4cGFu+fLlTriWVOoq4F+Aj4GjwJsM/J57MMK6Ymmih6LfAJYBVxYe/xrwr2xVxMyZM8PW1tZsc3PzsP8+adKksu61hoYGgiA452uCICh+r1htkSEpFmYCPwCuLzw+BZyNrpx4m+ih6Avgr4A/Ih+QppC/YaSKuPPOO3P33ntvrrGx8ZwwM2nSpLLO1dDQMGyImjp1atjW1papqSlr2SNJE8Mx4G+Ah4CbgTQQu30k42KiT7T+r8IhjZo1a9ZkOjo6aj788EMymYHOnMbGxrLO09DQQFjY+LMonU6Hra2t2a+bvyRpwjoArIm6iGox0XuKpFGXSCTYuHFjX0tLy6DgMlzv0YXU1NRQXM0a8j1NK1asyC1cuNB5RJJUAYYiaQyk02k2btyYaWpq6v9cuT1FQH9PUxAEzJ07N/fwww9nK1akJE1whiJpjMyePTtcs2ZNpvius3Q6XfaQV3FydktLS/jcc89lzvduNElS+QxF0hhatmxZ7q677sqlUikaGhpGdI7Jkyfz4osvDup1kiRdOkORNMbWr1+fmTVrVljuu88gvynsE088kbnuuuucWC1JFTbR330mjbmamhpeeumlvrq6urK/dtOmTZnLLrvMQCRJo2C89hR9SX4RxqHHv0VZlFTU1NTESNYVMhBJ0ugZr6FIkiSpLON1+Gxm1AVIkqTqYk+RJEkShiJJkiRg/A6fleMeoHQFvGTh4zeA3y75fCfw8VgVJUlShXwbKF0YrbjI2SwG/577P6BjrIqKI0MRvMnwPWZ/UjiKngXWj0E9kiRV0j+S/0N/qN8rHEX/BPzxmFQUU4Yi2M3FDSN+OdqFSJI0CjqA4xfxvM7RLiTuDEVwe9QFSJI0iu6LuoBq4URrSZIkDEWSJEmAoUiSJAkwFEmSJAGGIkmSJMBQJEmSBBiKJEmSAEORJEkSMI4WbwyC4PUtW7b0RV1Hqaeeeqq4jxqvfP55eltX1+wo6xnqYF9fbbHd3t4ePProozVR1jPUwYMH+9vZMORPOzpidf2GeuGFFxJNTU2x+kOjp6enf1+/nx8/Xh+3a9h15kx9sX3w4EHidg/29vYOevzDjz+eVRsE2YjKGVY2DPv/j1999dXEjh07YnUNP/300/76Pj51Khm3e/BoJjPoej322GOxun4AZ8+e7W/PmTPnwQcffHBRhOWcIwzDuVHXUCnjJhSFYRi7lamDYGCf2e7Tp1PdA5vwxc6JEyfYu3dv8PXPjEYI/G9vb2yvH0BnZ2dsrx/A8UwmcTyTie01PHPmTKzvQYBPTp1q+PpnRae7u5vu7u7YXsPebDaI++s47vdgXV3dlWEYXhl1HeNVrP6qlSRJikrV9hSlUqmDfX19P4y6jgu54YYbfuv06dP1AF1dXfu7u7u7o65piNnArEL7MLAvwlqG0wLML7RzwM4IazmfxSXtdi5u08Wx9JvAZYX2F8AvI6xlOLPI34cAp4D/jrCW4dQBd5Q83kW+zji5DWgstH8JdEVYy3CuB64qtI+Rf53EyRTgWyWP346qkPNZsGDBXUEQJAHq6ur2AkciLulCdkddwKWo2lD00EMPHQKeiboOTXixDuaSqt/777/f396zZ0+ElYx/Dp9JkiRhKJIkSQIMRZIkSYChSJIkCTAUSZIkAYYiSZIkwFAkSZIEGIokSZIAQ5EkSRJgKJIkSQIMRZIkSYChSJIkCTAUSZIkAYYiSZIkwFAkSZIEGIokSZIAQ5EkSRJgKJIkSQIMRZIkSYChSJIkCTAUSZIkAYYiSZIkwFAkSZIEGIokSZIAQ5EkSRJgKJIkSQIMRZIkSYChSJIkCTAUSZIkAYYiSZIkwFAkSZIEGIokSZIAQ5EkSRJgKJIkSQIMRZIkSYChSJIkCTAUSZIkAYYiSZIkwFAkSZIEGIokSZIAQ5EkSRJgKJIkSQIMRZIkSYChSJIkCYBU6YMwDK/dsmVLVLVIkiSNmTAMry19nBry7z8Lw3DsqpEkSYoJh88kSZIwFEmSJAHw/8Ve17voNSEvAAAAAElFTkSuQmCC"
+ }
+ },
+ "cell_type": "markdown",
+ "id": "dda0a371",
+ "metadata": {},
+ "source": [
+ "\n",
+ "![](\"attachment:g28304.png\")
\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9707f13b",
+ "metadata": {},
+ "source": [
+ "The implementation is given in the following cell. Note that we introduced an extra loop that represent\n",
+ "each one of the two phases."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c0cbe2d8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "function red_black_gauss_seidel(n,niters)\n",
+ " u = zeros(n+2)\n",
+ " u[1] = -1\n",
+ " u[end] = 1\n",
+ " for t in 1:niters\n",
+ " for color in (0,1)\n",
+ " for i in (n+1):-1:2\n",
+ " if color == mod(i,2)\n",
+ " u[i] = 0.5*(u[i-1]+u[i+1])\n",
+ " end\n",
+ " end\n",
+ " end\n",
+ " end\n",
+ " u\n",
+ "end"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "13ce270f",
+ "metadata": {},
+ "source": [
+ "Run the method for several values of `niters`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "59c1e7f8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "n = 5\n",
+ "niters = 1000\n",
+ "red_black_gauss_seidel(n,niters)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d1bacc26",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "Question: Which of the loops in the red-black Gauss-Seidel method are trivially parallelizable?\n",
+ "
\n",
+ "\n",
+ "```julia\n",
+ "for t in 1:niters\n",
+ " for color in (0,1)\n",
+ " for i in (n+1):-1:2\n",
+ " if color == mod(i,2)\n",
+ " u[i] = 0.5*(u[i-1]+u[i+1])\n",
+ " end\n",
+ " end\n",
+ " end\n",
+ "end\n",
+ "```\n",
+ "\n",
+ " a) All loops\n",
+ " b) Loop over t only\n",
+ " c) Loop over color only\n",
+ " d) Loop over i only\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d511cc02",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "answer = \"x\" # replace x with a, b, c or d\n",
+ "gauss_seidel_2_check(answer)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8946b83f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "gauss_seidel_2_why()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8ed4129c",
+ "metadata": {},
+ "source": [
+ "## MPI implementation\n",
+ "\n",
+ "We consider the implementation of the Jacobi method using MPI. We will consider the 1D version for simplicity.\n",
+ "\n",
+ "\n",
+ "\n",
+ "Note: The programming model of MPI is generally better suited for data-parallel algorithms like this one than the task-based model provided by Distributed.jl. In any case, one can also implement it using Distributed.jl, but it requires some extra effort to setup the remote channels right for the communication between neighbor processes. \n",
+ "
\n",
+ "\n",
+ "\n",
+ "We are going to implement the method in a function with the following signature\n",
+ "\n",
+ "```julia\n",
+ " u = jacobi_mpi(n,niters,comm)\n",
+ "```\n",
+ "\n",
+ "The signature will be very similar to the sequential function `jacobi`, but there are some important differences:\n",
+ "\n",
+ "1. The parallel one takes an MPI communicator object. This allows the user to decide which communicator to use. For instance, `MPI.COMM_WORLD` directly or a duplicate of this one (which is the recommended approach).\n",
+ "2. Function `jacobi_mpi` will be called on multiple MPI ranks. Thus, the values of `n` and `niters` should be the same in all ranks. The caller of this function has to make sure that this is true. Otherwise, the parallel code will not work.\n",
+ "3. The result `u` is NOT the same as the result of the sequential function `jacobi`. It will contained only the local portion of `u` stored at each rank. To recover the same vector as in the sequential case, one needs to gather these pieces in a single rank. We will do this later.\n",
+ "\n",
+ "The implementation of the parallel function is given in the following code snipped:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d31c1e41",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "code1 = quote\n",
+ " function jacobi_mpi(n,niters,comm)\n",
+ " # Initialization\n",
+ " u, u_new = init(n,comm)\n",
+ " for t in 1:niters\n",
+ " # Communication\n",
+ " ghost_exchange!(u,comm)\n",
+ " # Local computation\n",
+ " local_update!(u,u_new)\n",
+ " u, u_new = u_new, u\n",
+ " end\n",
+ " return u\n",
+ " end\n",
+ "end;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "04a38c37",
+ "metadata": {},
+ "source": [
+ "In the following cells, we discuss the implementation of the helper functions `init`, `ghost_exchange`, and `local_update`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "96b8d40e",
+ "metadata": {},
+ "source": [
+ "### Initialization\n",
+ "\n",
+ "Let us start with function `init`. This is its implementation:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b98a9348",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "code2 = quote\n",
+ " function init(n,comm)\n",
+ " nranks = MPI.Comm_size(comm)\n",
+ " rank = MPI.Comm_rank(comm)\n",
+ " if mod(n,nranks) != 0\n",
+ " println(\"n must be a multiple of nranks\")\n",
+ " MPI.Abort(comm,1)\n",
+ " end\n",
+ " load = div(n,nranks)\n",
+ " u = zeros(load+2)\n",
+ " if rank == 0\n",
+ " u[1] = -1\n",
+ " end\n",
+ " if rank == nranks-1\n",
+ " u[end] = 1\n",
+ " end\n",
+ " u_new = copy(u)\n",
+ " u, u_new\n",
+ " end\n",
+ "end;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1b9e75d8",
+ "metadata": {},
+ "source": [
+ "This function crates and initializes the vector `u` and the auxiliary vector `u_new` and fills in the boundary values. Note that we are not creating the full arrays like in the sequential case. We are only creating the parts to be managed by the current rank. To this end, we start by computing the number of entries to be updated in this rank, i.e., variable `load`. We have assumed that `n` is a multiple of the number of ranks for simplicity. If this is not the case, we stop the computation with `MPI.Abort`. Note that we are allocating two extra elements in `u` (and `u_new`) for the ghost cells or boundary conditions. The following figure displays the arrays created for `n==9` and `nranks==3` (thus `load == 3`). Note that the first and last elements of the arrays are displayed with gray edges denoting that they are the extra elements allocated for ghost cells or boundary conditions."
+ ]
+ },
+ {
+ "attachments": {
+ "fig.png": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6gAAAFzCAYAAAAzJyTtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7d15mBxVvf/xd08mmewkISEEIoTFsCMSBWRT2d0uCIr7DoGrXnC/qCigV38oCoKKTBCv4Mp1AwHZjIIiyBIBZQmIAUNICISELJBJMjP9++N0p2tmumeyzPQ5PfV+PU8/nOruTH/o7uqqb9Wpcwqtra1FJEmSJEmKrCl2AEmSJEmSwAJVkiRJkpSI5m7LRxcKhSdiBJEkSZIk5UuxWJwG3Fhe7lKgFgqFJ2bOnPlovUNJkiRJkvJn1qxZFIuVYZHs4itJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSkJz7ACS6ueyyy4b097e/ufYOXpz8cUXb7948eIWgFWrVi1ZtWrV0tiZupkITCi1XwQWRMxSzUhgamb50VhBejE9015AeB9TMpXwPgIsBZZEzFLNBML3EKANmB8xSzXDge0yy48BnZGy1LIzlYP0C4FVEbNUsw0wutR+HngmYpZqxgFbldprgSfiRempubl52MSJE6eVlz/+8Y/PGzduXHvESL0qFotfOPXUU6+JnUNKhQWqlCNNTU1DgJfFztGbZcuW8fTTT5cXX1K6pWoEsGXsEH1I+vMGXho7QB+2Ld1SNQIYHztEH/aKHaAPO8UO0IcRwJTYIXoxgsR+Z9rb27PbEYrF4h4R4/SpUChM6PtZUn7YxVeSJEmSlATPoEo5ViwWL2xqakqqi2pnZ+dXgaEAU6ZMeWTRokXfjxypu/+i0n3xReCsiFmqeS3w+szyp2MFqaEJ+Fpm+Vrg1khZavkS4awQwL+B70TMUs1MKmee1wKfj5ilmv2At2aWvwysiJSllvMy7T8Cv4sVpIYvAmNK7UXA+RGzVPMeYO9SuwM4I2KWHiZMmLDr0qVLP1ReXrly5UUTJkx4Mmam7orF4heAsbFzSCmyQJXy7cczZ868J3aIrClTppxDqUCdOnXqvEWLFn0jcqTu3kKlQG0DUss3hK4Famr5mulaoN4CfDNOlJo+R6VAXUh67+HRVArUdaSX74N0LVC/RyiyUvJ1oFBq30V67+EnqRSoS0gv3350LVCTyjdjxozX33zzzesL1Dlz5vz4K1/5yt0xM3XX2tr6cSxQpars4itJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSoIFqiRJkiQpCRaokjZXM7AX8AHCfJF/Bu4p3T5Z5ywFwjygrcADwEqgE3gauAb4jzrnqWYL4GTgt8ACYA2wGngE+DawQ7xo6+0BnAvcASwFisAywmf7n8CweNGAMA3RccCPgccI89GuA54Efgq8Kl609aYAnwJuBhYD7cALwH2EeVYnxou23kGEdfZ+wjylReAZ4HrgRCrTsMQyGngf8GtgPmFapzbCZ34JsEu8aOtNJ3yetxGmgykCy4HbgY9Tma4oliHAG4AfEn5jXiB8F58CfkH4vYxtEnA64Xv3NGFdfhH4B+F3aOtIuYYC+xJ+ry8hfKblbdsHImWScsF5UCVtjv8kzH83ssbjf65jlh2AK4FXVnlsMvDG0u3HwPsJc/fV2/uB84HxVR6bXrp9AHgXcHX9Yq3XAvwEOJ6exck44ODS7YPA6wg75PX2KuBHwE5VHpsKvAN4O3A2oXCI4RzgvwnvZ9ZI4GWl28mEwuFv9Y0GwDaEdeXgKo9NAo4p3d5JKFTX1i/aem8FvlvK091Opdv7gA8RDkrUW4HwPXwHPQ/2jyV8T19FyHcUYT7detuH8N7sVuWxbQhzOr+FMA/xp+qYK+tThHV1VLf7m4E9S7eTCAekbqtjrjNLt+7rcNmUOmaRcscCVdLmmEzP4vR5QjFTb7tRKU4fBq4lnGkZCxxGKKgA3l26/5x6BwTeRChO1wG/B24hnKHcHngvsB1hR+3nhB2zf9U530jghFL7aUKR/I/S8isIhfPQUvtHVN7TejqISnF6J3Aj4czpJMJO7H6E4uEcwvfgFxEynkjYsV0F/I5wJno1lQMQ4wlnhX4L7Fp6Xj3tQKU4/Rfhc36U8N07BDiW8B4eSziD9Yk654NQIE8iHEj6Q+m2hHAQ4l3AzsBw4HLgQcJZ4HoqlHJQynV1KUMH4QDE+wjfgT0I38GDCWdX6+mVVIrTewnfxSeALQkHRw4pPfZJYC7w/Trng/B7M4qwflwP/IWwPuxIOBA2iZD3KsK6Uq+DYlPoWpwWCb0MtqjT60u5ZoEqaXM8Q+h+N6d0uwd4DfDLSHn+AXwGuKHb/d8APkLozgihO9lXCF3d6qkIXAr8D6HLYtZXCUXrgYQd748QpzB4hnDm4HJ6njlrJRTVLYQCYk9CV+p6mw2cQfi+ZZ0LXAR8tLT8SeIUqGsIn/EFhAMQWV8H7iIcjNiWcLY3RmHwKGFduYbQDb7sfELh9ePS8qnAFwhdQ+vtx4Sza90P1HyVkPsown7M6YRipt6WAmcBlxEKrKzvEoqt0YR1+sDScr39hbCudD/7+DXCb+DnSsufJM73sJ3w+3we4bcn61zgr4Su3FsSeqB8o065FhJ6GZS3bX8jnC2/uE6vL+Wa16BK2hwXE46Af5VwJuu5iFn+CsygZ3Fa9l3g36X2eMIR+nr7CDCTnsUphB3cszLLM+qSqKsXCGdcLqV6t86/Eg5IlO1bj1DdXAEcQc/iFMIBgM9RKbheTpzrKI8mFHXdi1MI16N+M7Mc43N+kHCW72q6FqdlP6FyRnIE1buIDrTPAe+hei+CtYSDKGUx3sMi4X35Dj2LU4C/UynyIc66cjVwKLW7xp5NuNYTwtnJ7t1s6+EtwKfpWZxC6I3z/zLL9fycv0I4eHQe4ez983V8bSn3LFAlDRZLCV1ne/NUPYL0YnEfj2fzxSis1lK9qMqKnbHajmzWSkJXvLIYGfv6nBdk2jHyPU8YbKg3sT/n1NeV8oBSvYmd8RmqH4AoW0fX/wc/Z0lJsECVlBfbUDkC/yzweMQstbwp074zWorahgGvzyzfFStILw6hcg30PfS+gx5LdjTpFD/niVSuUV0FPBQxSy2prysFwqBsZSlm3Idw/TuEs+r1vhZ6Q6T+OUsaABaokvKghdDdrjzoxdfp+2xrvb0C+GKpvYLK9bIp+Tawe6n9a8IgRCmZRLgesOyrsYL04q2EAbEgdF+9MmKWaoYC/0cYXAzgW8S5/rQ3u1Pp+tlGuG42NWcC+5fas0mvuJpAuD67fFbyKxGz1HI08OFSeyFhqhxJOWCBKmmwawF+RWW+v6tIb4d2X8I1vKMIo4B+gMr1sqn4EuH6WQgjfp4SMUs1kwhzjr60tPw14Lp4cao6lnCgpEA4CPE2KtcApqCZMDpzeV35PfDleHGq2o2Qazyhm+2ppHeg5JNUpjj6N2FE35RsQVg3di4tXwL8LF6cqg4nHARrJlzj+3bCXMyScsBRfCUNZkMJOznlbqk3EkZiTKnb58uo7HB3EOZN/HWv/6L+ziYM+gPhrN9RxJkDtZYJhLNUe5WWLwY+Gy9OVW8inLEaSuhK+SbC6KCpGEIont9WWr4NeDNx5kCtZTrwR8L0VkXC6L2XR03U08epjDS7gDCgV+xr37NGE34Hy2d3f0wYvC0lryWM0jyCMCL2W6jvnNqSIrNAlTSYtVIpTq8HjqfvwWHqaRvCqMPjCUXz++k68mcKTqYyuvBjhJ3HBbWfXnfNhJ3ZcnH6HeA06j/nZG9eSejKO5TQXfZ11B5ZNZZvUClO/0SYJzOlaxInADcRilOA/yKMzJ2St1EZoXkBYcqtes9l3JsC4SBJuTi9gjA9T0oH7HYnjD5cLk6PJRTUknLELr6SBquDCV1lAe4mHIVPqTiFMFfm1qX2p0ivOB1H5WzQEsI1YSkVpxB2sA8sta8kveIUwrW7IwiFwFtIrzjdm/C+QRgs5z9IqziFcAa/PKDPl0ivOB1O+JzL3bePIa3iFML1z8eU2tcTemt0xItT1QXAmFL7fVicSrnkGVRJWV8ijNRazSLgwjpm2VzHZtqfJa1r/cqOK/13Hmm+t4dRGSznQkLO1GQ/50+QXnE6hcoZqxuoPU9vTP9B5YD1WcDyiFlqKX/OzxIO7KTmIMJ10BDmEX4wYpZajsu0Pwm0xwpSwxjCtacAt5PeAGKS6sQCVVLWJ6g9Wfv9pFlE1fKSTDu1QVQgvM/jS+1HSKubXdnUTDvF9xAqGZ8njPSZmkZ6DyHNKWUKVDI+RnojcEP67yFUMnYSBjpLzdaEa6Eh3XVFUh1YoErKupfQFbGaf9YzSD9oybTXREtR2/BMO7Wux2WNlDHFzxga4z3MrispZmwiXL8LaeaDxvqc15JeTwNojPdQUh1YoErKOiR2gAGS4s5YVur5GkEjvIeNkDF1voebrxHew0bIKGmAWKBKGqxOJVxnBaH7Z2qeB3YqtV+IGaQXs4BfltpPxwzSiyMIZ9dSG+yl7G4qn3OK30OAz1CZ7/TJmEFq6KDyHq6OGaQXPyPMwwvwTMwgvTiecBY11eLvESqf84qYQSTFZYEqabBaHDtAHzpIc9ChrOdJt6gqS7Ggymoj/c/52dItZam/hytIv6hKaT7WataS/ucsqQ6cZkbSYHUh4YzGzXQdMCkV21HJd0HkLLXMpJLx6MhZavkNId9NsYPUcAiV9/BjkbPU8hUqGXeNnKWaLankuzRyllreRSXj8ZGz1PJjKhlrjTUQ075U8p0ZOYukiDyDKmlzbEGlC2jZVpn28cCe3R4/h/rMA/kKKvNj1hqZOKZRhO6p0HVwkJS8lErGn8YM0ovXEOZrTXEUZAjrQ/k9TPXs0D5UMm4RM0gNLVTyPRAzSC92pJLxuphBenEQMK3UTnH/bwKV93BpzCCEOXe/3+2+7EjNHwBe2+3xj5Pu91NqKCn+QElqHMOo7FBUs13plvW9gYsjSdJmG03v27adS7escQMXR8oXC1RJm2Mt8PuN/Df1GkBkJbCs1E5xAJ0OKvlWxgzSi9VUMqY6jcvzhEFfUj2DupbKe/hizCC9WEUlY3vMIDV0UsmX6nWe2XUl1SlSllPJmOJASeuo5Is9cNwqNn7blvr1+lLDsECVtDmWA0fGDlHDMbED9OFRQpe2lH2xdEvZDrED9OEa0v+c3xY7QB+eJv338BulW8r2iR2gD7eSzuf8b9LdtkmDnoMkSZIkSZKSYIEqSZIkSUqCBaokSZIkKQkWqJIkSZKkJFigSpIkSZKSYIEqSZIkSUqC08xIOVYoFE5vbW1dHDtH1pe//OWh5fbjjz++K+lN3bBdpj2S9PLt1205tXyFbstvBKbECNKLkZn29qT3Hr400x5Gevn27Lb8ReLPa9ld9nv4WtJ7D8dk2lNIL9/emfYQEsv317/+tcv0U/vvv/9pb37zm5Pa1gFjYweQUmWBKuXbu2MH6K6pqdKxY8mSJTsAn4yXpk/DSTsfpJ/vNaVbqrYh7fdwKGnnAzg1doA+7EfPAzspmUjan/EQEsu3cuXKLssjRoxIblsnqTa7+EqSJEmSkuAZVClHhgwZsq5YLN4YO0dvJk+evFtzc/NIgGXLlj21bNmyp2Nn6mZbYOtSewXwz4hZqhlL1y6gc2IF6cWMTPtRYGWtJ0YynUoXy6eBpyJmqWZrwvcQQtfZuRGzVDMK2DWzfB/QESlLLfsQzvwB/At4PmKWanYCxpXazwLzI2apZhKVyx3agAcjZumhpaVl+LbbbrtH5q5/FIvFtdEC9aGpqSm13xgpKgtUKUfe+973vgAcEzuHJEkDZc2aNcybN2/98hlnnBExjaSNZRdfSZIkSVISLFAlSZIkSUmwQJUkSZIkJcECVZIkSZKUBAtUSZIkSVISLFAlSZIkSUmwQJUkSZIkJcECVZIkSZKUBAtUSZIkSVISLFAlSZIkSUmwQJUkSZIkJcECVZIkSZKUBAtUSaoff3M1kP4ELAVeEzmHBhd/tyTVlT86klQfw4GfAJ+KHUSD1hhgPDAsdhANGk3Ar4CzI+eQlCMWqJI08LYBbgPeDnyptCz1t47Sf4dETaHB5FjgOOAs4H/x4IekOrBAlaSBtTdwBzADeA54HbAwaiINVp2l/7ptV3/5DXAKsA54P/BHYFLMQJIGPzdikjRwXgf8GdgO+CdwIHBr1EQazCxQNRBmAW8Anif8ht0B7BY1kaRBzY2YJA2M04FrgbHA74H9gEejJtJgZ4GqgXIzcDDwOLAT8BfgsKiJJA1absQkqX81A98FvkX4jf0+8HrC2QdpIFmgaiA9CLySMFr0eOBG4MNRE0kalNyISVL/ye60dQBnACcTrt+SBpoFqgbac8BRwI+pHIy7EL9zkvqRPyiS1D92pNLtbRVwPPC1qImUNxaoqoc1wHsJB+A6gdOoXM4gSZvNjZgkbb7swCFPAa8Gfhs1kfLIAlX1UiQcgHs7sJquA8JJ0mZxIyZJm+ftwGxgK+A+4ADgb1ETKa8sUFVvvyD0GllMmFLrr8AroiaS1PDciEnSpikAZwM/A4YDvwIOAhZEzKR86yj9d0jUFMqbclF6HzAFuAV4c8xAkhqbBaokbbzhhEFCziotXwScCLwYLZHkGVTFswA4lHAt6ijCAbuzYwaS1LjciEnSxplCmGbhncBa4P2EOU87e/k3Uj1YoCqmlcBxhAN2BcIBvMuAYTFDSWo8bsQkacPtRRgM6ZXAUuBo4PKoiaQKC1TF1kE4YHcK0A58EPgDMClmKEmNxY2YJG2YY4DbgO2Bxwgj994SM5DUjQWqUjELeAOwnHBt/h3ArlETSWoYbsQkqW+nU5nnbzbhDOojURNJPVmgKiU3AQcDTwA7EeaJfm3MQJIagxsxSaqtGfgO8C3CyKiXEeb7ez5mKKkGC1Sl5gHCAb0/AxMIRet/Rk0kKXluxCSpujHA1cBHCJPSnwOcBKyLGUrqhQWqUrQEOBL4CeGg38XAhfg9lVSDPw6S1NOOhLn9Xg+8QJjT7+yYgaQNYIGqVK0B3kM40FcETgN+CYyMGUpSmtyISVJXryIM6LE7sBB4NeFMqpS6coE6JGoKqboi4UDfO4DVhAN/twMviZhJUoIsUCWp4m2EKRG2Au4HDgDmRE0kbbiO0n/dtitlVwKHA88ALyP0VpkRNZGkpLgRk6QwqfzZwM+A4cCvCdPIPBkxk7Sx7OKrRnEH4QDgQ8A2wK3AcVETSUqGGzFJedcC/Ag4i1CoXgS8FXgxZihpE1igqpE8TihSrwNGEQ4Mnh0zkKQ0uBGTlGcTgd8D7wLWAh8gzHna2ds/khJlgapGsxI4ljCdV4FwoPD7wNCYoSTF5UZMUl7tCdxDmEh+KXAM8MOYgaTNZIGqRtQB/BdwCtAOfIgwFsDEmKEkxeNGTFIeHQ3cBmwPPAYcBPwxaiJp81mgqpHNAt4IrCAcOLwD2CVqIklRuBGTlDczgWuBLQhF6quAuVETSf3DAlWN7kZCcfpvYGfCNDSviRlIUv25EZOUF0MIAyC1As3ADwhTHSyJGUrqRxaoGgz+QThweBcwgVC0vi9qIkl15UZMUh6MAa4mXOdUBM4hXOe0NmYoqZ9ZoGqwWAS8GvgpMIwwPsCF+N2WcsEVXdJgtwNhIvg3AC8Ax+NUBhqcOkr/dduuwaANeDfhgGIROA34P2BkzFCSBp4bMUmD2QGEgTZ2BxYSrmW6KmYgaQB5BlWDTZFwQPGdhIL1BOAvwNSImSQNMDdikgarEwlTFUwG/k64pumeqImkgWWBqsHq54QxA54B9iH0itk3aiJJA8aNmKTBpgD8N2GHZgRwPWFUyPkxQ0l1YIGqwex2woHGh4FtgVuBY6MmkjQg3IhJGkxagCuAcwmF6kWEefVWxgwl1YkFqga7ecD+hAOPo4FfEQ5IShpE3IhJGiwmAjcTBtVoBz4MnE5lp10a7CxQlQcrgf8AvkuYPuxc4FJgaMxQkvqPGzFJg8GewN3AIcAy4Gjge1ETSfVngaq8aAc+CnyMMHr1ScDvgHExQ0nqH27EJDW6o4DbgGnAv4ADCYMjSXljgaq8uZBwGccK4AjgLmB61ESSNpsbMUmNbCZwLbAFYeqBVwFzoyaS4ikXqEOippDq6wZC75n5wEsJgym9OmoiSZvFAlVSIxpCOHLeSrju6H+Bw4BnY4aSIvMMqvLq74R5r+8GtgRuBN4bNZGkTeZGTFKjGQ1cBZxGmMT9HOCDwNqYoaQEWKAqzxYBhwI/I4zofjnhQKbrg9RgXGklNZKpwJ8I1xy1Ae8Czo4ZSEqIBaryrrxdOKe0fBpwJTAyWiJJG82NmKRGcQBwD/Byuh4plxRYoEqhZ83ZVHrWvIUwcN7WETNJ2ghuxCQ1grcSdjAm0/VaI0kVFqhSRXZsgv2pHOCUlDg3YpJSVgD+G/g5MIKuozVK6soCVeoqO7r7toRLRN4UNZGkPrkRk5Sq8iAX5xJ+qy6iMt+dpJ4sUKWesvNjjwZ+QzjwKSlRbsQkpWhL4CbgPUA78FHgdKAjZigpcRaoUnXLgKOB7xGmKTsXmEWYpkxSYtyISUrNS4E7CIMgrQT+A/hu1ERSY7BAlWprBz4MfIywrpwMXAeMixlKUk9uxCSl5EjgLkKROo8wsMX1URNJjaPcw2BI1BRS2i4kXC6ykrDNuQ3YIWoiSV1YoEpKRfZo9u2EgS0ejppIaiyeQZU2zPXAwYQB9/YgjAp/aNREktZzIyYptu7XA/0cOBx4JmYoqQFZoEob7u+EA6H30HXcA0mRuRGTFFN2RMUicA7wTqAtZiipQVmgShtnIfAa4Cp6jhwvKRJXQEmxbAvcSpiTrg14N3A2oVCVtPEsUKWN9wJwPOEAafe5tyVF4EZMUgz7E7pV7QssIlz789OoiaTGZ4EqbZoi4QDph4C1wFsJ86ZOjphJyi03YpLq7S2EDf/WwD8I1wDdHTWRNDhYoEqb5weEMRCWAAcQDqTuEzWRlENuxCTV0+nAlcBI4EbCKIr/jppIGjwsUKXNdxvhwOlcYCrwZ8K0NJLqxI2YpHoYRhh84luE351ZhA3+ipihpEHGAlXqH48BBwF/JAzmdxVwWtREUo64EZM00MrD978X6AD+CzgFaI8ZShqEygXqkKgppMFhKXAM8EPCOnUh0Ao0R8wk5YIFqqSB9FLgduDVwErgWOA7URNJg8cBwLjMckfpv9ltewF4G6FbvaSNsxb4APAxwgGgmcB1wBYxQ0mDnQWqpIFyBHAXMB14nLAzfV3URNLg8g7gScL0GNvRtYvvWOBdhOvofkCleJW08S4kjOz7InAU4TrVaTEDSYOZBaqkTbUNcFaNx04Cfkc4u3MHoTh9qE65pLy4GFgDfJZwMOiS0v0HAI+WlqcDN5SeJ2nT/Ro4kHBQaE/C6POH1HjuZdgVWNpkFqiSNtX3gJO73VcgzCV3KTCUMGLv4cAzdU0m5cMjwLOEdW0yMKF0f0tpeTSwjHD2R9Lmu59wAGgOMBG4GXh3t+dMAt6PgypJm8wCVdKmmEEYhr+ZykTmo4HfEM6qFgndDt8BrI4RUMqJ79D7Oraa0B1RUv9YSBhX4WrCwaArgHMJB2gBjgTagE/jtarSJrFAlbQpfkA4SjyecJR4G+AWwiBIa4D3EM6kFuPEk3LjR8CqGo8VCdNjdNZ4XNKmeQF4M+FAbAH4b+DnwAhCYTqSMIL9ebECSo3MAlXSxnobYUAWCPObnkq4FmcGsIQwONJP4kSTcmcFcG+Nx5YQuuJL6n9FwoHYk4B1wImEdXFa6fGhwHHAThGySQ3NAlXSxmghHBHOTm0xgXAG9SHgldidUKq3bxCuNe1uBfBAnbNIeXMZ8DrCVGq70HX7OLH0uKSNYIEqaWN8jtBtqbsi4WyqoxZK9TebntehtgOXR8gi5dEIwpyp3RUII/6+tr5xpMZmgSppQ21F6M47sspjBWBn4HbCTvHL65hLyrtO4Bd0veZ7KeFacUkDYyjwJsJUapdT/eAtpftbcZ9b2mCuLJI21LepvQEum0S4DucqQkErqT4uBJZnlhcCT0XKIuXBqwiF6d5UpniqZWvglAFPJA0SFqiSNsTuhKHzh9R4fCVhrtNflZ43Dec+lerpcSrXoXYA342YRcqDPxGuMT0a+DWwmDC6bzVjCFOwja1PNKmxWaBK2hBXEqaUyVpHKELvAmYCLwHeQhgkyellpPr7GuFa1OcIXX4lDaxOwjbvBGAH4EOELr+LCdvIrK0I66ikPligSurLscC2meWlwDzgTMLgD/sT5n+rNkCEpPr5KWGn+AG6dveVNPBWEw7mHgjsC3wR+CdhmwlhrIbjge2jpJMaiAWqpN4MJQzu0Aw8DXyHUJDuBHwdeDZeNEndrARuAs6PHUTKuYXAucB0wtzgPy3dNwaYFTGX1BCcEkJSb7YHbgQuBf6CXXel1L01dgBJXdwLvIuwz3004VKYJkL3YElVWKBK6s1jwPtih5AkqcG1A9eVbpJ6YRdfSZIkSVISLFAlSZIkSUmwQJUkSZIkJcECVZIkSZKUBAtUSZIkSVISLFAlSZIkSUmwQJUkSZIkJcECVZIkSZKUBAtUSZIkSVISLFAlSZIkSUmwQJUkSZIkJaE5dgDly6xZs/YpFovXx87Rm89//vOT1qxZUwBYvXr18vb29jWxM3WzBdBSaq8GVkbMUs0IYEyp3Q4sjZilh6ampiGjRo3asrx85plnLhk3blxnzEy9KRaLHzv11FOvrMdrtba2PkXCBy5bPs0SoAAAHehJREFUW1vH/fOf/xwGsHbt2hfXrFmzKnambkYCo0vtdcCyiFmqaQYmZJafBYqRstQykcp3cDmQ1O/viBEjtmhubm4B2G+//VafeOKJqf3+Zj1+yimnHFiPF2ptbf0M8PF6vNameOyxx4Zecskl48vLK1eufCZmnhomAYVS+3lgbcQs1YwDhpXaLwJJ/f4edthhI1//+tePBmhqamofNWpUUvse3Q0ZMmSPk046KemMMVmgqq46OjqGNTU1bR07R29WrVpFW1tbeXF8b89NwKjSLVXDgKQ+787OTlaurOxTFovFrSLG2RAj6/haU6jsICVn9erV2c9uNJViMEUtJPbdr2Jy7AB9SO73d/Xq1evb69atS/3394V6vVChUBhTLBaT/b53/90n/XVzQt9PiSq539/m5mbGjCkfG09v36O7jo6OZA8Gp8A3R5IkSZKUBM+gKqpCoTATaOvzifV1Rbmx3XbbXTd//vy6dK/cCBdSObOwAPhcxCzVfATYv9RuBz4YMUsPW2211V7PPPPMp8vLS5cuPXvChAnzYmbqrlgsfov4R9C/XigUHoicoYt169ZdTOmo/dSpU+ctWLDg7KiBevoUsHepvRY4KWKWag4ETs0sfwJYEilLLZdTOYt/A/DTiFl6mDx58oWLFy8eD/DCCy88VSgUPhs7U1ZnZ+dhhULh/ZFjzC0UCl+NnKGLZ5999ijg3eXloUOHnrRu3brUutBekWn/Crg6VpAavkvl8p3HgbMiZulh/PjxHwdeDrB8+fK148aNS+r3t1gsbgf8T+wcjcICVVG1tbVdedppp62InSOrpaVl/Q7S9ttv/9D8+fN/FDlSd+dSKVCXA6nleyOVArWTxPLttddex8yePXt9gXrffffd8PWvf/3OmJm6a21t/SrxC9TZM2fOvClyhi6mTZt2AaUCdeLEiUsWLFiQ1HcLeAeVArWdxL77hOtNswXqVYQdzZRcnmk/TGLv4RZbbPH/ygXqmjVrls+cOTOpfJdccsko4P2RYzyT2vty+OGHjydToB5zzDE/u+aaa16MGKmabIF6P4l994HzqRSoz5FYvjFjxpxIqUBta2vrSO07WBqDxQJ1A9nFV5IkSZKUBAtUSZIkSVISLFAlSZIkSUmwQJUkSZIkJcFBkjSYbAnsS5hMGmAx8Kd4cSRJkiRtDAtUNbKphFH5ZgCvAKZ1e/z3wJF1ziRJkqTG1gKMLLXXAKmN+jyoWaCqke0P/L/YISRJktSwmoFdCCc8srfhpce/D5wcJ1o+WaBqMGgjzBk2B1gEfDluHEmSJDWI04FvxA6hCgtUNbI7gH2AB4H20n3TsUCVJEnSxusE/gk8CxwcOUtuWaCqkS0s3SRJkqRNcT/wKeAe4F5gBXAUcGPMUHlmgSpJkiQpr35fuikRzoMqSZIkSUqCBaokSZIkKQl28Y3vDcCepfYcNqyLwUFULtyeB/xiAHJJkiRJUl1ZoMZ3IvDeUvvbbFiBegRwdqn9OyxQJUmSJA0CdvGVJEmSJCXBM6hKyRh6/04uq1cQSZIkSfVngaqU3AQcUOOxDvy+SpIkSYOaXXwlSZIkSUnwjJRSchqwRY3HivUMIkmSJKn+LFCVkrtjB5AkSZIUj118JUmSJElJsECVJEmSJCXBLr5qdIcCwzLLUzPtCcAR3Z7/L+DxgQ4lSZIkaeNZoDamQuwACbkS2LrGY/sCN3e77yzgSwOaSJIkSdImsYtvfB2Z9oYeMKg10q0kSZKkDTecMFtE9nZj5vGTqjz+3jpnzBXPoMa3MtPe0MJz14EI0qCOBoZuxPMXDlQQSZIkSZvHAjW+pzPtPTfg+ROAQwYoSyP6e+wAkiRJaljtwBkb+W/+NhBBFFigxndvpr0XsAvwSC/PPxMYNaCJJEmSpHxoB74WO4QqvAY1vluB50vtAvA9oKXK8wrAx0o3SZIkSRp0LFDjW00oSsteC8wBTiV05X0t8FHgDuCC0vOvrnNGSZIkSRpwdvFNw5eBI4FXlJb3oGvRWrYWeF/p8WPrE02SJEmS6sMzqGlYDRwOXAF01njOA4Qzqr+sVyhJkiRJqifPoKZjBeHs6BeBo4DtCfMyPQncCfw189yvAReV2uvqmFGSJEmSBowFanr+DVzax3PaSjdJkiRJGjQsUBVVS0vLrNbW1qTOAp9++umFcvv+++9/PTAlYpxqxmfa2wI/ihWkhv0z7WYSy3f77bdvk10+6KCDzjrhhBOei5WnhgmxAwCfaW1tfU/sEFnf/OY3R5fbjz/++E4k9t0CXpZpDyO9fDt0Wz4fWBUjSC8KmfYxwKRYQapZuHDh+t/f0aNHb9va2praZ/zS2AGAXVN7X+6///5dZs+evX75uuuu+z7QES9Rn04ApscO0c2YTHsHEvt9e+aZZ15ebm+xxRbDUvsOFovF8X0/S2UWqIrtbbED9GbFihV7EAalStU44N2xQ/SiicTyrV69ustyS0vL6yJFSd3hsQN0N2TIkPXt5cuXb0li361umkk7H8BxsQP0YbfSLRmrVlXq+WHDhm1B+p9xDFuR2PsyYsSILsudnZ3viBRlQ72Mrge8UpPc729bW6Vj4fDhw4eQWD5tHAdJkiRJkiQlwTOoqqvm5uZni8XiFbFz9Gb69OkHtLe3DwFYuHDh3Oeffz617p+7Eo5eAjwN/CtilmqmADuW2quBv0XM0sPw4cNH7LjjjvuWlwuFwj2FQmFNzEy9aWpqeqxer1UsFi9vampK9sDllClT9igUCuMAli1b9tSiRYueiBqop22BaaX2KuD+eFGqGk3XszJ3Au2RstSyP5V9k0eAJRGz9LDddtvtOnr06C0Bxo8f/3ShUEjt93e9YrG4uF6v1dnZeW9TU1Oy2/bm5uaxu++++17l5blz597e2dlZjJmpigOpdHF/CFgWMUs1u1O5xOgp4Il4UXpatWrVtg8//PA0gGHDhr2w88473xc5Uq/WrVvnWDK9KLS2tq5fQQuFwi4zZ858NGYgSZIkSVI+zJo1a3qxWHykvJzskXJJkiRJUr5YoEqSJEmSkmCBKkmSJElKggWqJEmSJCkJFqiSJEmSpCRYoEqSJEmSkmCBKkmSJElKggWqJEmSJCkJFqiSJEmSpCRYoEqSJEmSkmCBKuXbZcBvgO1jB5E0oFqA/wOuBoZHziJp4B1MWOe/EjuItLGaYweQFNUbga2AL8QOImlAFYC3ltpDgbaIWSQNvG0J6/wfYweRNpZnUKV8ay/914NV0uDWkWm7vkuDX3mdHxI1hbQJLFClfHMDJuVDe6bt+i4Nfh6AVsOyQJXyzQJVyodi6QbusEp54PZdDcsCVco3j7BK+VFe391hlQa/coHq9l0NxwJVyjePsEr54fou5YcHpNSwLFClfPMMqpQfru9SfnhASg3LAlXKNzdgUn64vkv54QEpNSwLVCnf3GGV8sMdVik/3L6rYVmgSvnmDquUH+6wSvnh9l0NywJVyjd3WKX8cFRPKT/cvqthWaBK+eYRVik/HNVTyg8PSKlhWaBK+eYRVik/XN+l/PCAlBqWBaqUb+6wSvlhjwkpP9y+q2FZoEr55g6rlB/usEr54fZdDcsCVco3d1il/PCaNCk/3L6rYVmgSvnmDquUH16TJuWH23c1LAtUKd/cYZXywzMqUn64fVfDskCV8s0dVik/vCZNyg+372pYFqhSvrnDKuWHO6xSfrh9V8OyQJXyzR1WKT/cYZXyoyPTdhuvhmKBKuWbgyhI+eEBKSk/2jNt13k1FAtUKd8cREHKDw9ISfmRPYPqOq+GYoEq5ZtnVKT88ICUlB928VXDskCV8s1r0qT88ICUlB928VXDskCV8s0dVik/PCAl5YddfNWwLFClfLNAlfLD9V3Kj06gWGq7zquhWKBK+eYZFSk/HCRJyhfXeTUkC1Qp3zyjIuWHgyRJ+eI6r4ZkgSrlmxsvKT88ICXli+u8GpIFqpRvtbr/FOodRNKAq9Wl3/VdGpxc59WQLFClfLkGOA0YWlrufnR1BPB5YF6dc0nqf3cA76Syre++vo8DLgTurHMuSf3vZcBfgL0z93Vf53cFbgc+V8dc0kbzomkpXx4GzgPOAO6h8hvwCuBGwgZuPPDbKOkk9afFwPcJ6/xtwNTS/UcCrwd2ASaUHpfU2B4E9gRmA88BN1E5GH0m8HJgIjAKODlGQGlDWaBK+fId4H3AFOBNmfv3y7SXlp4nqbGdBxwMbAOcmLn/8Ey7XMRKamztwM8JxedEwgGosndl2vMIxayULLv4SvkyH3iqj+esBv5chyySBtZfgBf6eM5zwON1yCJp4J0PLOnl8Q7gR3XKIm0yC1Qpf84HVtZ4rAhcS5jgW1Lju5zKQCndrQUuq2MWSQPrEUIvqFqeA35YnyjSprNAlfLnl4SzpNUsAb5XxyySBtbF1N5hXQb8pI5ZJA28i6i9jX8GeKJ+UaRNY4Eq5U8bcDPhbGl3LwL31zeOpAH0NLW78D5JuAZV0uBxBbCiyv1rgUvrnEXaJBaoUj5dQOjqk9UB/CxCFkkD6zx67rC+CHw3QhZJA2sVYZT+7uwxoYZhgSrl0xx67rA+h9ejSYPR1YSCNGslobu/pMHnPHoehH68yn1SkixQpfxqpevgKc8Bj0XKImngtAPX0bVb/wOEMy2SBp8/0fU61DacPk4NxAJVyq/LqJxV6QR+EDGLpIF1AZX1fS1hIBVJg1ORcC1q+aBUG3BVvDjSxrFAlfLrOeChUns5zo0mDWYPAs+W2iuBGyJmkTTwLgaeL7Xvo+85kaVkWKBK+fYlQje/eTiapzTYfRNYB/yRcBZV0uD1FPAvQs+Jb0TOIm0UC1Qp324iXJ92cewgkgbcjwhd/rwWTcqH8wiX8NwUO4i0MZpjB5AUVQdwInB77CCSBtxy4K3An2MHkVQXVxF6Sa2LHUTaGBaokm6OHUBS3fw2dgBJdbMW+F3sENLGsouvJEmSJCkJFqiSJEmSpCRYoEqSJEmSkmCBKkmSJElKggWqJEmSJCkJFqiSJEmSpCQ4zYzq6uyzz26aMmXK8Ng5enPDDTeMaG9vLwA8+uijax555JGO2Jm6GUZl3V1HevObNRMyQpggvC1ilh4mT55c2G+//UaUlw899NDVY8eOLcbM1Jvx48evOfHEE+vyHTz//PNHjBo1qlCP19oUt956a8vKlSuHACxcuHDdnDlz/O5vnAIwIrP8YqwgvRhByAmwhjBXczJe85rXDBszZkwzwJQpU9bNmDEjte/gesOGDev8wAc+UJfvYGtr61BgaD1ea1MsWrSoac6cOev3Pa655prUv/tthN+QlLQAQ0rt5PY9dt555+bddtttGMDw4cM7jzjiiNR+f7s45ZRTUvwOJsMCVXU1efLkVwB3xs7Rm+uvv562tqR/17QZFi9ezDXXXLN++aCDDoqYpm9Lly79IPC/9XitUaNGvUBlByk59957Lw8//HDsGMqxW265ZX370EMPZcaMGfHC9GHt2rX/Anaux2sVCoUvFovFM+vxWpti5cqVXX73NfjsuOOOvPGNb4wdY4O1trZOOuWUU5bEzpEqu/hKkiRJkpJggSpJkiRJSoJdfBVVoVA4ksSugyoWi7dR6ua46667/mTu3LkXR47U3dXAxFL7ceDdEbNU8yXg8FK7HXh1xCw97LjjjgfMmzfvm+XlhQsXnjxhwoSHYmbqrlgs/gbYKnKMjxcKhbsiZ+hi7dq1vwO2ANh5550feuyxx06OHKm7bwIHlNptVNaDVBwNfDGz/BZgUaQstaz//QV+Dnw7YpYepk6detWCBQsmAaxYseKJQqHwrtiZsorF4nHApyPHuK9QKHwkcoYunnzyybcCHysvDx8+/LC2trY1ESNV85dM+1Lgh5Fy1LL+9xeYC3woYpYeJk2a9HXgIIClS5eu2XLLLQ+LHKmLYrH4UtL7TJNlgaqo2tra7jrttNNWxM6R1dLSsr49adKkhXPnzr09Ypxq1mbaLwKp5Xsu0+4ksXw77LDD2Hnz5q1fnjt37j8uuuiipK6Lbm1tXdv3swbcQzNnzkzqs5s2bVp7uT169OhVJPbdApZl2sl994Eduy3/jXCQK1WLSOw9HDly5Pp1s729/cXU1pFLLrlk70Ih+mXkK1J7Xw4//PBXZJePPPLIOxMdKKnsSRL77tN1UKTkfn9HjBix/vd33bp1nal9B2fNmvVisZjseIzJsYuvJEmSJCkJFqiSJEmSpCRYoEqSJEmSkmCBKkmSJElKgoMkabAYDuwNzADGlu57ArgyViBJkiRJG8cCNdgD2AuYAiwH/gncQZgiY1PtVfq7k4AisBi4E5i/WUmVtQvwKUJRuicwtNvjv8cCVZIkSRuuAOxM5YTHEuDf8eLkT3918b2UUIQVgUs28N98NvNvbu6nHL15PvN6x5TuezVwN/AA8DPgfOAy4E/AU8DGzrE3Cvg8sAD4e+lvXkSYx+3/CF/uuwhz0dXSAryQyXpAL8/dgjDsd/m5LxLOJNZyeOa5zwFD+vofStyewEnAy+lZnEqSJEl9mQy8HTgP+COhZngUuKd0OzNetHzK8xnUjwLfonaRthUwC9ieDfti7gFcC0zr43mvBG4Avg6cQSgWs9YQJio/qrR8OPDXGn/r1XT9DEcABwJ/qPH87KTxtwAdfWRtBJ2EM95zSrfncCJkSZIkbZh3A9+IHUIVeS1Q3wB8mHAG+U7gKkLX26HAq4D3Eoo9gM8BNxHOqtayB/BnYHxpuRO4Grix9HebgN2A95eeC/AZwpnSL1X5e7OpFKhHAF+p8bqHV7nvCDasQJ1d4zmN5PfAOGBl5r7pkbJIkiSpsS0mnPBYBZwYOUtu5bVA/SjhTOVJwI+7PXY58F3gVkLBWQA+Te0CdQThOsdycfoEcDxwb7fnXQdcQDhC87HSfV8knHX9W7fnZgvMVwEjCd13uysXnKsJZ4KHUb1ohVDIzajxGo1qeewAkiRJami3AMcRCtMFpfuOwgI1mjxPM3MaPYvTsn8A52SWjyEUidWcTOWs6ErCF7p7cVrWAXyCUKxCKCo/U+V59wJLS+0W4KAqz9ka2L3Uvo1wJhhCETquyvNfTaU781PA3BoZJUmSpLyYQ+j5uKCvJ6o+8lqgziUM7NSbn2bazYQpTLorAKdnls8nXA/ZmyLwhczym+lZ/HYQjuaUHVHl7xxWen0IXV3LXXaHAK+p8vzB1r1XkiRJ0iCT1wL11/QcnKi7Z4GFmeWpVZ6zC7BjZvlHG/j692b+9jDCwEndZbvgVuu2273gnF3jsWr3DYbuvZIkSZIGmbwWqA9s4POey7THVnk82/V2GfCvjcgwL9Peqcrj2YLz5cCEbo+XC86lhIL3TsIF3dnHyqYQBmmq9rclSZIkKQl5LVA3dHCddZl2tXk2s2dVxxKKxXYqc432djs482+7F58QuiE/VWo3Aa/NPLYTYfobCPM1dZaylgdy2g3YNvP8w6l0B34E+9hLkiRJSlBeC9TOfvo72cJyCGEk31rzqvZmRI37a3XbzbZ/X+P5h9V4vmdPJUmSJCUpr9PM9JeOTPspao8K3Je/1Lj/D4Q5WaF2gTq7RvsIKtfEZovVlK8/3RUYVeOxIj2n45EkSZI0iAxEgVro+ykADB+A1663JZn2WuCMfv772YJzOvASQvfccnff+XQdNfjvhMGdJlEpSl8KbFdqdxK6BKfqf4EDajzWgQdUJEmSpEGtv7r4vphp1+qu2t02/fTaMT2WaW8PTOznv7+AcM1o2eHAywgFKPTsrlukcoZ0KuGMZPZsa3Z+VUmSJElKSn+dkcoOOrRdzWd1dXDfT0neHwlFYYFQ7B8HfL+fX+MPhOlsIHTbzV73Wu160tnA20rtw+k6J2rK3XsB3gK0xA4hSZIkKY7+KlAfyrRfQbiO8IVenn8E4exeo3sWuBk4qrT8WeCndD2jvLlmA/9Zah9GpUAtUrtALTuSrgcCUh8g6am+nyJJkiRpsOqvLr5/oTIy7ijgo708d2tgVj+9bgrOIRSLADsSBiba0OtrW4BX9/Gc8jQyEOYzPbLUfhB4usrz5wGPl9pvALYstdcCt21gLkmSJEmqu/4qUJ+k63Qn/wOcBgzL3DeU0IXzr8AOVC+uGtHtwNcyy8cT/h/fRPUz1EMJAwGdBzwBfKGPv7+UcO1oWflv9nY2tPxY9vXvoPez2pIkSZIUVX+OivpJYA6hKG0GLiQUqo+WlncExpSeewPhbN7/9OPrx3QmYYCkk0rLLwN+SygIHyQUmcMIgxvtQtfC/cEN+Pt/AGZ0u6+vAvWkbvelfv3ppno3MDKzPDnT3haY2e359+B0NZIkSVKS+rNAfQA4Afg5lbksx9CzsPopcArwX/342rF1ACcTzlL+D6ErLoT3Yb9e/t0K4NYN+PuzgU9nltv7+HfZwZuyf2MwOo/Qbbya3YDWbvedhQWqJEmSggKhd2dWdt9yLOFEW9YzwKqBDJVn/T2v5LWEM4SfAI4BphHOFi4C7gR+AFxfeu4dVLrGPsbA+xaVa0PnbeC/+SFhECSA+zbg+T8gFOBvI1wruh/hCz4GWEM4k/pI6W/NLt1Wb8DfvY2u3YifIRS3tSwGPgeMKy0Xgbs24HUkSZKkPGkB/tXL4yeWblnvA64YsEQ5198FKoSRWD9ZuvXmltKtXs7ehH/z7U34N23A5aVbf3kBOGMj/825/fj6KduVjbuWekMOCEiSJEmKYCAKVKmelvf9FEmSJKmqdfQ8Q9oXeyYOIAtUSZIkSXnVAfwidghV9Nc0M5IkSZIkbRYLVEmSJElSElLs4rsvG98PvJbzCSPeSpIkSZISl2KBuhfw3/30t67AAlWSJEmSGkKKBeoaYFk//a2Ofvo7kiRJkqQBlmKB+vPSTZIkSZKUIykWqMqR4cOH39za2prUme7TTz+9UG7ffffd7wIOjhinmomZ9g7A7bGC1DA90x5KYvn+9Kc/bZFdPuSQQy494YQTVsXKU8NWsQMAF7S2tiY1z/AFF1yw/rObO3fu7iT23QJ2zbSHk16+id2Wf0notZSSQqb9duCAWEGqmT9//qRye+zYsTu0tram9hlPjh0A2Ce19+XBBx+cMnv27PXL11577R+AzniJ+nQy8LrYIboZl2nvSmK/b4sWLVr/+zthwoSW1L6DxWJxZOwMjcQCVVEVi8X9YmfoTVtb2zbANrFz9GIk8KrYIXpRILF869at67I8dOjQvSJFSd3usQN0VyhUape2trbRJPbd6qaJtPNBGJQwZVNKt2S0tbWtbzc3N48g/c84hrEk9r4MHTq0y3KxWNw/UpQN9ZLSLVXJ/f5mt+1Dhw5thN9f9cJpZiRJkiRJSfAMqupq2LBhCzo6Os6KnaM3++yzz8Hr1q1rBpg/f/4Dzz777JLYmbrZAyh3M3sK+GfELNVsQ6Wb74vAXRGz9DBq1KiRu+666/oz901NTXcUCoXUujmuVywW/1bHlzurkD1NmZhp06btPXLkyAkAzz333PwnnnhiXuxM3bwE2KnUXgnMiZilmjHAjMzybUB7pCy1HES4NADgQeDZiFl6mD59+u5jxozZCmCrrbZ6qlAopPb7u16xWFxar9fq6OiYPWTIkHV9PzOOoUOHjpsxY8Y+5eV777331s7OzmLMTFUcSuXE0d+Bun1+G2hvYEKpPR9I6vd3xYoVL5kzZ85OAC0tLav22muve2Jn6s2qVateiJ0hZYXW1tb1K2ihUNhl5syZj8YMJEmSJEnKh1mzZk0vFouPlJft4itJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSoIFqiRJkiQpCRaokiRJkqQkWKBKkiRJkpJggSpJkiRJSoIFqiRJkiQpCRaokiRJkqQkNGcXisXitFmzZsXKIkmSJEnKkWKxOC273Nzt8RuLxWL90kiSJEmSVGIXX0mSJElSEixQJUmSJElJ+P/lhrmotJ793gAAAABJRU5ErkJggg=="
+ }
+ },
+ "cell_type": "markdown",
+ "id": "756b8d01",
+ "metadata": {},
+ "source": [
+ "\n",
+ "![](\"attachment:fig.png\")
\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3f0e49e4",
+ "metadata": {},
+ "source": [
+ "### Communication\n",
+ "\n",
+ "Once the working arrays have been created, we can start with the iterations of the Jacobi method. Remember that this is implemented in parallel by updating the ghost cells with data from neighboring processes and then performing the sequential Jacobi update. See figure:"
+ ]
+ },
+ {
+ "attachments": {
+ "fig15.png": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6gAAAFACAYAAACxyVHuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15fFT19f/x12cmIQFkE1wANxBFrTu4b1itimQC2mKrtfrtRtXazbbWrmJba+1m7S5fv9Wqrbb5VWBugFatpe4buGvdFVcU2cISSOZ+fn+cO2QyTCAsmXszeT8fj/vIvXPv3Dm5M3PnnvvZ3DXXXOMRERERERERiVkq7gBEREREREREQAmqiIiIiIiIJERV0fLJzrlX4whEREREREREehbv/W7AP/PL7RJU59yrU6ZMeb7cQYmIiIiIiEjPM23aNLxv6xZJVXxFREREREQkEZSgioiIiIiISCIoQRUREREREZFEUIIqIiIiIiIiiaAEVURERERERBJBCaqIiIiIiIgkghJUERERERERSQQlqCIiIiIiIpIISlBFREREREQkEZSgioiIiIiISCIoQRUREREREZFEUIIqIiIiIiIiiaAEVURERERERBKhKu4ARDbGe+8aGxsvizuODbnuuusOWLhw4SCAhQsXvvbiiy++EndMRXYBRkbzy4H5McZSSn/g4ILlu4FcTLGUdPjhhx+dTqerAE455ZSnDjzwwEVxx9QR59yDdXV1s8rxWtls9nPOueHleK3NMXv27BFPPPHErgArV65c8thjjz0ed0xFBgEHFCzPjSmODRlXMP84sCSmODpyAHYcARYAL8cYy3pGjBixy7Bhw0YCbLvttsunTJmStPNvO+l0+vJTTz11TVe/ThAEewFndfXrbK7m5ub0L3/5y2Pyy88+++z8xYsXL48zphIOxn4/wT73C2KMpZQRwK7R/FLgsRhjWc/gwYMH7rXXXgfmly+55JK5MYazUWEY3jBx4sQX446jHJSgSuJddtllbsyYMd+NO44NeeGFF3jyySfzi8dsaNuEmBB3ABuRuGP4wAMPrJsfN25c4uIr5L3/LVCWBBX4DDC2TK+1yd577z3uvffewofq44qlkxL92SL58SXOK6+8wiuv2D3LUaNGQcLPv9XV1T8HujxB9d6Pds4l+re96NyR9M9+0uMDyMQdQKH333+/3XschuExqVRyK5em0+l7gR6RoCb3XRAREREREZEeRSWo0h09jlUVSYwwDI8i+j717j1w5erVPBdzSEVW7AGt/Ww+HUK/RFWzgTVDYPUubcsDnwRaYgunBOeWHey9B2DNmjULgKRV494XGBxzDK+QsCpma9euPYioClxNTZ/cmjW9ElbFt3lHaB7WtjwwgdU/lxZUv+/7JlQvjC+WUpYdAD5t871WQp9EnX/79HGjVq1a0h/Ae58D7ok5pGL9aN/EIg4h1rQjMcIw7AUckV+uqdl2wZo1YcKadiw7EHxU2FTTBL1fiDeeYk17Qm4bm0/noF+izr+1tentm5vf3ym/7L2/C/AxhlTKcXEHEAclqNLteO8vqq+vvzPuOAqNGjVqCTAQYOTII196+ulZ58ccUpHBv4bFh9p87RpYkrD4jq+Hud9uW/7zxXDq4vjiWV91de8H165tBuCpp576y89+9rNvxhxSO42NjTO89xNjDuN/M5nMFTHH0M6RRx55P3A4wHbbjVj1xhtPJeyzP+YTMP/CtuWkfTcB3INt80dOh9tujC+WUqpvh9aoHd6Or8BriTqGu+/+4auffPLWwwFaWlqaM5nMuJhDaiebzR7inHso5jASd1zOPPPMHYB38sujR3/xxieeuDQbY0glVM2FXG+b3+m/8OKFG9y87AZNg6VRG/u+q5N2fttrr0vOfOyxK7+cX/7zn//8oYaGhrVxxlQsCIIcPbDGa4/7h0VERERERCSZlKCKiIiIiIhIIihBFRERERERkURQgioiIiIiIiKJoARVREREREREEkEJqoiIiIiIiCSChpmRnqwa2A/YPVp+Bng6vnBERERERHo2JajS0+wBXASMAfYHagrWfR+4NI6gRERERKTbGhlNAK8Dz8UYS7enBFV6mn2B8wqWffTXxRCLiIiIiHQ/w4AvYAUeY4FBBeuuBr4cR1CVQgmq9DSLgZuBedE0H3iV9icWEREREZGO7AVcEncQlUoJqvQ0/4kmEREREZHNsRKYQ1uBxzzgLmC3GGOqGEpQRUREREREOu9B4NS4g6hUGmZGREREREREEkEJqoiIiIiIiCSCElQRERERERFJBCWoIiIiIiIikghKUEVERERERCQRlKBKpZiHjXFaPP0tzqBERERERKTzNMxM5doD+EfB8geA5o08pwZ4pmB5PPD8Vo6rqwwABpV4fJtyByIiIiIiIptHCWrl6gWMLFh2nXiOK3pOr60aUdcaQ+kaAS3lDkRERERERDaPElSpFMviDkBERERERLaM2qCKiIiIiIhIIqgEVXqig2h/cyYd/R2GVRXOewd4s1xBiYiIiIj0dEpQpSe6B+hT4vHPRFPelcAlZYlIRERERLqTXWkr5IC2vGoA7ft0WQ4sKldQlUAJqvRErwC9O7Hd4q4ORERERES6pQeAHUs8/j/RlPcH4PwyxFMxlKBKT7Rv3AGIiIiISLe2DBuicWNWdXUglUYJqoiIiIiIyKbZK+4AKpV68RUREREREZFEUIIqIiIiIiIiiaAEVQp1ph69iIiIiIhIl1CCWrlWFy13ptfaUj2RiYiIiIiIlIUS1Mq1rGh5eCeec0RXBCIiIiIiItIZSlAr1/u0HxT46I1s74DPdV04IiIiIiIiG6YEtbLdVzD/eaB6A9teBBzeteGIiIiIiIh0TAlqZftTwfwHgFuAIUXbDAGuBn4KLC1TXCIiIiIiIuupijsA6VIzgLuAY6Pl04EJwCNYG9XtgQOxz0ELcBYwu/xhbhrn3KFBENTGHUehr33ta+u+S8uXv90PPnZknPGsb+3AtvlcKnnxvT6q/fLvDoEbmuKJpbQwDNfNb7/99iODIDg1xnDW473fIe4YgNFJOy5XXXXVus9+c3NTOnmf/UW7tl9OWnzFXt81eTH6gmuZVYk7/y5b9vqg/HwqlUon7TsC7BF3AEDijsvChQsH3nzzzeuWFy26f1TSPlvgCwqaVg5IXnxr+7fNtybu/Lto0ZIRhcuTJ08+5ZxzzmmNKx5powS1soXAGcA/gQOix2qAo4q2WwKcC/y7fKFtkSviDqBYr1691s2//vqjI+DRq2IMZyOaa+CvCY4PYNb3446gWGvBT9bOO+98BvbdkvbOjabE2HbbbdfNL1q0oA8sSPhnP+nfzf9mbEqqRbsm7RguWNA2X1VVVQvMii2Y5KohYcdlwIAB7ZbfeuufHwU+Gk80nfHOnkn77Le3qnfS4nvjjfbLNTU1M+OJRIopQa18C4HDsDaoZwIHY1W7Q2AB8Dfgt9F8NTCt4LmLyxqpiIiIiIj0aEpQe4Y1wC+iKQUMBJqwar2FWkhgT76XXnqpb2xsXBN3HBuSTqerq6urUwC5XC4XhmHSqoikafu+h6z/3sfNAb0KlhP3fldXV/fC4gQ7fuEGNo+V975s728qlVrjvU/c+5WXSqWqqqur0wDe+7C1tTVpn/0U7TuwS+KxrCmYT+Jnv5q2PjVyQKLOv6lUKp1Op6sAqqqqknj+baelpcWX6aVyJPPzvk51dfW6z35ra+ta7325jk1nFX72W7FjmiSJvvZwzqWqqqqqC5YT/Xkkee9vl3HXXHPNui+bc270lClTno8zIBEREREREekZpk2btqf3/rn8snrxFRERERERkURQgioiIiIiIiKJoARVRCqZ2/gmIiLr6JwhIhIzJagiUomqgG8Ac9B5TrrGsLgDkK3uKOBRYP+4A5GKNAjoHXcQIt2BLtxEpBLtBnwXOBn4WryhSAUaDTwP3AD0jTkW2XouwcYMvwmojTkWqSwO+1w9jg39JyIboARVRCrRi8AXo/kfAofHGItUlhrgZiwx3QlYHW84shV9Cngb2A/4acyxSGX5CnAqsDM6Z4hslBJUEalUfwRuwcaJuwVVyZSt4yfAQcBC4CySNyaobL73gHOw9/TzwP/EGo1UijHAFdH8F4EnYoxFpFtQgioilew84FlgV2A2MCDecKSbm4JdYHrgM8A78YYjXeAO4AdYlcxpwPh4w5FubjgwHegF/B3433jDEekelKCKSCVbhl1gvoW1LbsVq6IpsqkmAr+L5r8LNMYYi3Sty4DrsNoXfwMOiTcc6aa2BW7DqvU+AXw63nBEug8lqCJS6V7DOktaCnwQK0ntF2tE0t0cg7U7TQN/AC6PNxzpYh74LDAT2AZLMo6ONSLpbvpin599gAVY+9NlsUYk0o0oQRWRnuApYBKwAktSb8PubotszAnYTY3eWKnpF+INR8okB5wN3A0MBP6JqvtK5wyi7abGMiADvBlrRCLdjBJUEekp/oMlp+9jvfreBewea0SSdGdiY+lugyWnZwCtsUYk5bQCq33RCPTBSsQ+GWtEknTDsN+WI4FFwIdQp0gim0wJqoj0JA9j1TVfBz4AzAc+EmtEkkQO+DY2bmF19Pd0NDxET7Qaq32Rb5P6R2z82z5xBiWJNBa4H9gXeAM4FvvNEZFNpARVRHqaZ4EjgHuB/lgnKFdivSyKbAsE2Pi5KeBq4FygJc6gJFY5rIOb70TznwDuAfaIMyhJlPOwz8QuwH+x6r3PxhqRSDemBFVEeqI3gXFYYgpwMVaaenhcAUkiHAM8AkzASs4+CXwZjXUq1nHS5Vib5HewsXAfB6ZiJavSM20L/Bn4PdZD/N+Bw7DO+URkMylBFZGeqhW4BPgwsBCr8nsv8Es0XmpPMwArKZ0LjABexG5WXB9fSJJQ/8Gqct6GdZx1KfBA9Jj0LBmsfelZtP2eTAaWxxmUSCVQgioiPd10YC9gGtb28EvAy8A3gNoY45Ku57COkJ4Bvhgt/wEYgzo2kY69iXWedAbWEc7BwENYc4FRMcYl5TEa6zgrCwwHngSOwmrk+BjjEqkYSlBFRGyM1M8BJwFPY9W2foy1Ifo0VnVLKsuJWHXev2A9b74UPXY+KgGRzmnAal78CUtMJmPnj98Au8UXlnSR7bCaFk9hzQBasKT0EOwGhYhsJUpQRUTa3AHsj5WMvIpdZF6LDbQ+FY2d2t05rOTrTuB2rORrGfAtrOfNO+MLTbqpd4H/AfbDEtZewOexGx4B1h5RurddgJ8Dr2A1LaqAGdh7fgmwJr7QRCqTElQRkfZC7EJzH+Cr2JA022NtzV7DEtYjYotONkcN1vPqo8A/gOOxi8qrsLFwrwCaY4tOKsEz2I2tcVj71BRQh7VPvRtLYvvGFJtsnoOAG7E26Rdh7999WA+9pwHPxReaSGVTgioiUtpq4BdYAnMO1mPnNliV3/uwqnwXY53qSDJ9AHsP38DGrjwAaMIS0z2xi873Y4tOKtF/sFL6A7Hxc1uwhOY64C2srfsHgXRcAcoGDcCaezyE9ex+NtZL8x3AKVhb03tji06kh1CCKiKyYS3YXfQDsWFIrgdWYiWsV2IdKj0CfDN6TOK1G/A14EGsrdhXgCFYSfglWHW9i7Bq2yJd5XGs1H5X7HP3PDbu8meBfwFvY8nqyagztrj1wXpz/wt2E+EPWLvSNdgQMgcDHwL+GVeAIj1NVdwBiIh0I/dE0xex6nwfw6r0jYmmH2FtV+dE039QhztdzWE3D8Zj1e4Kh/tYi7UDvBardqnxTKXc3sZuZP0Eu8H1cexzuh2WrH4WWIW1f56NtY1+MZZIe5btsZsDGeBU2le/fgo7Z9yEaliIxEIJqojIpmsC/i+ahgCTsDvw47ASvPOjKYcNV3I3bcnt22WPtvLshF3sn4glpkML1uWw8Uz/H/B34L1yBydSggfuiqYLsHPFR7B2qjtFf+uibd+m/TnjCexzLZuvD9Zh1TjsnDGG9rUIX8bOGQ1YjRgRiZESVBGRLbMIu9t+LXYRNA67ADoJa+d4UDR9Mdr+Jeyi827gYaxzldayRty9OKzq9NHRdAxWbbLQCqyN2Gysd00lpZJkOaya77+wG1n7Y+0bxwOHYzdczogmsFoY92FtH+/D2kYuLW/I3c4QrL3oMdHfMVhb0jwPzMM6TZsezYtIQihBFRHZelZhSdLsaHkH4FDsAulorF3T7tF0brRNC/ACdoGUn+ZH++qJdqAtqT8cO3aDi7ZpxY7T3dgF5t1YdV6R7uiJaPoJdl12AHa+OArrUGkwlsCeUvCct2l/zniEnls7owYb8uVgrIr/UcDe2M2tQguwc8UdWBOMhWWMUUQ2gRJUEZGusxBrAxlEy/2AI2lLWA8CBmIlhPtgnaqAJWDPYsOiPIl1uPIElXdBNYy29rv7YL3ulupoaiXwGFbyfC9WTXJZmWIUKaf8zZd5wNVYNdR9gWOx88ahwEislLWwWjBYBz+PYueK/DnjBSqrhsY2WJvz/PliDJaU1pTY9mXsfHEPlpS+XKYYRWQLKUEVESmfJqwnyMLeIEdid/4PKvi7A1YisF/R89+l7cIzPz2NlcImWW/sYvIA7H/aP5rftsS2OazH00exUqF7ovlKusgW6ayQtu/6b6LHBtJWyyB/3hiN3fAZBkwoeH4zdo7I7+NJ7GZP0jv/cdgQXvsXTAdg58tSI1C8i50n5gP3Y4np4rJEKiJbnRJUEZF4vUxbBx15w7CLzvxF2f7AHljPkx+Kpry12AXoY9H0KJbExtV78DDsovkA2sdeatzHfOzzsbjzsa8sS6Qi3dNS4N/RlNcH+64dSNv3bl9saJt8LYVCC2h/zngM64E8DvnYD8Di3x+7kdWvg+0X0Ha+yJ873uj6MEWkXJSgiogkz1vR1FjwWL4UMl+asB92MbctbSUpeR5Leh/FBpy/D6sy2LwVY0xhnUAdGE0HRX+372D7hbRVPXwymn8GtR0V2RpWAQ9EU15hKWRhzYXdsfGAdwHqC7ZfgiWq87ESyPuBd7ZynENoO1/lzx17UvoGVnHpb/78kfTSXxHZQkpQRUS6h9VYldfiIRDy7TgL22TtTVtnTB+JtmvFLu7uxZLVu9i0EpPtgCOwdnBHYheYfUtsl28/+xjt28JVWvtZ6YSJEyfuAwxLpVJPTJ8+/d244ylWV1c3Lp1OV82cOfOOuGPpAvkbVS9jvVvnbYNVCc6fL8ZgNTYGAcdH01ejbd+mre33POyGV2dvKvWK9ps/bxyGDalTylvYOeNxKrf9rIh0khJUEZHuLV/aGhQ8Noi2UoojsIRyKOtX9XsF69XydqwTkcLSkpG09SR6NKV7xVwBPIeVhKoHYllPLpf7qnPuU7lcbjLtq7EngnNuRhiGA7AaAT7ueMpkBW3f1xuix6qxm1wHYh0xHYmVug4FJkcTWDv6B7Ahcu7AammE0boBWE/l+fPGUVjNj2LqgVhENkgJqohI5VkC3BlNebthF4z5hHV/rPrfCOAc7OL8Laz951DWb/8VYtXt8mMxPoiVcISIJFB9ff3F3vsrvfdTGxsbL4s7noRroa308k/RY/1oGybriGgaQPt28MuxG1t9geEl9vsubZ0W3Y+Vkq7okv9ARCqGElQRkZ7h1Wj6c7TcH/gkcCZWalJD+wvMfML6b+AvWFKqoV1kk4Rh+H3n3O9I6BAfuVxunHMuTc8pPd0UTVhJ6b+i5RQ2LutnsWrA22Hnkf4Fz1mB1aL4K3Ab8GK5ghWRyqEEVUSk53BY9buPA5Ow4Wzy1mIXky1Y9d5+WMJ6NnZRejNwI1bCItIps2fPfg14Le44OjJ79uzH4o6hGxgOnAWcgTURKKzq/wZWSrodsDPWvvVY4HDgH9g5o5Gt20GbiFQ4JagiIpVvD6wa79lYVd+8xdjF40xsbNb88C7VwHHAaVjbs2FYpylfxdqM/Qa4hW520Tlx4sR9wjC8ELuA3h7rDfQVYGZra+v1c+bMWVOwuauvr/+o9/6TWPvbKu/9K865W1pbW6cVbUtdXd2hzrmvOefu8t7/EbgEuwmwHfAS8LMgCGYATJgwYe90Ov1t7/3hQB/n3APe+28FQfDfwn3W19eP9t7/wHv/xLBhw658++23LwI+ilXBft17/5vGxsYbACZNmrRba2vrt5xzx2LVMB8Lw/A7s2bNmle4zwkTJuybSqW+BzwUBMHPio9RJpM5GLjEOXdvNpu9uuDxk4DPOOeyNTU1M1avXv0t51wmOo4LvPfXNzY2/o6iksj6+vrzvffHA78IgqCwh9n8+uO9958FxmLje74LPOW9/2tjY+P0/Hbjx4/fqaqq6jSsaulI7ObK+8DD3vtfNzY2PlS03997748EcM5NzmQyH8ivc841ZLPZhuj/uh7oEwTBR0vEvkMYhhc558YDO2LVWR/03l9d/HrR9lO99/uk0+lvhmHY13v/LSxRqwYe9d5f3tjYeH/x8xKqL9bB2iew0tL82KM5rLruDOy8UVgyPgr7zH8Uez/ro2kx8H/A77Hvm4jIBpUa7FhERLq/FDAemIN1ZPQdLDldClwLnIBd5J8L3Er7sUdbsA5QPo+VntQDf8MS0jHAddhYhJfTvhQ2sTKZzGfCMHwcOB/rROoJrGfkY4E/1NTUDM1vO3Xq1FQmk7nJe38zdpzecs696Jw7GPhVVVXVv8ePH9+/6CWGA5O998di1aK/iZU0rcba8N2ayWTOzGQyR6dSqQe99xnn3FKg1nt/GnB3fX39sMIdOucGA5Odc8e//fbbWex493bOLQfGOuf+VF9f/6UJEybsm8vlHnbOnYV1UJUGTkmlUnMnTJiwd9E+d4j2eWSp4+ScGxr9H4cVrRoVPX5Mc3Pzfc65r0f/3zLgYOfcb+rr668s3l8YhmOj5xX33urq6up+7b2/E6tmHnrvH8c+t6c756YVblxVVfV14FdYDYA1WHvoPsDZzrl7M5nMmYXbe+/3o63H2HxP1/mp8DhPoq0DoHUmTpy4n/f+CefcxVhy+iT2vTjLOXd/XV3decXP8d4fB0xubW093Xv/ADDeObcIS1AnOOf+PXHixJLHPUFGAD8FXgeuxz7/AP/BqvbuiN28uor1q22/CPwM6yhpH+BHWOn5tsDXsTbrM6Lni4h0SAmqiEhl6QWcB/wXmA2cgpUMNWJV9IZiF5p30rkhHFqwHoI/il2cfhlry7od8C3sAvQa2l/0J0p9ff2xwB+wJOhTQRDsFATBiUEQHNzU1DTEe3+uc64pv/28efMuxKo0vgUcEATB4dls9tjq6uqRzrl5wBHpdPqqDl7udMCnUqmRQRDsFwTBSOfcFwDnnPsJcJNz7g9NTU2Ds9ns2Nra2uHOuX8CQ7z3X+tgn8cBuzjnPhAEwd7ZbHa0c+6jAN77y1Kp1M3AzNra2u2CIDh46NChw7Eq2dukUqnvbPkRbOdTwJtVVVU7BkGwbxAEezrn6rFje1Fxkt2RTCbzJefchcC7zrnjgiDYq7Gx8UNBEOxTVVU11Dn37aKn3OW9PzEIgsFBEIwJgmBcEAS7Ouc+gSXKfyi8aRAEwdHOuSsBvPdXB0Gwe34qLBkuZcqUKdVhGDZgpcPXtra27hwEwQlBEOztnDsTCJ1zv45Km9fjnPuR9/47Y8aMGZTNZsc2NTUNw24K1YRheHlnjk8MDsWSxxeBr2E3cV4Avo0lreOw/2FRJ/f3bPTckVipd0P0+ERgLjZ0TWarRC4iFUcJqohIZeiFVeN9FqtKtwdWJXEaNt5hBrtI3JJqucuAq7HStNOwjpNqgCnYhe0vsGqaieK9vwwrVby0sbHxOgqqcs6dO7e5sbHxhunTp78PVnoalQ7ivT8vCIKn89veeuutbwMfA3LOuXM6SMZawzA8a+bMma/nH8hms78FXvXe7+ScW5DNZi+eO3duK0BDQ8Nq59yl0aYdla6lnHPnZrPZ5wr22YCNSTkASDc1NZ3X0NCwGmDatGkt6XT6WxvZ5+ZaWVVVdW7+eEWxzMJuYqS994dubAfjxo2rxUr0SaVSH89ms3cVrp8+ffr72Wy2XQlqEAR/b2xs/Bftq+H6bDZ7U1Slun9VVdUpm/9vtXnrrbcmYuOEPjd06NALCqtzZ7PZW7Aq7lVYIldKY2Nj4y+mTp0aAsydO7c1nU5/HfvuHc76wzXF6XDsvXsASx4dVnviDKxq+4+w2hKbKyzY3z7A77DjcBSQxcZjPnwL9i8iFUgJqohI9+awTo9ewoaHGIlVz7sQKy39HFaaujXlsNKWo4BjsNLY3sBXoji+hCWEsZs8efIALEbCMLxmY9s//PDDo6PqqEvGjh07q3h9Npt9ERsuoyqq0lls/qxZs4qrPnrsxgHe+1uLn+C9fyaa3aWDsF7JZrOPlHg8/7xsPuHNmzFjxqtYdd/hkydP3prvxdzp06e/W+LxJ6K/u21sB/379z8CGAy8OHPmzDs25cXHjRtXW19fP6q+vv7YiRMnnjhx4sQTnXPNAM65fTZlXx1xzp0Yzf512rRpLcXrwzC8KZo9oXhd5G/FD8yYMWMpVtug9vTTT99xa8S5hfbEalXcD9RhtSn+COxFW4lnbiu/5vNYs4FdgSux6u/HYDe6/oZ1siQiogRVRKQbOwgrgbgJa2+3AKuCuyfwWyxB6Wr3YBfqJwFPYe3NfomNk1qyCmQ5rVy5cgSWLL87a9asJRvbPpVK7RbNvpovASvhpejviOIV3vuOeqxd2tH6bDbbhCUDgzp4bskSLO/9MgDn3IZesxrrWXVr6aiTmyVRLBstQffe7x7NPt/ZFz355JO3zWQyf+zXr99i7/0L3vv/hGF4exiGtwNfiPZb3C54szjndov+lhwap0+fPvmhU7afPHlyqWO7wWOUy+U6ep/LoS8wFbuhMAGrwn8jVsvi02zCe7IF3sU6Edsd6zzJY+2An8LOX7o2FenhdBIQEel+emNVbR/BOo1ZgV3w7RE9HkfvurdjCfPnsHZqY7Ak9cdYkhSLdDpdCxB1SLRR3vvaaHZDY77m19UWr3DOrVfiVqQz7X6Lldync84DhGG4OfvcLJ34/zqzj/xx69R7Mnny5HSvXr3mYOP2PoGV0E9yzh0XhuFY59wP87vehyroNQAAIABJREFU0tgAvPc1AGEYLu9gkxVEpYvNzc2lPgNlez82UR3WrvRSrGp+I1bt9pzo8XJ7G/gM1qnSvdh4qlcBd2PJq4j0UEpQRUS6l0OA+cAXsXN4I1b6cSU2lmmcWrE2r/tgVfaqgG9gPYCOiiMg7/370d9dpk6d2pnfvHwpa3Gvs+s45/JVERdvYXhll0+evPcd3TTo8tK9/HuCVfXcqObm5uOwTnzmDx069JggCH4VBMHMbDZ7VzSMzlatiuqcy5cGl/wMrF27dhhWKp9ramrqVJIdsz7YjassVu3/BSxZzWBtx+M2H6vqOwVowtpNz8MSZxHpgZSgioh0Dw7rWOY+rJ3Ym8DJ2EXmlnRi0hXew3r9PQNL4o7ALjjryh3I2LFjX8JKdGvnz5+/0SrH0VAnIbDraaedNrh4/dSpU1Pe+zHR4qNbNdgycM69Gc121Nvu/l0dQxiGD+Zfq4Mqsu147/eMZu8q1Sa04P0ofjzfNnWTSvC9949FsyU/L977Q6L9Plnc9jeBDsWGyPkiVpX2p8B+wHrtq2Pmgf/FPn//wTr/+hPWQ3hsNTBEJB5KUEVEkq8v1mnJD7BSyQbgAOC2OIPqhHycd2PV96Zj1TPLZurUqWHUyyve+5+NHz++poPtUgBRO9V/ANUtLS3fLd5u/vz5n8E6M1rQ1NR0b9dF3jVaWlpex0raD5g0aVK7apTjx4/fyXu/3vieW1vUidSdQL/Vq1dfUWqbwtLuVCr1VjS7XidI0RBC40vtwzn3RvR3t02JL5fL3YIlTB8rHkd23Lhxtd77bwOEYfiXTdlvDD6KDekyEuug6QTgYmwc2aR6FfggNkRNDitV/QfWqZaI9BBVcQcgIiIbtAswEzgQa1t6Hlay0F28AZyIddr0GawDpb2BC7CSyi6XSqV+6L3PAMdVVVXdV19ff1XUc+5AYF/n3Ccfe+yx07CLY1Kp1NfDMBznnPtSJpPZJgzDP1VVVa0NwzDjvb8YS16+1A1Kz9YzZ86cNZlM5m/A2blc7h/19fWXee/fwkquvoKVxn+gq+Nwzl3gvX/QOXdhJpPZ1Xv/v1Hp7g7AIfPmzZtEVILZ0tJyf1VV1SrgpEwm81vv/U2pVKrZe39ylCy+iA0L005ra+u8dDqd895PrqurW+Wcy3dudW8QBPd0FNvs2bOfz2QyVwNfTqVSd9bV1X3POTfPOTfUe//NKK5ne/fu/Zute1S2mhRwOVa93mE9bp+DVZ/tDkJseJvHgb9gCeuDWI2RlzbwPBGpECpBFRFJrg9gF2YHAu8Ax9O9ktO8tcBnsQ6UWqK/f6ZMN0mz2WxTa2vrcd77LHCw9/5GrMrxv4CrvffDwzBcnd9+5syZzzjnTgJeBj6dSqXuCsPwAaxUZznw8SAIZpQj9i7yFayDrVHRsfgX1jnNv4nGJ+1q2Wz2uVQqdQzW/jDjnMti78ls4LLCbefMmfOe9/5srHOiC5xz93nv5wNXOOf+BPys1GvMnj37Ne/9Z4ElzrnPYh12/bhgGJkO1dbWfs0591NgsHNuGjDPe9+IDa10Z3V19Qn5cWcTphq4Ges0DSxR/TDdJzktNAtrc/8c1mnSPZQoRReRyqMSVBGRZNofG+B+O+AxrK3pG7FGtOWmAW8B/w/4GFa6czale7b9OZYQbpUeiefMmfMeMPHUU0/ds6qq6mjv/XbAIu/9SytWrLinuDQ0m83eO2XKlL0WLlx4XBiG+2Cd4rxSW1t7R0NDw4ri/VdVVf27paVlbHV19fvF6yLfDsPw5wVDlBQ7NN8rb16vXr2eWLVq1VjnXMneZFOp1C9aW1tvSqVSJYeZcc6Nz+Vy1Vhit04QBIsmT558+KpVq050zn3AOdecSqXunjlz5pOTJ08esGrVqrGpVKpdB1Ctra0NqVTqwTAMF3bwWn/J5XJ3pdPptwsfD8Pw+86532HJfjszZ858EhibyWTGOufGhmG4DbDQOfdUEATzC7dtbGycnslkRgDHASOcc4tzudzcWbNmvZzJZIaEYfhoVVXVeuOzNjY2XgdcN2nSpIEtLS07ATWFMeZyuXHOuTRWKr5OQ0NDDrj49NNPv6q1tfUEYFgYhstTqdSD2Wy2ZNvjXC73Oedcv1Qq9Wyp9cAnwzDs29TUVHL4mq2gF3ALcBpWjfd/ouXu7HngWKyX8P2xquEnAE+X2PYUrHfimWWLTkS6hLvmmmvWnZSdc6OnTJlSjjGwRER6uv2AHwITS6w7CGtfOgQrVTqJbthj7AZMAv6KXVDfCJxL+wThbKykeB+s9EREzJ3Ap4iqoxfohX2nJmHJ6WQgKGtkXWsIdsPuAKw2yZG0H292EFbVeyZ2fESkG5k2bdqe3vt1v/eq4isiEo/RwKlYclZoFFZaMAS4HystqKTkFKxN3Iexqr+fAAo7IzoWq266gk4OQyLSg4zBOh8rrAHngJuw5HQV1lt2JSWnYD1xfxDrOXtH7P8bEK3bBqumPgCNnypSEZSgiojEYzR2kfkz7MISrBSgEeuxMt8pyLJYout6jVhbVICpWKnp14G/Y8n5NsBucQQmklD9gNVYJ2PTsVJTsN69J2Mlp3VYSWMlWozd1Hsda5//VyxhfxTYF6uGPzy26ERkq1EbVBGReOSHrxgCXAvUY8NBjMaGhJhI9+zYpDO2BXoD/8Sq+H4CuAG7+O4TbZOiDL3JinQju2KlpX2xmhVPYN+bb0XrP4t1dFWJqrAezZcCX8Da1p6M1bjoXbBdbflDE5GtTQmqiEg8RhXMD8aq+qawsf9uBcZhJSSPAU+WO7gudj5W6vMuVurhsQvvPkXb7VnmuESSbFfakrHe2M2sH2LfnYejv2dg36eGOALsQrtgCXkr1nFaOnq8d9F2VViSulU6VxOReChBFRGJx/ZFy/kmF2lsGJC12HiAx5UzqDK5HLuI/DxWrbkju5QnHJFuYSRW9b2Qi/4eAvwRO2dcS+UlqC8Dh2G1LjZWjXdn4IUuj0hEuozaoIqIxGNjVdEWA4cDD5Uhljh8FytFXbqBbfqVKRaR7mBf2hLSUlZgQzldUJ5wyu5p4BisCUTYwTa1qHM1kW5PCaqISPltQ8fnX491AnIM8HjZIorHVVhHSYs6WN+Ltqp8Ij3dHhtYtxi4FLiwTLHE5RXsxt1LlB4/uR8woqwRichWpwRVRKT88p2dFAuxqmxHYGP69QR/A84C3utg/dAyxiKSZDt18Pj7wJeAq8sYS5zeAQ4FnsGaQhRKYeMni0g3pgRVRKT8Cjs7yWsB/ou1s3qz7BHF63ZseIyFRY/XoHaoInnFnYiB3dj5H2wc1J5kKXYj7xGs9+9Co8sfjohsTUpQRUTKb3fad3ayGuut9zCsNKQnegj4IPBGwWP9UXsyEbDq7r2KHnsXOB0bU7gnWgUcD8ylfa+9o0puLSLdhhJUEZHyO4i2Kr5rsAuso7BOTnqyZ7C2t29Fy6quJ2J2ov1NrXeADwH3xBNOYqwFJgA3Y0N0wfo9HYtIN6MEVUSk/PKlgi3YmKd10bzAq8DB2DAROWCvWKMRSYZdaWtv+S5wNDYuqFjHcp8CfgoswYZQVOdqIt2YElQRkfLbGVgG/BbrIKijIRN6qoVYJyj/RVV8RcC+B9tgN272x3qxlfa+iSWpQ1DnaiLdWlXcAYiI9EADgB9Hk5S2FEtSPxR3ICIJsAfwLFZyuizmWJLsCuBJdIxEujUlqCIi5XcYVpVVNmwVMDPuIEQS4NfAD2jfGZCU1lM7jRKpGEpQRUTK79W4AxCRbuWduAMQESkXJaiSeN5719jY+ELccWzIFVdcscMLL7xQA7B69erlK1asWBp3TEX6AwOj+bUk72KnBtihYPl1rOOLxBgyZMjOzjkHcP755793yCGHFI+9lyQ3ZjKZy8rxQtls9u/Ouf3L8Vqb469//evA2267bQBAS0tL85IlS4rHWo1bLe0/+6/FFcgGFLYDXkjySvF2wI4jWNXORJ1/+/btO6Bv374DAXbZZZc1l156adLOv+2sXr36oDPOOKPLexQPguAE4A9d/Tqba82aNe6CCy5YNw7z0qVL31m7du2aOGMqYUfs9xPsc5+0qs0DsSYtYOeNRJ1/a2pqagcMGLDu/Hvttde+Fv3MJ9UnM5lMj+i5WwmqJN5ll13mxowZs3vccWxIU1MT7733Xn5xcDQlVTU2DmeSjYw7gGKLFi1aN5/L5YbFGEpnbFfG19qFBI872NzczLvvvptfrAb6xRhOZyT2WEZ2ijuAjRgSTYmxcuVKVq5cCUD//v2rSfh7PGjQoLL0gOu938Y5l9hj4b0vPHeAdW6XZNtR3nP/pkrc+XfNmjXt3mPv/agkJ6jOub5xx1Au6sVXREREREREEkElqNId/cI591zcQRRqaWm5CugDsMMOe725cOF2N8QcUpF5H4NVI2y+Zi0c+vN44yn2ygHwxqlty4f9CnqtjC+e9VVVPfDN1lYbqnTBgpX39+8/eG68EbU3YsTSD/fundszzhiWLq35z5tv9rk/zhiKLVmy+hPAcIBBg4Y1L1ky8pcxh1TkxcPgnePblo9OYM/O91zSNr/bv2GnB+OLpZT7vgxhVMV30FvwgUSdf3fccelH33nnqREAuVzY8vTTgxJ1/u3fv2XozjuvODfeKFzL008PTNRxaW5e2hf4Qn55yJDD5yxaVPV4jCGVcN9FEPay+cGvwt63xBrOeuadDaujWhe1a2DsVfHG097226895N13Hzohv9za2vr56urqXJwxFXPO/R5IbrFuF1GCKt2O935WJpO5M+44Co0aNepKogR1yJCR7y9cOGtGzCEVGXxCW4JalYO7Ehbf8WH7BPV7c+DUxfHFs75Uqvc3wRLUV19d9fyNNx6RqGN40023HxV3grp4cc3Tl1xy1PQ4Yyh22GFrJhAlqH37DmpZsuTuRMUHY7Zpn6AmLT4AV5Cg7vE03JawGKsvaEtQ+y1O2jHcbrsPjytIUHNJ+4586lPP7BN3guo9iTsuO+10/7YUJKjDhp0yf9GiS7MxhlRC1Zfa5ge+l7TPPgwa35ag9mpJWnzDhl1SW5ig3nLLLdc2NDSsjTOmYkEQ/I4emKCqiq+IiIiIiIgkghJUERERERERSQQlqCIiIiIiIpIISlBFREREREQkEZSgioiIiIiISCIoQRUREREREZFE0DAzItAX2ANIA6uBZ+INR0REREQSLAWMBsYCY4C9sOvIZ4AvbeB50glKUKWnupC2k8re2EkF4Elg/7iCEhEREZFE6wMsBLYpsa5vmWOpSEpQpaf6ddwBiIiIiEi3k6ItOV0AzAP2wUpUZStQG1TpqZ4ArsNKUo8Abo43HBERERHpBtYApwI7ALsCpwOPxxpRhVEJamX6GtamEqABuKMTzzkXODKa/w/wly6IK0kOKFr+bCxRiIiIiEh30gLMiTuISqYEtTJlgGOj+efoXIJ6HPDJaL6Vyk9QRUREREQkYVTFV0RERERERBJBCaqIiIiIiIgkghJUERERERERSQQlqCIiIiIiIpIISlBFREREREQkEdSLr1SSbwGf7mDdPti4VSIiIiIiklBKUKWSbAuM7GCdK2cgIiIiIiKy6ZSgSiW5AvhdB+tUeioiIiIiknBKUKWSvB9NIiIiIiLSDamTJMlLxx2AiIiIiIj0bCpBrUytBfPVnXzOtl0RSIL1BXoVLNdEf9PAoILHc8DycgUlIiIiIol3AXBAwfLY6O9I4JqCxxcAl5crqEqhBLUyFSZUnU089+uKQBJsGnBWicf3ARYXLD8PjC5LRCIiIiLSHZwM1Jd4fAdgSsHyfJSgbjIlqJXp9YL5gzqx/eHArl0Ui4iIiIhIJbkBuK8T2y3s6kAqkRLUyvRIwfw4YGfaJ62FqoCfd3VACfTxaBIRERER2RR/jzuASqZOkipTI9AczVcDf8LaXBbbFmgAjgR8eUITEREREREpTSWolWkx8HvgK9Hy8cB/gb8ALwK1WMPuDwMDgeeAZ4DTyh6piIiIiIhIRAlq5foucBhWOgqwE3Bxie1eATLAN8sUl4iIiIiISEmq4lu5VgInAT8DVnew/lrgYOAFYBWwJJpWlSlGERERERGRdVSCWtlWAl8HvoeVpA4HHNZh0iO0H47mwmgSERERERGJhRLUnmE18K+4gxAREREREdkQVfEVERERERGRRFCCKiIiIiIiIomgBFVEREREREQSQW1Qpdtxzn0zCIJPxR1HoW984xt98vOvvz5/Zxh+WZzxrK9p97b5Nb2SF9/Kndovf/rrkFobTyyltba2rJvff/+Bx3z603fsEGM46+nTp2WvuGMYOnTliddff8fIuOMo1NDQd+f8/OLFb9Qm77O/fET75aTFV+yRD8HwUXFH0V6ud9v8op2Sdgxfe82vO15VVenq66+/I1HxVVfnBsQdA/heSTsuq1ev6HX++W3LL730fxNh2pj4Iiol7NU2/87IpH32YcUubfOrE3f+ffHF1G6Fy2efffb155xzThhTOB1xcQcQByWo0h2dGHcAxdLp9Lr55cvfGQScEl80G9OahrcSHB/AOx+MO4JiYcFP1sCB1SMHD25OVCKWBL175/bs3Tu3Z9xxFOrdu3rd/KpVy6phWcI/+0n/bi4ZbVNSrRoIqxJ1DJcX9JefSrn04MHNiYovCZwjlbTj0tzc2m555crX9wX2jSeazlg5GFYm6hi211KVtPPbihXrPXRmDGFICariKyIiIiIiIomgElRJvEsvvdQ3NjZeF3ccG7LLLrvsV1VVNQDg/ffff33BggWvxR1TkZ2BXaP5JuDxGGMppR9wQMHy/UAuplhKOuCAA45IpVJpgJqammeB92MOaUPuL9cLpVKprPf+yXK93qYaNGjQbgcddNBOAKtWrVr63HPPPRV3TEUG0r5U5p64AtmAowvmnwKWxhVIB/bFjiPAG8Cr8YWyvuHDh++0/fbb7waw/fbbrwAeizeiDVuyZElZmlek0+nXwjBM8m97+qCDDjoiv/Diiy8+3tTU1BRnQCUcgP1+AryGjXOfJLti1x8Ay4BE/VYMHDhwwIgRI/bLL6dSqSSef9cJw/CNuGMoF3fNNdf4dQvOjZ4yZcrzcQYkIiIiIiIiPcO0adP29N4/l19WFV8RERERERFJBCWoIiIiIiIikghKUEVERERERCQRlKCKiIiIiIhIIihBFRERERERkURQgioiIiIiIiKJoARVREREREREEkEJqoiIiIiIiCSCElQRERERERFJBCWoIiIiIiIikghKUEVERERERCQRlKCKyJb4CfAI8JG4AxGRLrMf9j2fEXcgItKtjMHOHQ1xByLdS1XcAYhIt7Yr9gO0Y9yBiEiX6Y19zwfHHYiIdCt9sHPHNnEHIt2LSlBFZEvkor/pWKMQka7UGv3VTW0R2RQ6d8hmUYIqIlsin6Dqx0ekculGlIhsDp07ZLMoQRWRLZG/O6ofH5HKpRtRIrI5dO6QzaIEVUS2hO6OilQ+3YgSkc2hc4dsFiWoIrIl1L5EpPLpRpSIbA6dO2SzKEEVkS2hHx+RyqcbUSKyOXTukM2iBFVEtoSq74hUPt2IEpHNoXOHbBYlqCKyJdQBgkjlUymIiGwOnTtksyhBFZEtobujIpVP33MR2Rw6d8hmUYIqIltCd0dFKl/+ItOh6wYR6TzVspLNoh8aEdkSujsqUvlaC+b1XReRzlI/FbJZlKCKyJbQj49I5csVzKskREQ6S7UvZLPoh0ZEtoSq74hUPpWgSrdXV1d3aDqd7h+G4X1BEKyKO54eovjcEcYViHQvupshIltCVXxFKp9KUKXbS6VSvwvD8Hbv/fC4Y+lBdO6QzaIPi4hsCXWSJFL5VIIqW0UmkxkCvAe8GgTBiK213/r6+tO99393zt2QzWbP3Vr7lS2mc4dsFl1UisiWUAmqSOULAY+1I9N3XbqlXC73CaBPGIYL4o6lByksQdW5QzpNCaqIbAklqCI9Qw67ZtB1g3RLs2bNejbuGHogVfGVzaIPi4hsCVXxFekZ8gmqbkZtwNSpU1OPPPLIZOfc2cD+QG/gHeAR59yN2Wz234Xbn3zyydv26tXrIqAOGA6sBh4GfhsEwZ3F+89kMl8DDvXeX55Op8MwDL8DHAHgvb+3qqrqOzNmzHgJoK6u7uPOufOAPYAVwP9ramqaOnfu3OaifX4GOMk595tcLvdeKpX6DnAUUAM84b3/SWNj479KxPJnwAVBcFapY5HJZP7mnFudr3KbyWTO8N5/3DkHsF0mk/lbweaLgiC4INpuiPf+NOfcKcBoYAdgJfCY935aY2Pj7MLXqa+vnxqG4Yecc3jvjyva791BEPw62u+PgZHV1dVfuvXWW98u3Mf48eP7V1dXf8l7PwnY2Tm3xnv/mHPuD9lsdlbx/1ZXV3eec+6DqVTqly0tLW+m0+mpwHHANs65/3rvfx4EwcxSx6WHyaHaF7IZ1EmSiGwJlaCK9Ay6GbURkydP7j1v3rzAOXcLcCqwCHjSOVcDnOO9/37h9pMmTdq9V69ejwLfBnYGnnDOvQ+cDvwrk8l8u8TLHAFMBiaFYfggcBLwNpYYfSyXy91dX1+/QyaT+YVz7iYs6X0r2v83+vfvf2OJfR4U7XNiKpV6GDgeeNo59w6WuN6eyWQ+XeJ5pwMf3tAhiRK+vGHOuQOi+RpgTMG0X8F2ZznnpgEnYp+7J5xza4CJzrlZxcfFe7+nc273aHFw0X7XtXN1zp0ITF67du02hc+vq6sbXl1d/XD0/uwOPAUsBCZ47xszmczPiv8x59zY6P87KZ1OzwcmO+cWA6u998cA0zOZTMnEvQdSb/+yyZSgisiW0EWrSM+gm1Eb0dzc/EssMX3aObdPEARjgiA4IZvNjvbe7wr8rmBzl8vlbgZ2Af4K7Bxte5BzbjxWkvqDurq6E0q9lnPuu8BlY8aMGRwEwWFr1qzZGfgXMNR7Px04BzghCIKRQRAcHIbhGGCp9/4jEyZMGFNqn977LwMNtbW1uwVBMCGbzR4EfAR7739TX18/akuOTxAEvwTGRotvBEGwe8F0TMH/9qRzbmJTU9PgIAgOLDiGxwPLgO+feuqpexbs9yzn3AXRc28t2u9FnQjteu/9nsDs1tbWXYIg+GA2mx0LHAssB76ayWRKJuLe++8Af2ptbR2czWbHBkGwK/BNrMTwx1OnTtV1ts4dshn0xRGRLaEfHpGeQaUgG5DJZEYAnwbWAJlsNvtc4frGxsY3gyC4uWD744FDgDeBTxWOy5nNZv8B/ApwzrmLO3jJ/wRBcOXUqVNDgNtuu21lGIaXReuOcM59q7CK8KxZs54CbgRIp9NHdLDPpWvWrLmwoaFhbf6BIAj+7r3/A1Drvf98Jw7FFstms//OZrPZuXPnFvYAS2Nj41zv/eVAKp1Of2RrvFYmkzk4KlldEobh2XPmzFmeXxcEwT3AFdHiNzrYxRO1tbVfnzNnzpr8A7W1tT8FXgN2fvjhh0dvjTi7OZ07ZJMpQRWRLaEfHpGeIZ8s6GZUCVF7yTRwexAEr3TiKSdEz5tRmJzmhWE4LZo9dvLkyb1KPP/vxQ+kUql1nQB5728tXu+9fyaa3aWDmG657bbbVpbY7y2FMZfL5MmTe02YMGHkhAkTjpk4ceKJEydOPNE5lwZwzu2zNV4jSk5xzs2eNWvWkuL1YRheE82OnTRp0sDi9d77WxsaGgo7AiJafgoglUrtvDXi7OZ07pBNpgRVRLaEfnhEegbVltgA7/3uAM655zv5lN0AwjB8udTKQw455FVgLVC7cuXKocXrnXOvFT/W1NS0NJpdGQTBohLPWRa95nqJVrS+ZCxAPuHeauOWbsi4ceNqM5nML5qbm99LpVIvpVKpu8IwvD0Mw9uJSjS99/23xmtFVa/x3pf836Ok9T3ArV27drfi9alU6tUOdr0EwDlXuzXi7OZ07pBNpgRVRLaEfnhEega1N9+AgkRk6QY3bFMTPW95qZVR1d0VhdsWyuVyrcWPFdjQug1pKvVgTU1NPsayJFv9+vW7AfgK8Ib3/mJgknPuuDAMxwLnRZu5rfRyNQDe+5L/e2Q5QHV19XrvA5t/rHsSnTtkk+nDIiJbQj88Ij2DbkZt2CKAMAx36+T2+RK2nUqtPOmkk/oC2wKk0+nFWyG+jfLeDy/1eHNzcz7G94tWtQK9p06dmsq3hc2bMGHCoM2JYdKkSbvlcrnJwFu1tbVHNjQ0LCtcn8lktkrV3gL5ar0l/3csER4OkMvlyvI+VCCdO2STqQRVRLaEfnhEegbdjNoA7/2DAM65I+hc6d6j0fNK9qjbq1ev46LZ10tV1+0K3vsjO1h1VPT30aLH3wTSDz300PbFTygYTqad2tra/BispdrV4r3P9xQ8rzg5jdaXPF5hGDZHf0vudwMejeIdW2plXV3d4VjJ8bKxY8e+tIn7FqNzh2wyJagisiX0wyPSM+hm1AYMHTr0DuBVYO9MJvOFUtsUDjninPs71uPv+Lq6una96o4bN67KOfedaPHPXRPx+pxzJ2YymcOLYqkFvhqtv6XoKS8BpFKpdkOwTJkypdo5dxklNDQ0rMCGihlSX1/fr8Qmb0V/9xo3bly735VTTz11T+dcqfFYSaVSb0Qxjiy1viOtra2NWNXmo+rr608pXDd16tSUc+570eItxaXE0mk6d8gm00WliGwJ/fCI9AzqsXsDpk2b1pLJZD4N/AP4ZSaTOdA59zfv/ftYFdHj5s2bNwrIAGSz2YWZTOYHwA+dc7Pq6+svC8PwfmCIc+4i4AhgQTqdvrKM/8ZLQLa+vv47wAPAMO/9t4G9gUeWL1/eLll2zt3ovT/VOffT+vr6/mEYPgTs8vbbb1/Ahtur3g+c4r2fU1dXd7tzrtk5tzybzf5+2bJlz/fr128BsEe/fv0a6uvrf9Xa2roklUodFSWLbwB7Fe9w+fLlz/fr128xcGhdXd12xFpHAAAVo0lEQVTNzrmnsBuoTwdB0NhRIHPmzFmeyWQuAX7rvW/IZDI/TKVSc8MwHDBv3rwLgVOAd3O53NROH0UppnOHbDKVoIrIltAPj0jPoB67NyIad/QU4GXgk977OcBDwHTgy0Sd7RRs/yPg29gYo790zj3onJuFDedyfzqdPm7GjBmd7XRpa7gCuMN7f433/vEo/qO99/c55+qKxyXNZrO3AL+O4v+Rc+4O59wfnXNVuVyurqMXCcPw8865u4EjnXNTgR977y8BmDt3bmsqlToDWAhM8t7fmU6nH3XO/Qa42zn3lVL7nDt3brNz7mPAi9HfHwI/Bj66sX86CILfAV+MFn8chuEDwD+xmwmPhmE4bvbs2e9sbD/SIZ07ZJMpQRWRTVGciHb0w7N9iW1FpPso/v6Wqi2RAnYsTzjdQxAEdzY1Ne3lnDsO+DzwFefcmc65PYIg+HjR5j4Igh9VVVXt7L3/mPf+q977851zhwRBcNSMGTNeLd5/Op2+OAzDsVFpaztz587NhWE4NpfLjSsV29q1a/8ZPffyUuu99y1BEJzlvT8QOM8592Xn3HFjx449JpvNLuzg//1iGIb7R3F/OZVKfaimpmbs7NmzX4tiOa74ObNmzXo5m80eW1tbW5tOp0eEYTg2lUqtS2hnzpz54Jo1a3b33k/AevP9TBiG+wVB8JGampp7wjAcWypRzWaztwdBsAfQF9g76vU3X0WXXC73iSimBSX+j1+n0+mdgY8AF2Hv3RFjxowZO2vWrGeLt8/lcj8Iw3DsmjVrbit1XIDvRf/XXR2sr2SdOXek0blDNsBdc801ft2Cc6OnTJnS2TG8RKTn+QlwLHA+1rnEOODfwDPAB4BBWHulr2LVwl6NI0gR2SLbAi8AlwH/C6wGHgbGAhOAOcAHgd8DdwMl2wVK95DJZH4LXOC9P7exsfGGuOORbm1H4GngO8B1QDPwBLAfcCJwJ/Ah7NzxT+CCeMKUpJk2bdqe3vvn8ssq4RCRTfFn7Afln9hF69vR4ztjP0LbYxe3r6Hk9P+3d+9Rctb1Hcffs5ck5LZgkAYwgDXREBOMiEhpvWCItVYkeOmRHoxaFeyBY1UsJV6jHEtVopYeL8mxogVKKZATslC1hFBPhMZCIiRcAkFNsoZLICHJJiHZS57+8ZvJPjM7M3thd57f7L5f58zZ32+eZ2e+2cw+O5/5/Z7fI9WrXUAb8A1gEeG8v1Py2/6REFqPApqBn9S+PEmRegbYASwBvgJsAwqXKbqG8B7hKEL++GkWBao+GFAlDcRDhFBauBTASfmvkwifkAJ0Aj+qcV2ShtY1hFGOqRRPxUtfPmQ78KtaFiUpekuA7wJ/lL8VzE212wjnZ0tleQ6qpIH6AeHyCJXswk9GpXp3K3CgyvYEaM1/VX37DXBLQ0PDlqwL0YhwE7C/yvYEuA2PHarCEVRJA/Vj4AqKPxlN206Y5iOpfh0EVgF/XWH788D3a1eOhktra+uPcNaLhs5+4F7gggrbnyOcJiBV5AiqpIHaDTxcYduL+KZVGim+Qwii5ewHNtawFkn14xpgZ4VtewgLK0oVGVAlDcY3gBfK3L+XMDVQUv17gJJrd+Z1A/9e41ok1Y/7gH1l7u8Crq9xLapDBlRJg3E3YbS01CbCp6OSRoal9D7nfCdOCZVU3XVAR8l9O/P3S1UZUCUNxmHCCEp36r524NpsypE0TP6VMK0/7Tng9xnUIql+/JDex45nCJetkqoyoEoarO9R/MdnH3BnRrVIGh47gcdS/S5CaJWkap4FfpvqdxJmZEh9MqBKGqwthD9ABb+m+uVnJNWnbxHeXEKY2n9jhrVIqh/fpPjYcXOGtaiOGFAlvRTXEFbz3IPTe6WR6ueE6xsDbAZ2ZFiLpPpxBz0LKj5Kz3FEqsqAKuml+A/C1N4DwC8zrkXS8DhMGPnoxA+iJPVfF7CCcOz454xrUR0xoEp6KV4EVgP/RXgTK2lkupawKNptWRciqa58hxBQb8+6ENWPpqwLkFT3/g7DqTTS/RZ4PeWvbShJlWwC3kD5S9NJZRlQJb1Uz2VdgKSa2JR1AZLqkscODYhTfCVJkiRJUTCgSpIkSZKiYECVJEmSJEXBgCpJkiRJioIBVZIkSZIUBQOqJEmSJCkKXmZGdWHFihXTsq6hmhUrVkxpb28fC7Bjx472NWvWtGddU4mJwOR8+xCwM8NaymkGXp7qPw0kGdVS1nvf+96puVyuAWDu3Lm75syZczDrmipJkqT9ggsu2F2L51q+fPlxDQ0NY2vxXIOxZs2aSVu2bJkEsHfv3kN33XVXbK/9McCxqf5TWRVSxQmp9vNAR1aFVDAFKLwG2/O3aJx++ukTX/nKV04GmDRp0qEFCxbE9hoscv755/8hl8sN+/H3nnvuGbdnz56X971nNjo6OnI333zz8YX+/fff/9y2bds6s6ypjPRrfy/xXac46vce06ZNG3PmmWceOf5+6EMfivH4e0RLS8tz55xzTrTvPYaSAVXRW7x4cUNjY+O2rOuoZt26dWzcuDHrMjSMli9ffqQ9c+ZMGhsbM6ymT98DLqvFEzU1Nd2Zy+XOqMVzDcbWrVu57bbbsi5Do9j69etZv349ANOnT+d973tfxhVVt2rVqqOBPcP9PO3t7X/e2Ni4YrifZ7ByuZzHjhGura2Ntra2I/2FCxfS0BDv5NL9+/e/E/hF1nXUQrz/C5IkSZKkUcWAKkmSJEmKglN8VXeSJFmYy+XWZl1H2sGDBx8gf57F9Olv2fTkk+d/MeOSSnzty7DntNAefxCuuijbekpdPw8e/Nue/mc/BicO+xSzgWhu/vytnZ2HANi8efPSs846a0nGJZX6PnBuxjUsAZZmXEORHTt2/CcwF2Dq1Ffve+aZSz6SbUWlli2Ax1O/j0ven10tlVx+a0/7jTfAByOblnnFT6B7Ymif8ARc/vlMyykxY8bPvrh586q5AJ2dnS8Cr8u4pCK5XG5OkiRZz2WN7ueyZcuWY4H7Cv3p0z/6gyefnH13hiWV8fc3wOFxoT1tI3z6q9nWU2rxVdB+amhP2A9f+3C29RSbMePBd2/efP1HCv329vbXtrS0xHae8SZG4YCiAVX1aPt55523Oesi0qZPn3640B47dmIHfLat2v619/XUSfW5JL76WncV9+c9Be/aVX7fbORyXzjS3rt37wuxvQbvuOOO/UmS+bpSO2P7uZx99tlHXvtNTc0RvvZvLFnMKrb6AC5PtY/eHV+N/3C4p90U3fF33Lh7j7wGkyQ5HNvvyMqVK4/O5XJZl5HE9nO58MIL96b748efvCu21xZckTrojzkYX31XpRZUa4zu+DthwpVFx9/bb7/9yVtuuSWqReBaW1uzLiEToy6RS5IkSZLiZECVJEmSJEXBgCpJkiRJioIBVZIkSZIUBQOqJEmSJCkKBlRJkiRJUhS8zIxGo0bgNcAbgDPyX4/Pb/sEsDqjuiRJklR/JhFyVRfQnnEtdc+AqtHoKmBRhW3ja1mIJEmS6koDMJ/igY6T8tv+Fzg7o7pGDAOqRrOtwHqgDfhUxrVIkiQpfuOBn2ddxEhmQB2cBuCPgROBHPA08PgQPOargOOAo4AXgCeBPS/xcdXbUuDbwPP5/gwMqJIkSeqffcCDwLr87SPA27MsaCTJKqCeCXw/394HvK0f33MCsDLVPxfYPbRlFfkCcEG+fQewGBgDfA64GDi5ZP82YAnwPcL88/56NXAl8JeEcJrWBdwHXAO0VnmMqwlTDQBuAL5bZd9vAeek+ldQ/ZzLXwBT8u3PAGuq7FsvtmZdgCRJkurSfuBooDt137szqmVEyiqgTibM14b+jxCOTX0PDH/tJ6WebyNhEZ0VhHBdzjRCMDwH+ADQ2cfj54CvEIJwpX9LE/CW/O0WYCFwsMx+21O1dlE5oObyj5EOwudROaBOB96Rbx8GHq2wnyRJkjQaJBSHUw0xp/j2zzhgOSGcdgD3EELrAcL00POAifl9zyeMSn69j8dcBnw81d8N3EkIgfsJI8bvBmblt3+AMPX3PYRfjLR0wDyD8KlOudHlOfQepZ1Xpcb0tgeBnVX2lSRJkqSXxIDaP+8n/Kx+RRiB/H3J9hMJJ0vPzvevAL5DCLDlfJLicPovhJHU0mWpFwGXEc6XbCQE1svy+6c9CjxFCLWNwFuB28s8bzpwHiKMSs8GpgLP9LH/3RX+LZIkSZI0JBqyLqBONBFGTN9B73AKYYrthfSMbE4G3lXhsY4hnAda8G3CAj3lrpl0GLgW+FLqvkWEYFkqPYp6boXnTgfOpfmvOcqf1N1A8bmqBlRJkiRJw8qA2n+fBl6ssv1hYG2qf0aF/S6mZzpwG2HktC9LCCEYwrmw7yyzTzpAlpu220w4lxXCisM/7WP/1wHH5tsdhNFjSZIkSRo2TvHtn6eovtJtwQPAn+Tbp1TYZ0GqfSPlFz0q1UFYxfeT+f6b6T2FNx1QTyVMO96euu9MYFJq38I5pVMoH1DT960lnBcbs6OofKmYezFgS5IkSdEzoPbPb/q5345Uu6XM9vEUr0R83wBq2JRqzyqzvQ14gnDZGgjTdq9PbS89n/QwYbGn9xMumTMD2Jza5+0l+8duPPBPFbZ9FQOqJEmSFD0Dav+80M/9OlLtMWW2v4Iw1bbgm4RLzfTHlFT7ZRX2WU1PQJ1H+YDaTQimAKsIAbWwvRBQx9AzHRjqI6C+CHyjwrZ7a1mIJEmSpMExoPZP1xA9zjEl/ZmDfJwJFe6/m55pwOkR0wnAWfn2enoCd3ra8jzgh/n2m1LP0Q783yDrrKUDwJVZFyFJkiRp8AyotdVY0r+D6gsvVdJW4f57CFN3GwijtTMJU4P/jJ4R3fRo6GZgK2GK7zn57ztMcbhdA3QOokZJkiRJGpB6CqjlpszWm+dL+l+m/+e39sdOwuJHp+f78wgBtdr1TFcDHyVMIZ5LGGH1+qeSJEmSai6ry8ykV64d18/vOW44CqmxpyieLjxnGJ6jdNpu+utBep+Puapk/4mEKb4FIzGgngLclbr9JLXtqpJtb65xbZIkSdKoldUI6p5Ueyxh0Z9dfXzPm/rYXg/2AffTcyma84F/G+LnuBv4XL79NkKwn5vv30fvKcWrgQTIAecCj9CzkNNzwIYhri8GEwn/1nLmlvR/WHYvSZIkjVY3An+R6hfWbnkjxZlmA+H9uAYgq4D6O8JqsoVzMs8mnI9ZSQPwN8NdVI3cSnFAnQ08PISPv4awmvAYwqJMl9MzUr6qzP7PEELpbMK5qk+kthXC60jzFHBJP/ddP5yFSJIkqe5MpPfipxCyVfr+SbUpZ2TJKqDuJ3yi8Pp8/1KqB9QvAacOd1E1spSw2uzLCQH9ZsI00r5GkAumEa63eqjC9v3AWnouE3NZalul6bp3EwLqeIo/CFhdfve6twtYlnURkiRJqkuXAJ/px36V3q+riqzOQYXia3S+E7iGMN037RjgWmAxYXrsSLAf+DhhtVyAWYRRuguo/P8xhjCN4Abgt/T9aUw6iI7Pf30BWDeA/UvvlyRJkhRmIP6uH7ftWRVYz7JcxXcpYeT0Vfn+5cCHCedJ7iNcJuVNhNB6EPgEcFPtyxwWKwn/3iWEUHoysJwwMvpr4A+EAHs0MB04DThqAI+/GvhqyX3/Q5hWXc4vCYs3pV8PWwhhWJIkSZJqIsuAegBYAPw3cHz+vmOB95Tstwu4iHC5lJHku8DjhEV4TsrfdxxwXh/ft4W+pwv8mhDyJ6buqzYaupfixZv62l+SJEmShlzW10F9mDA6uAj4AOH8yoJnCSOm3wbaCOHtltT24Z7TvS71fPf383s2pb6nPwsf/QyYQQjgCwjnjbaU7LMHeJQwyvkzwiJIfS1c1EkYnZ1V8lzVLCWM3BaMlNFqSZIkSXUi64AK8DxhuuvlwBRCQNtJ8aVoIEx//asa1rWMgS+kszJ/G4gO4Mf5G8Bkws8hRxg93j3AxytYPMD9f5q/SZIkSVImYgioaTvzt9Fsb/4mSZIkSaNKlqv4SpIkSZJ0hAFVkiRJkhQFA6okSZIkKQqxnYM6GI3Ax4bosdYDDwzRY2mY5HK5m1tbWzuyriNt0aJFkwvtTZtWnQrj7syynt46J/e0D4yLr76uscX9BTdAQ1+rVddUZ2fPS27u3LmXXXrppQszLKeXJElelnUNwJWtra2XZV1E2nXXXXdsof30049PjPC1X3KN69jqK3XPR2HcB7OuolhX6pJq22fG9jN87DGOHH/HjBkzvrW1dXuW9ZTRnHUBwFGx/VwOHDjQcNNNPRc0eOSRqz8FV1+SYUlldI/raW+ZG9trHzpSV6ZonxBbfRs3Mi7dv+iii36/cGFUf9phlA4mjoSA2ky4RMpQ+BoG1HpwbN+71FYulzvS7u7uaCbCGnskOTgUcX0AnVOyrqBUkorLjY2NEym+zrCCyflbNBoaev62d3d35aAr8td+7L+bXRPCLVbdTdAd1c+wq6uomwNOyKaSqEX3c0kfOwC6uw9NAiZlU01/dDfH9tovFt97j+7uXndF9RoczUZlKpckSZIkxWckjKAeAoZqatvBIXocDaHFixcfXrlyZXRzLtJmzZo1e+rUqS0Azz77bNuGDRu2ZV1TiVcAJ+fb7cCGDGspZxJwWqq/Fuj92WaG5s2bd1ZDQ0MjwPjx4x9LkmRX1jVVsamGz/XlJEmi+lQ87fjjjz95/vz5rwBob2/fvXbt2keyrqlECzA71b83q0Kq+NNU+2F6X6c8a68Fjs63/wBszbCWXmbOnHnitGnTTgFoaWlpT5IktuNvkWOOOeZALZ6nqalpXVdXV7R/25MkaZw/f/5Zhf5DDz20YceOHe1Z1lTGafSM6m4D2jKspZyTgGn59h7C8SMaU6dObZkzZ86R42+SJPclSRLV6UVpzc3NUR87hlJu6dKlR/4jcrncay6++OInsixIkiRJkjQ6LFu27NVJkjxe6DvFV5IkSZIUBQOqJEmSJCkKBlRJkiRJUhQMqJIkSZKkKBhQJUmSJElRMKBKkiRJkqJgQJUkSZIkRcGAKkmSJEmKggFVkiRJkhQFA6okSZIkKQoGVEmSJElSFAyokiRJkqQoGFAlSZIkSVEwoEqSJEmSomBAlSRJkiRFwYAqSZIkSYqCAVWSJEmSFAUDqiRJkiQpCgZUSZIkSVIUDKiSJEmSpCgYUCVJkiRJUTCgSpIkSZKiYECVJEmSJEXBgCpJkiRJioIBVZIkSZIUBQOqJEmSJCkKBlRJkiRJUhQMqJIkSZKkKBhQJUmSJElRMKBKkiRJkqJgQJUkSZIkRcGAKkmSJEmKQlO6kyTJKcuWLcuqFkmSJEnSKJIkySnpflPJ9l8kSVK7aiRJkiRJynOKryRJkiQpCgZUSZIkSVIU/h9zhnJWRK6mqgAAAABJRU5ErkJggg=="
+ }
+ },
+ "cell_type": "markdown",
+ "id": "6b214777",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "![](\"attachment:fig15.png\")
\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c1bd8413",
+ "metadata": {},
+ "source": [
+ "The communication step happens in function `ghost_exchange!`. This one modifies the ghost cells in the input vector `u` by importing data using MPI point-to-point communication. See the following code:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "688638a1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "code3 = quote\n",
+ " function ghost_exchange!(u,comm)\n",
+ " load = length(u)-2\n",
+ " rank = MPI.Comm_rank(comm)\n",
+ " nranks = MPI.Comm_size(comm)\n",
+ " if rank != 0\n",
+ " neig_rank = rank-1\n",
+ " u_snd = view(u,2:2)\n",
+ " u_rcv = view(u,1:1)\n",
+ " dest = neig_rank\n",
+ " source = neig_rank\n",
+ " MPI.Sendrecv!(u_snd,u_rcv,comm;dest,source)\n",
+ " end\n",
+ " if rank != (nranks-1)\n",
+ " neig_rank = rank+1\n",
+ " u_snd = view(u,(load+1):(load+1))\n",
+ " u_rcv = view(u,(load+2):(load+2))\n",
+ " dest = neig_rank\n",
+ " source = neig_rank\n",
+ " MPI.Sendrecv!(u_snd,u_rcv,comm;dest,source)\n",
+ " end\n",
+ " end\n",
+ "end;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5bf5408c",
+ "metadata": {},
+ "source": [
+ "Note that we have used `MPI.Sendrecv!` to send and receive values."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d2ce54e4",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "Question: Is this other implementation based on `MPI.Send` and `MPI.Recv!` correct?\n",
+ "
\n",
+ "\n",
+ "```julia\n",
+ " function ghost_exchange!(u,comm)\n",
+ " load = length(u)-2\n",
+ " rank = MPI.Comm_rank(comm)\n",
+ " nranks = MPI.Comm_size(comm)\n",
+ " if rank != 0\n",
+ " neig_rank = rank-1\n",
+ " u_snd = view(u,2:2)\n",
+ " u_rcv = view(u,1:1)\n",
+ " dest = neig_rank\n",
+ " source = neig_rank\n",
+ " MPI.Send(u_snd,comm;dest)\n",
+ " MPI.Recv!(u_rcv,comm;source)\n",
+ " end\n",
+ " if rank != (nranks-1)\n",
+ " neig_rank = rank+1\n",
+ " u_snd = view(u,(load+1):(load+1))\n",
+ " u_rcv = view(u,(load+2):(load+2))\n",
+ " dest = neig_rank\n",
+ " source = neig_rank\n",
+ " MPI.Send(u_snd,comm;dest)\n",
+ " MPI.Recv!(u_rcv,comm;source)\n",
+ " end\n",
+ " end\n",
+ "```\n",
+ "\n",
+ " a) It is correct.\n",
+ " c) It is incorrect, but it might provide the right result depending on the MPI implementation.\n",
+ " b) It is incorrect, and it is guaranteed that it will result in a dead lock.\n",
+ " d) This implementation does not work when distributing over just a single MPI rank.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8498c7e2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "answer = \"x\" # replace x with a, b, c or d\n",
+ "sndrcv_check(answer)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d863cb89",
+ "metadata": {},
+ "source": [
+ "\n",
+ "Question: How would you fix the implementation above, while still using `MPI.Send` and `MPI.Recv!` instead of `MPI.Sendrecv!` ?\n",
+ "
\n",
+ "\n",
+ "Think about the answer. To uncover the the solution run next cell."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "629a3356",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sndrcv_fix_answer()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "82e21f4e",
+ "metadata": {},
+ "source": [
+ "### Local computation\n",
+ "\n",
+ "Once the ghost values have the right values, we can perform the Jacobi update locally at each process. This is done in function `local_update!`. Note that here we only update the data *owned* by the current MPI rank, i.e. we do not modify the ghost values. There is no need to modify the ghost values since they will updated by another rank, i.e. the rank that own the value. In the code this is reflected in the loop over `i`. We do not visit the first nor the last entry in `u_new`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f43560e5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "code4 = quote\n",
+ " function local_update!(u,u_new)\n",
+ " load = length(u)-2\n",
+ " for i in 2:(load+1)\n",
+ " u_new[i] = 0.5*(u[i-1]+u[i+1])\n",
+ " end\n",
+ " end\n",
+ "end;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "47861c19",
+ "metadata": {},
+ "source": [
+ "### Running the code\n",
+ "\n",
+ "Not let us put all pieces together and run the code. If not done yet, install MPI."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "53674411",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "] add MPI"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c966375a",
+ "metadata": {},
+ "source": [
+ "The following cells includes all previous code snippets into a final one. Note that we are eventually calling function `jacobi_mpi` and showing the result vector `u`. Run the following code for 1 MPI rank, then for 2 and 3 MPI ranks. Look into the values of `u`. Does it make sense?"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f91e4759",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "using MPI\n",
+ "code = quote\n",
+ " using MPI\n",
+ " MPI.Init()\n",
+ " $code1\n",
+ " $code2\n",
+ " $code3\n",
+ " $code4\n",
+ " n = 9\n",
+ " niters = 200\n",
+ " comm = MPI.Comm_dup(MPI.COMM_WORLD)\n",
+ " u = jacobi_mpi(n,niters,comm)\n",
+ " @show u\n",
+ "end\n",
+ "run(`$(mpiexec()) -np 1 julia --project=. -e $code`);"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8e4fe21d",
+ "metadata": {},
+ "source": [
+ "### Checking the result\n",
+ "\n",
+ "Checking the result visually is not enough in general. To check the parallel implementation we want to compare the result against the sequential implementation. The way we do the computations (either in parallel or sequential) should not affect the result. However, how can we compare the sequential and the parallel result? The parallel version gives a distributed vector. We cannot compare this one directly with the result of the sequential function. A possible solution is to gather all the pieces of the parallel result in a single rank and compare there against the parallel implementation.\n",
+ "\n",
+ "\n",
+ "The following function gather the distributed vector in rank 0."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "49a220c6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "code5 = quote\n",
+ " function gather_final_result(u,comm)\n",
+ " load = length(u)-2\n",
+ " rank = MPI.Comm_rank(comm)\n",
+ " if rank !=0\n",
+ " # If I am not rank 0\n",
+ " # I send my own data to rank 0\n",
+ " u_snd = view(u,2:(load+1))\n",
+ " MPI.Send(u_snd,comm,dest=0)\n",
+ " u_root = zeros(0) # This will nevel be used\n",
+ " else\n",
+ " # If I am rank 0\n",
+ " nranks = MPI.Comm_size(comm)\n",
+ " n = load*nranks\n",
+ " u_root = zeros(n+2)\n",
+ " # Set boundary\n",
+ " u_root[1] = -1\n",
+ " u_root[end] = 1\n",
+ " # Set data for rank 0\n",
+ " lb = 2\n",
+ " ub = load+1\n",
+ " u_root[lb:ub] = view(u,lb:ub)\n",
+ " # Receive and set data from other ranks\n",
+ " for other_rank in 1:(nranks-1)\n",
+ " lb += load\n",
+ " ub += load\n",
+ " u_rcv = view(u_root,lb:ub)\n",
+ " MPI.Recv!(u_rcv,comm;source=other_rank)\n",
+ " end\n",
+ " end\n",
+ " # NB onle the root (rank 0) will\n",
+ " # contain meaningfull data\n",
+ " return u_root\n",
+ " end\n",
+ "end;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e66bad1b",
+ "metadata": {},
+ "source": [
+ "Run the following cell to see the result. Is the result as expected?"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4a9f4a54",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "code = quote\n",
+ " using MPI\n",
+ " MPI.Init()\n",
+ " $code1\n",
+ " $code2\n",
+ " $code3\n",
+ " $code4\n",
+ " $code5\n",
+ " n = 9\n",
+ " niters = 200\n",
+ " comm = MPI.Comm_dup(MPI.COMM_WORLD)\n",
+ " u = jacobi_mpi(n,niters,comm)\n",
+ " u_root = gather_final_result(u,comm)\n",
+ " @show u_root\n",
+ "end\n",
+ "run(`$(mpiexec()) -np 3 julia --project=. -e $code`);"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "dc9b2b51",
+ "metadata": {},
+ "source": [
+ "Now that we have collected the parallel vector into a single array, we can compare it against the sequential implementation. This is done in the following cells."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "cb053382",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "code6 = quote\n",
+ " function jacobi(n,niters)\n",
+ " u = zeros(n+2)\n",
+ " u[1] = -1\n",
+ " u[end] = 1\n",
+ " u_new = copy(u)\n",
+ " for t in 1:niters\n",
+ " for i in 2:(n+1)\n",
+ " u_new[i] = 0.5*(u[i-1]+u[i+1])\n",
+ " end\n",
+ " u, u_new = u_new, u\n",
+ " end\n",
+ " u\n",
+ " end\n",
+ "end;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ffe6c605",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "code = quote\n",
+ " using MPI\n",
+ " MPI.Init()\n",
+ " $code1\n",
+ " $code2\n",
+ " $code3\n",
+ " $code4\n",
+ " $code5\n",
+ " $code6\n",
+ " n = 12\n",
+ " niters = 100\n",
+ " comm = MPI.Comm_dup(MPI.COMM_WORLD)\n",
+ " u = jacobi_mpi(n,niters,comm)\n",
+ " u_root = gather_final_result(u,comm)\n",
+ " rank = MPI.Comm_rank(comm)\n",
+ " if rank == 0\n",
+ " # Compare agains serial\n",
+ " u_seq = jacobi(n,niters)\n",
+ " if isapprox(u_root,u_seq)\n",
+ " println(\"Test passed 🥳\")\n",
+ " else\n",
+ " println(\"Test failed\")\n",
+ " end\n",
+ " end\n",
+ "end\n",
+ "run(`$(mpiexec()) -np 3 julia --project=. -e $code`);"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "73cd4d73",
+ "metadata": {},
+ "source": [
+ "Note that we have used function `isapprox` to compare the results. This function checks if two values are the same within machine precision. Using `==` is generally discouraged when working with floating point numbers as they can be affected by rounding-off errors."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d73c838c",
+ "metadata": {},
+ "source": [
+ "\n",
+ "Question: What happens if we use `u_root == u_seq` to compare the parallel and the sequential result?\n",
+ "
\n",
+ "\n",
+ " a) The test will still pass.\n",
+ " b) The test will fail due to rounding-off errors.\n",
+ " c) The test might pass or fail depending on `n`.\n",
+ " d) The test might pass or fail depending on the number of MPI ranks."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "cd2427f1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "answer = \"x\" # replace x with a, b, c or d\n",
+ "jacobitest_check(answer)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "790e7064",
+ "metadata": {},
+ "source": [
+ "Run cell below for an explanation of the correct answer."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "72ed2aa1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "jacobitest_why()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c9aa2901",
+ "metadata": {},
+ "source": [
+ "## Latency hiding\n",
+ "\n",
+ "Can our implementation above be improved? Note that we only need communications to update the values at the boundary of the portion owned by each process. The other values (the one in green in the figure below) can be updated without communications. This provides the opportunity of overlapping the computation of the interior values (green cells in the figure) with the communication of the ghost values. This technique is called latency hiding, since we are hiding communication latency by overlapping it with computation that we need to do anyway. The actual implementation is left as an exercise (see Exercise 1)."
+ ]
+ },
+ {
+ "attachments": {
+ "fig16.png": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6gAAADyCAYAAABAvOgkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7d17nFxlYf/xz9lssrmQm4EQkhACRi7h1kD4YSIqEGuwEi4tSNEakWpqBWuLtEKrgtW2WkDbn1VJ0Cra3w8lKMKGCIJc1BBCxCgghGsSJIEk5ELut93pH89s9uxkZnez7M7z7O7n/XqdV84zc2bmm9nNZL97Lk82e/bsApIkSZIkRVYTO4AkSZIkSWBBlSRJkiQlorZkPD3LsuUxgkiSJEmSepdCoTAeuKdp3KKgZlm2fNasWc9WO5QkSZIkqfeZM2cOhULzZZE8xFeSJEmSlAQLqiRJkiQpCRZUSZIkSVISLKiSJEmSpCRYUCVJkiRJSbCgSpIkSZKSYEGVJEmSJCXBgipJkiRJSoIFVZIkSZKUBAuqJEmSJCkJFlRJkiRJUhIsqJIkSZKkJFhQJUmSJElJsKBKkiRJkpJgQZUkSZIkJcGCKkmSJElKggVVkiRJkpQEC6okSZIkKQkWVEmSJElSEiyokiRJkqQkWFAlSZIkSUmwoEqSJEmSkmBBlSRJkiQlwYIqSZIkSUqCBVWSJEmSlAQLqiRJkiQpCRZUSZIkSVISLKiSJEmSpCTUxg4gSZKSMxB4b3F9PrA1YpZq619cdtO7/t6SlAQLqiRJKjUCuLW4/mbgxQ4+Tw3wFmAyMAkYXLz9CtItf/8IfBa4H5gWOYsk9ToWVEmS1BW+AXwAGFLmvqtJt6DG8F7gHOA54PrIWSQpKguqJEkqtRG4qri+voPPMZHmcrqy+JzHvsFc1fAH4BHg91V8zZOAWcAvqF5BPRQ4ObccCWTAbcCnq5RBkvZhQZUkSaU2A19+g89xE3Ad8BjwKvAh4Ltv8Dmr4abi0pNNBhZXuO+gagaRpFIWVEmS1BX+X+wA3ci3CBej2rKfjzsSOJFwjvBjHXjdDcCvi4/9IDCmA88hSZ3KaWYkSVKpQ4FCcTkicpZquwR4ALihiq/5CqEkPrOfj5tBuJjVrP183LOEi1+9CXg34Zzgdfv5HJLUJdyDKkmS1OwI4HSgsYqvOZFwfu5a4MEqvN6m4iJJyXEPqiRJUlx/RtgT+vnYQSQpNguqJEmSJCkJFlRJkiRJUhI8B1WSJLXXUGBChfuew/MaJUlvkAVVkiS11zuAOyvcdxZwTxWz9BZHUPkqvacU/zwV+FKFbW4HFnV2KEnqKhZUSZLUXhupPN+me0+7xjjg021sc2JxKWcZFlRJ3YgFVZIktdcvgcmxQ/QyLwFfrnDfKcCZwO+Auyts89uuCCVJXcWCKkmSlK4Xgasq3PcpQkFd1Mo2ktSteBVfSZIkSVIS3IMqSZK6womECyc1OSm3/klgW278dWBLNUJJktJmQZUkSV2htSvLfq5k/D0sqJIkLKiSJGlfW4A5xfWOXp336dxztGVb25tUzRZgLbAhdpAu9h1gbG58RPHP6cC9udsfAP61WqEkyYIqSZJKbQD+6g0+xy+LS3fz78Wlp5sCHFXm9tHFpcna6sSRpMCCKkmSFNePgKXsfxmsJ0xD82IHXvNvgcHt2O6lDjy3JHWYBVWSJCmup4rL/nq2uHREpXlTJSkqp5mRJEmSJCXBgipJkiRJSoIFVZIkSZKUBAuqJEmSJCkJFlRJkiRJUhIsqJIkSZKkJDjNjJJXKBSy+vr6R2PnaM0NN9xw2LJlywYCbN68+bUNGzakNrH5gcBBxfXtwPJ4UcoaAIzPjZ8BGuNEKW/cuHFHUfyl3iWXXPLy5MmTN0eOVFGWZbfOmDHjumq8Vn19/XcLhcKx1Xitjpg7d+7Ihx56aATAzp07t65evTq1OR0HAeNy46djBWnFMbn1l4CtsYJUMI7wPgKsA9ZEzLKPYcOGjRgyZMhIgDFjxuy4+uqrl8XO1Jq6urrTp0+f3uVf47vuuusdDQ0NN3T163TUjh07aq688sqjmsarV69evnPnzu0xM5UxnvD/J4Q5dF+LF6Wsgwg/fwBsA1ZEzLKP/v37Dxw5cuRhTeOvfe1rT2dZFjNSWz5xzjnnPBI7RDVYUJW8z3/+89nJJ588OXaO1qxfv56XXtr7c+8g4LBWNo9tEM3/YaTqpNgBSuW+vjQ0NByV+H9ii6r1QoVC4dgsy5L997l9+/bSf5sjI8Zpj2Tfy6Jj2t4kqtLCH93GjRvZuHEjAP369RuUZdmIyJFaVVNTU5WfDRsaGoan/NmRZVmLz30g2V/EFQ2i5S96UzOI5l+UJ2HHjh2lX+PJKf/fnmXZ0NgZqsVDfCVJkiRJSXAPqrqjm7IseyF2iLzdu3dfQ/EwmzFjxqxYuXLlNyNHKvWXwFuK67uAz0XMUs5k4ILc+IvAlkhZyqqtrf3Snj17AFi/fv0DWZbdEzlSqZmFQmFizACFQuHumpqaB2NmKLV58+aPU9yjNmLEiO3r1q37fORIpd4B/ElufFWsIK34UvPqmIfh4CXxopTz249AY11x8BLwjZhpSo0dO/bSl19++UiAhobCnhdfHDY7dqa8gQN3jRw1atuFkWPszrLss5EztLBz584DgM80jUeNGnXbq6+++uuIkcr5Z6Bfcf154FsRs5TzMZr36u4Aro2WpIzRo0eftmrVqrObxnv27Pmnurq6hpiZShUKhX8D0t2t20UsqOp2CoXCD2bMmHF/7Bx5EyZMuIpiQR03btzKlStXfjlypFLvormg7gZSy/cRWhbU/wJWR8pSVp8+ffYW1GXLli06++yzk3oP582bNwWIWlCzLPtFau/L1KlTz6NYUIcPH75j3bp1SeUjnGudL6j/DhQiZakkV1An/gZ+9v14Ucrp+8FcQV1FYp9vRxxxxJnNBbVhzyc/edr3YmfKu/TSpyaef/6L0Qtqap8dF1988cHkCuopp5xyT319fWoF8DM0F9QVJPa9D8ygZUFNKt/JJ5+8K19Qf/CDH1w/d+7cXTEzlaqvr/9XemFB9RBfSZIkSVISLKiSJEmSpCRYUCVJkiRJSbCgSpIkSZKSYEGVJEmSJCXBgipJkiRJSoIFVZIkSZKUBAuqJEmSJCkJFlRJkiRJUhIsqJIkSZKkJFhQJUmSJElJsKBKkiRJkpJgQZUkSZIkJcGCKkmSJElKggVVkiRJkpQEC6okSZIkKQkWVEmSJElSEmpjB1CXGQBMzI2XAI1tPKYGmJQbPwVs7+RckiRJklSWBbXnOgL4dW48kLbLZr+SxxwPPNnJuSRJkiSpLA/xlSRJkiQlwYIqSZIkSUqCBVWSJEmSlAQLqiRJkiQpCRZUSZIkSVISLKiSJEmSpCRYUCVJkiRJSbCgSpIkSZKSYEGVJEmSJCXBgipJkiRJSoIFtecqlIyzdjzmgK4IIkmSJEntYUHtubaUjNtTPg/riiCSJEmS1B4W1J5rPS33ok5ox2Pe1UVZJEmSJKlNFtSeawuwLDc+q43tBwOXdV0cSZIkSWqdBbVnuy+3fjkwpsJ2dcDNwKFdnkiSJEmSKrCg9mxzgMbi+nDgl8AFNJ+PeiBwEfAocD7w22oHlCRJkqQmtbEDqEs9BnwN+GRxfDgwt7jeSMtfUKwilNfnq5au4y6rr68/N3aIvH/4h38Y0LT+/PPPHw78Z8Q45RyZW+9HevmOLRl/EdgWI0gle/bs2bs+YcKEd9XX1w+MGGcfhULh+NgZgLPr6+tHxQ6R981vfnN80/ratWsHkN73/kkl4/+IkqLdfvt2mHBQ7BQtNfbPDcaT2Nd46dKlRzet9+lT0/emmx64ImaeUn377nlT7AxAv/r6+qS+bps2bRp4yy237B0vWLDgz4EUPmfz6nLrR5LY9z7h584myX3+Lly4cFJ+/P73v/+GmTNnNlbaPpL2zMLR41hQe75PFf/8BC0LaX79V8CfA+uqFeqNyLLsT2NnKFVb2/xPae3atYcAfxMvTZv6knY+gI/EDlCqoaFh7/rQoUMnA5PjpUnW1OKSjIEDm3+P8Prrr/cn/e/9xPOtPTEsyRpFYu/hmjVr9q736VPTZ9SorRdFjJOqWhL7uvXr16/FeP369dOAaXHStMuhJPYelqgjsXyvvfZai3FNTc3lkaKohAW152sA/hb4NqGEngQcRLjK7wuEPaoPEvao9gHel3vsH6oZVJIkSVLvZkHtPZ4oLq1poPkQ4GRcc801hXnz5t0ZO0drRo8efXRDQ8MQgA0bNqx65ZVXXo6dqcRoYGxxfQvwVMQs5RwATMyNf03z+dNJOProo0+uqanpA9C3b9/ngA2RI1WUZdnjVXythwinCCRpyJAhYydOnDgaYPv27ZuWLVu2NHamEkOAo3PjR2MFacX/ya0vBTbFClLBUcDQ4vorJPbL1YMPPviQESNGHFpc3wr8PnKkVu3YsWN3NV4ny7JXgJT/b6+ZOHHi3iNlVqxY8dTWrVtL55iPbSLN1xV5mfQ+i8cSfv6A8LmR1OfvAQccMHjcuHHHNI0LhcJiWk7RmJSGhoY1bW/VM2SzZ8/e+4XIsuyoWbNmPRszkCRJkiSpd5gzZ86RhULhmaaxV/GVJEmSJCXBgipJkpS+fm1vIkndnwVVkiQpfZ8lXHX/wNhBJKkrWVAlSZLS9yvgZOBx4MzIWSSpy1hQJUmS0rcQeB04BLiLMH1c36iJJKkLWFAlSZLStwnYU1zvD7wfWAIcES2ROqp/7ABSyiyokiRJ3cP9ufX+hHkwFwJ/ESeOOqA/4XDt2XjhK6ksC6okSVL3cCewNTfOgJHAfxXvGxgjlPbLjYRzic8DDoqcRUqSBVWSJKl7eAjYUub2ocB04GnghKom0v74OPAhoIGw13tl3DhSmiyokiRJ3cNaYFeF+/oB4wiHAV9VtURqrynAV4vrnwbujZhFSpoFVZIkqft4rI373wR8Abi9ClnUPgcDtxJ+iXA78JW4caS01cYOIEmSpHa7A3gPUFdy+zbgeeA24DfA+irnUnm1wA+BscAzwCVAIWYgKXUWVEmSpO7jQWAjYa8chEN++xGuDnslHjqamn8H3glsBs4nTBckqRUe4itJktR9LCdcZAfCFX1/Cnyb8DPd94FD4sRSGRcBf0fYY/phwkWsJLXBgipJktS9PEUoPU8DfwZ8AniCsFf1Zvz5LgWTCb84APg34EcRs0jdih9gkiRJ3cudhL2oFxT/3E7YW7cV+GPgc/GiCRgP1AODgPn49ZD2iwVVkiSpe5kP/F9gRe62p4GPFdc/B5xX7VACYDjh6zOKcLGqi2g+JFtSO1hQJUmSupcXgE+Vuf1/CHNtZoTzUY+rZijRD5gLHAOsBM4FtkRNJHVDFlRJkqSe4++Bu4EDgB8Dw+LG6TUy4FvANMKVev8EeDlqIqmbsqBKkiT1HA3AXwDLgLcA/x9/3quGLwIfJEz7cz7weNw4UvflB5YkSVLPso5wDupW4D3A1+PG6fH+EvhHwpWVZwH3x40jdW8WVEmSpJ7nceASoJFw8aQroqbpud4L3Fhcv4YwzY+kN8CCKkmS1DPdRjgnFeA64MKIWXqiMwnvcS3w38AX4saRegYLqiRJUs/1FeA/CT/zfQ84LW6cHmMKcAfQH/gJ8Fdx40g9hwVVkiSpZ7uCcEXfpjLl9DNvzCnATwlXSp5PmOt0T9REUg9iQZUkSerZGglX9n0YGAHcCxwVNVH3dSrwM2Ao4WJIFxCu3Cupk1hQJUmSer7thAv6/AYYBfwceHPURN3PaYRyOgz4BXAu4X2V1IksqJIkSb3DRmAasAQYAzwAjI8ZqBt5B+Fw3iHAQ4SyvyVqIqmHsqBKkiT1HhsJc6MuBQ4F7sOS2pZzgLuBwcU/34PlVOoyFlRJkqTeZTXwLuB5wmG+vwSOiZooXR8lXGBqAOGqvefhYb1Sl7KgSpIk9T4rgbcDvwPGAguAt0ZNlJ5PA3OAPsB3CRdE2hkzkNQbWFAlSZJ6p1eBM4FFwHDCBYDOiJooDXXAd4AvFcf/DHwYp5KRqsKCKkmS1HutJxzuex/N51heGjVRXAcTpo+5hFBIPwZcEzOQ1NtYUCVJknq3LcDZwA+BfsC3gRsIh7b2JpOAR4GphOL+HmB21ERSL2RBlSRJ0k7gYuAqoBG4AriLMOdnbzAT+BUwDngOeBthr7KkKrOgSpIkCaAAfBl4H7AVmA4sBibHDNXFhgFzgZuBgcCdwCmEaXgkRWBBlSRJUt6PCFf4XQ5MIFzh91NAFjFTVzgd+A3h6ry7gL8jTCPzesRMUq9nQZUkSWr2l4R5Qb8aO0hkSwjnZN5KOC/1emA+cEjMUJ3kAOC/CBdDOhx4gXBI738Q9iJLisiCKkmS1Oww4DTghNhBErARuAj4KLANOAt4Gvg43fdnyLOAx4HLiuMbCUX819ESSWqhu364SJIkqTq+RTgP9RFgKPB1wgWFjo8Zaj+9hXB+6U8Je02XEabX+Wtgc8RckkpYUCVJkpp9CTgIOD92kMQ8TTgM9jLCOZpTCOdv3gSMjZirLQcRDk9+EphBONf0esIe8vsj5pJUgQVVkiSp2TbgNWBT7CAJagS+AUwkXPm2FvgIYVqW64ED40Xbx0jgOsKe0k8RzqO9i7DX9+8Jc79KSpAFVZIkqdkUwhygF8QOkrBVhKlopgAPAP0JJXAFMIe4h/5OKmZYBlwJDAIWEabMORt4Nl40Se1hQZUkSWr2HuAGwrmJat0jwJnAuwklcCDhgkqPE4rrTMI8o13toOLrPkI47PijxSyPAH8CvBX4WRVySOoEtbEDSJIkqVu7t7hMBf4G+FPCHKOnE875vBf4MfAQYUqXN6oGOBY4g3Cu8NuBPsX7dhLmcb2RMF2QpG7GgipJkqTO8HBxGU2YT/YCwsWI3ltcAF4FFhD2dL5QXFYA6wjnuOb1J1w1+AjgSMKVeCcRinB+z2wBWEyYs/VmYG3n/rUkVZMFVZIkSZ1pFfCF4nIUoaieRZiqZhTwZ8Wl1G6aL140mNZ/Tt0MLCRMG/Nj4KXOCC4pPguqJEmSusozwL8UlzpCSZ0KHAO8mbB3dAyQAX2B4SWPbySUz+cJVwt+irAH9nGgoevjS6o2C6okSZKqYSehXC4oub0PMAQYQDisNyNM97MVp/uReh0LqiRJkmJqADYUF0m9nNPMSJIkSZKSYEGVJEmSJCXBgipJkiRJSoIFVZIkSZKUBAuqJEmSJCkJFlRJkiRJUhKcZkaSJKnZbOAu4PXYQSSpN7KgSpIkNVtZXCRJEXiIryRJkiQpCRZUSZIkSVISLKiSJEmSpCRYUCVJkiRJSbCgSpIkSZKSYEGVJEmSJCXBaWbULdx+++3jY2doTX19/YEbNmzoD7Bq1apNixYt2hQ7U4nBwNDi+i5gTcQs5fQDRubGK4FCpCxlnXvuuaNrampqAE466aR1xx133PbYmSoZMGDAprPOOmt9NV7r7rvvPmT79u111XitjliwYMGQF154YQjAli1bdtx7772vxc5Uog44KDd+OVaQVozNra8FdsYKUsGBQP/i+mYSm790ypQpg0eNGjUUYPDgwbvOP//81D5/WzjvvPNWZFnW5Z+/t95664C+ffse3NWv01G7d+/ObrnlljFN40cffXTNypUrd8XMVMZIwv+fEL7vN0fMUs6Q4gLhc2NtxCz7OPzww+v+6I/+aO/n78yZM1P8/N1r2LBhr55xxhk7YueoBguqknfttdfW1NbWLoudozWLFy/miSeeiB1DXeiOO+7Yuz5x4kRqa9P9+Ny9e/fXgcur8Vq7du26s7a2dnI1Xqsjli9fzu233x47hnqxhQsX7l2fMGECF154YcQ0bbvvvvuGUYWS379//3dnWfaTrn6djtqzZ4+fHT3csmXLWLas+cfLSy65hOLvoZO0devWs4B7YueohnS/CpIkSZKkXsWCKkmSJElKQrrHqEmV/UWWZQvb3qx6duzYsYTieRaTJk16bMmSJe+LHKnUzcBpxfVtwPERs5TzPuDfcuNTgaTOFayrq3th585w6t3SpUtvnDJlynWRI5W6sVAo/HHkDNdlWXZj5AwtrFmz5jZgEsD48eNfX758+UmRI5X6KHDV3tFX+Xi8KBX8Hd/Yu/5Obuc87o2YZl9/z/XsYWAYjH4W/vGq1h9QXUceecdnn3323kkAu3fv3p5l2XGxM+U1NjYen8Chtsm9L3/4wx8OBBY1jU899dSrFy1adGvESOU8AU3f+ywAZkbMUs4PgaZTQDZR/CxOxdSpUy99+OGH/6lpvGnTpqOHDx++O2amUoVC4Tl64Q5FC6q6nUKh8MqMGTNejJ0jb8KECY1N6/37998JJJUPyJ9UXyC9fKVldAWwOkaQ9ti6devGs88+O6n3cN68edtiZwA2pPa+TJ06de8FfWpraxtJ73u/5cWszmQNNWldIKyFYWzhXYn928xobB7U7oLLVsYLs6+6uvv3fg8WCoXG1P6N3HnnnSNiZwAKqb0vF1988db8eOTIka+R3udH7nufHaSXL39BteQ+f0eMGLEuP77zzjuXzZ07N6kLYdXX18eOEEWva+SSJEmSpDRZUCVJkiRJSbCgSpIkSZKSYEGVJEmSJCXBgipJkiRJSoIFVZIkSZKUBAuqJEmSJCkJFlRJkiRJUhIsqJIkSZKkJNTGDtDDZcCbgTFAf2AV8BTQ8AafdwIwChgMbAReBFa/weeUJEmSpKhS34M6Hnghtwxux2NqSh5zXFeFK/p47rX+J5fhE8BS4DngQeBu4HFCkfwCMHA/X+dQ4OvAy8Xn/CUwH3gYeAX4DfBhWv+aXp3L+p9tvN4/0fJ9/HAb29+W2/ZDbWwrSZIkSftIfQ9qX+CI3Li9hTr/mLrOi1PW8NzrLSeU6NuAd1fYfgTwGeBdwHRgUzte4zLgesJe2HIyYBLw34QieR6wvsx2T+ayXgj8LVCo8Jzn0/J9PAf4ToVtDwBmAP2K499W2E6SJEmSKkq9oHY3NYS9qO8mHMb7EPAYsBk4DDgXOLC47VsJpXNWG895LXBNbrwV+CmwBHgdGEkouqcW7387cC/wNmBHyXM9BOwhfN0PASYCvy/zmm8iFN6804E+lD88+e00l9O1hD3FkiRJkrRfLKid6zTCe/p74CL2LX9XAj8CziyOLwW+CLxU4flmAJ/LjX9I2Ju6rmS7awh7RL9LOHT4JOBfgE+VbLcJWAxMKY6nlckIcAbNe6t3EvZCDwNOBh4ts/203Pr9VN4rK0mSJEkVpX4OandTC6wkFLxyxW8j8H5gS3Hch1Asy+lLOOc0K45vBS5m33LaZC7wkdz444S9q6V+nlufVub+0ttv3M/t76+wjSRJkiS1yoLa+T5DOMy1ktXAvNx4coXtLiRcGAlCob2ctvdM3kLYQwrhfNWLy2yTL5DvpPxe9KbCuRn4MtBYcnvegcAJufHPy2wjSZIkSW2yoHau3YQ9nW15LLc+vsI25+TW76D10pv349z628vc/zCwvbg+FDil5P5DgSOL678gXCH4d8Xx24ABJdufSfP30XLCVXwlSZIkab9ZUDvXM8C2dmy3Jrc+tMI2+XK5cD8yLM2tH1Pm/p3Ar3Lj0r2i+fF9xT+b9or2B6a2sr17TyVJkiR1mBdJ6lzlpnYpZ1duvV+Z++uA0bnxFbQ9DymEKW7yc8W+qcJ2Pwf+uLg+jXChJnLj/HZNf16Zuz9fRM8ss70kSZIk7TcLaufa00nPM7xkfETZrdo2qMLt+fNQpxCu/Nu057epcL5KmDcV4JeEUt2PlgX2MGBCcb0APNDBnJIkSZJkQU1Un5Lx/VS+em9rtle4/TfABkIRriNMj/MzwryoTXtu89PFbAUeAd5BmGpmePHx+bL6e0KplSRJkqQO6YkFtdwhs91NaRn9GvCTTnz+BuBB4PzieBqhoLZ2uO7PCQW1D3A6cDuefypJkiSpE6V+kaQdJeP+7XhMubk/u5sdtNwbeUKlDd+AcvOhlrtAUqXtMzz/VJIkSVInSr2gbioZH9KOx5zaFUEiyJ/PeU7FrTouXygnEYr96cXxc8BLJds/SpgXFUJBPRYYVRzvAR7qgoySJEmSepHUC+rrwOrc+G3teMxHuyhLtf0ot34y8J5Ofv6lwMrieg3hKr3DiuNye0N3E+ZFBTgamJm779fs+8sESZIkSdovqRdUaDkH6Mdo/bzZD9M8fUp3dzvhwkNN/hs4fD8efxDNhbOS/NV8L8+tlx7e2yRfXC+vcLskSZIkdUh3KKjfz60fB9wMDCnZZiDwOeAmYEuVcnW1RkLhbjoPdxSwGLiUyheC6gOcAcwBVgBvbuM18sVyQO51K00XU2770tslSZIkqUO6w1V8fwIsoPnw3vcDZxdve52wp/CthDk/GwmHnv64+jG7xGLC3+d7hAtEjQC+DXyFsGf5ZUKBHUbYu3oicMB+PH+5YrkEWF9h+yeANbS8ENV2Wu7lliRJkqQO6Q4FtRG4CLgXOKZ42xD2PSdzG/DXwB3Vi1YVc4HlwHcJ85QCDAXOauNxrwAb29jmZeAZ4KjcbZUO74UwL+r9wJ/nblvAvldbliRJkqT91h0KKoSL+ZwCXAF8gJaFagOhxP0H8DRh+pO5Jfd3padzr/dkOx/zUu4xq1vbsGgxcDzwp8AFhMN4S6fT2UK48NEvgHsIe0cb2vHcXwPemRv/qNKGRd8nHErc3u0lSZIkqV26S0EF2Ap8obgMJhzau5F9D0ctAO+rYq4fs/+HFD9cXPZHI3BbcYFwSPMIwtew3PvQXl8vLu01v7hIkiRJUqfqTgU1bzPNc3L2VluLiyRJkiT1CN3hKr6SJEmSpF7AgipJkiRJSoIFVZIkSZKUhO56DmpHfRAY0AnP8yzwYCc8jzogy7Jb6+vrd8XOkXf11VcPaVpfvHjxZGBVxDjlvCm3PpD08pX+u/wd4cJgydi1q/lbbtKkSZdfdtllH4oYZx+FQmF47AzA1fX19Z+IHSLvO9/5zoim9WXLlg0lve/9QS1Gp3JTpBztM58LuIezY8doYU/+PVx5NPS7K16YfT31VDa0ab1fv34D6+vrNL98BAAAAwFJREFUU/se7Bs7ADAgtfdl27ZtNbfccsve8fz5868D/jleorLynx+nkd7n24jcenKfv/Pnz2/x+fuBD3xg+cyZM2PFqaRX7kzsbQX1OuDgTniem7GgxjSi7U2qK8uyvet79uzpBxwSL02bMtLOB53z77RTFQqFves1NTUHAAfES5OswcUlGTU1zf+3NzQ01JD69/4OUvhFQ2W7GcDuTvlFbxdpqIWGA2OnyGtoOeFbd/j8jSG59yX/2QHQ0NAwDBgWJ0271JHYe1giua9xQ+k/zixLKl9v1itbuSRJkiQpPb1tD+pRdE4pT+rw0p7u2muvbZw3b94HY+dozQknnHDs6NGjhwGsXLnyD08++eRLsTOVGAscVlzfDDweMUs5g4ETcuNHgIYK20Yxbdq0t9bW1vYBGDRo0NNZlnV07uEu19jY+Ey1XqtPnz6fLRQKSe2xyhs9evRh06dPHwuwadOmjQsXLvx97EwlhgLH5cYLYgVpxdty608Cr8cKUsGxNO/ZehlYETHLPo455pgx48aNGw8wbNiwzVmWpfb528Lw4cO3VeN1amtrH2tsbEz2//Ysy/pMnz79rU3jJUuWPL5mzZrUpjg8Hmg6xWgF4fs/JeOAQ4vrrxM+P5IxatSooSeeeOLez98syx7OsqzQ2mNiqq2tTfqzozNls2fP3vuFyLLsqFmzZj0bM5AkSZIkqXeYM2fOkYVCYe8v1z3EV5IkSZKUBAuqJEmSJCkJFlRJkiRJUhIsqJIkSZKkJFhQJUmSJElJsKBKkiRJkpJgQZUkSZIkJcGCKkmSJElKggVVkiRJkpQEC6okSZIkKQkWVEmSJElSEiyokiRJkqQkWFAlSZIkSUmwoEqSJEmSkmBBlSRJkiQlwYIqSZIkSUqCBVWSJEmSlAQLqiRJkiQpCRZUSZIkSVISLKiSJEmSpCRYUCVJkiRJSajNDwqFwvg5c+bEyiJJkiRJ6kUKhcL4/Li25P57CoVC9dJIkiRJklTkIb6SJEmSpCRYUCVJkiRJSfhf0SSRmgN3HHEAAAAASUVORK5CYII="
+ }
+ },
+ "cell_type": "markdown",
+ "id": "7d66b1a2",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c2d48c45",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "Question: Which MPI directives would you use to implement latency hiding in the communication the ghost values?\n",
+ "
\n",
+ "\n",
+ " a) MPI_Send and MPI_Recv\n",
+ " b) MPI_Bsend and MPI_Recv\n",
+ " c) MPI_Isend and MPI_Irecv \n",
+ " d) MPI_Sendrecv"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "29ade8f6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "answer = \"x\" # replace x with a, b, c or d\n",
+ "lh_check(answer)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9bd014b0",
+ "metadata": {},
+ "source": [
+ "## Conclusion\n",
+ "\n",
+ "- We learned how to parallelize the Jacobi method efficiently.\n",
+ "- Using block partitions the communication overhead is small if the problem size is larger than the number of processors.\n",
+ "- In addition, one can use latency hiding to further reduce the overhead caused by communication.\n",
+ "- Both are not true if we use cyclic data partitions.\n",
+ "- Gauss-Seidel is a simple variation of Jacobi but is much more challenging to parallelize.\n",
+ "- One needs to consider a special order (red-black) when updating the values to parallelize the method.\n",
+ "- We learned how to implement a 1D Jacobi method with MPI.\n",
+ "- Once needs to be careful when using blocking directives to avoid dead locks."
+ ]
+ },
{
"cell_type": "markdown",
"id": "47643bf6",
@@ -1005,12 +1722,152 @@
},
{
"cell_type": "markdown",
- "id": "0edeee84",
+ "id": "6527d24f",
"metadata": {},
"source": [
"### Exercise 1\n",
"\n",
- "Transform the parallel implementation of the 1d Jacobi method (function `jacopi_mpi`) to use latency hiding (overlap between computation of interior values and communication)."
+ "The following code implements the 1D Jacobi method studied in this notebook, but using non-blocking sends and receives. Modify this code so that you overlap the communication of the ghost values with local computations as explained above in the section \"latency hiding\". You only need to modify function `jacobi_mpi_latency_hiding`. Copy the code below to a file called `ex1.jl`. Modify the file (e.g. with vscode). Run it from the Julia REPL using the `run` function as explained in the [Getting Started tutorial](https://www.francescverdugo.com/XM_40017/dev/getting_started_with_julia/#Running-MPI-code)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9b57f876",
+ "metadata": {},
+ "source": [
+ "\n",
+ "```julia\n",
+ "# file ex1.jl (begin)\n",
+ "\n",
+ "using MPI\n",
+ "MPI.Init()\n",
+ "\n",
+ "function jacobi_mpi_latency_hiding(n,niters,comm)\n",
+ " u, u_new = init(n,comm)\n",
+ " load = length(u)-2\n",
+ " rank = MPI.Comm_rank(comm)\n",
+ " nranks = MPI.Comm_size(comm)\n",
+ " nreqs = 2*((rank != 0) + (rank != (nranks-1)))\n",
+ " reqs = MPI.MultiRequest(nreqs)\n",
+ " for t in 1:niters\n",
+ " ireq = 0\n",
+ " if rank != 0\n",
+ " neig_rank = rank-1\n",
+ " u_snd = view(u,2:2)\n",
+ " u_rcv = view(u,1:1)\n",
+ " dest = neig_rank\n",
+ " source = neig_rank\n",
+ " ireq += 1\n",
+ " MPI.Isend(u_snd,comm,reqs[ireq];dest)\n",
+ " ireq += 1\n",
+ " MPI.Irecv!(u_rcv,comm,reqs[ireq];source)\n",
+ " end\n",
+ " if rank != (nranks-1)\n",
+ " neig_rank = rank+1\n",
+ " u_snd = view(u,(load+1):(load+1))\n",
+ " u_rcv = view(u,(load+2):(load+2))\n",
+ " dest = neig_rank\n",
+ " source = neig_rank\n",
+ " ireq += 1\n",
+ " MPI.Isend(u_snd,comm,reqs[ireq];dest)\n",
+ " ireq += 1\n",
+ " MPI.Irecv!(u_rcv,comm,reqs[ireq];source)\n",
+ " end\n",
+ " MPI.Waitall(reqs)\n",
+ " for i in 2:load+1\n",
+ " u_new[i] = 0.5*(u[i-1]+u[i+1])\n",
+ " end\n",
+ " u, u_new = u_new, u\n",
+ " end\n",
+ " return u\n",
+ "end\n",
+ "\n",
+ "function init(n,comm)\n",
+ " nranks = MPI.Comm_size(comm)\n",
+ " rank = MPI.Comm_rank(comm)\n",
+ " if mod(n,nranks) != 0\n",
+ " println(\"n must be a multiple of nranks\")\n",
+ " MPI.Abort(comm,1)\n",
+ " end\n",
+ " load = div(n,nranks)\n",
+ " u = zeros(load+2)\n",
+ " if rank == 0\n",
+ " u[1] = -1\n",
+ " end\n",
+ " if rank == nranks-1\n",
+ " u[end] = 1\n",
+ " end\n",
+ " u_new = copy(u)\n",
+ " u, u_new\n",
+ "end\n",
+ "\n",
+ "function gather_final_result(u,comm)\n",
+ " load = length(u)-2\n",
+ " rank = MPI.Comm_rank(comm)\n",
+ " if rank !=0\n",
+ " u_snd = view(u,2:(load+1))\n",
+ " MPI.Send(u_snd,comm,dest=0)\n",
+ " u_root = zeros(0)\n",
+ " else\n",
+ " nranks = MPI.Comm_size(comm)\n",
+ " n = load*nranks\n",
+ " u_root = zeros(n+2)\n",
+ " u_root[1] = -1\n",
+ " u_root[end] = 1\n",
+ " lb = 2\n",
+ " ub = load+1\n",
+ " u_root[lb:ub] = view(u,lb:ub)\n",
+ " for other_rank in 1:(nranks-1)\n",
+ " lb += load\n",
+ " ub += load\n",
+ " u_rcv = view(u_root,lb:ub)\n",
+ " MPI.Recv!(u_rcv,comm;source=other_rank)\n",
+ " end\n",
+ " end\n",
+ " return u_root\n",
+ "end\n",
+ "\n",
+ "function jacobi(n,niters)\n",
+ " u = zeros(n+2)\n",
+ " u[1] = -1\n",
+ " u[end] = 1\n",
+ " u_new = copy(u)\n",
+ " for t in 1:niters\n",
+ " for i in 2:(n+1)\n",
+ " u_new[i] = 0.5*(u[i-1]+u[i+1])\n",
+ " end\n",
+ " u, u_new = u_new, u\n",
+ " end\n",
+ " u\n",
+ "end\n",
+ "\n",
+ "n = 12\n",
+ "niters = 100\n",
+ "comm = MPI.Comm_dup(MPI.COMM_WORLD)\n",
+ "u = jacobi_mpi_latency_hiding(n,niters,comm)\n",
+ "u_root = gather_final_result(u,comm)\n",
+ "rank = MPI.Comm_rank(comm)\n",
+ "if rank == 0\n",
+ " u_seq = jacobi(n,niters)\n",
+ " if isapprox(u_root,u_seq)\n",
+ " println(\"Test passed 🥳\")\n",
+ " else\n",
+ " println(\"Test failed 😢\")\n",
+ " end\n",
+ "end\n",
+ "\n",
+ "# file ex1.jl (end)\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1af362aa",
+ "metadata": {},
+ "source": [
+ "### Exercise 2\n",
+ "\n",
+ "In the parallel implementation of the Jacobi method, we assumed that the method runs for a given number of iterations. In function, `jacobi_with_tol` at the beginning of the notebook shows how the Jacobi iterations can be stopped when the difference between iterations is small. Implement a parallel version of this function. Start with the in Exercise 1 and add the stopping criterion implemented in `jacobi_with_tol`. Use a text editor and the Julia REPL. Do not try to implement the code in a notebook."
]
},
{
diff --git a/notebooks/julia_mpi.ipynb b/notebooks/julia_mpi.ipynb
index ef666e6..035ed72 100644
--- a/notebooks/julia_mpi.ipynb
+++ b/notebooks/julia_mpi.ipynb
@@ -1497,46 +1497,46 @@
"function matmul_mpi_3!(C,A,B)\n",
"```\n",
"\n",
- "Assume that the input matrices `A` and `B` are given only on rank 0, the other ranks get dummy matrices with zero rows and zero columns to save memory. You need to communicate the required parts to other ranks. For simplicity you can assume that `A` and `B` are square matrices and that the number of rows is a multiple of the number of processes (on rank 0). The result `C` should be overwritten only on rank 0. You can use the following cell to implement and check your result."
+ "Assume that the input matrices `A` and `B` are given only on rank 0, the other ranks get dummy matrices with zero rows and zero columns to save memory. You need to communicate the required parts to other ranks. For simplicity you can assume that `A` and `B` are square matrices and that the number of rows is a multiple of the number of processes (on rank 0). The result `C` should be overwritten only on rank 0. You can use the following cell to implement and check your result. Copy the code below to a file called `ex1.jl`. Modify the file (e.g. with vscode). Run it from the Julia REPL using the `run` function as explained in the [Getting Started tutorial](https://www.francescverdugo.com/XM_40017/dev/getting_started_with_julia/#Running-MPI-code)."
]
},
{
- "cell_type": "code",
- "execution_count": null,
- "id": "265c4593",
+ "cell_type": "markdown",
+ "id": "4fa53366",
"metadata": {},
- "outputs": [],
"source": [
- "code = quote\n",
- " using MPI\n",
- " MPI.Init()\n",
- " function matmul_mpi_3!(C,A,B)\n",
- " # Implement here\n",
- " end\n",
- " function testit(load)\n",
- " comm = MPI.COMM_WORLD\n",
- " rank = MPI.Comm_rank(comm)\n",
- " if rank == 0\n",
- " P = MPI.Comm_size(comm)\n",
- " N = load*P\n",
- " else\n",
- " N = 0\n",
- " end\n",
- " A = rand(N,N)\n",
- " B = rand(N,N)\n",
- " C = similar(A)\n",
- " matmul_mpi_3!(C,A,B)\n",
- " if rank == 0\n",
- " if !(C ≈ A*B)\n",
- " println(\"Check not passed\")\n",
- " else\n",
- " println(\"Check passed!\")\n",
- " end\n",
- " end\n",
- " end\n",
- " testit(100)\n",
+ "```julia\n",
+ "# ex1.jl (begin)\n",
+ "using MPI\n",
+ "MPI.Init()\n",
+ "function matmul_mpi_3!(C,A,B)\n",
+ " # Implement here\n",
"end\n",
- "run(`$(mpiexec()) -np 4 julia --project=. -e $code`);"
+ "function testit(load)\n",
+ " comm = MPI.COMM_WORLD\n",
+ " rank = MPI.Comm_rank(comm)\n",
+ " if rank == 0\n",
+ " P = MPI.Comm_size(comm)\n",
+ " N = load*P\n",
+ " else\n",
+ " N = 0\n",
+ " end\n",
+ " A = rand(N,N)\n",
+ " B = rand(N,N)\n",
+ " C = similar(A)\n",
+ " matmul_mpi_3!(C,A,B)\n",
+ " if rank == 0\n",
+ " if !(C ≈ A*B)\n",
+ " println(\"Test failed 😢\")\n",
+ " else\n",
+ " println(\"Test passed 🥳\")\n",
+ " end\n",
+ " end\n",
+ "end\n",
+ "testit(100)\n",
+ "end\n",
+ "# ex1.jl (end)\n",
+ "```"
]
},
{
@@ -1548,7 +1548,7 @@
"\n",
"Implement this \"simple\" algorithm (the telephone game):\n",
"\n",
- "Rank 0 generates a message (an integer). Rank 0 sends the message to rank 1. Rank 1 receives the message, increments the message by 1, and sends the result to rank 2. Rank 2 receives the message, increments the message by 1, and sends the result to rank 3. Etc. The last rank sends back the message to rank 0 closing the ring. See the next figure. Implement the communications using MPI. Do not use `Distributed`.\n"
+ "Rank 0 generates a message (an integer). Rank 0 sends the message to rank 1. Rank 1 receives the message, increments the message by 1, and sends the result to rank 2. Rank 2 receives the message, increments the message by 1, and sends the result to rank 3. Etc. The last rank sends back the message to rank 0 closing the ring. See the next figure. Implement the communications using MPI. Do not use `Distributed`. Use a text editor and the Julia REPL. Do not try to implement the code in a notebook.\n"
]
},
{
diff --git a/notebooks/mpi_collectives.ipynb b/notebooks/mpi_collectives.ipynb
index fbba57f..7555343 100644
--- a/notebooks/mpi_collectives.ipynb
+++ b/notebooks/mpi_collectives.ipynb
@@ -898,77 +898,6 @@
"After learning this material and the previous MPI notebook, you have a solid basis to start implementing sophisticated parallel algorithms using MPI."
]
},
- {
- "cell_type": "markdown",
- "id": "c6b23485",
- "metadata": {},
- "source": [
- "## Exercises"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "90dc58bb",
- "metadata": {},
- "source": [
- "### Exercise 1\n",
- "\n",
- "In the parallel implementation of the Jacobi method in previous notebook, we assumed that the method runs for a given number of iterations. However, other stopping criteria are used in practice. The following sequential code implements a version of Jacobi in which the method iterates until the norm of the difference between u and u_new is below a tolerance.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "0fcb0cd6",
- "metadata": {},
- "outputs": [],
- "source": [
- "function jacobi_with_tol(n,tol)\n",
- " u = zeros(n+2)\n",
- " u[1] = -1\n",
- " u[end] = 1\n",
- " u_new = copy(u)\n",
- " increment = similar(u)\n",
- " while true\n",
- " for i in 2:(n+1)\n",
- " u_new[i] = 0.5*(u[i-1]+u[i+1])\n",
- " end\n",
- " increment .= u_new .- u\n",
- " norm_increment = 0.0\n",
- " for i in 1:n\n",
- " increment_i = increment[i]\n",
- " norm_increment += increment_i*increment_i\n",
- " end\n",
- " norm_increment = sqrt(norm_increment)\n",
- " if norm_increment < tol*n\n",
- " return u_new\n",
- " end\n",
- " u, u_new = u_new, u\n",
- " end\n",
- " u\n",
- "end"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "dbf0c3b8",
- "metadata": {},
- "outputs": [],
- "source": [
- "n = 10\n",
- "tol = 1e-12\n",
- "jacobi_with_tol(n,tol)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "aab1455e",
- "metadata": {},
- "source": [
- "Implement a parallel version of this algorithm. Recommended: start with the parallel implementation given in the previous notebook (see function `jacobi_mpi`) and introduce the new stopping criteria. Think carefully about which MPI operations you need to use in this case."
- ]
- },
{
"cell_type": "markdown",
"id": "5e8f6e6a",