seb/XM_40017
1
0 Fork 0
mirror of https://github.com/fverdugo/XM_40017.git synced 2026-11-15 05:47:17 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

More changes in ASP and LEQ

Browse Source
This commit is contained in:
Francesc Verdugo committed 2024-09-19 10:37:09 +02:00
1 parent d7b9ff635a
commit 70f7004cc3
3 files changed
+3607 -288
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

notebooks/LEQ.ipynb
+80 -277
View File
@@ -22,7 +22,7 @@
"In this notebook, we will learn\n",
"\n",
"- How to parallelize Gaussian elimination\n",
"- How to fix static load imbalance"
"- What is load imbalance, and how to fix it in this algorithm"
]
},
{
@@ -37,26 +37,15 @@
},
{
"cell_type": "code",
"execution_count": 18,
"execution_count": null,
"id": "7e93809a",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"ge_lb_answer (generic function with 1 method)"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"using Printf\n",
"function answer_checker(answer,solution)\n",
" if answer == solution\n",
" \"🥳 Well done! \"\n",
" \"🥳 Well done!\"\n",
" else\n",
" \"It's not correct. Keep trying! 💪\"\n",
" end |> println\n",
@@ -70,7 +59,8 @@
"function ge_lb_answer()\n",
" msg = \"It is a form of static load balancing. We know in advance the load distribution and the partition strategy does not depend on the actual values of the input matrix\"\n",
" println(msg)\n",
"end"
"end\n",
"println(\"🥳 Well done!\")"
]
},
{
@@ -105,14 +95,14 @@
"1 \\\\\n",
"35 \\\\\n",
"\\end{matrix}\n",
"\\right]\n",
"\\right].\n",
"$$\n",
"\n",
"This is just a small example with three unknowns, but practical applications need to solve linear equations with large number of unknowns. Parallel processing is needed in these cases.\n",
"\n",
"### Problem statement\n",
"\n",
"Let us consider a system of linear equations written in matrix form $Ax=b$, where A is a nonsingular square matrix, and x and b are vectors. The goal of Gaussian elimination is to transform the system $Ax=b$, into a new system $Ux=c$ such that\n",
"Let us consider a system of linear equations written in matrix form $Ax=b$, where $A$ is a nonsingular square matrix, and $x$ and $b$ are vectors. $A$ and $b$ are given, and $x$ is unknown. The goal of Gaussian elimination is to transform the system $Ax=b$, into a new system $Ux=c$ such that\n",
"- both system have the same solution vector $x$,\n",
"- the matrix $U$ of the new system is *upper triangular* with unit diagonal, namely $U_{ii} = 1$ and $U_{ij} = 0$ for $i>j$.\n",
"\n",
@@ -167,7 +157,7 @@
"\\right]\n",
"$$\n",
"\n",
"The most challenging part of solving a system of linear equations is to transform it to upper triangular form. Afterwards, the solution vector can be obtained easily with a backward substitution."
"The most challenging part of solving a system of linear equations is to transform it to upper triangular form. Afterwards, the solution vector can be obtained easily with a backward substitution. For this reason, we will study here the triangulation step only."
]
},
{
@@ -179,7 +169,7 @@
"\n",
"### Augmented system matrix\n",
"\n",
"In practice, vector $b$ is added as an additional column to A forming the so-called *augmented* matrix $A^* = [A | b]$.\n",
"In practice, vector $b$ is added as an additional column to A forming the so-called *augmented* matrix $A^* = [A b]$. The augmented matrix in the example above is\n",
"\n",
"$$\n",
"\\left[\n",
@@ -211,10 +201,10 @@
"1 & 2 & -1 & 1 \\\\\n",
"3 & 11 & 5 & 35\\\\\n",
"\\end{matrix}\n",
"\\right]\n",
"\\right].\n",
"$$\n",
"\n",
"With this new notation, the goal of Gaussian elimination is to find the augmented matrix containing $U$ and $c$, namely $U^*= [U | c]$, given the augmented matrix $A^* = [A | b]$.\n",
"With this new notation, the goal of Gaussian elimination is to find the augmented matrix containing $U$ and $c$, namely $U^*= [U c]$, given the augmented matrix $A^* = [A b]$. These are $A^*$ and $U^*$ in our example:\n",
"\n",
"$$\n",
"A^*=\n",
@@ -232,7 +222,7 @@
"0 & 1 & 2 & 8\\\\\n",
"0 & 0 & 1 & 4\\\\\n",
"\\end{matrix}\n",
"\\right]\n",
"\\right].\n",
"$$\n",
"\n"
]
@@ -245,41 +235,32 @@
"### Serial implementation\n",
"\n",
"\n",
"The following algorithm computes the Gaussian elimination on a given augmented matrix `B`, representing a system of linear equations.\n",
"The following algorithm computes the Gaussian elimination on a given augmented matrix `B`, representing a system of linear equations. The result is given by overwriting `B`, avoiding the allocation of an additional matrix.\n",
"\n",
"- The outer loop is a loop over rows.\n",
"- The first inner loop in line 4 divides the current row by the value of the diagonal entry, thus transforming the diagonal to contain only ones. Note that we skip the first entries in the row, as we know that these values are zero at this point. The cells updated in this loop at iteration $k$ are depicted in red in the figure below.\n",
"- The second inner loop beginning in line 8 substracts the rows from one another such that all entries below the diagonal become zero. The entries updated in this loop are depicted in blue in the figure below."
"- The second inner loop beginning in line 8 subtracts the rows from one another such that all entries below the diagonal become zero. The entries updated in this loop are depicted in blue in the figure below."
]
},
{
"cell_type": "code",
"execution_count": 13,
"execution_count": null,
"id": "e4070214",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"gaussian_elimination! (generic function with 1 method)"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"function gaussian_elimination!(B)\n",
" n,m = size(B)\n",
" @inbounds for k in 1:n\n",
" for t in k:m\n",
" for t in (k+1):m\n",
" B[k,t] = B[k,t]/B[k,k]\n",
" end\n",
" B[k,k] = 1\n",
" for i in (k+1):n \n",
" for j in k:m\n",
" for j in (k+1):m\n",
" B[i,j] = B[i,j] - B[i,k]*B[k,j]\n",
" end\n",
" B[i,k] = 0\n",
" end\n",
" end\n",
" B\n",
@@ -288,17 +269,17 @@
},
{
"attachments": {
"g30822.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAzAAAAJsCAYAAAAiM0YjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15nBx1nf/xV/X0zOS+LxJOEw4JEORGEMEFFEXwRF1wFQWPXd0VcfHADU0WVERWRV1XQMRjXcFdEfH4KaCcchiCAoKAAiEhCQFykWuurt8fNQM9ne6eqU73VNXM6/l49CNd1d/v1LtnJj396W/V9xsgSRru8sDZJdt/Aa7rvd8OnAy8G9gD2AkoAo8DvwIuBZ6u8nX3A84AXg3sBowCVgP3AN8Hrm3kk5AkSZI0MowGwpLbj3r37wM8XPZY+W0D8LayrzcK+E+gZ4C+PwfGNOk5SZIkSRqmKhUwBwGbqF2A9N26geN7v9Yo4KZB9guBXwJBk5+fJEmSpGGkvID5DfBE7/2twOVEoyyHA8cAnwZWlfVZ2vt1vl2y7z6iU9OOBQ4FTiE6Na28iDmt+U9RkiRJ0nBRXsD03R4Fdq/SZ2bv46Xtf9D7bxH4VyBXpe8ny/rd14gnIUmSJGlkqFTArCG6YL+WEyr0C4FFA/QLgHvL+uxZZ3ZJkiRJI0ylAuZjg+iXY9tTyZYBrYPo+7Gyfn8fO7UkSRVUG/6XJA1fPcBVg2hXBJaU7fs+0DWIvn8o23YERpLUEBYwkjTyPAisG2Tb5WXbd9TZb9Ig+0mSVJMFjCSNPNUWpqxkY9n2ikH221S2PT7GMSVJqsoCRpJGnvKipJZinX3L+7kWjCSpISxgJGnkCZMOIElSvSxgJEmSJGWGBYwkSZKkzLCAkSRJkpQZFjCSJEmSMsMCRpIkSVJmWMBIkiRJygwLGEmSJEmZYQEjSZIkKTMsYCRJkiRlhgWMJEmSpMzIJx1AktR0IbC2ZHtTjL5byvr21HnMzTGOKUmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJA2N0cCpwM+BtoSzSJKk+KYCvwPeTfR3XZKGnQA4CrgWWAVsBf4v0USSJGl73Eb09/wZ4Drg1UAu0USS1ADzgEuAZcCzQNh7WwXslGAuSZK0fQ6g/9/2NcBK4GvAHgnmkqTYJgMfAf5MVKh089KLW9/twcTSSZKkRnmKbf/GF4HngUeBjwFTEkunYS9IOoAyrRU4AVhI9KnLqN591dwOfH8IckmSpOb5GPDyGo93A1uAB4AvAr8EuoYgl0YICxjV6zDg10QX5I9KOIskSUqnTqJrZg4BHkk4i4YJCxjVKwccA3wcOAIYB7QM0Odxok9jJElSdh0KzBqgTRewFvgFcCnwx2aHkqQ4RgHvAH4PrCZ60So/NzYE7kgqoCRJapjHqPx3fgvRBf13AqdQ+7RySUqNHYHzgSfpP0tJ3yxkYxNLJkmSttdsor/nfX/be4imVH4Q+DAwMblokrT9DgK+R/RpzHqic2A/mWgiSZK0Pf6LqGh5jmi5hIuBlyWaSJKaoBU4GbiFaIrFtmTjSJKkOkwluq7lx0TXv3ottaQRoT3pAJIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZI0kMnAmpLbN5KNI0lSOuSTDiBJqiggKmL6jEkqiCRJaZJLOoAkSZIkDZYFjCRJkqTMsICRJEmSlBkWMJIkSZIywwJGkiRJUmZYwEiSJEnKDKdRlqThZ0dgZtm+B4GOBLJIkiRJGgGmAGHJ7TuD7HccsL6kXxEoNCGfJEmSJL2ongLmvUBnSZ8O4O+blE+SJEmSXhSngAmIRllK268Fjm5qQkmSJEnqNdgCJg9cVtb2aWDBEGSUJEmSJGBwBcw44Jdl7R4AdhqijJIkSZIEDFzA7AAsKWtzIzBxCDNKkiRJElC7gJkPLC17/LtA6xBnlCRJkiSgegFzDNEF+qWPfZXoQn5JkiRJSkSlAubtwJaSfd3Ah5MKKEmSJEl9yguYlUSLUvZtbwTekFg6SZIkSSpRXsCU324A2hJLJ0mSJEklyguYO4hGYUr3/QIYlVRASZIkSepT6RqY3YDHy/bfDIxPJqIkSZIkRarNQrYz8FjZY7cBExLIKEmSJElA7XVgZgEPlD2+GJg6xBklSZIkCahdwADMAP5Y1ubPwA5DmFGSJEmSgIELGIDJwF1l7R4G5gxRRkmSJEkCBlfAAEwEbi9r+zjRBf+SJEmSNCQGW8AAjCFaF6a0/VJgXpMzSpIkSRIQr4ABaAd+WtZnJbBPEzNKkiRJEhC/gAFoA/63rN8zwIImZZQkSZIkoL4CBqAFuKqs7xrg0MZHlCRp6AVJB5AkVZQDdi3ZfgF4dpB9W4BdyvZtBlZtfyxJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJUnq1AlOSDiFJkiQpnfJJByjzQeBNwLFJB9FghbsBeySdQpIkpdrtEGxKOoSGhyDpAGUeAmYC84C1CWfRoISfBL6QdApJkpRq8yF4KOkQGh5ySQcoMQ+YBowFTks4iwYUjoZwAvz0mKSTSJKktPv0CdH7hrA16STKvjSNwPwHcFbv/SeAlyWYRQMKrwNOSjqFJEnKlDMhuCLpEMq2tIzABMDbS7Yn4nUVkiRJksqk5SL+VwGjS7YnE13Qf3YycVSHJcA/JB1iWwf8HHb+X/jpVUknqezYT8OGfeCeU5NOUtm3doKvXwsPHJR0kur2uxve8R449y9JJ6ns0KtgzFL43XlJJ6nslFPgsffDfa9NOkl1+/4JHliQdIrq9r8RXv4N+J9rk05S2asvhI4ZcNeZSSep7NwFcP234P7Dkk5S3b73wr+dAKesTjpJZQf/CCYvht98KekkZcYDdyYdQmqWnwBh2W056TrFTf2E10EYltxuSzpRFY+Q7kL4UtL94j4XSPusMRuA/ZMOUcPNwOVJh6jhQ0Sn7aZZV9IBBvAUcHrSIWq4Crgh6RA1HAasSzrEALYAc5IOUcM9wMVJh9hWOKnsvUII4RlJp1L2peEUslHAKyvsHwMcOcRZJEmSJKVYGgqYtxLNPFZuMvCxIc4iSZIkKcXSUMB8DBhX5bEjiEZiJEmSJCnxAmYmsFONx8cBbx6iLJIkSZJSLulZyN5PdKpYNWOBfwH+e2jiKGEzgb2Blt7tR4BlycWRJElS2iRdwJxJNAr0DNFMRzsCbcDjRMXLWGB3YBawKqGMap4jgGOAg4ADiX7+pT4KfH2oQ0mSJCm9kjyFbD6wA/BlYAHRdK3P9j72dmAXojewPUBK18jQdjoP+HfgZLYtXiRJkqRtJDkC8zjwMmBFlcc7iOauvwaYOESZlIxO4H7gXqLC9XXJxpEkSVJaJTkCs4XqxUupzcDKJmdRMr5EdOrYOOBgogX17ko0kSRJklIt6WtgNLL9JukAkiRJypakp1GWJEmSpEGzgJEkSZKUGRYwkiRJkjLDAkaSJElSZljASJIkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFjCRJkqTMsICRJEmSlBkWMJIkSZIywwJGkiRJUmZYwEiSJEnKDAsYSZIkSZlhAaMknQyEZbdCyeNfq/D4kUMbUZIkSWliASNJkiQpM/JJB9CI9hDwqZh9nmxCDkmSJGWEBYyS9BhwUdIhJEmSlB2eQiZJkiQpMyxgJEmSJGWGBYwkSZKkzLCAkSRJkpQZFjCSJEmSMsMCRpIkSVJmWMBIkiRJygzXgVGDLJ8FfCjpFBVMAY4CNiUdpIpXAHNI5/cOYCbQSnrzAbQD7wAOSzpIFTsDo0nv9/A1wETSmw+iD9vSnG8CcBzR72IazQcmkd7v4Tyi711a80H0fuk9wJqkg1SxA3Awqfse7j46WvJNaqwg6QBllhO9mTsQWJJwFtUUXgec9NL2PVvh0FWJxaluJ2AjsDbpIFVMI/rD/XTSQapoBWYDS5MOUsMuwCqgI+kgVcwGuoHVSQepYmLv7amkg9SwK/Bkwhlq2RlYB2xIOkgVM4jegK9IOkgVo4g+LEnz68yuRO9RuhPOUc0cotfA55IO0t+UHDy/c9nOMyG4IpE4UpMsB0LggKSDaCDhdRCGJbfbkk5UxSPA2UmHqOFS4M6kQ9Qwl/SOXvXZAOyfdIgabgYuTzpEDR8Cnkg6xAC6kg4wgKeA05MOUcNVwA1Jh6jhMKICMM22EBUJaXUPcHHSIbYVTip7rxBCeEbSqZR9XgMjSZIkKTMsYCRJkiRlhgWMJEmSpMywgJEkSZKUGU6jLA1sR2APXir47yKa2UySJElDzAJG2tYbiObTP4hoSu9ZZY/vD/xpqENJkiTJAkaq5H+J1iWQJElSyngNjFTdFqLTxb6BSwlLkiSlgiMw0rY+CPwReIiXVl2+Adg9sUSSJEkCLGCkSr6XdABJkiRV5ilkkiRJkjLDAkaSJElSZljASJIkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFjCRJkqTMsICRJEmSlBkWMJIkSZIywwJGkiRJUmZYwEiSJEnKDAsYSZIkSZlhASNJkiQpMyxgJEmSJGVGPukAUgotAj5Stm9cyf1bgZ6S7RCY2uxQkiRJsoCRKhkNTK7x+ISy7bCJWSRJklTCAkba1u/oP8IyEAsYSZKkIWIBI23rl703SZIkpYwX8UuSJEnKDAsYSZIkSZlhASNJLine truncated
"g27851.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAzAAAAJsCAYAAAAiM0YjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15nBx1nf/xV/X0zOS+LxIIwYRDDkFuFBFUUFbE+wQPFBRXXQVcL1xss3ggrgfeeKKuq7grIh4/BeUSOQxBbgEFQgIkAZJJyDVX1++PmoGeTnfPVKe7q2rm9Xw8+pGu6u936t09k5n+9Lfq+w2QJI12eeCsku2/A5cO3O8EXgG8GdgD2BkoAvcDvwMuAB6u8nWfBZwKPB94xsDXWgvcBPwQ+GUjn4QkSZKksWE8EJbcfjqwf1/g7rLHym8bgdeUfb1xwNeB/mH6/hqY0KTnJEmSJGmUqlTAHAxspnYBMnjrA44b+FrjgD+OsF8I/BYImvz8JEmSJI0i5QXMH4AHBu5vA75NNMpyBHAM8FFgdVmfFQNf57sl+24hOjXtRcBhwOuITk0rL2JObv5TlCRJkjRalBcwg7d7gd2r9Jk78Hhp+x8P/FsE/h3IVen74bJ+tzTiSUiSJEkaGyoVMOuAXYbpd3yFfiGwdJh+AXBzWZ8968wuSZIkaYypVMB8YAT9cmx/KtlKoH0EfT9Q1u9NsVNLklRBteF/SdLo1Q/8YATtisDysn0/AnpH0PevZduOwEiSGsICRpLGnjuArhG2XVW2fV2d/aaNsJ8kSTVZwEjS2FNtYcpKNpVtPzLCfpvLtifHOKYkSVVZwEjS2FNelNRSrLNveT/XgpEkNYQFjCSNPWHSASRJqpcFjCRJkqTMsICRJEmSlBkWMJIkSZIywwJGkiRJUmZYwEiSJEnKDAsYSZIkSZlhASNJkiQpMyxgJEmSJGWGBYwkSZKkzLCAkSRJkpQZ+aQDSJKaLgTWl2xvjtF3a1nf/jqPuSXGMSVJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkqTWGA+cDFwGtCecRZIkxTcTuBJ4M9HfdUkadQLgKOASYDWwDfjfRBNJkqQdcS3R3/M1wKXA84FcookkqQGWAF8AVgKPAeHAbQ2wc4K5JEnSjjmQoX/b1wGPAl8B9kgwlyTFNh14L3An0WhLH0//chu83ZFYOkmS1CgPsf3f+CLwBHAv8AFgRmLpNOoFSQdQprUDxwPnEH3qMo7a17f8GfhRC3JJkqTm+QDwzBqP9wFbgduBzwG/BXpbkEtjhAWM6nU48AeigmVcwlkkSVI69RBdM3MocE/CWTRKWMCoXjngGOBM4LnAJKBtmD73E30aI0mSsuswYN4wbXqB9cBvgAuAvzU7lCTFMQ54PfAXYC3RL63yc2ND4LqkAkqSpIa5j8p/57cSXdB/PfA6XDZBUkbsDCwFHmToLCUh0cX9ExNLJkmSdtR8or/ng3/b+4lmGb0DeDcwNblokrTjDia6aH81sIHoHNgPJ5pIkiTtiG8SFS2PEy2XcD7wjEQTSVITtAMvB64mmmLRIWVJkrJnJtF1LT8nuv7Va6kljQmdSQeQJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSpOFMB9aV3L6WbBxJktIhn3QASVJFAVERM2hCUkEkSUqTXNIBJEmSJGmkLGAkSZIkZYYFjCRJkqTMsICRJEmSlBkWMJIkSZIywwJGkiRJUmY4jbIkjT47A3PL9t0BdCeQRZIkSdIYMAMIS27fH2G/Y4ENJf2KQKEJ+SRJkiTpKfUUMG8Dekr6dANvalI+SZIkSXpKnAImIBplKW2/Hji6qQklSZIkacBIC5g8cGFZ24eB/VuQUZIkSZKAkRUwk4DflrW7HdilRRklSZIkCRi+gNkJWF7W5gpgagszSpIkSRJQu4DZB1hR9vhFQHuLM0qSJEkSUL2AOYboAv3Sx75MdCG/JEmSJCWiUgHzWmBryb4+4N1JBZQkSZKkQeUFzKNEi1IObm8CXppYOkmSJEkqUV7AlN8uBzoSSydJkiRJJcoLmOuIRmFK9/0GGJdUQEmSJEkaVOkamN2A+8v2XwVMTiaiJEmSJEWqzUK2ELiv7LFrgSkJZJQkSZIkoPY6MPOA28seXwbMbHFGSZIkSQJqFzAAc4C/lbW5E9iphRklSZIkCRi+gAGYDtxQ1u5uYEGLMkqSJEkSMLICBmAq8OeytvcTXfAvSZIkSS0x0gIGYALRujCl7VcAS5qcUZIkSZKAeAUMQCfwy7I+jwL7NjGjJEmSJAHxCxiADuB/y/qtAfZvUkZJkiRJAuorYADagB+U9V0HHNb4iJIktV6QdABJUkU5YFHJ9pPAYyPs2wbsWrZvC7B6x2NJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJktKrHZiRdAhJkiRJ6ZRPOkCZdwGvAF6UdBCNVLgbsEfSKSRJUqr9GYLNSYfQ6BAkHaDMXcBcYAmwPuEsGpHww8Bnk04hSZJSbR8I7ko6hEaHXNIBSiwBZgETgZMTzqJhheMhnAK/PCbpJJIkKe0+enz0viFsTzqJsi9NIzBfAM4YuP8A8IwEs2hY4aXAiUmnkCRJmXIaBN9JOoSyLS0jMAHw2pLtqXhdhSRJkqQyabmI/3nA+JLt6UQX9J+VTBzVYTnwlqRDbO/A38DCi+GXFyWdpLIXfRQ27gs3nZR0ksq+sRC+/gu4/eCkk1T3rBvh9W+Fs/+edJLKDvsBTHgQriwkHKSK170O7nsH3PLipJNUt9+tcPv+Saeo7oAr4Jlfg/+5JOkklT3/09A9G244LekklZ29P1z2Lbjt8KSTVLffzfAfx8Pr1iadpLJDfgrTl8EfPp90kjKTgeuTDiE1yyVAWHZbRbpOcdMQ4aUQhiW3a5NOVMW9wJlJh6jhAtL9y30JkPZZYzYCByQdooargAuTDlHD6USn7aZZb9IBhvEQcErSIWq4CLg86RA1HA50JR1iGFuBBUmHqOEm4PykQ2wvnFb2XiGE8NSkUyn70nAK2TjgiAr7JwBHtjiLJEmSpBRLQwHzGqKZx8pNBz7Q4iySJEmSUiwNBcz7gUlVHnsu0UiMJEmSJCVewMwFdqnx+CTglS3KIkmSJCnlkp6F7FSiU8WqmUg0QvPfrYmjhM0F9gbaBrbvAVYmF0eSJElpk3QBcxrRKNAaopmOdgY6gPuJipdJwO7APGB1QhnVPM8FjgEOBg4i+v6Xeh/w1VaHkiRJUnoleQrZvkSFyReB/YHFwGMDj70W2BV4L9APpHSNDO2gTwD/Cbyc7YsXSZIkaTtJjsDcD+wGPFrl8W7gB8DFwNQWZVIyeoDbgJuJCteXJBtHkiRJaZXkCMwWqhcv9bRT9nye6NSxScAhRAvq3ZBoIkmSJKVa0tfAaGz7Q9IBJEmSlC1JT6MsSZIkSSNmASNJkiQpMyxgJEmSJGWGBYwkSZKkzLCAkSRJkpQZFjCSJEmSMsMCRpIkSVJmWMBIkiRJygwLGEmSJEmZYQEjSZIkKTMsYCRJkiRlhgWMJEmSpMywgJEkSZKUGRYwkiRJkjLDAkZJejkQlt0KJY9/pcLjR7Y2oiRJktLEAkaSJElSZuSTDqAx7S7gIzH7PNiEHJIkScoICxgl6T7gvKRDSJIkKTs8hUySJElSZljASJIkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFjCRJkqTMsICRJEmSlBmuA6MGWTUPOD3pFBXMAJ4PbEk6SBXPBhaQztcOYC7QTnrzAXQCrwcOTzpIFQuBcaT3NXwBMJX05oPow7Y055sCHEv0s5hG+5Du7/ESotcurfkger/0VmBd0kGq2Ak4hNS9hruPj5Z8kxorSDpAmVVEb+YOApYnnEU1hZcCJz69fdM2OGx1YnGq2wXYBKxPOkgVs4j+cD+cdJAq2oH5wIqkg9SwK7Aa6E46SBXzgT5gbdJBqpg6cHso6SA1LAIeTDhDLQuBLmBj0kGqmAu0AY8kHaSKcUQZ0/x7ZhHRe5S+hHNUs4Dod+DjSQcZakYOnlhYtvM0CL6TSBypSVYBIXBg0kE0nPBSCMOS27VJJ6riXuDMpEPUcAFwfdIhalgCbE46xDA2AgckHaKGq4ALkw5Rw+nAA0mHGEZv0gGG8RBwStIhargIuDzpEDUcTlQAptlWoiIhrW4Czk86xPbCaWXvFUIIT006lbLPa2AkSZIkZYYFjCRJkqTMsICRJEmSlBkWMJIkSZIyw2mUpeHtDOzB0wX/DUQzm0mSJKnFLGCk7b2UaD79g4mm9J5X9vgBwK2tDiVJkiQLGKmS/yVal0CSJEkp4zUwUnVbiU4X+xouJSxJkpQKjsBI23sX8DfgLp5edflyYPfEEkmSJAmwgJEq+WHSASRJklSZp5BJkiRJygwLGEmSJEmZYQEjSZIkKTMsYCRJkiRlhgWMJEmSpMywgJEkSZKUGRYwkiRJkjLDAkaSJElSZljASJIkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFjCRJkqTMsICRJEmSlBn5pANIKbQUeG/Zvkkl968B+ku2Q2Bms0NJkiTJAkaqZDwwvcbjU8q2wyZmkSRJUgkLGGl7VzJ0hGU4FjCSJEktYgEjbe+3AzdJkiSljBfxS5IkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFLine truncated
}
},
"cell_type": "markdown",
"id": "ffb84b53",
"id": "cc6ce65e",
"metadata": {},
"source": [
"<div>\n",
"<img src=\"attachment:g30822.png\" align=\"left\" width=\"300\"/>\n",
"</div>\n"
"<img src=\"attachment:g27851.png\" align=\"left\" width=\"300\"/>\n",
"</div>"
]
},
{
@@ -307,7 +288,7 @@
"metadata": {},
"source": [
"<div class=\"alert alert-block alert-info\">\n",
"<b>Note:</b> This algorithm is not correct for all matrices: if any diagonal element <code>B[k,k]</code> is zero, the computation in the first inner loop fails. To get around this problem, another step can be added to the algorithm that swaps the rows until the diagonal entry of the current row is not zero. This process of finding a nonzero value is called <b>pivoting</b>. We are not going to consider pivoting here for simplicity. \n",
"<b>Note:</b> This algorithm is not correct for all nonsingular matrices: if any diagonal element <code>B[k,k]</code> is zero, the computation in the first inner loop fails. To get around this problem, another step can be added to the algorithm that swaps the rows until the diagonal entry of the current row is not zero. This process of finding a nonzero value is called <b>pivoting</b>. We are not going to consider pivoting here for simplicity. \n",
"</div>"
]
},
@@ -321,24 +302,10 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": null,
"id": "eb30df0d",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"3×4 Matrix{Float64}:\n",
" 1.0 3.0 1.0 9.0\n",
" 0.0 1.0 -1.0 1.0\n",
" 0.0 0.0 1.0 35.0"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"A = Float64[1 3 1; 1 2 -1; 3 11 5]\n",
"b = Float64[9,1,35]\n",
@@ -373,15 +340,15 @@
"\n",
"```julia\n",
"n,m = size(B)\n",
"for k in 1:n\n",
" for t in k:m\n",
" B[k,t] = B[k,t]/B[k,k]\n",
" end\n",
" for i in (k+1):n \n",
" for j in k:m\n",
" B[i,j] = B[i,j] - B[i,k]*B[k,j]\n",
" end\n",
"for t in (k+1):m\n",
" B[k,t] = B[k,t]/B[k,k]\n",
"end\n",
"B[k,k] = 1\n",
"for i in (k+1):n \n",
" for j in (k+1):m\n",
" B[i,j] = B[i,j] - B[i,k]*B[k,j]\n",
" end\n",
" B[i,k] = 0\n",
"end\n",
"```"
]
@@ -403,37 +370,21 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": null,
"id": "078e974e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"🥳 Well done! \n"
]
}
],
"outputs": [],
"source": [
"answer = \"a\" # replace x with a, b, c, or d \n",
"answer = \"x\" # replace x with a, b, c, or d \n",
"ge_par_check(answer)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"execution_count": null,
"id": "1169c86e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The outer loop of the algorithm is not parallelizable, since the iterations depend on the results of the previous iterations. However, we can extract parallelism from the inner loops.\n"
]
}
],
"outputs": [],
"source": [
"ge_par_why()"
]
@@ -445,7 +396,7 @@
"source": [
"### Data partition\n",
"\n",
"Let start considering a row-wise block partition, as we did for the previous algorithms.\n",
"Let start considering a row-wise block partition, as we did in previous algorithms.\n",
"\n",
"In the figure below, we use different colors to illustrate which entries are assigned to a CPU. All entries with the same color are assigned to the same CPU."
]
@@ -501,7 +452,7 @@
"<b>Definition:</b> *Load imbalance*: is the problem when work is not equally distributed over all processes and consequently some processes do more work than others.\n",
"</div>\n",
"\n",
"Having processors waiting for others is a wast of computational resources and affects negatively parallel speedups. The optimal speedup (speedup equal to the number of processors) assumes that the work is perfectly parallel and that it is evenly distributed. If there is load imbalance, the last assumption is not true anymore and the speedup will be suboptimal.\n"
"Having processors waiting for others is a waist of computational resources and affects negatively parallel speedups. The optimal speedup (speedup equal to the number of processors) assumes that the work is perfectly parallel and that it is evenly distributed. If there is load imbalance, the last assumption is not true anymore and the speedup will be suboptimal.\n"
]
},
{
@@ -511,9 +462,9 @@
"source": [
"### Fixing load imbalance\n",
"\n",
"In this application is relatively easy to fix the load imbalance problem. We know in advance which data is going to be processes at each CPU and we can design a more clever data partition.\n",
"In this application, is relatively easy to fix the load imbalance problem. We know in advance which data is going to be processes at each CPU and we can design a more clever data partition.\n",
"\n",
"We can consider row-wise cyclic partition to fix the problem. See figure below. In this case, the CPUs will have less work as the value of $k$ increases, but amount of work will be better distributed than with the block partition."
"We can consider row-wise cyclic partition to fix the problem. See figure below. In this case, the CPUs will have less work as the value of $k$ increases, but amount of work will be better distributed than with the previous row block partition."
]
},
{
@@ -555,18 +506,10 @@
},
{
"cell_type": "code",
"execution_count": 19,
"execution_count": null,
"id": "a6741a25",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"It is a form of static load balancing. We know in advance the load distribution and the partition strategy does not depend on the actual values of the input matrix\n"
]
}
],
"outputs": [],
"source": [
"ge_lb_answer()"
]
@@ -578,46 +521,46 @@
"source": [
"### Data dependencies\n",
"\n",
"Using a cyclic partition, we managed to distribute the work uniformly. But we still need to study the data dependencies in order to implement it efficiently.\n",
"Using a cyclic partition, we managed to distribute the work uniformly. But we still need to study the data dependencies in order to implement this algorithm in parallel efficiently.\n",
"\n",
"Look again to the algorithm\n",
"\n",
"```julia\n",
"n,m = size(B)\n",
"for k in 1:n\n",
" for t in k:m\n",
" B[k,t] = B[k,t]/B[k,k]\n",
" end\n",
" for i in (k+1):n \n",
" for j in k:m\n",
" B[i,j] = B[i,j] - B[i,k]*B[k,j]\n",
" end\n",
"for t in (k+1):m\n",
" B[k,t] = B[k,t]/B[k,k]\n",
"end\n",
"B[k,k] = 1\n",
"for i in (k+1):n \n",
" for j in (k+1):m\n",
" B[i,j] = B[i,j] - B[i,k]*B[k,j]\n",
" end\n",
" B[i,k] = 0\n",
"end\n",
"```\n",
"\n",
"Note that all updates on the loop over i and j we do the following update \n",
"Note that all updates on the loop over i and j we do the following: \n",
"\n",
"```julia\n",
"B[i,j] = B[i,j] - B[i,k]*B[k,j]\n",
"```\n",
"\n",
"As we are using a row-wise partitions, the CPU that updates `B[i,j]` will also have entry `B[i,k]` in memory (both are in the same row). However, `B[k,j]` is in another row and it might be located on another processor. We might need to communicate `B[k,j]` for `j=k:m`. This corresponds to the cells marked in red in the figure below. These red entries are the data dependencies of this algorithm. The owner of these entries will send them to the other processors. This is very similar to the communications seen in previous notebook for Floyd's algorithm. There is a key difference however, in the current case we do not need to send the full row, only the entries beyond column $k$ (the red cells in the figure).\n"
"As we are using row-wise partitions, the CPU that updates `B[i,j]` will also have entry `B[i,k]` in memory (both are in the same row). However, `B[k,j]` is in another row and it might be located on another processor. We might need to communicate `B[k,j]` for `j=(k+1):m`. This corresponds to the cells marked in red in the figure below. These red entries are the data dependencies of this algorithm. The owner of these entries has to send them to the other processors. This is very similar to the communication pattern seen in previous notebook for Floyd's algorithm. There is a key difference however. In the current case, we do not need to send the full row, only the entries beyond column $k$ (the red cells in the figure).\n"
]
},
{
"attachments": {
"g30822.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAzAAAAJsCAYAAAAiM0YjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15nBx1nf/xV/X0zOS+LxJOEw4JEORGEMEFFEXwRF1wFQWPXd0VcfHADU0WVERWRV1XQMRjXcFdEfH4KaCcchiCAoKAAiEhCQFykWuurt8fNQM9ne6eqU73VNXM6/l49CNd1d/v1LtnJj396W/V9xsgSRru8sDZJdt/Aa7rvd8OnAy8G9gD2AkoAo8DvwIuBZ6u8nX3A84AXg3sBowCVgP3AN8Hrm3kk5AkSZI0MowGwpLbj3r37wM8XPZY+W0D8LayrzcK+E+gZ4C+PwfGNOk5SZIkSRqmKhUwBwGbqF2A9N26geN7v9Yo4KZB9guBXwJBk5+fJEmSpGGkvID5DfBE7/2twOVEoyyHA8cAnwZWlfVZ2vt1vl2y7z6iU9OOBQ4FTiE6Na28iDmt+U9RkiRJ0nBRXsD03R4Fdq/SZ2bv46Xtf9D7bxH4VyBXpe8ny/rd14gnIUmSJGlkqFTArCG6YL+WEyr0C4FFA/QLgHvL+uxZZ3ZJkiRJI0ylAuZjg+iXY9tTyZYBrYPo+7Gyfn8fO7UkSRVUG/6XJA1fPcBVg2hXBJaU7fs+0DWIvn8o23YERpLUEBYwkjTyPAisG2Tb5WXbd9TZb9Ig+0mSVJMFjCSNPNUWpqxkY9n2ikH221S2PT7GMSVJqsoCRpJGnvKipJZinX3L+7kWjCSpISxgJGnkCZMOIElSvSxgJEmSJGWGBYwkSZKkzLCAkSRJkpQZFjCSJEmSMsMCRpIkSVJmWMBIkiRJygwLGEmSJEmZYQEjSZIkKTMsYCRJkiRlhgWMJEmSpMzIJx1AktR0IbC2ZHtTjL5byvr21HnMzTGOKUmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJA2N0cCpwM+BtoSzSJKk+KYCvwPeTfR3XZKGnQA4CrgWWAVsBf4v0USSJGl73Eb09/wZ4Drg1UAu0USS1ADzgEuAZcCzQNh7WwXslGAuSZK0fQ6g/9/2NcBK4GvAHgnmkqTYJgMfAf5MVKh089KLW9/twcTSSZKkRnmKbf/GF4HngUeBjwFTEkunYS9IOoAyrRU4AVhI9KnLqN591dwOfH8IckmSpOb5GPDyGo93A1uAB4AvAr8EuoYgl0YICxjV6zDg10QX5I9KOIskSUqnTqJrZg4BHkk4i4YJCxjVKwccA3wcOAIYB7QM0Odxok9jJElSdh0KzBqgTRewFvgFcCnwx2aHkqQ4RgHvAH4PrCZ60So/NzYE7kgqoCRJapjHqPx3fgvRBf13AqdQ+7RySUqNHYHzgSfpP0tJ3yxkYxNLJkmSttdsor/nfX/be4imVH4Q+DAwMblokrT9DgK+R/RpzHqic2A/mWgiSZK0Pf6LqGh5jmi5hIuBlyWaSJKaoBU4GbiFaIrFtmTjSJKkOkwluq7lx0TXv3ottaQRoT3pAJIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZIkSZI0kMnAmpLbN5KNI0lSOuSTDiBJqiggKmL6jEkqiCRJaZJLOoAkSZIkDZYFjCRJkqTMsICRJEmSlBkWMJIkSZIywwJGkiRJUmZYwEiSJEnKDKdRlqThZ0dgZtm+B4GOBLJIkiRJGgGmAGHJ7TuD7HccsL6kXxEoNCGfJEmSJL2ongLmvUBnSZ8O4O+blE+SJEmSXhSngAmIRllK268Fjm5qQkmSJEnqNdgCJg9cVtb2aWDBEGSUJEmSJGBwBcw44Jdl7R4AdhqijJIkSZIEDFzA7AAsKWtzIzBxCDNKkiRJElC7gJkPLC17/LtA6xBnlCRJkiSgegFzDNEF+qWPfZXoQn5JkiRJSkSlAubtwJaSfd3Ah5MKKEmSJEl9yguYlUSLUvZtbwTekFg6SZIkSSpRXsCU324A2hJLJ0mSJEklyguYO4hGYUr3/QIYlVRASZIkSepT6RqY3YDHy/bfDIxPJqIkSZIkRarNQrYz8FjZY7cBExLIKEmSJElA7XVgZgEPlD2+GJg6xBklSZIkCahdwADMAP5Y1ubPwA5DmFGSJEmSgIELGIDJwF1l7R4G5gxRRkmSJEkCBlfAAEwEbi9r+zjRBf+SJEmSNCQGW8AAjCFaF6a0/VJgXpMzSpIkSRIQr4ABaAd+WtZnJbBPEzNKkiRJEhC/gAFoA/63rN8zwIImZZQkSZIkoL4CBqAFuKqs7xrg0MZHlCRp6AVJB5AkVZQDdi3ZfgF4dpB9W4BdyvZtBlZtfyxJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJUnq1AlOSDiFJkiQpnfJJByjzQeBNwLFJB9FghbsBeySdQpIkpdrtEGxKOoSGhyDpAGUeAmYC84C1CWfRoISfBL6QdApJkpRq8yF4KOkQGh5ySQcoMQ+YBowFTks4iwYUjoZwAvz0mKSTSJKktPv0CdH7hrA16STKvjSNwPwHcFbv/SeAlyWYRQMKrwNOSjqFJEnKlDMhuCLpEMq2tIzABMDbS7Yn4nUVkiRJksqk5SL+VwGjS7YnE13Qf3YycVSHJcA/JB1iWwf8HHb+X/jpVUknqezYT8OGfeCeU5NOUtm3doKvXwsPHJR0kur2uxve8R449y9JJ6ns0KtgzFL43XlJJ6nslFPgsffDfa9NOkl1+/4JHliQdIrq9r8RXv4N+J9rk05S2asvhI4ZcNeZSSep7NwFcP234P7Dkk5S3b73wr+dAKesTjpJZQf/CCYvht98KekkZcYDdyYdQmqWnwBh2W056TrFTf2E10EYltxuSzpRFY+Q7kL4UtL94j4XSPusMRuA/ZMOUcPNwOVJh6jhQ0Sn7aZZV9IBBvAUcHrSIWq4Crgh6RA1HAasSzrEALYAc5IOUcM9wMVJh9hWOKnsvUII4RlJp1L2peEUslHAKyvsHwMcOcRZJEmSJKVYGgqYtxLNPFZuMvCxIc4iSZIkKcXSUMB8DBhX5bEjiEZiJEmSJCnxAmYmsFONx8cBbx6iLJIkSZJSLulZyN5PdKpYNWOBfwH+e2jiKGEzgb2Blt7tR4BlycWRJElS2iRdwJxJNAr0DNFMRzsCbcDjRMXLWGB3YBawKqGMap4jgGOAg4ADiX7+pT4KfH2oQ0mSJCm9kjyFbD6wA/BlYAHRdK3P9j72dmAXojewPUBK18jQdjoP+HfgZLYtXiRJkqRtJDkC8zjwMmBFlcc7iOauvwaYOESZlIxO4H7gXqLC9XXJxpEkSVJaJTkCs4XqxUupzcDKJmdRMr5EdOrYOOBgogX17ko0kSRJklIt6WtgNLL9JukAkiRJypakp1GWJEmSpEGzgJEkSZKUGRYwkiRJkjLDAkaSJElSZljASJIkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFjCRJkqTMsICRJEmSlBkWMJIkSZIywwJGkiRJUmZYwEiSJEnKDAsYSZIkSZlhAaMknQyEZbdCyeNfq/D4kUMbUZIkSWliASNJkiQpM/JJB9CI9hDwqZh9nmxCDkmSJGWEBYyS9BhwUdIhJEmSlB2eQiZJkiQpMyxgJEmSJGWGBYwkSZKkzLCAkSRJkpQZFjCSJEmSMsMCRpIkSVJmWMBIkiRJygzXgVGDLJ8FfCjpFBVMAY4CNiUdpIpXAHNI5/cOYCbQSnrzAbQD7wAOSzpIFTsDo0nv9/A1wETSmw+iD9vSnG8CcBzR72IazQcmkd7v4Tyi711a80H0fuk9wJqkg1SxA3Awqfse7j46WvJNaqwg6QBllhO9mTsQWJJwFtUUXgec9NL2PVvh0FWJxaluJ2AjsDbpIFVMI/rD/XTSQapoBWYDS5MOUsMuwCqgI+kgVcwGuoHVSQepYmLv7amkg9SwK/Bkwhlq2RlYB2xIOkgVM4jegK9IOkgVo4g+LEnz68yuRO9RuhPOUc0cotfA55IO0t+UHDy/c9nOMyG4IpE4UpMsB0LggKSDaCDhdRCGJbfbkk5UxSPA2UmHqOFS4M6kQ9Qwl/SOXvXZAOyfdIgabgYuTzpEDR8Cnkg6xAC6kg4wgKeA05MOUcNVwA1Jh6jhMKICMM22EBUJaXUPcHHSIbYVTip7rxBCeEbSqZR9XgMjSZIkKTMsYCRJkiRlhgWMJEmSpMywgJEkSZKUGU6jLA1sR2APXir47yKa2UySJElDzAJG2tYbiObTP4hoSu9ZZY/vD/xpqENJkiTJAkaq5H+J1iWQJElSyngNjFTdFqLTxb6BSwlLkiSlgiMw0rY+CPwReIiXVl2+Adg9sUSSJEkCLGCkSr6XdABJkiRV5ilkkiRJkjLDAkaSJElSZljASJIkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFjCRJkqTMsICRJEmSlBkWMJIkSZIywwJGkiRJUmZYwEiSJEnKDAsYSZIkSZlhASNJkiQpMyxgJEmSJGVGPukAUgotAj5Stm9cyf1bgZ6S7RCY2uxQkiRJsoCRKhkNTK7x+ISy7bCJWSRJklTCAkba1u/oP8IyEAsYSZKkIWIBI23rl703SZIkpYwX8UuSJEnKDAsYSZIkSZlhASNJLine truncated
"g27851.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAzAAAAJsCAYAAAAiM0YjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAewgAAHsIBbtB1PgAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAACAASURBVHic7N15nBx1nf/xV/X0zOS+LxIIwYRDDkFuFBFUUFbE+wQPFBRXXQVcL1xss3ggrgfeeKKuq7grIh4/BeUSOQxBbgEFQgIkAZJJyDVX1++PmoGeTnfPVKe7q2rm9Xw8+pGu6u936t09k5n+9Lfq+w2QJI12eeCsku2/A5cO3O8EXgG8GdgD2BkoAvcDvwMuAB6u8nWfBZwKPB94xsDXWgvcBPwQ+GUjn4QkSZKksWE8EJbcfjqwf1/g7rLHym8bgdeUfb1xwNeB/mH6/hqY0KTnJEmSJGmUqlTAHAxspnYBMnjrA44b+FrjgD+OsF8I/BYImvz8JEmSJI0i5QXMH4AHBu5vA75NNMpyBHAM8FFgdVmfFQNf57sl+24hOjXtRcBhwOuITk0rL2JObv5TlCRJkjRalBcwg7d7gd2r9Jk78Hhp+x8P/FsE/h3IVen74bJ+tzTiSUiSJEkaGyoVMOuAXYbpd3yFfiGwdJh+AXBzWZ8968wuSZIkaYypVMB8YAT9cmx/KtlKoH0EfT9Q1u9NsVNLklRBteF/SdLo1Q/8YATtisDysn0/AnpH0PevZduOwEiSGsICRpLGnjuArhG2XVW2fV2d/aaNsJ8kSTVZwEjS2FNtYcpKNpVtPzLCfpvLtifHOKYkSVVZwEjS2FNelNRSrLNveT/XgpEkNYQFjCSNPWHSASRJqpcFjCRJkqTMsICRJEmSlBkWMJIkSZIywwJGkiRJUmZYwEiSJEnKDAsYSZIkSZlhASNJkiQpMyxgJEmSJGWGBYwkSZKkzLCAkSRJkpQZ+aQDSJKaLgTWl2xvjtF3a1nf/jqPuSXGMSVJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkqTWGA+cDFwGtCecRZIkxTcTuBJ4M9HfdUkadQLgKOASYDWwDfjfRBNJkqQdcS3R3/M1wKXA84FcookkqQGWAF8AVgKPAeHAbQ2wc4K5JEnSjjmQoX/b1wGPAl8B9kgwlyTFNh14L3An0WhLH0//chu83ZFYOkmS1CgPsf3f+CLwBHAv8AFgRmLpNOoFSQdQprUDxwPnEH3qMo7a17f8GfhRC3JJkqTm+QDwzBqP9wFbgduBzwG/BXpbkEtjhAWM6nU48AeigmVcwlkkSVI69RBdM3MocE/CWTRKWMCoXjngGOBM4LnAJKBtmD73E30aI0mSsuswYN4wbXqB9cBvgAuAvzU7lCTFMQ54PfAXYC3RL63yc2ND4LqkAkqSpIa5j8p/57cSXdB/PfA6XDZBUkbsDCwFHmToLCUh0cX9ExNLJkmSdtR8or/ng3/b+4lmGb0DeDcwNblokrTjDia6aH81sIHoHNgPJ5pIkiTtiG8SFS2PEy2XcD7wjEQTSVITtAMvB64mmmLRIWVJkrJnJtF1LT8nuv7Va6kljQmdSQeQJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSJEmSpOFMB9aV3L6WbBxJktIhn3QASVJFAVERM2hCUkEkSUqTXNIBJEmSJGmkLGAkSZIkZYYFjCRJkqTMsICRJEmSlBkWMJIkSZIywwJGkiRJUmY4jbIkjT47A3PL9t0BdCeQRZIkSdIYMAMIS27fH2G/Y4ENJf2KQKEJ+SRJkiTpKfUUMG8Dekr6dANvalI+SZIkSXpKnAImIBplKW2/Hji6qQklSZIkacBIC5g8cGFZ24eB/VuQUZIkSZKAkRUwk4DflrW7HdilRRklSZIkCRi+gNkJWF7W5gpgagszSpIkSRJQu4DZB1hR9vhFQHuLM0qSJEkSUL2AOYboAv3Sx75MdCG/JEmSJCWiUgHzWmBryb4+4N1JBZQkSZKkQeUFzKNEi1IObm8CXppYOkmSJEkqUV7AlN8uBzoSSydJkiRJJcoLmOuIRmFK9/0GGJdUQEmSJEkaVOkamN2A+8v2XwVMTiaiJEmSJEWqzUK2ELiv7LFrgSkJZJQkSZIkoPY6MPOA28seXwbMbHFGSZIkSQJqFzAAc4C/lbW5E9iphRklSZIkCRi+gAGYDtxQ1u5uYEGLMkqSJEkSMLICBmAq8OeytvcTXfAvSZIkSS0x0gIGYALRujCl7VcAS5qcUZIkSZKAeAUMQCfwy7I+jwL7NjGjJEmSJAHxCxiADuB/y/qtAfZvUkZJkiRJAuorYADagB+U9V0HHNb4iJIktV6QdABJUkU5YFHJ9pPAYyPs2wbsWrZvC7B6x2NJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJkiRJktKrHZiRdAhJkiRJ6ZRPOkCZdwGvAF6UdBCNVLgbsEfSKSRJUqr9GYLNSYfQ6BAkHaDMXcBcYAmwPuEsGpHww8Bnk04hSZJSbR8I7ko6hEaHXNIBSiwBZgETgZMTzqJhheMhnAK/PCbpJJIkKe0+enz0viFsTzqJsi9NIzBfAM4YuP8A8IwEs2hY4aXAiUmnkCRJmXIaBN9JOoSyLS0jMAHw2pLtqXhdhSRJkqQyabmI/3nA+JLt6UQX9J+VTBzVYTnwlqRDbO/A38DCi+GXFyWdpLIXfRQ27gs3nZR0ksq+sRC+/gu4/eCkk1T3rBvh9W+Fs/+edJLKDvsBTHgQriwkHKSK170O7nsH3PLipJNUt9+tcPv+Saeo7oAr4Jlfg/+5JOkklT3/09A9G244LekklZ29P1z2Lbjt8KSTVLffzfAfx8Pr1iadpLJDfgrTl8EfPp90kjKTgeuTDiE1yyVAWHZbRbpOcdMQ4aUQhiW3a5NOVMW9wJlJh6jhAtL9y30JkPZZYzYCByQdooargAuTDlHD6USn7aZZb9IBhvEQcErSIWq4CLg86RA1HA50JR1iGFuBBUmHqOEm4PykQ2wvnFb2XiGE8NSkUyn70nAK2TjgiAr7JwBHtjiLJEmSpBRLQwHzGqKZx8pNBz7Q4iySJEmSUiwNBcz7gUlVHnsu0UiMJEmSJCVewMwFdqnx+CTglS3KIkmSJCnlkp6F7FSiU8WqmUg0QvPfrYmjhM0F9gbaBrbvAVYmF0eSJElpk3QBcxrRKNAaopmOdgY6gPuJipdJwO7APGB1QhnVPM8FjgEOBg4i+v6Xeh/w1VaHkiRJUnoleQrZvkSFyReB/YHFwGMDj70W2BV4L9APpHSNDO2gTwD/Cbyc7YsXSZIkaTtJjsDcD+wGPFrl8W7gB8DFwNQWZVIyeoDbgJuJCteXJBtHkiRJaZXkCMwWqhcv9bRT9nye6NSxScAhRAvq3ZBoIkmSJKVa0tfAaGz7Q9IBJEmSlC1JT6MsSZIkSSNmASNJkiQpMyxgJEmSJGWGBYwkSZKkzLCAkSRJkpQZFjCSJEmSMsMCRpIkSVJmWMBIkiRJygwLGEmSJEmZYQEjSZIkKTMsYCRJkiRlhgWMJEmSpMywgJEkSZKUGRYwkiRJkjLDAkZJejkQlt0KJY9/pcLjR7Y2oiRJktLEAkaSJElSZuSTDqAx7S7gIzH7PNiEHJIkScoICxgl6T7gvKRDSJIkKTs8hUySJElSZljASJIkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFjCRJkqTMsICRJEmSlBmuA6MGWTUPOD3pFBXMAJ4PbEk6SBXPBhaQztcOYC7QTnrzAXQCrwcOTzpIFQuBcaT3NXwBMJX05oPow7Y055sCHEv0s5hG+5Du7/ESotcurfkger/0VmBd0kGq2Ak4hNS9hruPj5Z8kxorSDpAmVVEb+YOApYnnEU1hZcCJz69fdM2OGx1YnGq2wXYBKxPOkgVs4j+cD+cdJAq2oH5wIqkg9SwK7Aa6E46SBXzgT5gbdJBqpg6cHso6SA1LAIeTDhDLQuBLmBj0kGqmAu0AY8kHaSKcUQZ0/x7ZhHRe5S+hHNUs4Dod+DjSQcZakYOnlhYtvM0CL6TSBypSVYBIXBg0kE0nPBSCMOS27VJJ6riXuDMpEPUcAFwfdIhalgCbE46xDA2AgckHaKGq4ALkw5Rw+nAA0mHGEZv0gGG8RBwStIhargIuDzpEDUcTlQAptlWoiIhrW4Czk86xPbCaWXvFUIIT006lbLPa2AkSZIkZYYFjCRJkqTMsICRJEmSlBkWMJIkSZIyw2mUpeHtDOzB0wX/DUQzm0mSJKnFLGCk7b2UaD79g4mm9J5X9vgBwK2tDiVJkiQLGKmS/yVal0CSJEkp4zUwUnVbiU4X+xouJSxJkpQKjsBI23sX8DfgLp5edflyYPfEEkmSJAmwgJEq+WHSASRJklSZp5BJkiRJygwLGEmSJEmZYQEjSZIkKTMsYCRJkiRlhgWMJEmSpMywgJEkSZKUGRYwkiRJkjLDAkaSJElSZljASJIkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFjCRJkqTMsICRJEmSlBn5pANIKbQUeG/Zvkkl968B+ku2Q2Bms0NJkiTJAkaqZDwwvcbjU8q2wyZmkSRJUgkLGGl7VzJ0hGU4FjCSJEktYgEjbe+3AzdJkiSljBfxS5IkScoMCxhJkiRJmWEBI0mSJCkzLGAkSZIkZYYFLine truncated
}
},
"cell_type": "markdown",
"id": "44639340",
"id": "0050b6c4",
"metadata": {},
"source": [
"<div>\n",
"<img src=\"attachment:g30822.png\" align=\"left\" width=\"300\"/>\n",
"</div>\n"
"<img src=\"attachment:g27851.png\" align=\"left\" width=\"300\"/>\n",
"</div>"
]
},
{
@@ -625,113 +568,26 @@
"id": "cf189776",
"metadata": {},
"source": [
"### Implementation\n",
"### Parallel implementation\n",
"\n",
"The parallel implementation of this method is closely related to the Floyd's algorithm. But there are some differences.\n",
"\n",
"At iteration $k$,\n",
"\n",
"1. The CPU owning row $k$ does the loop over $t$ to update row $k$\n",
"2. This CPU sends the "
]
},
{
"cell_type": "markdown",
"id": "9c553ada",
"metadata": {},
"source": [
"The computation of the complexity of computation and communication is left as an exercise.\n"
]
},
{
"cell_type": "markdown",
"id": "6b17aee4",
"metadata": {},
"source": [
"1. **Block-wise row partitioning**: Each processor gets a block of subsequent rows. \n",
"2. **Cyclic row partitioning**: The rows are alternately assigned to different processors."
]
},
{
"attachments": {
"fig-asp-1d-partition.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAABL4AAAI0CAYAAAATGeCIAAAACXBIWXMAAB7CAAAewgFu0HU+AAAAGXRFWHRTb2Z0d2FyZQB3d3cuaW5rc2NhcGUub3Jnm+48GgAAIABJREFUeJzs3XmcXFWZ//HvU1XdnU5nIyEhCYRVGBYBWRwQERFk2JRVUBxEXAYXXEcUB3QE11FREcF12BQFQWRRGJBNVFBQdhAwkIXsezq9d1Xd5/fHrfzsVN1Oeq17cvN5v155Kae2p+69ffv09557jrm7AGy+zOxsSVdXNX/N3S+oYw3Nkjqrmue4+y71qmFzYmbtklr6NLW5+4S06gEAAEA2mdmnJX2jqvk8d/9WGvUAaSikXQC2XGZ2uqRJfZpud/elI/Teh2tgx3e7pF5JayV1S+p097UjUQMAAACyzcxOkbR1n6Y73X3hCL33oZLGDOCpHZJ6FPdnexT3Z9eMRA3YcpnZREn/lfDQbe7+53rXAwwHwRdSYWY5SVdpw1Evz0kakeBL0i3aMFQbMDNbJ+kFSX+X9Lyk+yQ97gyPBAAAwIZ+KGlqn/8+RtKIBF+SfiFp1lBeaGZtkl7UP/uz90v6K/1ZDMJ4SecntC+URPCFzQrBF9JyqDYMvUIyQdK/Vv6tt8DMbpf0fXf/ezplAQAAIBRmtr82DL1CMl7SgZV/6y02s99I+oG7P5VOWQBQf7m0C8CWx8yaJH0t7ToGaZakcyU9Y2a/NLM90y4IAAAA6TCzBtXOmxS6mZI+IOkJM7vFzPZNuyAAqAeCL9SVmc2U9FtJh6RdyxDlJJ0u6XEz+5SZWdoFAQAAoH7MbKqkX0s6Mu1ahsgknSTpb2Z2oZnl0y4IAEYTtzpi1JjZBMWTfU6TtI+kNyn+JTuQSTpHw/skJQ3rbpHUKGmypB0k7SLp1ZIOUv8/I02SLpH0ejM7w917Rr5cAAAApMnMxkuaImkbxf3DwyWdrPSm7PiokudXGqu4f7qV/tmf3UvSwZIa+nmvgqQvSzrUzE519+oVurFl65D044T25+pdCDBcBF8YFWY2W9Kr0q6jyovu/thAn1xZyeTNkj6k/q/onSzpDjN7q7t3jUCNAAAACICZPSHpNWnXUWX2IPuz4xX3Y89RPPF+0t0Kx0j6nZkd6+5tI1MmNneVlUE/kHYdwEjgVkeMls3+2HL3Vne/2d3fLOm1ild3THKkpCu57REAACBTstCfbXP3W939OEn7Sbqzn6e+XtJ1lZXXASBTOLEBA+Duf5N0lKSPSColPOUMSZ+oa1EAAADAALn7U+5+vOLpP5Km6ThB0oX1rQoARh+3OmK03KB4fq/+nKR47q/Nhru7pCvMbK7iCU2bqp7yZTO71d3n1r+6gTOzaZK2U7yyT0HSMknz3H1JqoX1ozLh6s6Kj5epklzSCklLFdcd1bGOrSv/xkhaK2k5twQAAJBZN0v6y0YeP17StnWqZcS4+1WV/uxvFc8N1tfnzOxX7v58CqVtkpmNUdwXmyqpLGm1pBVZnG/XzJr1z75nJr+rmTXqn98xr39+R+abw4gi+MKocPeNXi0ys320mQVf67n7nWb2QUlXVz00VtK3JJ1S/6o2rhJ2fVjxyLTd+nnO3yT9QtIP3L27juUl1VJQHI6erHjeicn9PHWZmd2hOIi8sxJOjmQdhyju1B6t+PaAmlGyZjZPcaf4Pkm3uPuqkaxhE/Vto3ikYfVttt2SvkGnAQCAoXP3L27scTO7R5th8CVJ7v6AmZ0t6ZfasB/RKOm7kv4tjbqqVYKuExTXc7Tii7fVSmb2tOJJ/++QdK+7FxPe61TF05f09aK7V/fph1Ln8ZLeUNX8F3e/dRDv0awNv2vSsVXs811/K+n+pO86UszsI6rd5l9z99Yhvl9e8Xc7VvH3TPq7JDKz5yU9Iun/FPfx6dNiWAi+gCFw92vM7C2STq166EQz+xd3fzGNuqqZ2ThJn5J0nqRxm3j6gZV/55jZe909acWgUWdmR0u6XANbHGEbSe+t/HvSzD7g7o+OQA2vl/QVSW8cwNN3rPx7h6QfmNndkr7r7vcMt46NMbOtJd2reIWpvroknUQHAQAAbIy731QJbN5d9dBRZra/uz+eRl2SZGYNivt3n1Ny2NVXQdL+lX/nSlptZtcp7o/N6fO8LknnV722s3LHxpph1JqTdJniOxT66m9xrOrXNyq+/fRCbTpIbZB0QOXfRyStMrOfSbpslO46ebfivw/6ulzSoIKvylzIp0r6oqQ9NvH0nOIVSfdSfAx0mNmvJF3q7k8O5nOB9ZjjCxi6/5RUfYUlJ+ljKdRSw8wOlvS0pIu06dCrr90l3W9mR4xGXRtjZp9RfGVnKCuCvkbSn8zsXcP4/LyZXSrpTxpY6FWtoHiE2O/M7KKh1rEpZraVpHuUHHqd6O6/G63PBgAAmXK+4v5DtY/Xu5D1zGxbxaPpf6hNh15JJivuj//DzPbt036XpJernjtW0tlD+Iy+jlFt6PWcpAc29UIzm6V4ZNP3NbTRg1MUj/6fbWbV/cIgmNkExaPTbtKmQ68kLYoDuCfM7OSRrA1bDoIvYIjc/RXFt9hVOymAFR5PlvRHSTsN8fVjJN1mZnVbwtvMviHp60peZluK5zZYqnhOsv5uaWyQdK2ZDXqhgcpy37dr5Dp6LSP0Phsws4mS7lbt8uqdkk4Y7ZFmAAAgO9x9meKpLqq9tTL1RF2Z2f6SHlU8emu48pKa1/9HZV7YHyY874PD7Lufm9B2+aam4DCzAxWHXiPR384r7r8Hxcx2kPSQpONG6C0HczEf+P8IvoDh+XlC20yNzC/r4dhdtbcyr5V0naTPSnqXpLMq//9+xaFStXGSflKPZa3N7CxJn054qFvSpZJeJ6nR3We4+3TFAdfhkn6i2lU2TdK3zOyoQXx+QdKvlPxLuaz4CtW/S9pT8VXEqZL2lnS64rnelg/0s4ajcuvqnaqdn2J96HVvPeoAAACZktSf3UrS6+tZhJntJul3ivvS1ZZL+rbieaF2kjRB8WiwgxQHT3creaXKalepdoTbbpLePMSad1Y84quvtZJ+tonX7a74u85IeHiZ4nmDj1I8ncb673qwpI9WXhf8BPdmNkXJdyhIUrukHyue02w3xd9xhuJbON+jeGGJ9vpUii0Bc3wBw/N7xbc7NlS1HyzpsbpXk+yvkr6qeGLI3oTHv14ZBn6dan8xHSjp/Yp/MY0KM9tJ0vcSHnpa0inuXj0kXe5elvSgpAfN7PuSbpW0Q5+n5CRdY2b7DHCy+c8peRLXRyS9392fTXhspaRnJd1UmYfi3xUHif8ygM8bNDMbq3iY+CFVD3VKequ73z8anwsAADLvIUkdqh2tfrDi/taoq8xz9UvFt+5Vu0zSfydMqN4maZHiEWLfryz685+SPiRpfNLnuPtqM7te8dxRfX1YcUgzWB9S7WCSq9y9o78XmFmTpBsVh4vVLpX0BXdfV9W+/rs+IulyM5uueB7fDyrcUVBXSto1of1Xkj7u7our2tsU393xuOJ+/HjF++U/tZkuioZwMOILGAZ3b5OUNJH9nvWuJcFixZOc/6u739pP6CVJcvenJB0hKWlSzNGe4+Griq/y9PWCpCOTQq9qlUkuD5e0pOqhmYonCd0oM3uVpAsSHvqdpCP6Cb2qayi6+zWK9/snlTxXxpBVVjS6TbXzjnVIeguhFwAAGKpKH/GZhIfq2Z/9mJJv+fuku398IKsIuvsydz9f8XxbN27kqVcktL2lMt/WgFVWYXxPVXOkeL6ujfmk4jsH+nJJH3P3TyaEXjXcfam7f1rSLopHRwXFzE6UdGLCQ5dLOj0h9Krh7m3u/nXFI/y+q3jbAkNC8AUM3+yEtqSleevpAUmvdvfbBvoCd1+h+KpVtT0r8y2MuMrkpdUrY0aKR1mtHOj7uPs8JS8q8N7K1aKN+ZxqR+wtlPT2wa6M6O6Ru18qaT/FV+SGrXIF9GbVDsHvkHS8u29y4lQAAIBNSOrPJo3WGXGVAOmzCQ9dU+lXDYq7r3T3t0t6u6Sakf+V1SqrVy8vSDpnkB/1DtWOULtzYxduzaxF0mcSHrrS3ZPugNgod1/u7m+T9E5JQ16ZchRcnND2R8UjvTY691k1d+9090+o/4v0wCYRfAHDtyyhbVLdq9jQX4ayLLO7363kWzTfNvySEp2j2tDpN+7+0GDfyN1/pfi2zr4mKp7PLJGZTVLcKar2CXdfO9ga+tTyYqWeYancQvlL1c491i7pOHevy+0HAAAg85L6s0m34o2GU1UbIK1VvFrhkLn7je6eFOhJyaOy3l+54DhQH05o21R4dZpqt+tqxbctDpm7Xz+QOyXqobKy/L5VzesvbA951Ja7P+jufxpWcdhiEXwBw5c08eKmRhmFLGmlygNG6bOS5tW6bhjvl/TajU1yf7hqV8BZovi2wlSZWV7StZJOqnqoQ/FE9n+of1UAACCjkuakqld/9tiEtp8P5PbGYbhRtWHfdEmnDOTFlXDnwKrmf2jT84QlfdefDeT2xs1I9WT/knSPu/+j7pUAFQRfwPAlrYi4OS8ckbQy4Egss7yByrxV+1U1u+JVeYbqjoS2QzayRHX1nFmS9Et3r14psq4qK2leI+mMqofaJB3D7Y0AAGCEJfV96tWfPSyh7YbR/MDKvGZXJjyUNO1HkqTnXTaA2/jq/l1TkPQdr697FUAfBF/A8FWvgCNJm/NVmxcS2qZVliQeSa+R1FTVNreyYMBQzVEcDvU1TfGkmEkOTWh7dBifP2yVkO4nks6seqhN0rEM8QYAAKMglf5sZXXv7aqaS4pX9httP1LtBezDzKx6lfMNmNlUSadXNa+T9NNNvG5XxaPK+uqV9MSmS908VKbpODjhoerpSIC6IvgChi9p/oPRHJo9qipDrVckPDTS8zzMTGhLCt0GrHKVLWmVzW37eUl1R0uSnhpODcNRCb2uUO0S2+sUj/Qa9NxnAAAAA5BWfzapLine truncated
}
},
"cell_type": "markdown",
"id": "c518f863",
"metadata": {},
"source": [
"<div>\n",
"<img src=\"attachment:fig-asp-1d-partition.png\" align=\"left\" width=\"400\"/>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"id": "746e37f6",
"metadata": {},
"source": [
"### Block-wise partition"
]
},
{
"cell_type": "markdown",
"id": "102d6fa2",
"metadata": {},
"source": [
"To evaluate the efficiency of both partitioning schemes, consider how much work the processors do in the following example. \n",
"In any iteration k, which part of the matrix is updated in the inner loops? \n",
"1. The CPU owning row $k$ does the loop over $t$ to update row $k$.\n",
"2. The CPU owning row $k$ sets $B_{kk} = 1$.\n",
"2. This CPU sends the red cells in figure above to the other processors.\n",
"3. All processors receive the updated values in row $k$ and do the loop over i and j locally (blue cells).\n",
"\n",
"### Block-wise partition"
]
},
{
"cell_type": "markdown",
"id": "d9d29899",
"metadata": {},
"source": [
"It is clear from the code that at any given iteration `k`, the matrix is updated from row `k` to `n` and from column `k` to `m`. If we look at how that reflects the distribution of work over the processes, we can see that CPU 1 does not have any work, whereas CPU 2 does a little work and CPU 3 and 4 do a lot of work. "
]
},
{
"cell_type": "markdown",
"id": "6409890d",
"metadata": {},
"source": [
"### Load imbalance\n",
"\n",
"The block-wise partitioning scheme leads to load imbalance across the processes: CPUs with rows $<k$ are idle during any iteration $k$. The bad load balance leads to bad speedups, as some CPUs are waiting instead of doing useful work. \n",
" \n",
"### Data dependencies\n",
" \n",
"\n"
]
},
{
"attachments": {
"fig-asp-data-updated.png": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAtcAAAJNCAYAAAD6c1l4AAAACXBIWXMAAB7CAAAewgFu0HU+AAAAGXRFWHRTb2Z0d2FyZQB3d3cuaW5rc2NhcGUub3Jnm+48GgAAIABJREFUeJzs3XeYJFXVx/Hv6ZnZvLC7LGFZwrLkDIpECRJEEMVXl/waEQyor6IoBhAFEwZEBUFUVEREEQwgIkEyIkGQKJkl7i5h8+ykPu8ft4btrqmeTtVTNTO/z/P0A32n6tbZ6urqU7du3Wvujohkx8w6gXHR2053n5BlPKOFmc0GHi8pusvdd8ggjo2B24GpJcW3APu6+4qhjkdkuDCz04AvlhQd5+5nZxVP3pjZGGBtYDxQBJYBL7p7bw3rfhU4KVb8Pnf/ZeqBjkDtWQdQjZl9B5g8yCLdhANmafT/c4GHgYf1wyQieWZmqwB/pDyxvhk4sNL5y8zWAy6NFX/C3W9tTZTSKmb2e2CDkqJL3f3rWcUjw5uZjQPeARwI7AasD7TFFltuZr90948OVpe7n2xmHcCJJcXnmNmD7n5HmnGPRLlProF3A2s0sF7RzO4B/kQ4Yd2fbljpMrMdgXeWFN3g7ldmFY+Mbmb2JmD/kqK/uvuNWcUzgp0DbFHy/r/A2919ySDrjANeHytbtdLCZjaZ8ta9+e7+vXoDlXJmtivw9pKia9396jqr2YLyz//OpgOTUcfMJgL/BxwPrFZl8QnAZrXU6+6fN7OZhDwMwrnnEjPbxt0XNRrvaDAckutGFYDXRa+vmNkVwCnunteT13bA52JlSq4lK7tSfjy+BCi5TpGZHQIcUVK0kJBYv5rypiZR/lk+CCi5bt7rKd+vXUC9ybVIU8xsF+DXwOwWbeJYYBNgp+j9eoTzx9Et2t6IMByT6yVAaX+hVRh42yPJW4EDzOx04GR372lFcCIi1ZjZNOCsWPHH3P2RLOIRkeHHzOYQEuuxsT8VgbuAfwILCF1n1wVmAXvWsw13X2FmRwL3sLKL7gfM7HfuflXj0Y9swzG5PsDdbyktMDMDpgBrARsRWhTeDOxIeeJdIPQf2tbM5rj78qEJWUSkzMnA6iXvL3H3C2tc91lgv1jZv1OJSobaB4GJJe+fyyoQGV7MbF/gN0BHSXEn4aL92+4+v8J6HcDG9WzL3Z8ws+OB80qKv2dm29bycORoNByT6wE8DHnyavR6CPgLcIqZbQR8BvgA5QfgAcAfzexAHRgiMpTMbBOg9GGi5cAna10/ahS4Ju24ZOi5+21ZxyDDj5mtxcDE+ingIHd/YLB1o7v2Dzaw2Z8RuoLsHL3fgtBlRKOzJChkHUAruftj7v5h4A2EpLvUfsC3hj4qERnlTqT8R/EMd1eLpYjU6kzK73w9A+xaLbFuRtSI+ZlY8YlRS7jEjIiW62rc/V4z2w34O1A6ju0nzez37v7PWusys3ZC36X1CE/ljgH6CK1P8whDAC5NLfgmmNkkwjBPMwl9pdoIsb4CPA086e592UWYPTPbENgSWIewb54HbnP3l1KqfzPCFf46QA/h7sp/CMdJMYX6OwjH4rrAdMJ3uo8wNOWL0XY6m91OmsysDdieMEzUmoSL/JeBJwhjTTd9NynqKrY94UGcGYTv5wLg3+7+ZLP1NxHX2sCRJUULgdMzCicVZrYusA3hx34K4RhfQPgepf1wZt2ic3bpd6SD8B1Zxspz9rLsImyemW0AbEX4DCYT9v+zwL9aNSRttM2tCfvVCeeb29z9hVZsTwIz2wk4pKSoB5gzFPvd3W8xs8uBg6KidYHDgQtave3hZlQk1wDu/qqZHUzom9g/tF8B+C5hPMiKoocG9iK0gG9HSKgHW/4e4DLgR+7+yiDLTQRuiN5Oj/35vVGfqkqucvfS4bUws9WA90dxvoHy8VOTdJrZ1YQvxh98CGYUMrNbWbn/utx90H0fW/cA4NSSot+6+3cGWf47hM+t37HufneUkB4NHEMYTSau18yuAk5w9/gdj1ribAc+RLj1v0WFxZ43s4uBHzRQ/3sIx+wbCD+og7UcFM3sLuD3wI8Hu/Azs3UIYy5DSEhL/Z+ZHT7Idgb9LKL6NyIMCXcQA4/3fovM7BLgNHd/arD6KmxjInAC4XuwXoVlHiE8BPS3eutPwbGUP3z0C3dfXG8lZnYl5S1Xu7l7V2yZnxCO7/jxMdvMBhs1abm771Fl++OA44D3EBLrJH1m9k/gW+7+l8HqK6n357H65pQeB9Ht8P8Bdif8sI8jjGRzm7t/tWS5w4E9CN+RbRn8O+Jm9m/C2OFnufvCQeKbysoRQeJDxB5jZm8dZDuXu/spsfpOBOaUFH2x1ofEoglCjiN0e9yqwmLLomPlVHf/Ty31RnX/kdAg0G9vd18cndveTzi/xYeDhLAvrwc+l/exkM3sy8DbSoqcEPd1GYVUq88AVvL+Z+7+ryHc/pmsTK4BPoGS64HcPdcvQsuCl7x2a7K+98fqc2CXKus8kLBOLa+XCT8OlepdpcF6HfhNQn07N1HfLcDaQ/B5dpZsc3md6/5vLOYzqyx/WWz5PQkPuj5W4z7pBP63zhg3A+6rY7/3EJ7srnmfAPMb/IyfB/YbpN4Nmzh+Kn4WhDsmpxMmeaq1vi7gA3Xu+zcRWutq3UY8njuH4Ph/pGR7fcBGDdbzXCz28QnLXN/gZ7mkyrb3I9z5qqfOPwOTavh33RJbb7OofE3Cg1qVjqG7Y/U83uC/fT5w8CDxrd5gvQ6cn1DfWbFljqjx89+B2s9jThhh6/tAW431x+ueBuwTO36rfX8PG4Lv02mx7X60xvW+HFuvSJjdsaXxpvDvnRb7DvQC6wxxDEbos126/zbOet/k7TWi+1xXcAHhB7jUUS3a1jTgt2Z2aIvqT9OuwI1R6/dI9XlCa+WGNS4/Dji/SmvUa8zs9cBNVG5FStJOeStEK80A/lLljkiqopbkywityfHWw2eBewnPQ8QnJBgD/MzMPlHjdt5BGBd+Zh3hDWlfQTPbgfKn9G9298eGMoZmmdn7gSsYeFdgEeFzvIeB51cILYRXm1l8yLBatnkoYXKdj9L6z2x1wiQZB7d4Ow2LJni6juTzmBO6QMW1ESYZ+X0jnwHwVUK3ylpHmRgD/NLMtm5gWy1lZp8DTikpcsIwmPGhMfPorZR/B25y96TvW8t4yLDjU6AfNpQxDAejpltIP3fvNbPfUt4x/8AaV19OaFm5mdA6+QRh3O0xhOmL1yXManckITGDcFI7y8yudfeXY/V1sfKhym2Bt5T87XZCy1Ml91aJ9RngH8C/CC3v86LtTSIk/VsRLip2LllnwyieD1ape7gqnXGwmzDiwnWEZKBA+PcfSuhH2K8d+KmZbeqD3L6Pxi3+IwO7O9xIeKr7fkL/2tUIx8mmhBNlUreUWi2J6r8lqv9pQl/rsYTjcQPCsX0IK0/IY4HzzGxLHzgU5UJWHo+7EG6r9+s/liqpNPX2eZTfel1I6Ir1S3d/pr8w6oe9I6HbSOnFzLfN7FYfZPInM9sCuJDy7hZO+DwuJbTCLSFcXMwgdO16O2HYzqH05tj7mrpKNOEiwji3kwjdB/q9RHjyv5LupEIz2x/4KeUPwl9K+Dxv95LnN6K+5R8kTLIyISremXB81TwyCmGyigPqWD5uGeF83X/Oforyc/b6rDxn93dXaydM83y9D5yFbjkrvyOvB0ovVPt/Gyq5q+F/RcTM1ifs88klxYuB7xC6fj3q7n1mtgrhTt3/EVqc+/0P8E3gU3VuuvT46SV0jbmG8DvTRvguHUZ5w8JY4NuU/65lysw+Sfj393NCi/WPq6zXRjhWWqnTq/eb3jH2vuwcEnUV2pxwkTiVcNH7KvCgp/tcwZ8p349vJtxFkH5ZN53XcAsi1W4hUZ1vYeBtrOmDLP8V4GBgbI31b0N4oKS0/o9XWefY2PLfbODftTHwNWCbOtb5Umy73cCUFn6eWXYLccKP7enA6hXWKRASvPh6p1XZ1sWx5Tup4bYoIdErvc1XS7eQ/h+sjhr3266EpLs0vsOrrBPfB59p4LM+OlbHf4ENq6xjhGStdL2rBlm+jdBaWrr8i9XOE9F25sTWa2m3EEKLb+n2NmmirqrdQkqWnRFb9oEGtrcW5d2ReoH317DeTtF3rn+9HmD9QZaPdwspfT1F+AHfi3AhvD4hmTsIOCpWz9cIF3W1nrNfR3jQu3R7x1ZZ5+Ox5b/SwH6tq1sIA7v6PEyVbgGEu0al6xSBfaqsk9TlpAv4ETCjwjptwI9j6/QN9nmn8J2quVsI8BHKu+AVgQ/XuJ3VBjku03pdU0McN8fW2ZfQkPceQkNRZ4W6+wh3VT9AjV2Daojl0ZL6O2v9ro2WV+YB1PABtiK5Xjfh4Ns95bg/GKv/r1WWbzq5biLWW2PbPqSF28oyub6FGvuGJfxIPFvppES4mIqftN82FPukjm18Pvbv+VWV5ZtKrgmtVs+XrL+QKol1ybrtlCfMRWDzCsseFotzGbBVjduZHVu3Zck1IZl/tWRbzzVZ31An199p9HgAPh1b91uDLJuUXM8DPgy0t+rzibZ9XGy7f6yy/JAm14SW6NJllwKzatzOj2Lr3lBl+XhyfRewZQ3biX/vHXh3Cz+zmpJrQlIZP0d/qI7tZJ5cExp+lsTW+TL1PWfihHNrTefIKvGcF6t30GfXRttrNPa5hnAwxodBWzPlbfwx9r7ayB1Z+lPs/awsghgCX3D3R2tc9mSgdAirmYQRCpLEn96+0GscHWEIxY/HWS3e3pGUjzjyNXd/vJYVPQzF98OSImPgjIT9Toi9P83d7685yqGzBmGYun7VunXlhpmtShhZp9/9wBl1VHEu5f2A491jBnMV4U7cOd76Cb/yfh78WOz9d732EXW+QOg+0m8PM9u2jm3v7zWMoexhxJrLY8WVRpMZEmZ2FCER7D9HO6HF+tzsomrIaoQuXqVOob7nTCB0Qb3JzAYdEagG8XPYpk3WN6KMuj7XLine truncated
}
},
"cell_type": "markdown",
"id": "a67e0aad",
"metadata": {},
"source": [
"<div>\n",
"<img src=\"attachment:fig-asp-data-updated.png\" align=\"left\" width=\"250\"/>\n",
"</div>"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e0565e92",
"metadata": {},
"outputs": [],
"source": [
"answer = \"x\" # replace x with a, b, c, or d \n",
"ge_dep_check(answer)"
"As you probably see, the parallel implementation of this method is closely related to Floyd's algorithm. But there are some differences. \n",
"\n",
"1. The process that owns row $k$ updates its values before sending them.\n",
"2. We do not send the full row $k$, only the entries beyond column $k$.\n",
"3. We need a cyclic partition to balance the load properly.\n",
"\n",
"A key similarity between the two algorithms, however, is that they both suffer from synchronization problems. We need to make sure that the rows arrive in the right order. The strategies discussed for Floyd's algorithm also apply in this current case.\n",
"\n",
"The actual implementation of the parallel algorithm is left as an open exercise."
]
},
{
@@ -740,54 +596,9 @@
"metadata": {},
"source": [
"## Conclusion\n",
"Cyclic partitioning tends to work well in problems with predictable load imbalance. It is a form of **static load balancing** which means using a pre-defined load schedule based on prior information about the algorithm (as opposed to **dynamic load balancing** which can schedule loads flexibly during runtime). The data dependencies are the same as for the 1d block partitioning.\n",
"\n",
"At the same time, cyclic partitioning is not suitable for all communication patterns. For example, it can lead to a large communication overhead in the parallel Jacobi method, since the computation of each value depends on its neighbouring elements."
]
},
{
"cell_type": "markdown",
"id": "20982b04",
"metadata": {},
"source": [
"## Exercise\n",
"The actual implementation of the parallel algorithm is left as an exercise. Implement both 1d block and 1d cyclic partitioning and compare their performance. The implementation is closely related to that of Floyd's algorithm. To test your algorithms, generate input matrices with the function below (a random matrix is not enough, we need a non singular matrix that does not require pivoting). "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a65cf8e6",
"metadata": {},
"outputs": [],
"source": [
"function tridiagonal_matrix(n)\n",
" C = zeros(n,n)\n",
" stencil = [(-1,2,-1),(-1,0,1)]\n",
" for i in 1:n\n",
" for (coeff,o) in zip((-1,2,-1),(-1,0,1))\n",
" j = i+o\n",
" if j in 1:n\n",
" C[i,j] = coeff\n",
" end\n",
" end\n",
" end\n",
" C\n",
"end"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "31d8586a",
"metadata": {},
"outputs": [],
"source": [
"n = 12\n",
"C = tridiagonal_matrix(n)\n",
"b = ones(n)\n",
"B = [C b]\n",
"gaussian_elimination!(B)"
"\n",
"We studied the parallelization of an algorithm that leads to load imbalance. We fixed the problem using a cyclic partition. This is a form of static load balancing since we were able to distribute the load in advance without using runtime values. This is opposed to dynamic load balancing which can schedule loads flexibly during runtime. Note however that cyclic partitioning is not suitable for all communication patterns. For example, it can lead to a large communication overhead in the parallel Jacobi method, since the computation of each value depends on its neighboring elements.\n"
]
},
{
@@ -801,14 +612,6 @@
"\n",
"This notebook is part of the course [Programming Large Scale Parallel Systems](https://www.francescverdugo.com/XM_40017) at Vrije Universiteit Amsterdam and may be used under a [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) license."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8ab22f67",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
notebooks/asp.ipynb
+5 -5
View File
@@ -647,7 +647,7 @@
"- On the receive side $O(N)/O(N^2/P) = O(P/N)$\n",
"\n",
"\n",
"In summary, the send/computation ratio is $O(P^2/N)$ and the receive/computation ratio is $O(P/N)$. The algorithm is potentially scalable if $P^2<<N$. Note that this is worse than for matrix-matrix multiplication, which is scalable for $P<<N$."
"In summary, the send/computation ratio is $O(P^2/N)$ and the receive/computation ratio is $O(P/N)$. The algorithm is potentially scalable if $P^2<<N$. Note that this is worse than for matrix-matrix multiplication, which is scalable for $P<<N$. I.e., you need a larger problem size in the current algorithm than in matrix-matrix multiplication."
]
},
{
@@ -755,7 +755,7 @@
"### Distributing the input matrix\n",
"\n",
"Since only rank 0 receives the input matrix C, this rank needs to split it row-wise and\n",
"send the pieces to all other ranks. This is done in the function below. We start by communicating the problem size N to all ranks (at the start only rank 0 knows the problem size). This is trivially done with an `MPI.Bcast!`. Once all ranks know the problem size, they can allocate space for their local part of `C` , called `myC` in the code. After this, rank 0 sends the pieces to all other ranks. We do it here with `MPI.Send` and `MPI.Recv!`. This can also be done with `MPI.Scatter!`, but it is more challenging since we are using a row partition and Julia stores the matrices in column major order. Note that this algorithm can also be implemented using a column-wise matrix partition. In this case, using `MPI.Scatter!` would provably be the best option.\n"
"send the pieces to all other ranks. This is done in the function below. We start by communicating the problem size N to all ranks (at the start only rank 0 knows the problem size). This is trivially done with an `MPI.Bcast!`. Once all ranks know the problem size, they can allocate space for their local part of `C` , called `myC` in the code. After this, rank 0 sends the pieces to all other ranks. We do it here with `MPI.Send` and `MPI.Recv!`. This can also be done with `MPI.Scatter!`, but it is more challenging since we are using a row partition and Julia stores the matrices in column major order. Note that this algorithm can also be implemented using a column partition. In this case, using `MPI.Scatter!` would probably be the best option.\n"
]
},
{
@@ -894,7 +894,7 @@
"source": [
"### Collecting back the results\n",
"\n",
"At this point, we have solved the ASP problem, but the solution is cut in different pieces, each one stored on a different MPI rank. It is often useful to gather the solution into a single matrix, e.g., to compare it against the sequential algorithm.The following function collects all pieces and stores them in $C$ on rank 0. Again, we implement this with `MPI.Send` and `MPI.Recv!` as it is easier as we are working with a row-partition. However, we could do it also with `MPI.Gather!`."
"At this point, we have solved the ASP problem, but the solution is cut in different pieces, each one stored on a different MPI rank. It is often useful to gather the solution into a single matrix, e.g., to compare it against the sequential algorithm.The following function collects all pieces and stores them in $C$ on rank 0. Again, we implement this with `MPI.Send` and `MPI.Recv!` as it is easier as we are working with a row partition. However, we could do it also with `MPI.Gather!`."
]
},
{
@@ -1024,7 +1024,7 @@
"source": [
"## Is this implementation correct?\n",
"\n",
"In the cell above, the result of the parallel code was provably the same as for the sequential code. However, is this sufficient to assert that the code is correct? Unfortunately no. In fact, the parallel code we implemented is not correct. There is no guarantee that this code computes the correct result. This is why:\n",
"In the cell above, the result of the parallel code was probably the same as for the sequential code. However, is this sufficient to assert that the code is correct? Unfortunately, it is not. In fact, **the parallel code we implemented is not correct!** There is no guarantee that this code computes the correct result. This is why:\n",
"\n",
"In MPI, point-to-point messages are *non-overtaking* between a given sender and receiver. Say that process 1 sends several messages to process 3. All these will arrive in FIFO order. This is according to section 3.5 of the MPI standard 4.0.\n",
"Unfortunately, this is not enough in our case. The messages could arrive in the wrong order *from different senders*. If process 1 sends messages to process 3, and then process 2 sends other messages to process 3, it is not granted that process 3 will receive first the messages from process 1 and then from process 2 (see figure below)."
@@ -1113,7 +1113,7 @@
"\n",
"### Exercise 1\n",
"\n",
"Modify the `floyd_iterations!` function so that it is guaranteed that the result is computed correctly. Use `MPI.Bcast!` to solve the synchronization problem. Note: only use `MPI.Bcast!`in `floyd_iterations!`, do not use other MPI directives. "
"Modify the `floyd_iterations!` function so that it is guaranteed that the result is computed correctly. Use `MPI.Bcast!` to solve the synchronization problem. Note: only use `MPI.Bcast!`in `floyd_iterations!`, do not use other MPI directives. You can assume that the number of rows is a multiple of the number of processes."
]
},
{
notebooks/figures/fig_jacobi.svg
+3522 -6
View File
File diff suppressed because it is too large. Load diff
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 6.7 MiB

After

Width:  |  Height:  |  Size: 6.8 MiB