{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e3666bcd",
   "metadata": {},
   "source": [
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/3/39/VU_logo.png/800px-VU_logo.png?20161029201021\" width=\"350\">\n",
    "\n",
    "### Programming large-scale parallel systems\n",
    "\n",
    "# Traveling sales person"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d49d8e7",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "In this notebook, we will learn\n",
    "\n",
    "- How to parallelize the solution of the traveling sales person problem\n",
    "- The concept of search overhead\n",
    "- A dynamic load balancing method\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "480af594",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Note:</b> Do not forget to execute the cell below before starting this notebook! \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e93809a",
   "metadata": {},
   "outputs": [],
   "source": [
    "using Printf\n",
    "function answer_checker(answer,solution)\n",
    "    if answer == solution\n",
    "        \"🥳 Well done! \"\n",
    "    else\n",
    "        \"It's not correct. Keep trying! 💪\"\n",
    "    end |> println\n",
    "end\n",
    "tsp_check_2(answer) = answer_checker(answer, 4)\n",
    "tsp_check_3(answer) = answer_checker(answer, \"d\")\n",
    "tsp_check_4(answer) = answer_checker(answer, \"a\")\n",
    "function q_superlinear_answer(bool)\n",
    "    bool || return\n",
    "    msg = \"\"\"\n",
    "    Negative search overhead can explain the superlinear speedup in this algorithm. The optimal speedup (speedup equal to the numer of processors) assumes that the work done in the sequental and parallel algorithm is the same. If the parallel code does less work, it is possible to go beyond the optimal speedup. Cache effects are not likely to have a positive impact here. Even large search spaces can be represented with rather small distance matrices. Moreover, we are not partitioning the distance matrix.\n",
    "    \"\"\"\n",
    "    println(msg)\n",
    "end\n",
    "println(\"🥳 Well done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1d637cb",
   "metadata": {},
   "source": [
    "## The traveling sales person (TSP) problem\n",
    "\n",
    "\n",
    "In this notebook, we will study another algorithm that works with graphs, the [traveling sales person (TSP) problem](https://en.wikipedia.org/wiki/Travelling_salesman_problem). The classical formulation of this problem is as follows (quoted from Wikipedia) \"Given a list of cities and the distances between each pair of cities, what is the shortest possible route that visits each city exactly once?\" This problem has applications in combinatorial optimization, theoretical computer science, and operations research. It is very expensive problem to solve (NP-hard problem), which makes parallel computing often required to solve it.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Note:</b> There are two key variations of this problem. One in which the sales person returns to the initial city, and another in which the sales person does not return to the initial city. We will consider the second variant for simplicity.\n",
    "</div>\n",
    "\n",
    "\n",
    "### Problem statement\n",
    "\n",
    "Our version of the TSP problem can be formalized as follows. Given a graph $G$ with a distance table $C$ and an initial node  in the graph, compute the shortest route that visits all nodes exactly once, without returning to the initial node. The nodes on the graph can be interpreted as the \"cities\", and the solution is the optimal route for the traveling salesperson to visit all cities. The following figure shows a simple TSP problem and its solution."
   ]
  },
  {
   "attachments": {
    "fig-tsp-intro.png": {
     "image/png": "[encoded data, 92016 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "bf3dd0e6",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"attachment:fig-tsp-intro.png\"  align=\"left\" width=\"450\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd4f87fc",
   "metadata": {},
   "source": [
    "### Sequential algorithm (branch and bound)\n",
    "\n",
    "A well known method to solve this problem is based on a [branch and bound](https://en.wikipedia.org/wiki/Branch_and_bound) strategy. It consists in organizing all possible routes in a tree-like structure (this is the \"branch\" part from the branch and bound strategy). The root of this tree is the initial city. The children of each node in the graph are the neighbor cities that have not been visited in the path so far. When all neighbor cities are already visited, the city becomes a leaf node in the tree. See figure below for the tree associated with our TSP problem example. The TSP problem consists now in finding which is the \"shortest\" branch in this tree. The tree data structure is just a convenient way of organizing all possible routes in order to search for the shortest one checking one by one, from left to right, using a [depth-first search](https://en.wikipedia.org/wiki/Depth-first_search). We refer to it as the *search tree* or the *search space*."
   ]
  },
  {
   "attachments": {
    "g25091.png": {
     "image/png": "[encoded data, 78328 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "9797bb92",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"attachment:g25091.png\"  align=\"left\" width=\"525\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c303dddf",
   "metadata": {},
   "source": [
    "### Nearest city first heuristic\n",
    "\n",
    "When building the search tree we are free to choose any order when defining the children of a node. A clever order is using the *nearest city first heuristic*. I.e., we sort the children according to how far they are from the current node, in ascending order. This heuristic increases the chances that the the optimal route is located at the left of the tree, which will allow us to speed-up the search using pruning (next section). The figure above used the nearest city first heuristic. In blue you can see the distance between cities. The first child is always the one with the shortest distance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a40faff",
   "metadata": {},
   "source": [
    "### Pruning the search tree\n",
    "\n",
    "The basic idea of the algorithm is to loop over all possible routes (all branches in the search tree) and find find the one with the shortest distance. One can optimize this process by \"pruning\" the search tree. We visit all possible routes in the tree from left to right. In this process, we keep track of the shortest route visited so far. While visiting a new node in a route, we check if the distance traveled in this route is already larger than the shortest route traversed so far. If this is the case, we can skip all routes that continue from this city as we now for sure that a route will not be in this part of the tree. This is called \"pruning\" since we are cutting branches of the tree, and is the \"bound\" part of the branch and bound strategy.  \n",
    "For example, in the following graph, only 3 out of 6 possible routes need to be fully traversed to find the shortest route. In particular, we do not need to fully traverse the second branch/route (figure below left). When visiting the third city in this branch, the current distance is already equal to the full previous route. It means that the solution will not be in this part of the tree.  In figure below (right), the gray nodes are the ones we do not visit because the minimum distance had been exceeded before completing the route."
   ]
  },
  {
   "attachments": {
    "fig-tsp-pruning.png": {
     "image/png": "[encoded data, 186948 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "ecf89636",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"attachment:fig-tsp-pruning.png\"  align=\"left\" width=\"700\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9da5f5ae",
   "metadata": {},
   "source": [
    "###  Computation complexity\n",
    "\n",
    "The total number of routes is $O(N!)$, where $N$ is the number of cities. This comes from the fact that the number of possible routes is equal to the number of possible permutations of $N$ cities. Thus the cost of the algorithm is $O(N!)$, which becomes expensive very quickly when $N$ grows.\n",
    "\n",
    "In practice, however, we will not need to traverse all $O(N!)$ possible routes to find the shortest one since we consider pruning. The nearest city first heuristic also makes more likely that the shortest route is among the first routes to be traversed (left part of the tree), thus increasing the chance to prune routes afterwards. However, the solution can be anywhere in the search tree, and the number of routes to be traversed is $O(N!)$ in the worse case scenario (the optimal route is the right-most one)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fe017a4",
   "metadata": {},
   "source": [
    "## Serial implementation\n",
    "\n",
    "Let's implement the serial algorithm.\n",
    "\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Note:</b> The implementation of this algorithm is rather challenging. Try to understand the key ideas (the explanations) instead of all code details. Having the complete functional implementation is useful to analyze the actual performance of our parallel implementation at the end of the notebook, and to check if it is consistent with the theory.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2b70f85",
   "metadata": {},
   "source": [
    "### Nearest-city first heuristic\n",
    "\n",
    "The first step is preprocessing the distance table to create a new data structure that takes into account the nearest city first heuristic. This is done in the following function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a50706bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "function sort_neighbors(C)\n",
    "    n = size(C,1)\n",
    "    map(1:n) do i\n",
    "        Ci = C[i,:]\n",
    "        cities = sortperm(Ci)\n",
    "        distances = Ci[cities]\n",
    "        collect(zip(cities,distances))\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fc1a398",
   "metadata": {},
   "source": [
    "Execute the next cell to understand the output of `sort_neighbors`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6dd0288e",
   "metadata": {},
   "outputs": [],
   "source": [
    "C = [\n",
    "    0 2 3 2\n",
    "    2 0 4 1\n",
    "    3 4 0 3\n",
    "    2 1 3 0    \n",
    "]\n",
    "C_sorted = sort_neighbors(C)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a63dc266",
   "metadata": {},
   "source": [
    "The output is a vector of vector of tuples. The outer vector is indexed by a city id, for instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00608e1d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "city = 3\n",
    "C_sorted[city]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f769627c",
   "metadata": {},
   "source": [
    " This returns a vector of tuples that contains information about the connections to this city of the form (destination, distance). In this case, city 3 is connected to city 3 at distance 0 (itself), then with city 1 at distance 3, then with city 4 at distance 3, and finally with city 2 at distance 4. Note that the connections  are sorted by their distance in ascending order (here is where the nearest city first heuristic is used). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44025bd5",
   "metadata": {},
   "source": [
    "### Loop over all paths"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c91a99f",
   "metadata": {},
   "source": [
    "Next, we write an algorithm that traverses the whole search tree and prints all the possible paths. To this end, the tree is traversed in [depth-first order](https://en.wikipedia.org/wiki/Depth-first_search) using a recursive function call. Before we go to a neighbouring city, we also have to verify that it has not been visited on this path yet. If we reach a leaf node, we print the complete path and continue searching."
   ]
  },
  {
   "attachments": {
    "fig-tsp-output-1.png": {
     "image/png": "[encoded data, 131352 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "987a0273",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"attachment:fig-tsp-output-1.png\"  align=\"left\" width=\"600\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ddc2ec1",
   "metadata": {},
   "outputs": [],
   "source": [
    "function visit_all_paths(C_sorted,city)\n",
    "    # Initialize path\n",
    "    num_cities = length(C_sorted)\n",
    "    path=zeros(Int,num_cities)\n",
    "    hops = 1\n",
    "    path[hops] = city\n",
    "    visit_all_paths_recursive!(C_sorted,hops,path)\n",
    "end\n",
    "function visit_all_paths_recursive!(C_sorted,hops,path)\n",
    "    num_cities = length(C_sorted)\n",
    "    if hops != num_cities\n",
    "        city = path[hops]\n",
    "        next_hops = hops + 1\n",
    "        connections = C_sorted[city]\n",
    "        for (next_city,distance_increment) in connections\n",
    "            # Check if next_city is already on path\n",
    "            already_visited = (next_city in view(path,1:hops))\n",
    "            if ! already_visited\n",
    "                path[next_hops] = next_city\n",
    "                visit_all_paths_recursive!(C_sorted,next_hops,path)\n",
    "            end\n",
    "        end\n",
    "        return nothing\n",
    "    else\n",
    "        println(\"I just completed route $path\")\n",
    "        return nothing\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "723a0f1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "city = 1\n",
    "visit_all_paths(C_sorted,city)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f68093d",
   "metadata": {},
   "source": [
    "### Serial implementation without pruning\n",
    "\n",
    "Now, we know how to traverse all possible routes. We just need a minor modification of the code below to solve the TSP problem (without pruning). We add a new variable called `min_distance` that keeps track of the distance  of the shortest route so-far. This variable is updated at the end of each route, i.e., when a leaf node is visited. After traversing all routes, `min_distance` will contain the distance of the shortest route (the solution of the ASP problem)."
   ]
  },
  {
   "attachments": {
    "fig-tsp-output-2.png": {
     "image/png": "[encoded data, 144188 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "881076fa",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"attachment:fig-tsp-output-2.png\"  align=\"left\" width=\"600\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ac4875c",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Note:</b> We could further modify the function so that we also return a vector containing the cities in the shortest route. However, in this notebook, we will only return the distance of the shortest route (a single value) for simplicity.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5989f0ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "verbose::Bool = true\n",
    "function tsp_serial_no_prune(C_sorted,city)\n",
    "    num_cities = length(C_sorted)\n",
    "    path=zeros(Int,num_cities)\n",
    "    hops = 1\n",
    "    path[hops] = city\n",
    "    distance=0\n",
    "    min_distance = typemax(Int)\n",
    "    tsp_serial_no_prune_recursive!(C_sorted,hops,path,distance,min_distance)\n",
    "end\n",
    "function tsp_serial_no_prune_recursive!(C_sorted,hops,path,distance,min_distance)\n",
    "    num_cities = length(C_sorted)\n",
    "    if hops != num_cities\n",
    "        city = path[hops]\n",
    "        next_hops = hops + 1\n",
    "        connections = C_sorted[city]\n",
    "        for (next_city,distance_increment) in connections\n",
    "            already_visited = (next_city in view(path,1:hops))\n",
    "            if ! already_visited\n",
    "                # Add new city to path\n",
    "                next_distance = distance + distance_increment\n",
    "                path[next_hops] = next_city\n",
    "                min_distance = tsp_serial_no_prune_recursive!(\n",
    "                    C_sorted,next_hops,path,next_distance,min_distance)\n",
    "            end\n",
    "        end\n",
    "        return min_distance\n",
    "    else\n",
    "        # Set new minimum distance in leaf nodes\n",
    "        min_distance = min(distance,min_distance)\n",
    "        verbose && println(\"I just completed route $path. Min distance so far is $min_distance\")\n",
    "        return min_distance\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1be2bfc",
   "metadata": {},
   "outputs": [],
   "source": [
    "city = 1\n",
    "verbose = true\n",
    "min_distance = tsp_serial_no_prune(C_sorted,city)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb9b8353",
   "metadata": {},
   "source": [
    "### Final serial implementation\n",
    "\n",
    "Finally, we add the pruning to our algorithm. Anytime the current distance exceeds the minimum distance, the search in this path is aborted and continued with another path. By running the function below, you will see that only three routes will be traversed thanks to pruning as shown in next figure."
   ]
  },
  {
   "attachments": {
    "fig-tsp-output-3.png": {
     "image/png": "[encoded data, 135532 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "3c160cf9",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"attachment:fig-tsp-output-3.png\"  align=\"left\" width=\"600\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8241e0df",
   "metadata": {},
   "outputs": [],
   "source": [
    "function tsp_serial(C_sorted,city)\n",
    "    num_cities = length(C_sorted)\n",
    "    path=zeros(Int,num_cities)\n",
    "    hops = 1\n",
    "    path[hops] = city\n",
    "    distance=0\n",
    "    min_distance = typemax(Int)\n",
    "    tsp_serial_recursive!(C_sorted,hops,path,distance,min_distance)\n",
    "end\n",
    "function tsp_serial_recursive!(C_sorted,hops,path,distance,min_distance)\n",
    "    # Prune this path if its distance is too high already\n",
    "    if distance >= min_distance\n",
    "        verbose && println(\"I am pruning at $(view(path,1:hops))\")\n",
    "        return min_distance\n",
    "    end\n",
    "    num_cities = length(C_sorted)\n",
    "    if hops != num_cities\n",
    "        city = path[hops]\n",
    "        connections = C_sorted[city]\n",
    "        next_hops = hops + 1\n",
    "        for (next_city,distance_increment) in connections\n",
    "            already_visited = (next_city in view(path,1:hops))\n",
    "            if ! already_visited\n",
    "                # Add city to path \n",
    "                next_distance = distance + distance_increment\n",
    "                path[next_hops] = next_city\n",
    "                min_distance = tsp_serial_recursive!(\n",
    "                    C_sorted,next_hops,path,next_distance,min_distance)\n",
    "            end\n",
    "        end\n",
    "        return min_distance\n",
    "    else\n",
    "        # Set new minimum distance in leaf nodes\n",
    "        min_distance = min(distance,min_distance)\n",
    "        verbose && println(\"I just completed route $path. Min distance so far is $min_distance\")\n",
    "        return min_distance\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "998087f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "city = 1\n",
    "verbose = true\n",
    "min_distance = tsp_serial(C_sorted,city)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f0a5d5c",
   "metadata": {},
   "source": [
    "### Performance\n",
    "Let's have a look at how well our code performs. In the following block, we create another random matrix and time the execution for both versions with and without pruning. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1eb74d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 11 # It is safe to test up to n=11 on a laptop\n",
    "using Random\n",
    "using Test\n",
    "Random.seed!(1)\n",
    "C = rand(1:10,n,n)\n",
    "C_sorted = sort_neighbors(C)\n",
    "city = 1\n",
    "verbose = false\n",
    "@time min_no_prune = tsp_serial_no_prune(C_sorted,city)\n",
    "@time min_prune = tsp_serial(C_sorted,city)\n",
    "@test min_no_prune == min_prune"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6088ddc9",
   "metadata": {},
   "source": [
    "You can observe that, especially for larger numbers of cities (n=11), the performance of the algorithm with pruning is much better than the performance of the algorithm without pruning. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "733ce76e",
   "metadata": {},
   "source": [
    "## Parallel algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "732a3ffb",
   "metadata": {},
   "source": [
    "### Where can we extract parallelism ?\n",
    "\n",
    "All branches of the search tree can be traversed in parallel. Let us discuss how we can distribute these branches over several processes.\n",
    "\n"
   ]
  },
  {
   "attachments": {
    "g25091.png": {
     "image/png": "[encoded data, 78328 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "5e853346",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"attachment:g25091.png\"  align=\"left\" width=\"525\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c72df0e",
   "metadata": {},
   "source": [
    "### Option 1\n",
    "\n",
    "We can (at least in theory)  assign a branch of our search tree to each process. However, as mentioned in an earlier section, the number of branches in the search tree can be up to $O(N!)$. This would require an unfeasibly large amount of processors which each do only very little work. Thus, we skip this option as it is impractical."
   ]
  },
  {
   "attachments": {
    "g18142.png": {
     "image/png": "[encoded data, 67600 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "73b7633a",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"attachment:g18142.png\"  align=\"left\" width=\"300\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "052a5676",
   "metadata": {},
   "source": [
    "### Option 2\n",
    "\n",
    "Instead of assigning one branch per worker, we can assign a fixed number of branches to each worker. This would be a good strategy if we do not consider pruning. However, it is not efficient if we include pruning (which is essential in this algorithm). "
   ]
  },
  {
   "attachments": {
    "g18910.png": {
     "image/png": "[encoded data, 58748 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "2701cc6a",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"attachment:g18910.png\"  align=\"left\" width=\"300\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3af7def",
   "metadata": {},
   "source": [
    "### Performance issues:  Load balance\n",
    "\n",
    "Pruning is essential in this algorithm but makes challenging to evenly distribute the work over available processors. Image that we assign the same number of branches per worker and that the workers use pruning locally to speed up the solution process. It is not possible to know in advance how many branches will be fully traversed by each worker since pruning depends on the values in the input distance matrix (runtime values). It might happen that a worker can prune many branches and finishes fast, whereas other workers are not able to prune so many branches and they need more time to finish. This is a clear example of bad load balance. We will explain a strategy to fix it later in the notebook.\n",
    "\n",
    "\n",
    "### Performance issues: Search overhead \n",
    "\n",
    "Another disadvantage of this kind of parallel search is that the pruning is now less effective. The workers each run their own version of the search algorithm and keep track of their local minimum distances, which are  different from the global minimum found so far. This means that less nodes will be pruned in the parallel version than in the serial version. The parallel code might search more routes than the sequential ones. This is called *search overhead*."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9edf5c1",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>Question:</b> How many routes are fully traversed in total when we assign two branches to each worker? Look at the illustration below. Assume that each worker does pruning locally and independently of the other workers.\n",
    "</div>"
   ]
  },
  {
   "attachments": {
    "g20390.png": {
     "image/png": "[encoded data, 118580 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "673fb192",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"attachment:g20390.png\"  align=\"left\" width=\"300\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95e952af",
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = -1 # Replace with any integer value \n",
    "tsp_check_2(answer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0bc4fdd",
   "metadata": {},
   "source": [
    "In this example, the parallel algorithm traverses more routes (1 more) then the serial version because not all workers are able to use the global minimum distance as a pruning bound. Remember that the sequential code only traverses 3 routes completely. See figure:"
   ]
  },
  {
   "attachments": {
    "g26375.png": {
     "image/png": "[encoded data, 109924 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "f6329f5d",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"attachment:g26375.png\"  align=\"left\" width=\"300\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c8e5eb7",
   "metadata": {},
   "source": [
    "In order to minimize search overhead, workers need to collaboratively keep track of a global minimum distance. However, this needs to be done carefully to avoid race conditions. We show how to do this later in the notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e4cee1a",
   "metadata": {},
   "source": [
    "### Negative search overhead\n",
    "\n",
    "The parallel algorithm might search more branches than the sequential one when we parallelize the pruning process. However, it is also possible that parallel algorithm searches less branches that the sequential one for particular cases. Imagine that the optimal route is on the right side of the tree (or the last route in the tree in the limit case). The parallel algorithm will need less work than the sequential one in this case. The last workers might find the optimal route very quickly and inform the other workers about the optimal minimum, which can then prune branches very effectively. Whereas the sequential algorithm will need to traverse many branches in order to reach the optimal one. If the parallel code does less searches than the sequential one, we say that the search overhead is negative. \n",
    "\n",
    "Negative search overhead is very good for parallel speedups, but it depends on the input values. We cannot rely on it to speed up the parallel execution of the algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6190b00a",
   "metadata": {},
   "source": [
    "### Option 3: Dynamic load balancing with replicated workers model\n",
    "\n",
    "In this third option, we explain a strategy to improve load balance based using the [*replicated workers model*](https://en.wikipedia.org/wiki/Thread_pool) also known as *worker pool* or *thread pool*. In this model, the main processes (aka master or coordinator process) sends jobs to a job queue. Then, workers take one available job from the queue, run it, and take a new job when they are done. In this process, workers never wait for other workers thus fixing the load balance problem. It does not matter if there are some jobs that are larger than others as long as there are enough jobs to keep the workers busy. The main limiting factor of this model is the number of jobs and speed in which the main process is able to generate jobs and send them to the queue. This is an example of **dynamic load balancing**: the load is distributed among the workers at runtime.\n",
    "\n",
    "\n",
    "In our parallel implementation, we will use a coordinator process and several worker processes. The coordinator process will search the tree up to a certain maximum depth given by a number of hops/levels _maxhops_. When _maxhops_ is reached, the coordinator will stop searching the tree and will let any available worker to continue searching in the subtree. In the figure below, the master process will only visit the nodes in the top green box. The worker processes will search in parallel the subtrees below.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {
    "fig-tsp-master-worker-2.png": {
     "image/png": "[encoded data, 290908 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "c6d51e00",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"attachment:fig-tsp-master-worker-2.png\"  align=\"left\" width=\"600\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "497bad13",
   "metadata": {},
   "source": [
    "This avoids the performance issues of the previous approach: the workers are not idle since they get new work every time they finish a job. To avoid the search overhead, it is also important that the workers communicate the minimum distance each time they update their local minimum distance. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e345393",
   "metadata": {},
   "source": [
    "### Performance impact of  maxhops\n",
    "\n",
    "We introduced a new parameter `maxhops`.  Which is then the optimal value for it? When choosing `maxhops`, there is a trade off between load balance and communication overhead.\n",
    "\n",
    "- A small `maxhops` will reduce the number of jobs communicated to the workers (less communication), but reducing the number of jobs is bad for load balance. In the limit, we might generate even less jobs than the number of workers.\n",
    "\n",
    "- A large `maxhops` will increase the number of parallel jobs, improving dynamic load balance, but it will lead to more communication.\n",
    "\n",
    "\n",
    "The optimal value of `maxhops` will depend on the given system, the number of workers, problem size, and also the particular input values. It is not possible to determine it in advance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e4fdbe1",
   "metadata": {},
   "source": [
    "## Implementation of the parallel algorithm \n",
    "\n",
    "We will implement this algorithm using the task-based programming model provided by Distributed.jl as it is convenient to implement the replicated workers model.\n",
    "\n",
    "First, let's add our worker processes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "354f5abb",
   "metadata": {},
   "outputs": [],
   "source": [
    "using Distributed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f8f6702",
   "metadata": {},
   "outputs": [],
   "source": [
    "if workers() == procs()\n",
    "    addprocs(3)\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26188ad3",
   "metadata": {},
   "source": [
    "### Visit paths in parallel\n",
    "\n",
    "Next we will write a simplified version of the algorithm which visits all paths in parallel. The communication between the workers and the coordinator is implemented using a `RemoteChannel`. The coordinator calls the function `generate_jobs` which recursively traverses the search tree until _maxhops_ is reached. Then, it copies the current path to the channel. The workers repeatedly take jobs from the channel until the channel is empty. The workers receive the path from the job and continue the search. When they reach a leaf node in the search tree, they print the path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c0603a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "function visit_all_paths_dist(C_sorted,city,max_hops)\n",
    "    T = typeof((0,Int[]))\n",
    "    jobs_chnl = RemoteChannel(()->Channel{T}(1))\n",
    "    @sync begin\n",
    "        for w in workers()\n",
    "            @spawnat w consume_jobs(C_sorted,jobs_chnl)\n",
    "        end\n",
    "        generate_jobs(C_sorted,city,max_hops,jobs_chnl)\n",
    "        close(jobs_chnl)\n",
    "    end\n",
    "end\n",
    "function generate_jobs(C_sorted,city,max_hops,jobs_chnl)\n",
    "    num_cities = length(C_sorted)\n",
    "    path=zeros(Int,num_cities)\n",
    "    hops = 1\n",
    "    path[hops] = city\n",
    "    generate_jobs_recursive(C_sorted,hops,path,max_hops,jobs_chnl)\n",
    "end\n",
    "function generate_jobs_recursive(C_sorted,hops,path,max_hops,jobs_chnl)\n",
    "    num_cities = length(C_sorted)\n",
    "    if hops == max_hops\n",
    "        @show path\n",
    "        put!(jobs_chnl,(hops,copy(path)))\n",
    "        return nothing\n",
    "    else\n",
    "        city = path[hops]\n",
    "        connections = C_sorted[city]\n",
    "        next_hops = hops + 1\n",
    "        for (next_city,distance_increment) in connections\n",
    "            already_visited = (next_city in view(path,1:hops))\n",
    "            if ! already_visited\n",
    "                path[next_hops] = next_city\n",
    "                generate_jobs_recursive(C_sorted,next_hops,path,max_hops,jobs_chnl)\n",
    "            end\n",
    "        end\n",
    "        return nothing\n",
    "    end\n",
    "end\n",
    "@everywhere function consume_jobs(C_sorted,jobs_chnl)\n",
    "    while true\n",
    "        try\n",
    "            hops,path = take!(jobs_chnl)\n",
    "            consume_jobs_recursive(C_sorted,hops,path)\n",
    "        catch\n",
    "            break\n",
    "        end        \n",
    "    end\n",
    "end\n",
    "@everywhere function consume_jobs_recursive(C_sorted,hops,path)\n",
    "    num_cities = length(C_sorted)\n",
    "    if hops != num_cities\n",
    "        city = path[hops]\n",
    "        connections = C_sorted[city]\n",
    "        next_hops = hops + 1\n",
    "        for (next_city,distance_increment) in connections\n",
    "            already_visited = (next_city in view(path,1:hops))\n",
    "            if ! already_visited\n",
    "                path[next_hops] = next_city\n",
    "                consume_jobs_recursive(C_sorted,next_hops,path)\n",
    "            end\n",
    "        end\n",
    "        return nothing\n",
    "    else\n",
    "        @show path\n",
    "        return nothing\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02801b66",
   "metadata": {},
   "outputs": [],
   "source": [
    "C = [\n",
    "    0 2 3 2\n",
    "    2 0 4 1\n",
    "    3 4 0 3\n",
    "    2 1 3 0    \n",
    "]\n",
    "C_sorted = sort_neighbors(C)\n",
    "city = 1\n",
    "max_hops = 2\n",
    "min_distance = visit_all_paths_dist(C_sorted,city,max_hops)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3bbf609",
   "metadata": {},
   "source": [
    "### How to track the global minimum distance?"
   ]
  },
  {
   "attachments": {
    "fig-tsp-master-worker.png": {
     "image/png": "[encoded data, 290908 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "d85c8e7c",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"attachment:fig-tsp-master-worker.png\"  align=\"left\" width=\"600\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d579e3d",
   "metadata": {},
   "source": [
    "\n",
    "### Simplified example\n",
    "\n",
    "We will demonstrate how the workers communicate the minimum distance with each other with a short example. Each worker generates a random value and updates a globally shared minimum. The variable for the global minimum is stored in a `RemoteChannel`. The buffer size of the channel is 1, such that only one worker can take and put new values to the channel at a time, thus solving the race condition problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0303392",
   "metadata": {},
   "outputs": [],
   "source": [
    "buffer = 1 # Very important\n",
    "min_distance_chnl = RemoteChannel(()->Channel{Int}(buffer))\n",
    "put!(min_distance_chnl,typemax(Int))\n",
    "@sync for w in workers()\n",
    "    @spawnat w begin\n",
    "         sleep(rand(1:3))\n",
    "         min_distance = take!(min_distance_chnl)\n",
    "         @show min_distance\n",
    "         distance = rand(5:10)\n",
    "         @show distance\n",
    "         min_distance = min(distance,min_distance)\n",
    "         put!(min_distance_chnl,min_distance)\n",
    "    end\n",
    "end\n",
    "min_distance = take!(min_distance_chnl)\n",
    "@show min_distance\n",
    "close(min_distance_chnl)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fabefcb",
   "metadata": {},
   "source": [
    "### Final parallel implementation\n",
    "Finally, we add the communication of the minimum distance to the first simplified version. Like in the serial implementation, the workers stop the search if the distance of their current path exceeds the global minimum distance. In total, we need two channels: one to communicate the jobs and one to communicate the minimum distance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96c649b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "function tsp_dist(C_sorted,city,max_hops)\n",
    "    T = typeof((0,Int[],0))\n",
    "    jobs_chnl = RemoteChannel(()->Channel{T}(1))\n",
    "    min_distance_chnl = RemoteChannel(()->Channel{Int}(1))\n",
    "    put!(min_distance_chnl,typemax(Int))\n",
    "    @sync begin\n",
    "        for w in workers()\n",
    "            @spawnat w consume_jobs(C_sorted,jobs_chnl,min_distance_chnl)\n",
    "        end\n",
    "        generate_jobs(C_sorted,city,max_hops,jobs_chnl)\n",
    "        close(jobs_chnl)\n",
    "    end\n",
    "    min_distance = take!(min_distance_chnl)\n",
    "    close(min_distance_chnl)\n",
    "    return min_distance\n",
    "end\n",
    "function generate_jobs(C_sorted,city,max_hops,jobs_chnl)\n",
    "    num_cities = length(C_sorted)\n",
    "    path=zeros(Int,num_cities)\n",
    "    hops = 1\n",
    "    path[hops] = city\n",
    "    distance = 0\n",
    "    generate_jobs_recursive(C_sorted,hops,path,max_hops,jobs_chnl,distance)\n",
    "end\n",
    "function generate_jobs_recursive(C_sorted,hops,path,max_hops,jobs_chnl,distance)\n",
    "    num_cities = length(C_sorted)\n",
    "    if hops == max_hops\n",
    "        #@show path, distance\n",
    "        put!(jobs_chnl,(hops,copy(path),distance))\n",
    "        return nothing\n",
    "    else\n",
    "        city = path[hops]\n",
    "        connections = C_sorted[city]\n",
    "        next_hops = hops + 1\n",
    "        for (next_city,distance_increment) in connections\n",
    "            already_visited = (next_city in view(path,1:hops))\n",
    "            next_distance = distance + distance_increment\n",
    "            if ! already_visited\n",
    "                path[next_hops] = next_city\n",
    "                generate_jobs_recursive(\n",
    "                        C_sorted,next_hops,path,max_hops,jobs_chnl,next_distance)\n",
    "            end\n",
    "        end\n",
    "        return nothing\n",
    "    end\n",
    "end\n",
    "@everywhere function consume_jobs(C_sorted,jobs_chnl,min_distance_chnl)\n",
    "    while true\n",
    "        min_distance = take!(min_distance_chnl)\n",
    "        put!(min_distance_chnl,min_distance)\n",
    "        try\n",
    "            hops,path,distance = take!(jobs_chnl)\n",
    "            consume_jobs_recursive(C_sorted,hops,path,min_distance_chnl,distance,min_distance)\n",
    "        catch\n",
    "            break\n",
    "        end\n",
    "    end\n",
    "end\n",
    "@everywhere function consume_jobs_recursive(C_sorted,hops,path,min_distance_chnl,distance,min_distance)\n",
    "    if distance >= min_distance\n",
    "        return min_distance\n",
    "    end\n",
    "    num_cities = length(C_sorted)\n",
    "    if hops != num_cities\n",
    "        city = path[hops]\n",
    "        connections = C_sorted[city]\n",
    "        next_hops = hops + 1\n",
    "        for (next_city,distance_increment) in connections\n",
    "            already_visited = (next_city in view(path,1:hops))\n",
    "            next_distance = distance + distance_increment\n",
    "            if ! already_visited\n",
    "                path[next_hops] = next_city\n",
    "                min_distance = consume_jobs_recursive(\n",
    "                        C_sorted,next_hops,path,min_distance_chnl,next_distance,min_distance)\n",
    "            end\n",
    "        end\n",
    "        return min_distance\n",
    "    else\n",
    "        min_distance = take!(min_distance_chnl)\n",
    "        min_distance = min(min_distance,distance)\n",
    "        put!(min_distance_chnl,min_distance)\n",
    "        #@show path, distance, min_distance\n",
    "        return min_distance\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67888155",
   "metadata": {},
   "outputs": [],
   "source": [
    "city = 1\n",
    "max_hops = 2\n",
    "min_distance = tsp_dist(C_sorted,city,max_hops)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48b54da1",
   "metadata": {},
   "source": [
    "## Testing the parallel implementation\n",
    "Next, we will test the correctness and performance of our parallel implementation by comparing the results of the parallel algorithm to the results of the serial algorithm for multiple problem instances. Run it for different values of `n` and `max_hops`. Try to explain the impact of these values on the parallel efficiency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d950905",
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 18 # Safe to run up to 18 on a laptop\n",
    "using Random\n",
    "Random.seed!(1)\n",
    "C = rand(1:10,n,n)\n",
    "C_sorted = sort_neighbors(C)\n",
    "city = 1\n",
    "verbose = false\n",
    "T1 = @elapsed min_serial = tsp_serial(C_sorted,city)\n",
    "max_hops = 2\n",
    "P = nworkers()\n",
    "TP = @elapsed min_dist = tsp_dist(C_sorted,city,max_hops)\n",
    "println(\"Sequential time = \", T1)\n",
    "println(\"Parallel time = \", TP)\n",
    "println(\"Speedup = \", T1/TP)\n",
    "println(\"Optimal speedup = \", P)\n",
    "println(\"Efficiency = \", 100*(T1/TP)/P, \"%\")\n",
    "@test min_serial == min_dist"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92e68978",
   "metadata": {},
   "source": [
    "### Super-linear speedup\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a724509",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>Question:</b> For some values of `n` and `max_hops` the parallel efficiency can be above 100% (super-linear speedup). For example with `n=18` and `max_hops=2`, I get super-linear speedup on my laptop for some runs. Explain a possible cause for super-linear speedup in this algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eebe7e9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "uncover = false\n",
    "q_superlinear_answer(uncover)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19835531",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "- We studied the solution of the TSP problem using a branch and bound strategy\n",
    "- The problem is $O(N!)$ complex in the worse case scenario, where $N$ is the number of cities.\n",
    "- Luckily, the compute time can be drastically reduced in practice using the nearest city first heuristic and branch pruning.\n",
    "- Pruning, however, introduces load imbalance in the parallel code. To fix this, one needs a dynamic load balancing strategy as the actual work per worker depends on the input matrix (runtime values).\n",
    "- A replicated workers model is useful to distribute work dynamically. However, it introduces a trade-off between load balance and communication depending on the value of `maxhops`.\n",
    "- The parallel code might suffer from positive search overhead (if the optimal route is on the left of the tree) or it can benefit from negative search overhead (if the optimal route is on the right of the tree).\n",
    "- In some cases, it is possible to observe super-linear speedup thanks to negative search overhead.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c789dc7a",
   "metadata": {},
   "source": [
    "# License\n",
    "\n",
    "\n",
    "\n",
    "This notebook is part of the course [Programming Large Scale Parallel Systems](https://www.francescverdugo.com/XM_40017) at Vrije Universiteit Amsterdam and may be used under a [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) license."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.10.0",
   "language": "julia",
   "name": "julia-1.10"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
