Day 15 - Wend and weave our winding way

I love me a path finding algorithm and this feels like a path finding algorithm problem.

This is how I got my teeth cut as a child programming computer games. I loved the earliest Hack, Nethack, Angband and Rogue games, which were available on AMInet for my Amiga for free, and since I couldn't draw, the idea of making games that just had little symbols being printed in different locations was very attractive to me. But the moment you start writing that kind of game, you'll get to a point where you need to find the path from point A to point B, or know whether it's even possible, or find a random path or similar. By the time I was 17, and I opted for the Decision and Discreete mathmatics in my A-Level, we moved onto path anaylsis and I realised that I'd been solving these problems for years, but had no formal training to understand the difference between a breadth first search and a depth first search, or who Djikstra was. I went back and devoured everything I could find on this stuff, and since then I've written these algorithms, especially A*, multiple times.

But that aside, let's properly read the days description.

So there's a few things we need to pick up here. The first is that we cannot move diagonally, so from any given location, we can only move in 4 directions, and one of those is backwards the way we came. The second is that we count nodes as we enter them.

There's a bit of mental thinking here, when we make graphs like this, we can either think of the cost being associated with the node, so the node at (1,1) has cost or risk level 3. Or we can think of the cost as being on the path that connects the node, so traversing from (1,0) or (0,1) to (1,1) will cost 3. Sometimes it doesn't make any difference, and sometimes we might start looking at things like doors, hills or other issues that occur between nodes. It can then sometimes be useful to think of the cost as being between the nodes rather than on the nodes themselves. In this case, we're going to say that the cost of moving from X node to Y node will always be the danger level of Y node. (At this point, I'm also thinking, what's the part 2 here, and I wonder if the part 2 is going to add some additional cost when we move. It would be nice to have a cost function that calculates the cost to get to a node, but it's probably an early optimisation, so I'm not going to do it).

Anyway, the final thing here is that we are looking for the path with the lowest total risk. This can also be described as the path with the lowest cost, or the shortest path. Djikstra is going to be our friend here.

There are a couple of ways we can do this, the easiest is a "breadth first search", we just brute force our way through every node, seeing if we can get there. We're going to need to think about some interesting cases, the main one being that given the following layout:

ab
cd

We can get from A to D either through b or c. Our alogorithm is going to work out teh cost to get from a->b and then from a->b->d. It'll store that cost at d becuase we've got there. Later it might try a->c->d. It will see that we've already got to d via another route, if the total cost via our route is lower than the previous routes cost, then it can update the cost on the node.

This means that as our algorithm winds it way through the map, it will keep updated on each node, the cheapest or fastest way to get from the start to that node. This will mean that when we check for the next node, we can always work out which of its cheapest neighbours it can be connected from.

But, we're only going to have partial knowledge at any given time, so as we explore the potential route, we'll be continually updating our information as we go.

Now we've got a second decision to make in this whole process, how do we select which neighbour to try visiting next? Well this is the difference that an A* algorithm normally makes. It uses an estimation of cost function, and it selects the node with the lowest estimated cost. In our case, we will estimate the cost as being 1 per node travelled, and since from any node at (x1,y1) to get to (x2,y2), if every node was a 1, the estimated cost would be by moving x2-x1 nodes across and y2-y1 nodes down. We call this the Manhatten distance, becuase the Manhatten road system is based on grids, so the cost to get from any intersection is always something like 4 up and 3 across for 7 total.

We're going to use the same approach as we used in Day 9 more or less. We can have an open list, these are all the possible locations we haven't visited yet, which are on our path. We're going to pick the location from the open list that has the lowest estimated cost. We're going to examine the location, and we're going to calculate the cost of getting here as being the cost of our predecessor, plus the cost of the location. We then need to check to see if the new cost is lower than an existing stored cost. If not, we leave it alone. If it's lower, then we change the cost and the predecessor as to where we came from. Then we add all the neighbours of the node to the open list, with an estimated cost of that node being the cost so far plus the estimated cost to the end. We keep doing this until we hit the end node, at that point, we can check the predecessor, and follow the path back trhough the cheapest routes until we hit the start and add up the cost as we go.

Final point, I note that we have no example mazes here. I'm going to use a couple of examples because it should stress test our algorithm, first the simple maze, we've added an obvious bump to it

111
331
331

The total here is 4, going right, right, down, down

Now we add one with a deliberate giant bump, so we expect the path to go round

111
339
331

The total here is likely using path right, down, down, right for 8 total.

Finally lets have a path which should involve backtracking

11111
99191
11191
19999
11111

The ideal path here is right, right, down, down, left, left, down, down, right, right, right, right, right, for 13 cost, beating right, right, right, right, down, down, down, down at 15 cost.

First thing first, let's reuse our Grid cost from Day 9

Ok, we've got a grid from Day 9, so let's write out basic pathfinding function on it.

We're going to use python's heapq module, which maintains a cheap and easy sorted list, called a heap. Every node we put into it is stored as (cost, node), and heap[0] will be the smallest item. We can use heappush to add an item, and heappopto remove the smallest item. It just makes it easy to always get the smallest item.

It's been a long time since I last implemented that, so I've left the debugging lines in so you can see how I played with it. Couple of gotchas for me in that process.

  1. the last node is at height-1, width-1 not height-width, that made it search the whole system and decide there was no path (hence the ran out of nodes, that "looking for {end}" was me trying to work out why I kept running out)
  2. The heap holds a tuple of (estimatedcost, coordinate) but everything else looks stuff up by coordinate. that estimated cost I now ignore when popping from the heap because it's one and only purpose is to help us try to find the most effective route, so we don't use it for anything else.2
  3. I had to refactor thiscost, estcost out becuase I kept using the wrong one.

Guess we can try that on our other two test paths, and then I'll rewrite it without the debugging

Fabulous, that seems to work, lets try it on the test data and then the production data

Now for production data

A much bigger map

Ok, this should be simple for me, my A* algorithm tries optimially to explore downwards and right constantly, all I need to do is generate a much bigger grid and feed it in.

This map generation feels much harder than the actual path finding! Guess lets give it a go (and be thankful that I didn't prematurely optimise assuming that per node costs would change)

Note: for this I want to compare Grid objects, but I didn't add an __eq__ function, so I'm going to go back and add it

Ok, we can create a bigger grid, lets pathfind our way through the test data and see if our expanded path looks sensible

At this point, something worth pointing out, once the path comes off of the left side, my path goes 3->2->4->1->7->2->1. But the example given selects a slightly different path, 3->2->4->1->5->4->1. In this case the 5+4 is the same cost as going 7+2, so either path is valid. If the requirement needed us to know what the actual path was. But what we've found is that there are several equivalent paths.

Anyway, lets give it a try on the production data.

I'm not going to render this one, because the production data was significantly bigger than the test data and when expanded 5 times in each direction it will be far less interesting.