Usually what happens in cases like this is that the RL network ends up learning a slightly crappier version of A* Or finds a bug in the environment and learns

Usually what happens in cases like this is that the RL network ends up learning a slightly crappier version of A* Or finds a bug in the environment and learns to teleport, those are fun too