Thanks for the detailed explanation. It sounds like everything was done to fix the problem except try to figure out what the problem actually was. Why not use

Thanks for the detailed explanation. It sounds like everything was done to fix the problem except try to figure out what the problem actually was. Why not use tools to see what the program is doing, form a hypothesis, gather data to confirm or reject the hypothesis, repeat until cause found, and then take corrective action that by this point you have high confidence will work? I realize HN is more of a side project than a production service, but the goal is the same in both cases: to restore service quickly so you can move on to other things. It feels like a more rigorous approach would allow restoring service much faster than randomly guessing about what could be wrong and applying (costly) corrective action to see if it helps. Besides that, in many cases (including this one), you cannot randomly guess the appropriate corrective action without finding the root cause.