I debugged a few crazy bugs in the past. Example: we have a listing of some items sorted by ascending distance from the user and someone noticed the ordering is

I debugged a few crazy bugs in the past. Example: we have a listing of some items sorted by ascending distance from the user and someone noticed the ordering isn't always right. Luckily at some point I looked at the SQL server production logs and noticed that the SQL string is always the same, doesn't include the coordinates of the newly arrived user. This bug turned out to be there for years and nobody ever noticed it. The pitfall was that there was a constant reference to a template SQL string for doing the ordering, that was passed to an external plugin that filled the template with detailed data. Unfortunately, the plugin modified the string instead of making a copy, so the sorting criteria applied correctly only to the first user. More unfortunately, there were multiple application servers in a round robin, so when you did the first few tests you hit different servers and the behaviour was correct until you managed to hit the same server the second time with different coordinates.