3 ms·
Do you have munin monitoring on the production HN server? That would really make situations like this easier to debug. First, it can pinpoint exactly when som
by birken 14y ago
Do you have munin monitoring on the production HN server?
That would really make situations like this easier to debug. First, it can pinpoint exactly when something started happening, which in this situation might have helped you realize the problem was caused by your change. Secondly, in this specific situation it probably would have been easier to differentiate a situation where you are running low on memory vs this completely different situation.
As somebody who spent a lot of time professionally debugging large software systems when they were misbehaving (as a Google SRE), I can tell you that looking at graphs of many key metrics (disk IO, CPU, memory, then application specific things) was always the place to start when debugging a situation, because you can learn so many things right away. When did it start? Was it a slow buildup or an immediate thing? What is the general problem (Memory?, Disk IO?, CPU?, none of the above?)? Has a similar pattern happened in the past?
Then you can start to get fancy and plot things like "messages/minute" or something and then it becomes easy to see when issues are affecting the site performance and when they aren't.
- stcredzero 14y agoThat and something like the Smalltalk Change Log would have made this a no-brainer debug. (Yes, every REPL action in Smalltalk got logged by the same mechanism that logged every code change.) Such mechanisms aren't trivial, but they're not rocket science either, and they have tremendous ROI.