3 ms·
I have worked at a company where every 500 error is a reason to go check the logs and try to reproduce... We would literally page an engineer. I've also worke
by londons_explore 6y ago
I have worked at a company where every 500 error is a reason to go check the logs and try to reproduce... We would literally page an engineer.
I've also worked at a company where "as long as the error rate stays under 0.1%, it's fine". As long as each app server doesn't have more than 1 crash per week it's fine.
I can tell you that engineers at the first company end up finding all kinds of super rare bugs, usually in the OS, platform, malloc library, load balancer, etc. They then commit fixes for those bugs which ends up helping the latter company...
- treve 6y agoIt's an interesting take. We definitely don't have the capacity & skills to be in the first company, although it would be awesome. I think most places I worked we had no choice but to be in the second company, although I think the rate is a bit below .1%. I'm grateful for all the people hunting down these bugs and making the world better for the rest of us. I remember around 2006, we still found PHP segfaults that were critical enough I reported at least a dozen. Everything feels so much more stable today (at least on that layer).
- shinji97 6y agocurious if you can share a little more on the first company you mentioned, what was their area of business?
- khaless 6y agoWhile it's nice to get paged, and look at every 5xx error; it doesn't really scale all that well once you get past a certain point, particularly if your application is gracefully degrading. That said, I love the wisdom in your comment that you find all sorts of super rare bugs, or conditions that could seriously effect performance, or availability if they become more common (which they often do). Past a point, I've found that an approach which works well is to encourage engineers/operators to drive by metrics, and pay close attention over time to p100's (max), as you've suggested with your 500 errors. Lots of goodies can be hidden behind them, just like you've found with the 500 errors.