2 ms·
Most of this appears to be just end-to-end testing, and whether you're alerting on a failure of testing the entire application stack or just individual componen
by 0xbadcafebee 12y ago
Most of this appears to be just end-to-end testing, and whether you're alerting on a failure of testing the entire application stack or just individual components. He probably got paged by too many individual alerts versus an actual failure to serve data, which I agree would be annoying.
In a previous position, we had a custom ticketing system that was designed to also be our monitoring dashboard. Alerts that were duplicates would become part of a thread, and each was either it's own ticket or part of a parent ticket. Custom rules would highlight or reassign parts of the dashboard, so critical recurrent alerts were promoted higher than urgent recurrent alerts, and none would go away until they had been addressed and closed with a specific resolution log. The whole thing was designed so a single noc engineer at 3am could close thousands of alerts per minute while logging the reason why, and keep them from recurring if it was a known issue. The noc guys even created a realtime console version so they could use a keyboard to close tickets with predefined responses just seconds after they were opened.
The only paging we had was when our end-to-end tests showed downtime for a user, which were alerts generated by some paid service providers who test your site around the globe. We caught issues before they happened by having rigorous metric trending tools.
- robewaschuk 12y agoI don't think it's end-to-end testing because "testing" to me implies a synthetic environment. This is about instrumenting and monitoring the production system at scale, and learning about the right things at the right time. It certainly shares some things with end-to-end testing, and blackbox monitoring is very useful for finding high level problems with any complex networked system.
- peterwwillis 12y agoSo he's talking about system testing and not end-to-end testing. I suppose if your application is really simple, system testing is fine. But if your QA group ever starts automating tests, it's time to re-evaluate. Blackbox monitoring is (imho) only appropriate for 3rd parties. If it's part of your company, it shouldn't be a black box; that means someone got lazy and didn't demand the devs provide an API. Also, i'm sorry but this really gets to me: at what point are we talking about 'at scale' ? I think it's whenever tons of money is riding on your site's availability and an unexpected failure causes customers to complain. Immediately VPs start screaming "WE NEED TO SCALE UP!!" and then they mandate some half-assed implementation of the comprehensive monitoring solution they claimed was unnecessary just a month before. But maybe i'm just jaded.