3 ms·
What metrics do you alert on? How do you distinguish between error due to faulty database client vs error due to database disk failure?
by potamic 2y ago
What metrics do you alert on? How do you distinguish between error due to faulty database client vs error due to database disk failure?
- dullcrisp 2y agoDefine SLOs based on what can realistically be achieved with underlying infrastructure, only alert if those SLOs are breached?
- majewsky 2y agoTaking my managed container image registry service as an example. - The only critical alert that can actually page people is if the blackbox test fails. Every 30 seconds, it downloads a test image and if the contents don't match the expectation, an alert is raised (with some delay). - Warning alerts are mostly for any errors being returned from background tasks, but these are only monitored during business hours.
- perfect_wave 2y agoi dont see how that is separated from the underlying infra. If the network/server/some dependency goes down, the blackbox test will fail and you'll get paged.
- silisili 2y agoYou can test for this. For example, we had routines that were called on repeated HTTP failures that would then get 5 or so of the top US websites. If those fail too, it moves from an application error to an infra one.
- sgarland 2y agoIf your endpoint is failing, it might be you. If everyone’s endpoint is failing, it’s almost certainly not you.