3 ms·
In the context of alerting, I agree with TFA. You should not be alerting on bad request errors, because you might have no control over it. That said you might w
by breischl 5y ago
In the context of alerting, I agree with TFA. You should not be alerting on bad request errors, because you might have no control over it. That said you might want monitoring on it so you can check if the rate jumped at some important point (eg, after a deployment) but I wouldn't look at it on a regular basis.
I had something like that on an internal system. The 400 rate would jump all over the place because our edge systems had shitty input validation, and bots would crawl us with broken requests ("can I reserve this item starting last week?" kind of thing) with no rate throttling. After a few years the edge validation (and bot detection) got better, but alerting on that would've been worse than useless.
- cle 5y agoYeah I agree that false positives are a risk with monitoring 4xx rate. I've never seen a satisfactory "bulletproof" way to monitor 4xx rate, it's inherently difficult and simultaneously important to monitor. It's easy in retrospect to say "oh that was a waste of time because it was just bots" but you don't know that until you investigate. I ask myself "if I see elevated 4xx's, at what point do I start to care if they're caused by a bug?" and set monitor thresholds somewhere around there.
- 8note 5y agoThe client who's calling wrong might want some help getting it right. Its empathetic to keep an eye on surges in 4xxs
- closeparen 5y agoSure, but there’s a big world between keeping an eye on surges and waking someone up if it goes out of 3-4 nines.
- raynorelyp 5y agoAuthor here. This.