2 ms·
One of the challenges I have had particularly with small teams (aka startups) is deciding what is worthy of failure and how to avoid fatigue if being too aggres
by agentgt 10y ago
One of the challenges I have had particularly with small teams (aka startups) is deciding what is worthy of failure and how to avoid fatigue if being too aggressive with what failure is.
I have found if you are not aggressive with what a failure is (aggressive meaning classifying things that are not really fatal as outage... the system is up but there are lots of errors) it will bite you the in the ass in the long run. The small errors become frequent big errors.
The problem is if you are too aggressive you will eventually get alerting fatigue.
I don't have a foolproof solution. I have done things like fingerprinting exceptions and counting them to the extreme of failing really fast (ie crash on any error).
In large part this because small teams just don't have the resources to get this right but still have the demands to deliver more functionality.
I wish the article delved into this more because there are different levels of "its down".