Some complementary reading could be My Philosophy on Alerting ( https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q https://docs.goo

Some complementary reading could be My Philosophy on Alerting ( https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa... ) and https://how.complexsystems.fail/ https://how.complexsystems.fail/ In any case, not all signals are the same. Most systems have a lot of components interacting and what turns to be dangerous is usually a combination of factors, but in the end, what defines that it was or not is that the system is doing what it should. You can put some guessing thresholds, but you must contrast it with that the system works. And they should be actionable too, at least for alerts instead of slow day notifications, or metrics giving context to perceived problems that could take out the guessing from the thresholds.