I work on a system with notoriously noisy alarming and messaging logging - my first step is to tell the customer to ignore all of it. I tell them to do two thi

I work on a system with notoriously noisy alarming and messaging logging - my first step is to tell the customer to ignore all of it. I tell them to do two things to tell them how to check system health. 1. Is it working normally? if so, then nothing is wrong. 2. Walk by the hardware once a week and check for idiot lights, check the system dashboard if one of the idiot lights is on. If no error is found, its a hardware problem (failed supply, drive, fan), if an error is found it's a failed server - either way, call us. There are a handful of cases not fixed by this, but they're limited to time sync failures - which are critical, but generates no user facing alarms.