3 ms·
Greg Lindhal had created a really fascinating system that surfaced anomalies like that to ops so that the systems could be fixed. He had a cute name for them li
by ChuckMcM 3y ago
Greg Lindhal had created a really fascinating system that surfaced anomalies like that to ops so that the systems could be fixed. He had a cute name for them like 'fractures' or something (where it was a non-fatal failure but it was impacting overall performance). Things the system would catch were switch ports going bad, DIMMs going bad, and file system corruption outside of the set of released tools. Disks failure was fairly common and got caught, drive reformatted, and resumed quickly.By tracking the rate of disk FS errors for a particular drive we could pick up the "you need to replace this drive" signal as well.
- toast0 3y agoWe mostly relied on a couple of easy metrics to signal trouble and then debug from there. Total size of all the Erlang message queues above a threshold was usually an indication of trouble; replication delays of more than a couple seconds too. CPU %, swap % were also signs of trouble. Tracking down almost working networking without access to the switch metrics was kind of fun, ish. :P