2 ms·

Well to be fair if we started regularly getting UCE's on a stick of memory we did preemptively replace it. (where regularly was more frequently than one a week)

• toast0 3y ago
  Yeah, I'd guess a good number of the systems where we got one UCE, we got another one within a day after reboot. If so, off to repair. If not, it was pretty rare for that machine to cause trouble again; seemed worth the risk to check if repair was needed. The correctables were trickier to judge, because system halt on UCE is easy to recover from and easy to diagnose (system is down, check console, see UCE message); system slow as heck because of constant machine check exceptions is hard to diagnose and a slow system can disrupt a distributed system a lot more than a dead system.
  • ChuckMcM 3y ago
    Greg Lindhal had created a really fascinating system that surfaced anomalies like that to ops so that the systems could be fixed. He had a cute name for them like 'fractures' or something (where it was a non-fatal failure but it was impacting overall performance). Things the system would catch were switch ports going bad, DIMMs going bad, and file system corruption outside of the set of released tools. Disks failure was fairly common and got caught, drive reformatted, and resumed quickly.By tracking the rate of disk FS errors for a particular drive we could pick up the "you need to replace this drive" signal as well.
    • toast0 3y ago
      We mostly relied on a couple of easy metrics to signal trouble and then debug from there. Total size of all the Erlang message queues above a threshold was usually an indication of trouble; replication delays of more than a couple seconds too. CPU %, swap % were also signs of trouble. Tracking down almost working networking without access to the switch metrics was kind of fun, ish. :P