5 ms·

An excellent read. I have ECC RAM on my Threadripper board. One of the things that ops at Blekko found was that you needed to tell Intel boards (NOTE vendor cha

An excellent read. I have ECC RAM on my Threadripper board. One of the things that ops at Blekko found was that you needed to tell Intel boards (NOTE vendor change!) to actually REPORT correctable errors. The default was machine-check if there was an unrecoverable error, otherwise just go with the flow. With ~ 1600 192GB systems we would see correctable errors about once a week as I recall. I don't recall a single unrecoverable error in 6 years so that was pretty good. Greg will surely correct me if I missed one :-).

toast0 3y ago
Your luck was better than mine, but at least our systems reported correctable errors without us having to ask. We had a roughly similar amount of machines, probably about similar ram on average (many with less, and some with much more) We definitely had unrecoverable errors from time to time; maybe one or two a year, enough to have a policy: give it a shot to see if it was a one off, if it doesn't fail again shortly after it's probably fine; if it does fail again shortly after, swap the ram. Nicer server boards even have a led to tell you which ram module to replace. For correctable errors, we wouldn't swap unless the error counts got pretty big; some systems went for a long time with one or two errors a day, which is fine. Others went zero errors for a long time, then a couple days at a small number, and then big numbers. There was one system that managed to get thousands per hour and the system was unusable because handling machine check exceptions was too expensive; unfortunately the reporting interval was one hour, so we didn't realize why the system was unusable until the next report.
- ChuckMcM 3y ago
  Well to be fair if we started regularly getting UCE's on a stick of memory we did preemptively replace it. (where regularly was more frequently than one a week)
  - toast0 3y ago
    Yeah, I'd guess a good number of the systems where we got one UCE, we got another one within a day after reboot. If so, off to repair. If not, it was pretty rare for that machine to cause trouble again; seemed worth the risk to check if repair was needed. The correctables were trickier to judge, because system halt on UCE is easy to recover from and easy to diagnose (system is down, check console, see UCE message); system slow as heck because of constant machine check exceptions is hard to diagnose and a slow system can disrupt a distributed system a lot more than a dead system.
    - ChuckMcM 3y ago
      Greg Lindhal had created a really fascinating system that surfaced anomalies like that to ops so that the systems could be fixed. He had a cute name for them like 'fractures' or something (where it was a non-fatal failure but it was impacting overall performance). Things the system would catch were switch ports going bad, DIMMs going bad, and file system corruption outside of the set of released tools. Disks failure was fairly common and got caught, drive reformatted, and resumed quickly.By tracking the rate of disk FS errors for a particular drive we could pick up the "you need to replace this drive" signal as well.
      - toast0 3y ago
        We mostly relied on a couple of easy metrics to signal trouble and then debug from there. Total size of all the Erlang message queues above a threshold was usually an indication of trouble; replication delays of more than a couple seconds too. CPU %, swap % were also signs of trouble. Tracking down almost working networking without access to the switch metrics was kind of fun, ish. :P

toast0 3y ago

Your luck was better than mine, but at least our systems reported correctable errors without us having to ask. We had a roughly similar amount of machines, probably about similar ram on average (many with less, and some with much more) We definitely had unrecoverable errors from time to time; maybe one or two a year, enough to have a policy: give it a shot to see if it was a one off, if it doesn't fail again shortly after it's probably fine; if it does fail again shortly after, swap the ram. Nicer server boards even have a led to tell you which ram module to replace. For correctable errors, we wouldn't swap unless the error counts got pretty big; some systems went for a long time with one or two errors a day, which is fine. Others went zero errors for a long time, then a couple days at a small number, and then big numbers. There was one system that managed to get thousands per hour and the system was unusable because handling machine check exceptions was too expensive; unfortunately the reporting interval was one hour, so we didn't realize why the system was unusable until the next report.

ChuckMcM 3y ago
Well to be fair if we started regularly getting UCE's on a stick of memory we did preemptively replace it. (where regularly was more frequently than one a week)
- toast0 3y ago
  Yeah, I'd guess a good number of the systems where we got one UCE, we got another one within a day after reboot. If so, off to repair. If not, it was pretty rare for that machine to cause trouble again; seemed worth the risk to check if repair was needed. The correctables were trickier to judge, because system halt on UCE is easy to recover from and easy to diagnose (system is down, check console, see UCE message); system slow as heck because of constant machine check exceptions is hard to diagnose and a slow system can disrupt a distributed system a lot more than a dead system.
  - ChuckMcM 3y ago
    Greg Lindhal had created a really fascinating system that surfaced anomalies like that to ops so that the systems could be fixed. He had a cute name for them like 'fractures' or something (where it was a non-fatal failure but it was impacting overall performance). Things the system would catch were switch ports going bad, DIMMs going bad, and file system corruption outside of the set of released tools. Disks failure was fairly common and got caught, drive reformatted, and resumed quickly.By tracking the rate of disk FS errors for a particular drive we could pick up the "you need to replace this drive" signal as well.
    - toast0 3y ago
      We mostly relied on a couple of easy metrics to signal trouble and then debug from there. Total size of all the Erlang message queues above a threshold was usually an indication of trouble; replication delays of more than a couple seconds too. CPU %, swap % were also signs of trouble. Tracking down almost working networking without access to the switch metrics was kind of fun, ish. :P