3 ms·

Fascinating read. Kudos to the folks who no doubt had to work hard to identify these esoteric bugs. I do wonder if it is worth it for these companies to identif

Fascinating read. Kudos to the folks who no doubt had to work hard to identify these esoteric bugs. I do wonder if it is worth it for these companies to identify these issues - it sounds more economical to just isolate the faulty machine and get rid of it, instead of bothering to investigate. On the other hand, they perhaps can't rule out that the issue isn't something more widespread.

mikeytown2 5y ago
At scales this large, finding the root cause is the best choice. By finding the cause you can then do tests on all boxes via cron to make sure everything works as expected; allowing you to pull hardware before an issue becomes widespread; can then rerun the boxes inputs on another machine if it was handling more than serving http requests. Robust hardware validation and testing is a must when dealing with millions of cores.
- jeffbee 5y ago
  I once worked where a faulty machine out of a massive fleet was finally sent back to intel. They put it under some kind of exotic test fixture and confirmed it was fucked. It was a defect never seen again or before. Root-causing that stupid chip was a complete waste of everyone's time.
- DylanSp 5y ago
  That's sort of the opposite of what I've read from the companies operating at this scale; most seem to assume that failures are inevitable in a sufficiently large distributed system, that tracking the root cause of every failure is infeasible, and that the solution instead is to design resilient systems.
  - cestith 5y ago
    It's probably best to find a root cause for the first instance of a new class of problems. At first there was no way to be sure what part of the stack some of these were in. They ruled out their software, third-party libraries, and memory before focusing on a CPU and then which core of the CPU. Knowing that sometimes a single core goes bad means you can't randomly assign one test run to a CPU with no core affinity and reliably detect it. Yes, the fix is to disable that core or core complex if you have that kind of resolution, or to ditch the CPU if you must, or to pull the whole server if your scale demands it. But knowing it's a hardware error in the CPU and not chasing it through layers of code every time is invaluable.
  - piyh 5y ago
    You can design resiliency to specific faults. Server goes down, bad code deploy, cosmic rays flip a bit. Each resiliency design looks different.
cratermoon 5y ago
The issue at scale is that you don't know if there's an underlying flaw that will manifest all at once across enough machines to cause a real outage or major service disruption.
yuliyp 5y ago
What is a faulty machine? How do you find one? Some of the failures experienced happened with specific data and workloads assigned to specific cores of specific machines. How do you identify software bugs vs hardware issues? Given a suspect machine how do you figure out if it's actually the machine's fault? Is it the CPU? RAM? Motherboard that's faulty?

mikeytown2 5y ago

At scales this large, finding the root cause is the best choice. By finding the cause you can then do tests on all boxes via cron to make sure everything works as expected; allowing you to pull hardware before an issue becomes widespread; can then rerun the boxes inputs on another machine if it was handling more than serving http requests. Robust hardware validation and testing is a must when dealing with millions of cores.

jeffbee 5y ago
I once worked where a faulty machine out of a massive fleet was finally sent back to intel. They put it under some kind of exotic test fixture and confirmed it was fucked. It was a defect never seen again or before. Root-causing that stupid chip was a complete waste of everyone's time.
DylanSp 5y ago
That's sort of the opposite of what I've read from the companies operating at this scale; most seem to assume that failures are inevitable in a sufficiently large distributed system, that tracking the root cause of every failure is infeasible, and that the solution instead is to design resilient systems.
- cestith 5y ago
  It's probably best to find a root cause for the first instance of a new class of problems. At first there was no way to be sure what part of the stack some of these were in. They ruled out their software, third-party libraries, and memory before focusing on a CPU and then which core of the CPU. Knowing that sometimes a single core goes bad means you can't randomly assign one test run to a CPU with no core affinity and reliably detect it. Yes, the fix is to disable that core or core complex if you have that kind of resolution, or to ditch the CPU if you must, or to pull the whole server if your scale demands it. But knowing it's a hardware error in the CPU and not chasing it through layers of code every time is invaluable.
- piyh 5y ago
  You can design resiliency to specific faults. Server goes down, bad code deploy, cosmic rays flip a bit. Each resiliency design looks different.

cratermoon 5y ago

The issue at scale is that you don't know if there's an underlying flaw that will manifest all at once across enough machines to cause a real outage or major service disruption.

yuliyp 5y ago

What is a faulty machine? How do you find one? Some of the failures experienced happened with specific data and workloads assigned to specific cores of specific machines. How do you identify software bugs vs hardware issues? Given a suspect machine how do you figure out if it's actually the machine's fault? Is it the CPU? RAM? Motherboard that's faulty?