3 ms·

At scales this large, finding the root cause is the best choice. By finding the cause you can then do tests on all boxes via cron to make sure everything works

At scales this large, finding the root cause is the best choice. By finding the cause you can then do tests on all boxes via cron to make sure everything works as expected; allowing you to pull hardware before an issue becomes widespread; can then rerun the boxes inputs on another machine if it was handling more than serving http requests. Robust hardware validation and testing is a must when dealing with millions of cores.

jeffbee 5y ago
I once worked where a faulty machine out of a massive fleet was finally sent back to intel. They put it under some kind of exotic test fixture and confirmed it was fucked. It was a defect never seen again or before. Root-causing that stupid chip was a complete waste of everyone's time.
DylanSp 5y ago
That's sort of the opposite of what I've read from the companies operating at this scale; most seem to assume that failures are inevitable in a sufficiently large distributed system, that tracking the root cause of every failure is infeasible, and that the solution instead is to design resilient systems.
- cestith 5y ago
  It's probably best to find a root cause for the first instance of a new class of problems. At first there was no way to be sure what part of the stack some of these were in. They ruled out their software, third-party libraries, and memory before focusing on a CPU and then which core of the CPU. Knowing that sometimes a single core goes bad means you can't randomly assign one test run to a CPU with no core affinity and reliably detect it. Yes, the fix is to disable that core or core complex if you have that kind of resolution, or to ditch the CPU if you must, or to pull the whole server if your scale demands it. But knowing it's a hardware error in the CPU and not chasing it through layers of code every time is invaluable.
- piyh 5y ago
  You can design resiliency to specific faults. Server goes down, bad code deploy, cosmic rays flip a bit. Each resiliency design looks different.

jeffbee 5y ago

I once worked where a faulty machine out of a massive fleet was finally sent back to intel. They put it under some kind of exotic test fixture and confirmed it was fucked. It was a defect never seen again or before. Root-causing that stupid chip was a complete waste of everyone's time.

DylanSp 5y ago

That's sort of the opposite of what I've read from the companies operating at this scale; most seem to assume that failures are inevitable in a sufficiently large distributed system, that tracking the root cause of every failure is infeasible, and that the solution instead is to design resilient systems.

cestith 5y ago
It's probably best to find a root cause for the first instance of a new class of problems. At first there was no way to be sure what part of the stack some of these were in. They ruled out their software, third-party libraries, and memory before focusing on a CPU and then which core of the CPU. Knowing that sometimes a single core goes bad means you can't randomly assign one test run to a CPU with no core affinity and reliably detect it. Yes, the fix is to disable that core or core complex if you have that kind of resolution, or to ditch the CPU if you must, or to pull the whole server if your scale demands it. But knowing it's a hardware error in the CPU and not chasing it through layers of code every time is invaluable.
piyh 5y ago
You can design resiliency to specific faults. Server goes down, bad code deploy, cosmic rays flip a bit. Each resiliency design looks different.