5 ms·

Site is down, archive link: https://web.archive.org/web/20210602080638/https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s01-hochschild.pdf https://web

Site is down, archive link: https://web.archive.org/web/20210602080638/https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s01-hochschild.pdf https://web.archive.org/web/20210602080638/https://sigops.or... What stands out to me: - "Mercurial cores are extremely rare" but "we observe on the order of a few mercurial cores per several thousand machines". On average one core per 1000 machines is faulty? That's quite a high rate. - Vendors surely must know about this? If not by testing then through experiencing the failures in their company servers. - I've read the whole paper and I see no mention of them even reaching out to vendors about this issue. Their are strong incentives on both sides to solve or mitigate this issue so why aren't they working together?

thanatos519 5y ago
https://www.goodreads.com/quotes/691547-wherever-i-m-going-i-ll-be-there-to-apply-the-formula https://www.goodreads.com/quotes/691547-wherever-i-m-going-i...
ffff1312 5y ago
I highly assume that they must have reached out and reported these issues to, presumably, Intel as the biggest player here. Likely they're just not disclosing these numbers, and generally not many are talking about these and potentially other CPU issue in public due to NDAs. Either way, it's quite amazing to dig down so deep in the production stack that you must conclude that it's the CPU at fault here. I presume academic research might have a hard time on this given the scale needed to run into these issues, but hopefully we'll see more on this research in future.
- scottlamb 5y ago
  > Either way, it's quite amazing to dig down so deep in the production stack that you must conclude that it's the CPU at fault here. Google's internal production stack is much more amenable to that kind of digging than public cloud products: * You can easily find out what machine a given borg task was running on. In fact, not just your own borg job but anyone's. You can query live state, or you can use Dremel to look up history. * Similarly, even as a client of Bigtable or Spanner, you can find out the specific tabletservers/spanservers operating on a portion of your database and what machines they're running on. (Not as easy to cross this layer and get to the relevant D servers actually storing the data but I think it's all checksummed here anyway.) If your team has your own partition, you can see tabletserver/spanserver debug logs yourself also. * There's a convenient frontend for looking up a bunch of diagnostic info for the machine, including failures of borg tasks (were other people's tasks crashing at the same time mine did? what was their crash message?), syslog-level stuff, other machine diagnostics like ECC / MCE errors, and repair history (swapped this DIMM, next attempt will swap this CPU). It's not unusual for application teams to suspect a machine and basically vote it off the island (I don't want my jobs running here anymore, I cast a vote for it to be repaired / Office Spaced). It's more rare for them to really take the time to really understand the problem in detail like "core 34 sometimes returns incorrect results on this computation", although there's nothing in particular stopping them from doing so (other than lack of expertise and a long list of other things to do). The platforms team gets involved sometimes and really digs in—iirc in one bug they mentioned sending a CPU back to the vendor to examine with an electron microscope. I'm not sure what lessons that offers for a public cloud where that kind of transparency isn't realistic...
  - H8crilA 5y ago
    So much for non disclosure agreements. Or are they not respected at Google?
    - scottlamb 5y ago
      I'm not disclosing anything new. Google's official SRE book, research publications, and conference presentations describe the systems I mentioned in more detail.
redis_mlc 5y ago
Not sure why you're surprised. Ever seen displays with dead pixels? Same thing. Usually CPUs are speed-binned and tested for how many cores work, then given different part numbers (and prices.) Increasing temperature or stressing them will reveal more problems.
tails4e 5y ago
That rate sounds too high. Typically scan test gives 99.9% logic coverage. This me as random defects must hit this exact subset of logic to cause a fault undetectable by production test. Given defect rates are low, 1 in 1000 having a fault that got past theses tests seems too high. Unless of course Intel does not use scan test, and has a more functional type test method, though even then I'd imagine they must have a high coverage rate.

thanatos519 5y ago

https://www.goodreads.com/quotes/691547-wherever-i-m-going-i-ll-be-there-to-apply-the-formula https://www.goodreads.com/quotes/691547-wherever-i-m-going-i...

ffff1312 5y ago

I highly assume that they must have reached out and reported these issues to, presumably, Intel as the biggest player here. Likely they're just not disclosing these numbers, and generally not many are talking about these and potentially other CPU issue in public due to NDAs. Either way, it's quite amazing to dig down so deep in the production stack that you must conclude that it's the CPU at fault here. I presume academic research might have a hard time on this given the scale needed to run into these issues, but hopefully we'll see more on this research in future.

scottlamb 5y ago
> Either way, it's quite amazing to dig down so deep in the production stack that you must conclude that it's the CPU at fault here. Google's internal production stack is much more amenable to that kind of digging than public cloud products: * You can easily find out what machine a given borg task was running on. In fact, not just your own borg job but anyone's. You can query live state, or you can use Dremel to look up history. * Similarly, even as a client of Bigtable or Spanner, you can find out the specific tabletservers/spanservers operating on a portion of your database and what machines they're running on. (Not as easy to cross this layer and get to the relevant D servers actually storing the data but I think it's all checksummed here anyway.) If your team has your own partition, you can see tabletserver/spanserver debug logs yourself also. * There's a convenient frontend for looking up a bunch of diagnostic info for the machine, including failures of borg tasks (were other people's tasks crashing at the same time mine did? what was their crash message?), syslog-level stuff, other machine diagnostics like ECC / MCE errors, and repair history (swapped this DIMM, next attempt will swap this CPU). It's not unusual for application teams to suspect a machine and basically vote it off the island (I don't want my jobs running here anymore, I cast a vote for it to be repaired / Office Spaced). It's more rare for them to really take the time to really understand the problem in detail like "core 34 sometimes returns incorrect results on this computation", although there's nothing in particular stopping them from doing so (other than lack of expertise and a long list of other things to do). The platforms team gets involved sometimes and really digs in—iirc in one bug they mentioned sending a CPU back to the vendor to examine with an electron microscope. I'm not sure what lessons that offers for a public cloud where that kind of transparency isn't realistic...
- H8crilA 5y ago
  So much for non disclosure agreements. Or are they not respected at Google?
  - scottlamb 5y ago
    I'm not disclosing anything new. Google's official SRE book, research publications, and conference presentations describe the systems I mentioned in more detail.

redis_mlc 5y ago

Not sure why you're surprised. Ever seen displays with dead pixels? Same thing. Usually CPUs are speed-binned and tested for how many cores work, then given different part numbers (and prices.) Increasing temperature or stressing them will reveal more problems.

tails4e 5y ago

That rate sounds too high. Typically scan test gives 99.9% logic coverage. This me as random defects must hit this exact subset of logic to cause a fault undetectable by production test. Given defect rates are low, 1 in 1000 having a fault that got past theses tests seems too high. Unless of course Intel does not use scan test, and has a more functional type test method, though even then I'd imagine they must have a high coverage rate.