9 ms·
Cores that don’t count [pdf]
- guidovranken 5y agoSite is down, archive link: https://web.archive.org/web/20210602080638/https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s01-hochschild.pdf https://web.archive.org/web/20210602080638/https://sigops.or... What stands out to me: - "Mercurial cores are extremely rare" but "we observe on the order of a few mercurial cores per several thousand machines". On average one core per 1000 machines is faulty? That's quite a high rate. - Vendors surely must know about this? If not by testing then through experiencing the failures in their company servers. - I've read the whole paper and I see no mention of them even reaching out to vendors about this issue. Their are strong incentives on both sides to solve or mitigate this issue so why aren't they working together?
- thanatos519 5y agohttps://www.goodreads.com/quotes/691547-wherever-i-m-going-i-ll-be-there-to-apply-the-formula https://www.goodreads.com/quotes/691547-wherever-i-m-going-i...
- ffff1312 5y agoI highly assume that they must have reached out and reported these issues to, presumably, Intel as the biggest player here. Likely they're just not disclosing these numbers, and generally not many are talking about these and potentially other CPU issue in public due to NDAs. Either way, it's quite amazing to dig down so deep in the production stack that you must conclude that it's the CPU at fault here. I presume academic research might have a hard time on this given the scale needed to run into these issues, but hopefully we'll see more on this research in future.
- scottlamb 5y ago> Either way, it's quite amazing to dig down so deep in the production stack that you must conclude that it's the CPU at fault here. Google's internal production stack is much more amenable to that kind of digging than public cloud products: * You can easily find out what machine a given borg task was running on. In fact, not just your own borg job but anyone's. You can query live state, or you can use Dremel to look up history. * Similarly, even as a client of Bigtable or Spanner, you can find out the specific tabletservers/spanservers operating on a portion of your database and what machines they're running on. (Not as easy to cross this layer and get to the relevant D servers actually storing the data but I think it's all checksummed here anyway.) If your team has your own partition, you can see tabletserver/spanserver debug logs yourself also. * There's a convenient frontend for looking up a bunch of diagnostic info for the machine, including failures of borg tasks (were other people's tasks crashing at the same time mine did? what was their crash message?), syslog-level stuff, other machine diagnostics like ECC / MCE errors, and repair history (swapped this DIMM, next attempt will swap this CPU). It's not unusual for application teams to suspect a machine and basically vote it off the island (I don't want my jobs running here anymore, I cast a vote for it to be repaired / Office Spaced). It's more rare for them to really take the time to really understand the problem in detail like "core 34 sometimes returns incorrect results on this computation", although there's nothing in particular stopping them from doing so (other than lack of expertise and a long list of other things to do). The platforms team gets involved sometimes and really digs in—iirc in one bug they mentioned sending a CPU back to the vendor to examine with an electron microscope. I'm not sure what lessons that offers for a public cloud where that kind of transparency isn't realistic...
- redis_mlc 5y agoNot sure why you're surprised. Ever seen displays with dead pixels? Same thing. Usually CPUs are speed-binned and tested for how many cores work, then given different part numbers (and prices.) Increasing temperature or stressing them will reveal more problems.
- tails4e 5y agoThat rate sounds too high. Typically scan test gives 99.9% logic coverage. This me as random defects must hit this exact subset of logic to cause a fault undetectable by production test. Given defect rates are low, 1 in 1000 having a fault that got past theses tests seems too high. Unless of course Intel does not use scan test, and has a more functional type test method, though even then I'd imagine they must have a high coverage rate.
- sfvisser 5y ago> A deterministic AES mis-computation, which was “selfinverting”: encrypting and decrypting on the same core yielded the identity function, but decryption elsewhere yielded gibberish. Incredible
- formerly_proven 5y agoSounds like a corrupted S-box or something like that in the hardware implementing AES-NI
- twic 5y agoDepends on the mode. AES-CTR encrypts an increasing counter to make a keystream, then xors that with the plaintext to make the ciphertext, or with the ciphertext to make the plaintext. Any consistent error in encryption will lead to a consistently wrong keystream, which will round-trip successfully. It's possible other modes have this property because of the structure of the cipher itself, but that's way out of my league.
- muricula 5y agoThe intel AES instructions work at a block level. You build the mode on top of the block level primitives intel gives you. https://en.wikipedia.org/wiki/AES_instruction_set https://en.wikipedia.org/wiki/AES_instruction_set
- buildbot 5y agoThis specifically stood out to me - it implies it is possible to hide weakened/bad crypto via manufacturing bugs
- muricula 5y agoI'm not sure about that. I think it's just that the AES hardware was busted somehow and didn't actually perform the AES algorithm correctly. The intel AES hardware just deterministically performs the algorithm, so Intel can't just weaken the algorithm somehow, at least if you're not worrying about local side channels.
- dataflow 5y agoThis is fascinating. I feel like the most straightforward (but hardly efficient) solution is to provide a way for kernels to ask CPUs to "mirror" pairs of cores, and have the CPUs internally check that the behaviors are identical? Seems like a good way to avoid large scale data corruption until we develop better techniques...
- smallpipe 5y agoThat’s called dual core lockstep and it’s very common in automotive and other applications where reliability is paramount.
- dataflow 5y agoYeah I didn't know! And I just realized this is mentioned in the paper just a little further below where I paused. It seems like it would significantly affect anything shared (like L3 cache)... would Intel and AMD have appetite for adding this kind of thing to x86?
- yaantc 5y agoThe pair in lockstep is "close", in that it only includes the core and deterministic private resources like core private caches. Shared resources like a L3 cache are outside of the whole pair, and can be seen as accessed by the pair. All output is from the pair and checked for consistency (same for both cores in lockstep) before going out. Not directly related but some platforms supporting lockstep are flexible: you can use a pair as either 2 cores (perf) or a single logical one (lockstep).
- jacques_chester 5y agoTandem used to do this. By descent the technology wound up with HPE. Their Tech Reports are worth a sample and fortunately they're online: https://www.hpl.hp.com/hplabs/index/Tandem https://www.hpl.hp.com/hplabs/index/Tandem Probably the best one to start at: https://www.hpl.hp.com/techreports/tandem/TR-90.5.pdf https://www.hpl.hp.com/techreports/tandem/TR-90.5.pdf
- temac 5y agoI don't completely understand the perception that standard non hardened high-perf CPU, especially in an industry and more specifically in a segment that has been reported as consistently cutting a few corners in recent years (maybe somehow less than client CPUs, but still), should somehow be exempt of silent defaults, because... magic? If you want extremely high reliability, for critical applications, you use other CPUs. Of course, they are slower. So the only interesting info that remains is that the defect rate seems way too high and maybe the quality decreasing in recent years. In which case, when you are Google, you probably could and should complain (strongly) to your CPU vendors, because likely their testing is lacking and their engineering margins too low... (at least if that's really the silicon that is at fault, and not say for example the MB) Now of course it's a little late for the existing ones, but still the sudden realization that "OMG CPU do sometimes fail, with a variety of modes, and for a variety of reasons" (including, surprise(?!), aging) seems, if not concentrating on the defect rate, naïve. And the potential risk of sometimes having high rate errors was already a very well known esp. in the presence of software changes and/or heterogenous software and/or heterogenous hardware, due to the existence of logical CPU bugs, sometimes also resulting in silent data corruption, and sometimes also with non-deterministic-like behaviors (so can as well work on a core but not another because of "random" memory controller pressure and delays, and the next time with the two cores reversed)
- ng55QPSK 5y agoI think, the main point is: We have reached a time in which there are no guarantees anymore that your HW works (until recently we only had no guarantee that the SW works). Correctly.
- gbrown_ 5y agoI'd love to see more details on the defective parts, particularly counts of CPU model (anonymized if needs be) and counts of which part of the architecture exhibited faults. From working in HPC I've handled reports of things like FMA units producing incorrect results or random appearance of NaNs. Were it not for the fact that we knew these things could happen and customer's intimate knowledge of their codes I dread to think how'd "normal" operations would track these issues down. Bad parts went back to the CPU manufacturer and further testing typically confirmed the fault. But that end of the process was pretty much a black box to anyone but the CPU manufacturer. I'd be keen to know more about this too.
- smallpipe 5y agoModern embedded cores have self-testing code that detects anywhere from 50% to 90%[1] of faults in the hardware, including from ageing. If google and the other hyperscalers complain enough, there’s no reason Intel couldn’t give them some self test to run every hour or so. [1] depends on how complex the CPU is, how long you accept to run the self testing code, and how well it was done.
- gmueckl 5y agoThese periodic self tests are required for some safety critical applications. My reaction to this paper is that this approach might have to be used in data centers as well. Unfortunately, it can't be done without help from the CPU designers because the test sensitivity relies on knowledge of the exact implementation details of the underlying hardware units.
- freehrtradical 5y agoThe article references Dixit et al. for an example of a root cause investigation of a CEE which is an interesting read: https://arxiv.org/pdf/2102.11245.pdf https://arxiv.org/pdf/2102.11245.pdf > After a few iterations, it became obvious that the computation of 𝐼𝑛𝑡(1.153)=0 as an input to the math.pow function in Scala would always produce a result of 0 on Core 59 of the CPU. However, if the computation was attempted with a different input value set 𝐼𝑛𝑡(1.152)=142 the result was accurate.
- yummypaint 5y agoI wonder about the larger feedback loops between hardware error checking in software and the optimizations hardware manufacturers are making at the fab. Presumably more robust software would result in buggier cores being shipped, but would this actually result in more net computation per dollar spent on processors?
- yummypaint 5y agoFault tolerance seems to be the fundamental issue looming in the background of both traditional and quantum computing at the moment. Silicon is already at the point where there are only a dozen or so dopant atoms per gate, so a fluctuation of one or two atoms can be enough to impact behavior. It's amazing to me that with billions of transistors things work as well as they do. At some point it might be good to try to re-approach computation from some kind of error-prone analogue of the turing machine.
- minikites 5y agoIs silicon cheap enough that we can do what we do for spacecraft and have multiple processors (where the majority "wins") for fault tolerance?
- maweki 5y agoCheap is probably not the question. The synchronization and communication overhead is enormous. Would you check that every register assignment matches? Or every page write?
- mirker 5y agoYou can have a log of N register “transactions”, where N is large enough to hide core-to-core communication. If any of N transactions fail due to mismatch between cores, you roll back and throw exceptions. A lot of this logic is already in out of order execution (e.g., Tomasulo algorithm). Memory has ECC and is probably a different problem.
- jeffbee 5y agoThe economics will never favor this approach. Customers will not choose to pay double to avoid the 1-in-a-million chance of occasionally getting a slightly wrong answer.
- theevilsharpie 5y ago> Customers will not choose to pay double to avoid the 1-in-a-million chance of occasionally getting a slightly wrong answer. With today's high-speed multi-core processors, a 1-in-a-million chance of a computation error would mean tens to hundreds of thousands of errors per second.
- sehugg 5y agoMaybe we need something like SMART but for CPU cores?
- gbrown_ 5y agoWe have machine-check exceptions, the purpose of this paper is to draw attention to erroneous behavior that is silent.
- mywittyname 5y agoA lot of these errors are subtle, making it incredibly difficult to generate a test suite to find them.
- stevewodil 5y agoBut when they occur, something could increment a counter that tracks the error rate like hard drives do for SMART
- tylfin 5y agoCan't reproduce the issue after a few minutes? Sorry wont-fix, mercurial core. Joking aside, it's really neat to see the scale and sophistication of error detection appearing in these data centers.
- dekhn 5y agoI worked on this problem for the past year at Google. It's a fascinating problem. In my subarea I focused on accelerators (like GPUs) running machine learning training. Many users report problems like "NaN" during training- at some point, the gradients blow up and the job crashes. Sometimes these are caused by specific examples, or numerical errors on the part of the model developer, but sometimes, they are the result of errors from bad cores (during matrix multiplication, embedding lookup, vector op, whatever). ML is usually pretty tolerant of small amounts of added noise (especially if it's got nice statistcal properties) and some training jobs will ride through a ton of uncorrected and undetected errors with few problems. It's a very challenging field to work in because it's hard to know if your nan is because of your model or your chip.
- gbrown_ 5y agoAre you able to say anything about the distribution across hardware? E.G. Is there any correlation across such faulty parts with serials/production dates or is it very random/insufficient frequency to say?
- watersb 5y agoThis might not be a constructive observation for me to post this comment, but I can just see the IBM mainframe designers sitting back with a refreshing beverage, while we talk about identifying and handling hardware faults during runtime. https://www.ibm.com/support/pages/ibm-power-systems™-reliability-availability-and-scalability-ras-features https://www.ibm.com/support/pages/ibm-power-systems™-reliabi...