9 ms·
Intel's $475M error: the silicon behind the Pentium division bug
- coin 2y ago> He called Intel tech support but was brushed off I laughed when I read this. It’s hard enough to get support for basic issues, good luck explaining a hardware bug.
- huracan20094 2y ago[flagged]
- pieterr 2y agoReminds me of part 2 of day24. Some wrong wirings. ;-) https://adventofcode.com/2024/day/24 https://adventofcode.com/2024/day/24
- Unearned5161 2y agoFrom someone who had to mentally let go once you started talking about planes crossing each other, thank you for such an amazingly detailed writeup. It's not everyday that you learn a new cool way to divide numbers!
- ijustlovemath 2y ago> Curiously, the adder is an 8-bit adder but only 7 bits are used; perhaps the 8-bit adder was a standard logic block at Intel. I believe this is because for any adder you always want 1 bit extra to detect overflow! This is why 9 bit adders are a common component in MCUs
- kens 2y agoThe weird thing is that I traced out the circuitry and the bottom bit of the adder is discarded, not the top bit where overflow would happen. (Note that you won't get overflow for this addition because the partial remainder is in range, just split into the sum and carry parts.)
- chiph 2y agoI'm surprised they took the risk of extending the lookup table to have all 2's in the undefined region. A safer route would have been to just fix the 5 entries. Someone was pretty confident!
- justsid 2y agoIt actually seems like it becomes much easier to reason about because you remove a ton of (literal in the diagram) edge cases.
- CaliforniaKarl 2y ago> The explanation is that Intel didn't just fill in the five missing table entries with the correct value of 2. Instead, Intel filled all the unused table entries with 2. I wonder why they didn't do this in the first place.
- Panzer04 2y agoImplementation detail. Somone overspecified it and didn't realise that it didn't matter. Look at it again later, someone asks why not just fill everything in instead and everyone feels a bit silly XD.
- Jean-Papoulos 2y ago>Since only one in 9 billion values caused the problem, Intel's view was that the problem was trivial: "This doesn't even qualify as an errata." This sounds utterly insane. You are making a CPU, if any calculations are wrong it needs to be fixed ?? I supposed this only came to light very late into testing and it was very impractical to bin every cpu, so they rolled the dice.
- kens 2y agoAuthor here if anyone has Pentium questions :-) My Mastodon thread about the bug was on HN a few weeks ago, so this might seem familiar, but now I've finished a detailed blog post. The previous HN post has a bunch of comments: https://news.ycombinator.com/item?id=42391079 https://news.ycombinator.com/item?id=42391079
- ernst_mulder 2y agoThank you very much for this detailed article. I never realised this is how floating point division can be implemented. Actually funny how I didn't realise that multiple integer division steps are required to implement floating point division :-) In hindsight one could wonder why the unused parts of the lookup table were not filled with 2 and -2 in the first place.
- ksec 2y agoIn my view, this $475M was perhaps the best marketing spend for Intel. Because of the bug and recall, everyone including those not in tech knew about Intel. Coming from the 486 when people were expecting 586 or 686 but then suddenly "Pentium", this bug and recall built a reputation and good will that carried on later with Pentium MMX.
- gone35 2y agoTour de force, truly. Amazing work!
- mras0 2y agoGreat article and analysis as always, thanks! Somewhat crazy to remember that a (as you argue) minor CPU erretum made world wide headlines. So many worse ones out there (like you mention from Intel) but others as well, that are completely forgotten. For the Pentium, I'm curious about the FPU value stack (or whatever the correct term is) rework they did. It's been a long time, but didn't they do some kind of early "register renaming" thing that had you had to manually manage doing careful fxchg's?
- fourseventy 2y agoDidn't Intel have floating point division issues more recently as well?
- hinkley 2y agoInflation adjusted that’s over 1 billion today. And they do more mitigations with microcode these days.
- xattt 2y agoSome irony if internal calculations of financial damage estimates were under or over-estimated because they were done on a defective chip.
- anticensor 2y agoThen that immediately becomes a life or death situation.
- fuzztester 2y agoyes. their ”1 in a billion” (excuse) became $1 billion (cost to them). of course, the CEOs not only go scot-free, but get to bail out with their golden parachutes, while the shareholders and public take the hit. https://en.m.wikipedia.org/wiki/Golden_parachute https://en.m.wikipedia.org/wiki/Golden_parachute
- bolognafairy 2y agoThose evil CEOs pulling the wool over the eyes of the poor shareholders! This is the employment contract that was negotiated and agreed to by the board / shareholders.
- fuzztester 2y agoWhere yuh bin livin all yore laafe, pilgrim? Under a Boulder in Colorado, mebbe? Dontcha know dat contracts can be gamed, and hev bin fer yeers, if not deecades? Dis here ain't Aahffel, ya know? Come eat some chili widdus. Id'll shore put some hair on yore chest, and grey cells in yore coconut. # sorry, in a punny mood and too many spaghetti western movies
- hinkley 2y ago> Intel's whitepaper claimed that a typical user would encounter a problem once every 27,000 years, insignificant compared to other sources of error such as DRAM bit flips. > However, IBM performed their own analysis,29 suggesting that the problem could hit customers every few days. I bet these aren’t as far off as they seem. Intel seems to be considering a single user, while I suspect IBM is thinking in terms of support calls. This is a problem I’ve had at work. When you process a 100 million requests a day the one in a billion problem is hitting you a few times a month. If it’s something a customer or worse a manager notices, they ignore the denominator and suspect you all of incompetence. Four times a month can translate into “all the time” in the manner humans bias their experiences. If you get two statistical clusters of three in a week someone will lose their shit.
- kens 2y agoNo, IBM's estimate is for a single user. IBM figures that a typical spreadsheet user does 5000 divides per second when recalculating and does 15 minutes of recalculating a day. IBM also figures that the numbers people use are 90 times as likely to cause an error as Intel's uniformly-distributed numbers. The result is one user will have an error every 24 days.
- hinkley 2y agoAh. The other failure mode that occurred to me is that if a spread sheet is involved you could keep running the same calc on a bad input for months or even years when aggregating intermediate values over units of time. A problem that happens every time you run a calculation is very different from one that happens at random. Better in some ways and worse in others.
- jiggawatts 2y agoThat's also a clearly flawed analysis, because the numbers mostly don't change between re-computations of the spreadsheet cell values! E.g.: Adding a row doesn't invalidate calculations for previous rows in typical spreadsheet usage. The bug is deterministic, so repeating successful calculations over and over with the same numbers won't ever trigger the bug.
- dboreham 2y agoAnother great article from Ken. I remember this particularly because the first PC that I bought with my own money had an affected CPU. Prior to this era I hadn't been much interested in PCs because they couldn't run "real" software. But Windows NT changed that (thank you Mr. Cutler), and Taiwanese sourced low cost motherboards made it practical to build your own machine, as many people still do today. Ken touched on the fact that it was easy for users to check if their CPU was affected. I remember that this was as easy as typing a division expression with the magic numbers into Excel. If MS had released a version of Excel that worked around the bug, I suspect fewer users would have claimed their replacement device!
- urbandw311er 2y agoWhat an interesting and utterly dedicated analysis. Thank you so much for all your work analysing the silicon and sharing your findings. I particularly like how you’re able to call out Intel on the actual root cause, which their PR made sound like something analogous to a trivial omission. But, in fact, was less forgivable and more blameworthy, ie they stuffed up their table generation algorithm.
- ThrowawayTestr 2y ago>Smith posted the email on a Compuserve forum, a 1990s version of social media. I hate how this sentence makes me feel.
- throwawee 2y agoHe sent it via his Personal Computer, a precursor to the smartphone.
- lizzas 2y agoMy initial feeling is: that data is probably mostly unmined and lost. Lucky bastards!
- lotsofpulp 2y agoI like to use the 1900s instead of the 1990s.
- lazide 2y agoDoes it help, or make it worse, if you say it as ‘late 1900s’?
- tgma 2y agoIntel $475B error: not building a decent GPU
- lizzas 2y agoLack of clairvoyance? Missing out on mobile was more obvious tho.
- phire 2y agoMore explicitly. In 2006, Apple asked Intel to make a SoC for their upcoming product... the iPhone. At the time, Intel was one of the leading ARM SoC providers, their custom XScale ARM cores were faster than anything from ARM Inc themselves. It was the perfect line of chips for smartphones. The MBA types at Intel ran some sales projects and decided that such a chip wasn't likely to be profitable. There was apparently debate within Intel, the engineering types wanted to develop the product line anyway, and others wanting to win good-will from Apple. But the MBA types won. Not only did they reject Apple's request for an iPhone SoC, but they immediately sold off their entire XScale division to marvel (who did nothing with it) so they wouldn't even be able to change their mind later even if they wanted. With hindsight, I think we can safely say Intel's projections for iPhone sales were very wrong. They would have easily made their money back on just the sales from the first-gen iPhone, and Apple would probably gone back to intel for at least a few generations. Even if Apple dumped them, Intel would have a great product to sell to the rapidly market of Android smartphones in the early 2010s. ----------- But I think it's actually far worse than just Intel missing out on the mobile market. In 2008, Apple acquired P.A. Semi, and started work on their own custom ARM processors (and ARM SoCs). The ARM processors which Apple eventually used to replace Intel as suppler in laptops and desktops too. Maybe Apple would have gone down that path anyway, but I really suspect Intel's reluctance to work with Apple to produce the chips Apple wanted (especially the iPhone chip) was a huge motivating factor that drove Apple down the path of developing their own CPUs. Remember, this is 2006. Intel had only just switched to Intel in January because IBM had continually failed to deliver Apple the laptop-class powerpc chips they needed [1]. And while at that time, Intel had a good roadmap for laptop-class chips, it would have looked to Apple as if history was at risk of repeating itself, especially as they moved into the mobile market where low power consumption was even more important. [1] TBH, IBM were failing to provide desktop-class CPUs too. But the laptop cpus were the more pressing issue. Fun fact: IBM actually tried to sell the PowerPC core they were developing for the xbox 360 and PS3 to Apple as a low-power laptop core. It was sold to Microsoft/Sony as a low-power core too, but if you look at the launch versions of both consoles, they run extremely hot, even when paired with comically large (for the era) cooling solutions.
- Sniffnoy 2y agoGiven that the fixed table is a much simpler one (by letting out-of-bounds just return 2, rather than adding circuitry to make it return 0), I wonder why they didn't just do it that way in the first place?
- kens 2y agoReturning 0 for undefined table entries is the obvious thing to do. Setting these entries to 2 is a bit of a conceptual leap, even though it would have prevented the FDIV error and it makes the PLA simpler. So I can't fault Intel for this.
- garaetjjte 2y agoI would have expected that instead of manually picking a value they would be specified as "don't care". I guess optimizer software like Espresso should allow for that?
- Sniffnoy 2y agoIt's not really a conceptual leap if you've ever had to work with "don't care" cases before...
- mjevans 2y agoIt's a NULL / 'do not care' issue. 0 isn't a reserved out of band value, it's payload data and anything beyond the bounds should have been DNC. It's possible some other result, likely aligned to an easy binary multiple would still produce a square block of 2, and that allowing the far edges to float to some other value could yield a slightly more compact logic array. Back-filling the entire side to the clamped upper value doesn't cost that much more though, and is known to solve the issue. As pointed out elsewhere, that sort of solution would also be faster for engineering time, fit within the planned space budget, and best of all reduces conative load. It's obviously correct when looking at the bug.
- ajross 2y ago"Make it work first before you make it work fast". Fundamentally this is a software problem solved with software techniques. And like most software there's some optimization left on the table just because no one thought of it in time. And you can't patch a CPU of this era.
- evanmoran 2y agoThe bug is super fun, but I also find the Intel response to be fascinating on its own. They apparently didn’t replace everyone’s processor with a non faulty version who wanted it, resulting in a ton of bad press. To contrast, I’ve been thinking a lot about the Amazon Colorsoft launch, which had a yellow band graphics issue on some devices (mine included). Amazon waited a bit before acknowledging it (maybe a day or two, presumably to get the facts right). Then they simply quietly replace all of them. No recall. They just send you a new one if you ask for it (mine replacement comes Friday, hopefully it will fix it). My takeaway is that it’s pretty clear that having an incredibly robust return/support apparatus has a lot of benefits when launches don’t go quite right. Certainly more than you’d expect from analysis. Similarly I haven’t seen too many recent reports about the Apple AirPod Pros crackle issue that happened a couple years ago (my AirPods had to be replaced twice), but Apple also just quietly replaced them and the support competence really seemed something powerful that isn’t always noticed. Colorsoft: https://www.tomsguide.com/tablets/e-readers/amazon-kindle-colorsoft-yellow-stripe-defect-now-has-a-culprit https://www.tomsguide.com/tablets/e-readers/amazon-kindle-co... AirPods Pro: https://support.apple.com/airpods-pro-service-program-sound-issues https://support.apple.com/airpods-pro-service-program-sound-...
- lizzas 2y agoThat is default Amazon - you can return stuff no hassle for almost any reason.
- WalterBright 2y agoOnly up to a point. If one is abusing it, expect getting locked out. I buy enough stuff from Amazon that they don't mind me returning something once in a while.
- dan-robertson 2y agoI thought the response from intel was to invest a lot in correctness for a while and then deciding that AMD were not being punished for their higher defect rate and so, more recently, investing in other things to try to compete with AMD on other metrics than how buggy the cpu is.
- WalterBright 2y ago> It appears that only one person (Professor Nicely) noticed the bug in actual use. I recall a study done years ago where students were supplied calculators for their math class. The calculators had been doctored to produce incorrect results. The researchers wanted to know how wrong the calculators had to be before the students noticed something was amiss. It was a factor of 2. Noticing the error, and being affected by the error, are two entirely different things. I.e. how many people check to see if the computer's output is correct? I'd say very, very, very few. Not me, either, except in one case - when I was doing engineering computations at Boeing, I'd run the equations backwards to verify the outputs matched the inputs.
- wat10000 2y agoI used to tutor physics in college. My students would show a problem they worked and ask for feedback, and I’d tell them that they definitely went wrong somewhere since they calculated that the rollercoaster was 23,000 miles tall. Which is to say, it will depend a lot on the context and the understanding of the person doing the calculation.
- WalterBright 2y agoIt is institute policy at Caltech (at least when I attended) that obviously wrong answers would get you zero credit, even if the result came from a minor error. However, if you concluded after solving the problem that the answer was absurd, but you didn't know where the calculation went wrong, you'd get partial credit.
- kllrnohj 2y ago> Noticing the error, and being affected by the error, are two entirely different things. Only somewhat true. Take any consumer usage here for example. If you're playing a game and it hits this incorrect output but you don't notice anything as a result, were you actually affected? How much usage of FDIV on a Pentium was for numerically significant output instead of just multimedia?
- 2y ago
- WalterBright 2y agoI remember that bug. Because I could not control what CPU my customers were running on, I had to add special code in the library to detect the bad FPU and execute workaround code (this code was supplied by Intel). I.e. Intel's problem became my problem, grrrr
- stickfigure 2y agoReminds me of a joke floating around at the time that captures a couple different 90s themes: I AM PENTIUM OF BORG. DIVISION IS FUTILE. YOU WILL BE APPROXIMATED.
- rconti 2y agopretty sure that was in my tagline generator...
- fortran77 2y ago"At Intel, Quality is job 0.9999999999999999762"
- keshavmr 2y agoAt the 2012 Turning Award conference in San Francisco, Prof William Kahan mentioned that he had a newer test suite available in 1993 that would have caught Intel's bug. Still, Intel did not run that.. Prof. Kahan was actively involved in its analysis and further testing. (I'm stating this just from memory).
- hyperman1 2y agoHow did idiv work on the pentium. Was it also optimized, or somehow connected to fdiv, or just the old slow algorithm?