12 ms·
I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy
by nostrademons 6y ago
I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory."
Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the software being stored in that memory being robust against cosmic rays. And when you have thousands of machines with petabytes of RAM, those bitflips do happen. Google wasted many man-years tracking down corrupted GFS files and index shards before they finally bit the bullet and just paid for ECC.
- tyoma 6y agoFigure this is as good of a time as any to ask this: There are many various DRAMs in a server (say, for disk cache). Has Google or anyone who operates at a similar scale seen single bit errors in these components?
- deleted 6y ago[deleted]
- gh02t 6y agoThe supercomputing community has looked at some of the effect on different parts of the GPU. https://ieeexplore.ieee.org/abstract/document/7056044 https://ieeexplore.ieee.org/abstract/document/7056044
- bsder 6y agoThis is as old as computing and predates Google. When America Online was buying EV6 servers as fast as DEC could produce them, they used to see about about 1 double bit error per day across their server farm that would reboot the whole machine. DRAM has only gotten worse--not better.
- sitkack 6y agoYes. Bit flips (for all reasons) occur in buses, registers, caches, etc. Anything that has state can have state changed incorrectly. This is why filesystems like ZFS exist and storage formats have pervasive checksums.
- starfallg 6y agoRecent advances have blurred the lines a bit. The ECC memory that we all know and love is mainly side-band EEC, with the memory bus widened to accommodate the ECC bits driven by the memory controller. However as process size shrink, bit flips become more likely to the point that now many types of memory have on-die EEC, where the error correction is handled internally on the DRAM modules themselves. This is present on some DDR4 and DDR5 modules, but information on this is kept internal by the DRAM makers and not usually public. https://semiengineering.com/what-designers-need-to-know-about-error-correction-code-ecc-in-ddr-memories/ https://semiengineering.com/what-designers-need-to-know-abou... There has been a lot of debate regarding this that was summarised in this post - https://blog.codinghorror.com/to-ecc-or-not-to-ecc/ https://blog.codinghorror.com/to-ecc-or-not-to-ecc/
- wtallis 6y ago> This is present on some DDR4 and DDR5 modules, but information on this is kept internal by the DRAM makers and not usually public. On-die ECC is going to be a standard feature for DDR5. I'm not aware of any indication that anyone has implemented on-die ECC for DDR4 DRAM, and Hynix at least has made clear statements that on-die ECC is new for their DDR5 and was not present in their DDR4.
- starfallg 6y agohttps://scholar.google.com/scholar?cluster=4712032890316480599&hl=en&as_sdt=0,5&sciodt=0,5 https://scholar.google.com/scholar?cluster=47120328903164805... On-die ECC DRAM is already here with us.
- ksec 6y ago>I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Did they ( Google ) or He ( Craig Silverstein ) ever officially admit it on record? I did a Google search and results that came up were all on HN. Did they at least make a few PR pieces saying that they are using ECC memory now because I dont see any with searching. Admitting they made a mistake without officially saying it? I mean the whole world of Server or computer might not need ECC insanity was started entirely because of Google [1] [2] with news and articles published even in the early 00s [3]. And after that it has spread like wildfire and became a common accepted fact that even Google doesn't need ECC. Just like Apple were using custom ARM instruction to achieve their fast JS VM performance became a "fact". ( For the last time, no they didn't ). And proponents of ECC memory has been fighting this misinformation like mad for decades. To the point giving up and only rant about every now and then. [3] [1] https://blog.codinghorror.com/building-a-computer-the-google-way/ https://blog.codinghorror.com/building-a-computer-the-google... [2] https://blog.codinghorror.com/to-ecc-or-not-to-ecc/ https://blog.codinghorror.com/to-ecc-or-not-to-ecc/ [3] https://danluu.com/why-ecc/ https://danluu.com/why-ecc/
- djur 6y agoYour [3] has a footnote quoting a Google book that reads "Modern DRAM systems are a good example of a case in which powerful error correction can be provided at a very low additional cost... The following machine generation at Google did include memory parity detection, and once the price of memory with ECC dropped to competitive levels, all subsequent generations have used ECC DRAM"
- ksec 6y agoWell They made a big deal out of it and lots of PR out of not having use ECC, and only quietly adding back ECC and listed it out in a book that barely any one reads.
- sitkack 6y agoThe fact that ECC isn't the default across everything is a failure of human cognition and Capitalism.
- grishka 6y agoI remember reading how someone registered some google domains with a single bit flipped, and saw actual requests coming to them.
- andrewstuart2 6y agoIf you or anybody can remember the source article, that sounds like an interesting read! Edit: found one with a quick search. https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-squatting-dns-hijacking-without-exploitation/ https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-squa... And https://www.researchgate.net/publication/262273269_Bitsquatting_Exploiting_bit-flips_for_fun_or_profit https://www.researchgate.net/publication/262273269_Bitsquatt...
- garaetjjte 6y agoHere's original paper: http://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinaburg_Bitsquatting_WP.pdf http://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabur... But I'm not entirely convinced that most of these requests weren't just typos. Though those requests with mismatched Host header surely were true bitflips.
- tyoma 6y agoThere was some follow up work as well: http://blog.dinaburg.org/2013/09/bitsquatting-at-defcon21-and-more.html http://blog.dinaburg.org/2013/09/bitsquatting-at-defcon21-an... And I am beyond certain they were not typos; the requests were for long URLs that no one would be typing by hand. A few were for services that humans never enter urls into (like the Windows crash reporter).
- fart32 6y agoAlso here: https://www.youtube.com/watch?v=gXY3jm34RFU https://www.youtube.com/watch?v=gXY3jm34RFU I can't find it right now, but I also saw someone doing this with some version of Google Now (or what it was back then) and serving different assets into it.
- 6y ago
- finiteloop 6y agoOne of the best quotes in the Google quotes file an early Googler maintained (I am sure I am screwing it up): “I’ve heard of defensive programming, but never adversarial memory.” — Ben Gomes
- fragmede 6y agoClose! > I've never thought of defensive programming in terms of adversarial memory.
- itisit 6y agoNew Yorker article that credits Jeff Dean and Sanjay Ghemawat with discovering the company’s bitflip issue: https://www.newyorker.com/magazine/2018/12/10/the-friendship-that-made-google-huge https://www.newyorker.com/magazine/2018/12/10/the-friendship...
- maria_weber23 6y agoECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. So in theory the behavior of software under random bit flips is well... Random. You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. I could imagine that doing so would still be cheaper than using ECC ram, at least around 2000. Generally this goes against software engineering principles. You don't try to eliminate the chances of failure and hope for the best. You need to create these failures constantly (within reasonable bounds) and make sure your software is able to handle them. Using ECC ram is the opposite. You just make it so unlikely to happen, that you will generally not encounter these errors at scale anymore, but nontheless they can still happen and now you will be completely unprepared to deal with them, since you chose to ignore this class of errors and move it under the rug. Another intersting side effect of quorum is that it also makes certain attacks more difficult to pull off, since now you have to make sure that a quorum of machines gives the same "wrong" answer for an attack to work.
- sobriquet9 6y agoIf you use multiple computers doing the same calculation and then take the answer from the quorum, how do you ensure the computer that does the comparison is not affected by memory failures? Remember that all queries have to through it, so it has to be comparable in scale and power.
- rovr138 6y ago> how do you ensure the computer that does the comparison is not affected by memory failures? You do the comparison on multiple nodes too. Get the calculations. Pass them to multiple nodes, validate again and if it all matches, you use it.
- sobriquet9 6y ago> validate again Recursion, see recursion.
- gigatexal 6y agoI mean early on sure at a startup where you’re not printing money I can see how saving on hardware makes sense. But surely you don’t need an MBA to know that hardware will continue to get cheaper whereas developers and their time will only get more expensive: better to let the hardware deal with it than to burden developers with it ... I’d have made the case for ECC but hindsight being what it is ...
- colejohnson66 6y agoBut if you can save $1M+ now, then throw the cost of fixing it onto the person who replaces you, why do you care? You already got your bonus and jumped ship.
- gigatexal 6y agoSuch a short sighted incentivisation structure harms the business in the long run I think.