6 ms·

This paper is ~15 years old. It was published in 2009, but the data was collected 2006-2008. As densities for things like RAM increase, error rates have to decr

This paper is ~15 years old. It was published in 2009, but the data was collected 2006-2008. As densities for things like RAM increase, error rates have to decrease commensurately to hide errors from end users. Additionally, Google runs hardware in their datacenters in a much more pathological way than most consumers will (e.g. they are more likely to run hardware much hotter than typical consumers). The point is that the figures in the paper are an interesting starting point for discussion, but they're not necessarily applicable to your laptop.

wongarsu 3y ago
> they are more likely to run hardware much hotter than typical consumers I'd expect laptops and phones with their limited cooling capacity to run their hardware much hotter than datacenters with their expensive HVAC systems
p-e-w 3y ago
I'm not convinced by this argument. In practice, errors on consumer hardware seem to be several orders of magnitude less common than implied by the paper. If they weren't, we'd be hearing anecdotes of one-character diffs appearing out of nowhere all the time, among many other issues. It's hard to see how this could be explained away with the slight increase in RAM densities since 2008, or different temperatures in data centers (plenty of individuals run their systems crazy hot, e.g. when the tower is stuffed with dust after years without cleaning).
- alpaca128 3y ago
  Run any consumer device for more than 4 weeks and it'll slowly accumulate all kinds of weird bugs. I've seen smartphones unable to receive SMS because they simply ran too long without reboot, a Windows PC where selecting files wouldn't work anymore etc. > we'd be hearing anecdotes of one-character diffs appearing out of nowhere all the time On consumer devices it might simply be a small "attack" surface. A server running 24/7 serving countless requests would probably see more issues than a laptop running idle 95% of the time and loading a website every few minutes. Also a bit flip isn't guaranteed to cause a noticeable bug - and if it does it might not stand out from the usual noise of janky software oddities.
  - taneq 3y ago
    We're perfectly capable of being bad enough at software to do that without invoking cosmic rays and quantum weirdness. How long would you expect to have to run memtest on any desktop computer in good condition before you observe a memory error? Because I would expect the number to be somewhere between 'weeks' and 'forever.'
  - dspillett 3y ago
    > a Windows PC where selecting files wouldn't work anymore etc. That is just Windows Explorer being shite. They even have a specific “restart” button for it in Task Manager for when it leaks into itself so much it stops functioning usefully.
    - accoil 3y ago
      Isn't the restart there because explorer is also the desktop? If it was just killed, you'd have to know how to launch shutdown.exe from the task manager etc.
      - dspillett 3y ago
        I think it is a relatively recent (Win8?) addition, and explorer has been the shell since '95. I take adding a specific button for it as an admission that Explorer is getting increasingly buggy, so users are needing to restart it more. Before then you could do the same thing from the same task manager just in two steps: kill the existing explorer.exe and start a new one from the Run command in the menus.
  - semi-extrinsic 3y ago
    IDK what you mean by consumer, but I've routinely had desktop computers running 24/7 for several hundred days without noticeable glitches on Linux (BTW I run Arch).
  - sidewndr46 3y ago
    I find this kind of difficult to believe, because I have machines with >6 months of uptime and never see any issues. Always just running regular consumer grade hardware.
- RajT88 3y ago
  Somebody at Mozilla a while back had the idea of using the browser telemetry to find off-by-one errors due to bit flip, which were correlated with cosmic storms. Interesting, but not definitive results: https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-what-flips-your-bit/ https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-... I have access to some massive number of API logs for my job. So it was easy for me to do something similar - most bit flip errors would result in a 4xx of some sort, especially 404. Every time there's a huge magnetic storm, I run a query to see if 400's go up correlating with the storm. I even filter the logs for the geo hardest hit. Nope. No correlation. ¯\_(ツ)_/¯
  - k1t 3y ago
    ArenaNet did this for GuildWars when they encountered particularly inexplicable bugs: [Mike O’Brien] wrote a module (“OsStress”) which would allocate a block of memory, perform calculations in that memory block, and then compare the results of the calculation to a table of known answers. He encoded this stress-test into the main game loop so that the computer would perform this verification step about 30-50 times per second. On a properly functioning computer this stress test should never fail, but surprisingly we discovered that on about 1% of the computers being used to play Guild Wars it did fail! One percent might not sound like a big deal, but when one million gamers play the game on any given day that means 10,000 would have at least one crash bug. Our programming team could spend weeks researching the bugs for just one day at that rate! https://www.codeofhonor.com/blog/whose-bug-is-this-anyway https://www.codeofhonor.com/blog/whose-bug-is-this-anyway
    - sidewndr46 3y ago
      were they testing system RAM exclusively? If it's VRAM, I'd agree with this. Back in the day we'd even overclock VRAM until we got "snow" in the rendered graphics then backoff a bit.
    - mdip 3y ago
      Thanks for sharing that -- on its own it was a great read. At first, I was not surprised that they saw 1% of GuildWars players had these issues. My reasoning was given the time of the article (2012) and pointing to a more distant hazy point somewhere between StarCraft and Diablo[0] their user-base probably included a lot of "custom builds", a not-small portion of which would be "built intended for overclocking", in an era when the firmware/motherboard/processors would let you apply settings that would become extremely unstable under load[1] ... and I got about that far before I thought, only 1% of users managed to make their "let's see what happens if we alter the voltages/frequencies such that we run the DDR2-333 memory at DDR2-2048 (it's been a while), change the various "latency" settings throwing caution to the wind. Meanwhile all of this hardware is sitting in a case with a fan configuration that slightly lowers the air pressure in the case[2] ... Back in the day (less common, today) you could often put any value from a range of "Way Too Low" to "Way Too High" and steps in-between. If it could reach the boot sector,... "Weeeeeeeeee!!" I recall a case of a customer who completely trashed an Intel Pentium (2? 3?) chip via firmware settings "my. kid. was. F---in'. with![3]" Maybe I'm wrong and it's just a matter that the game involved attracts people who rarely OC/build and the 1% is "just those people", but anecdotally "in my life", everyone I knew back then who was a PC gamer either started with a build of their own or went that route the instant they had a job which allowed them buy their own computer (rather than sharing Dad's or getting his hand-me-downs if you were really lucky/have fewer years on you than I do). My experience, especially back in the 08-09 era, was that things were ridiculously forgiving. Depending on the use, overclocking would either "do nothing at all" or maybe let you get away with slightly better graphics settings while maintaining a playable game. Generally, I purchased the cheapest memory rated slightly better than what I intended to configure them for (DDR2-xxx slightly higher than where I will clock it) with whatever I decided were the right latency values[4], placebo-overclocked day-to-day, but would push it from time to time. There's been times the system simply doesn't boot but one "tiny notch south" and it'll pass a 72-hour memory stress-test. I've had crashes, locally, that I can't explain beyond "memory corruption" but I can't say if it's an OS failure, hardware failure or "something that's not memory corruption but malfunctioning other hardware" ... it passes a stress test, I usually find the "other hardware" a little while later (surprisingly often the PSU, for me). [0] Unclear if editions implied are "I" in both cases, and it's not important enough to matter since reasonably earlier than 2012 puts you in the category of "the firmware/motherboards will let you do incredibly stupid things". [1] Sometimes subtle "some character's armor texture is transparent" (making your enemies see-through, not naked, much to my teenage chagrin). Sometimes crazy like "Everything is shades of green with partial geometries". [2] Say, if you had to have the most sleek case where every available air-flow "vent" can take a fan facing inward or outward (where it can sometimes be a bad idea to not leave one empty) has all of the fans blowing out... I've seen "The Vacuum Computer". [3] So, this really happened to me (CompUSA, filling in at the Parts Department), and that's exactly the way I hear it, but you need to understand that it wasn't some form of "Jersey Shore" accent -- this guy was absolutely Korean with an accent so thickand not a day younger than...60? Aside from the shock of a person very much struggling with the English language dragging out the "F" word, you just don't see a plain-looking-harmless-old-man speak with such vigor and gusto! [4] This being "unimportant" information for me day-to-day, requires hours of "refreshing via Google" and learning every miserable thing that's changed, coming to a complete understanding, finding that the memory I want either doesn't exist, doesn't work with my mother board, or costs a fortune and settling somewhere pretty close to what is "typical" if not ... exactly typical.
- AlexandrB 3y ago
  > If they weren't, we'd be hearing anecdotes of one-character diffs appearing out of nowhere all the time, among many other issues. I think the amount of textual user data in memory is dwarfed by various kinds of program data (pointers, metadata, static strings, libraries, etc) as well as image data. It seems pretty likely that bit flips could result in a crash once in a while or could have no effect at all if the bit flip is in some rarely used piece of an executable loaded in memory. For example, would anyone notice that a letter in an error message string is corrupted? So errors may be happening at the rate that researchers report, but with only a tiny fraction of them happening in user data where they will be noticed.
- kimixa 3y ago
  One question is what proportion of the memory is being used for different things, and which of those you'd notice. The actual text representation is tiny compared to (for example) the framebuffer of said text rendered to the screen. I wouldn't be surprised if the size of my desktop background in RAM is larger than all the code I have written ever in my professional life. You probably wouldn't even notice a bit corruption in that, especially if it's a temporary image buffer being updated constantly. As you'd expect a random error to hit things proportional to their size in memory, there's likely orders of magnitude more running code in your editor than the actual text you're editing, which is more likely to simply crash than silently corrupt a diff. And on crashing, there's issues I've seen from user reports with backtraces and memory states that just don't make sense. I personally suspect that some, especially those without multiple examples, could quite easily be due to random memory corruption. And there's a long enough tail of these weird issues from that sampling I honestly wouldn't be surprised if such corruption events were a lot more common than many people think. And that's on a pretty locked down platform so no worries about consumers overclocking. I've heard it's even worse for some of my colleagues working on more gamer-adjacent products - some people's definition of "Stable" is rather loose. And as for the OG paper, I think it also said the errors weren't evenly distributed over all hardware - likely due to manufacturing differences some are run closer to the edge than others, but in a way that don't break quickly enough to easily test and be replaced as defective. So you might just be unlucky and have a weirdly unstable system.
- GTP 3y ago
  I think that the main difference is that the average person reboots his PC at least once a day, and in addition to this a good chunk if not most of your RAM is unallocated most of the time. Combine the two, and the chances of a bitflip actually affecting something important get very low.
YetAnotherNick 3y ago
But it should be applicable to EC2 instances, no? I have seen instances running for multiple months quite a bit of time. If there 1000s of random bit flip for any app, I think we should assume it is not running what we expect it to be running.
- p-e-w 3y ago
  It just dawned on me that EC2 is probably named that to avoid confusion with ECC.
- ericpauley 3y ago
  All EC2 instances use ECC: https://aws.amazon.com/ec2/faqs/ https://aws.amazon.com/ec2/faqs/
GTP 3y ago
> As densities for things like RAM increase, error rates have to decrease commensurately to hide errors from end users. But I would say that with more density you actually increase the error rate. Things like rowhammer are possible because higher density means smaller capacitors (as volume, and also as capacity) and so easier to casue bitflips. Or there is something in recent RAMs to compensate for the smaller caps?

wongarsu 3y ago

> they are more likely to run hardware much hotter than typical consumers I'd expect laptops and phones with their limited cooling capacity to run their hardware much hotter than datacenters with their expensive HVAC systems

p-e-w 3y ago

I'm not convinced by this argument. In practice, errors on consumer hardware seem to be several orders of magnitude less common than implied by the paper. If they weren't, we'd be hearing anecdotes of one-character diffs appearing out of nowhere all the time, among many other issues. It's hard to see how this could be explained away with the slight increase in RAM densities since 2008, or different temperatures in data centers (plenty of individuals run their systems crazy hot, e.g. when the tower is stuffed with dust after years without cleaning).

alpaca128 3y ago
Run any consumer device for more than 4 weeks and it'll slowly accumulate all kinds of weird bugs. I've seen smartphones unable to receive SMS because they simply ran too long without reboot, a Windows PC where selecting files wouldn't work anymore etc. > we'd be hearing anecdotes of one-character diffs appearing out of nowhere all the time On consumer devices it might simply be a small "attack" surface. A server running 24/7 serving countless requests would probably see more issues than a laptop running idle 95% of the time and loading a website every few minutes. Also a bit flip isn't guaranteed to cause a noticeable bug - and if it does it might not stand out from the usual noise of janky software oddities.
- taneq 3y ago
  We're perfectly capable of being bad enough at software to do that without invoking cosmic rays and quantum weirdness. How long would you expect to have to run memtest on any desktop computer in good condition before you observe a memory error? Because I would expect the number to be somewhere between 'weeks' and 'forever.'
- dspillett 3y ago
  > a Windows PC where selecting files wouldn't work anymore etc. That is just Windows Explorer being shite. They even have a specific “restart” button for it in Task Manager for when it leaks into itself so much it stops functioning usefully.
  - accoil 3y ago
    Isn't the restart there because explorer is also the desktop? If it was just killed, you'd have to know how to launch shutdown.exe from the task manager etc.
    - dspillett 3y ago
      I think it is a relatively recent (Win8?) addition, and explorer has been the shell since '95. I take adding a specific button for it as an admission that Explorer is getting increasingly buggy, so users are needing to restart it more. Before then you could do the same thing from the same task manager just in two steps: kill the existing explorer.exe and start a new one from the Run command in the menus.
- semi-extrinsic 3y ago
  IDK what you mean by consumer, but I've routinely had desktop computers running 24/7 for several hundred days without noticeable glitches on Linux (BTW I run Arch).
- sidewndr46 3y ago
  I find this kind of difficult to believe, because I have machines with >6 months of uptime and never see any issues. Always just running regular consumer grade hardware.
RajT88 3y ago
Somebody at Mozilla a while back had the idea of using the browser telemetry to find off-by-one errors due to bit flip, which were correlated with cosmic storms. Interesting, but not definitive results: https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-what-flips-your-bit/ https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-... I have access to some massive number of API logs for my job. So it was easy for me to do something similar - most bit flip errors would result in a 4xx of some sort, especially 404. Every time there's a huge magnetic storm, I run a query to see if 400's go up correlating with the storm. I even filter the logs for the geo hardest hit. Nope. No correlation. ¯\_(ツ)_/¯
- k1t 3y ago
  ArenaNet did this for GuildWars when they encountered particularly inexplicable bugs: [Mike O’Brien] wrote a module (“OsStress”) which would allocate a block of memory, perform calculations in that memory block, and then compare the results of the calculation to a table of known answers. He encoded this stress-test into the main game loop so that the computer would perform this verification step about 30-50 times per second. On a properly functioning computer this stress test should never fail, but surprisingly we discovered that on about 1% of the computers being used to play Guild Wars it did fail! One percent might not sound like a big deal, but when one million gamers play the game on any given day that means 10,000 would have at least one crash bug. Our programming team could spend weeks researching the bugs for just one day at that rate! https://www.codeofhonor.com/blog/whose-bug-is-this-anyway https://www.codeofhonor.com/blog/whose-bug-is-this-anyway
  - sidewndr46 3y ago
    were they testing system RAM exclusively? If it's VRAM, I'd agree with this. Back in the day we'd even overclock VRAM until we got "snow" in the rendered graphics then backoff a bit.
  - mdip 3y ago
    Thanks for sharing that -- on its own it was a great read. At first, I was not surprised that they saw 1% of GuildWars players had these issues. My reasoning was given the time of the article (2012) and pointing to a more distant hazy point somewhere between StarCraft and Diablo[0] their user-base probably included a lot of "custom builds", a not-small portion of which would be "built intended for overclocking", in an era when the firmware/motherboard/processors would let you apply settings that would become extremely unstable under load[1] ... and I got about that far before I thought, only 1% of users managed to make their "let's see what happens if we alter the voltages/frequencies such that we run the DDR2-333 memory at DDR2-2048 (it's been a while), change the various "latency" settings throwing caution to the wind. Meanwhile all of this hardware is sitting in a case with a fan configuration that slightly lowers the air pressure in the case[2] ... Back in the day (less common, today) you could often put any value from a range of "Way Too Low" to "Way Too High" and steps in-between. If it could reach the boot sector,... "Weeeeeeeeee!!" I recall a case of a customer who completely trashed an Intel Pentium (2? 3?) chip via firmware settings "my. kid. was. F---in'. with![3]" Maybe I'm wrong and it's just a matter that the game involved attracts people who rarely OC/build and the 1% is "just those people", but anecdotally "in my life", everyone I knew back then who was a PC gamer either started with a build of their own or went that route the instant they had a job which allowed them buy their own computer (rather than sharing Dad's or getting his hand-me-downs if you were really lucky/have fewer years on you than I do). My experience, especially back in the 08-09 era, was that things were ridiculously forgiving. Depending on the use, overclocking would either "do nothing at all" or maybe let you get away with slightly better graphics settings while maintaining a playable game. Generally, I purchased the cheapest memory rated slightly better than what I intended to configure them for (DDR2-xxx slightly higher than where I will clock it) with whatever I decided were the right latency values[4], placebo-overclocked day-to-day, but would push it from time to time. There's been times the system simply doesn't boot but one "tiny notch south" and it'll pass a 72-hour memory stress-test. I've had crashes, locally, that I can't explain beyond "memory corruption" but I can't say if it's an OS failure, hardware failure or "something that's not memory corruption but malfunctioning other hardware" ... it passes a stress test, I usually find the "other hardware" a little while later (surprisingly often the PSU, for me). [0] Unclear if editions implied are "I" in both cases, and it's not important enough to matter since reasonably earlier than 2012 puts you in the category of "the firmware/motherboards will let you do incredibly stupid things". [1] Sometimes subtle "some character's armor texture is transparent" (making your enemies see-through, not naked, much to my teenage chagrin). Sometimes crazy like "Everything is shades of green with partial geometries". [2] Say, if you had to have the most sleek case where every available air-flow "vent" can take a fan facing inward or outward (where it can sometimes be a bad idea to not leave one empty) has all of the fans blowing out... I've seen "The Vacuum Computer". [3] So, this really happened to me (CompUSA, filling in at the Parts Department), and that's exactly the way I hear it, but you need to understand that it wasn't some form of "Jersey Shore" accent -- this guy was absolutely Korean with an accent so thickand not a day younger than...60? Aside from the shock of a person very much struggling with the English language dragging out the "F" word, you just don't see a plain-looking-harmless-old-man speak with such vigor and gusto! [4] This being "unimportant" information for me day-to-day, requires hours of "refreshing via Google" and learning every miserable thing that's changed, coming to a complete understanding, finding that the memory I want either doesn't exist, doesn't work with my mother board, or costs a fortune and settling somewhere pretty close to what is "typical" if not ... exactly typical.
AlexandrB 3y ago
> If they weren't, we'd be hearing anecdotes of one-character diffs appearing out of nowhere all the time, among many other issues. I think the amount of textual user data in memory is dwarfed by various kinds of program data (pointers, metadata, static strings, libraries, etc) as well as image data. It seems pretty likely that bit flips could result in a crash once in a while or could have no effect at all if the bit flip is in some rarely used piece of an executable loaded in memory. For example, would anyone notice that a letter in an error message string is corrupted? So errors may be happening at the rate that researchers report, but with only a tiny fraction of them happening in user data where they will be noticed.
kimixa 3y ago
One question is what proportion of the memory is being used for different things, and which of those you'd notice. The actual text representation is tiny compared to (for example) the framebuffer of said text rendered to the screen. I wouldn't be surprised if the size of my desktop background in RAM is larger than all the code I have written ever in my professional life. You probably wouldn't even notice a bit corruption in that, especially if it's a temporary image buffer being updated constantly. As you'd expect a random error to hit things proportional to their size in memory, there's likely orders of magnitude more running code in your editor than the actual text you're editing, which is more likely to simply crash than silently corrupt a diff. And on crashing, there's issues I've seen from user reports with backtraces and memory states that just don't make sense. I personally suspect that some, especially those without multiple examples, could quite easily be due to random memory corruption. And there's a long enough tail of these weird issues from that sampling I honestly wouldn't be surprised if such corruption events were a lot more common than many people think. And that's on a pretty locked down platform so no worries about consumers overclocking. I've heard it's even worse for some of my colleagues working on more gamer-adjacent products - some people's definition of "Stable" is rather loose. And as for the OG paper, I think it also said the errors weren't evenly distributed over all hardware - likely due to manufacturing differences some are run closer to the edge than others, but in a way that don't break quickly enough to easily test and be replaced as defective. So you might just be unlucky and have a weirdly unstable system.
GTP 3y ago
I think that the main difference is that the average person reboots his PC at least once a day, and in addition to this a good chunk if not most of your RAM is unallocated most of the time. Combine the two, and the chances of a bitflip actually affecting something important get very low.

YetAnotherNick 3y ago

But it should be applicable to EC2 instances, no? I have seen instances running for multiple months quite a bit of time. If there 1000s of random bit flip for any app, I think we should assume it is not running what we expect it to be running.

p-e-w 3y ago
It just dawned on me that EC2 is probably named that to avoid confusion with ECC.
ericpauley 3y ago
All EC2 instances use ECC: https://aws.amazon.com/ec2/faqs/ https://aws.amazon.com/ec2/faqs/

GTP 3y ago

> As densities for things like RAM increase, error rates have to decrease commensurately to hide errors from end users. But I would say that with more density you actually increase the error rate. Things like rowhammer are possible because higher density means smaller capacitors (as volume, and also as capacity) and so easier to casue bitflips. Or there is something in recent RAMs to compensate for the smaller caps?