40 ms·

10% of Firefox crashes are caused by bitflips

• deleted 7mo ago
  [deleted]
• thegrim33 7mo ago
  A 5 part thread where they say they're "now 100% positive" the crashes are from bitflips, yet not a single word is spent on how they're supposedly detecting bitflips other than just "we analyze memory"?
  • tredre3 7mo ago
    > last year we deployed an actual memory tester that runs on user machines after the browser crashes. He doesn't explain anything indeed but presumably that code is available somewhere.
    • thatguy27 7mo ago
      [flagged]
    • hedora 7mo ago
      That, and 50% of the machines where their heuristics say it is a hardware error fail basic memory tests. I've seen a lot of confirmed bitflips with ECC systems. The vast majority of machines that are impacted are impacted by single event upsets (not reproducible). (I worded that precisely but strangely because if one machine has a reproducible problem, it might hit it a billion times a second. That means you can't count by "number of corruptions".) My take is that their 10% estimate is a lower bound.
  • rincebrain 7mo ago
    The simplest way to do this, what I believe memtest86 and friends do, is to write a fixed pattern over a region of memory and then read it back later and see if it changed; then you write patterns that require flipping the bits that you wrote before, and so on. Things like [1] will also tell you that something corrupted your memory, and if you see a nontrivial (e.g. lots of bits high and low) magic number that has only a single bit wrong, it's probably not a random overwrite - see the examples in [2]. There's also a fun prior example of experiments in this at [3], when someone camped on single-bit differences of a bunch of popular domains and examined how often people hit them. edit: Finally, digging through the Mozilla source, I would imagine [4] is what they're using as a tester when it crashes. [1] - https://github.com/mozilla-firefox/firefox/commit/917c4a6bfa66853cbf0df08973d48ea7cba6547f https://github.com/mozilla-firefox/firefox/commit/917c4a6bfa... [2] - https://bugzilla.mozilla.org/show_bug.cgi?id=1762568 https://bugzilla.mozilla.org/show_bug.cgi?id=1762568 [3] - https://media.defcon.org/DEF%20CON%2019/DEF%20CON%2019%20presentations/DEF%20CON%2019%20-%20Dinaburg-Bit-Squatting.pdf https://media.defcon.org/DEF%20CON%2019/DEF%20CON%2019%20pre... [4] - https://github.com/mozilla-firefox/firefox/blob/main/toolkit/crashreporter/client/app/src/memory_test.rs https://github.com/mozilla-firefox/firefox/blob/main/toolkit...
    • rendaw 7mo ago
      That would tell you if there's a bitflip in your test, but not if there's a bitflip in normal program code causing a crash, no? IIUC GP's questions was how do they actually tell after a crash that that crash was caused by a bitflip.
      • rincebrain 7mo ago
        The example I gave in there is of adding sentinel values in your data, so you can check the constants in your data structures later and go "oh, this is overwritten with garbage" versus "oh, this is one or two bits off". I would imagine plumbing things like that through most common structures is what was done there, though I haven't done the archaeology to find out, because Firefox is an enormous codebase to try and find one person's commits from several years ago in.
        • patrulek 7mo ago
          But it would be also possible that sentinel value used for comparison changed because of bitflip, not data structure used by program.
          • rincebrain 7mo ago
            Detecting that is the point, yes.
        • kevincox 7mo ago
          This doesn't always protect against out-of-bounds writes. Although if these sentinel values are in read only memory mappings it probably gets pretty close. (Especially if you consider kernel memory corruption a "bitflip".)
          • rincebrain 7mo ago
            Well, yes, the author described it as "a heuristic", which checking sentinel values would seem to match.
  • hexyl_C_gut 7mo ago
    It sounds like they don't know that the crashes are from bitflips but those crashes are from people with flaky memory which probably caused the crash?
  • wmf 7mo ago
    A common case is a pointer that points to unallocated address space triggers a segfault and when you look at the pointer you can see that it's valid except for one bit.
    • dboreham 7mo ago
      That tells you one bit was changed. It doesn't prove that single bit changed due to a hardware failure. It could have been changed by broken software.
      • LeifCarrotson 7mo ago
        Broken software causes null pointer references and similar logic errors. It would be extremely unusual to have an inadvertent ptr ^= (1 << rand_between(0,64)); that got inserted in the code by accident. That's just not the way that we write software.
        • vlovich123 7mo ago
          Except no one is claiming the bit flip is the pointer vs the data being pointed to or a non pointer value. Given how we write software there’s a lot more bits not in pointer values that still end up “contributing “ to a pointer value. Eg some offset field that’s added to a pointer has a bit flip, the resulting pointer also has a bit flip. But the offset field could have accidentally had a mask applied or a bit set accidentally due to the closeness of & and && or | and ||.
          • rockdoe 7mo ago
            I think that if you hit the crash in the same line of code many times, you can safely assume it's your own bug and not a memory issue. If it's only hit once by a random person, memory starts being more likely. (Unless that LOC is scanning memory or smth)
            • vlovich123 7mo ago
  • hrmtst93837 7mo ago
    [flagged]
    • gcp 7mo ago
      I don't think Firefox has the access permissions needed to read MCE status, and the vast majority of our users don't have ECC, let alone they're going to run memtest86(+) after a Firefox crash. If they did, we wouldn't be having this discussion to begin with!
• devy 7mo ago
  I wonder if Chrome dev team can corroborate on this finding in their crash reporting.
• kmoser 7mo ago
  The next logical step would be to somehow inform users so they could take action to replace the bad memory. I realize this is a challenge given the anonymized nature of the crash data, but I might be willing to trade some anonymity in exchange for stability.
  • titaniumtravel 7mo ago
    The easy solution for that is to just do that analysis locally... Firefox doesn't submit the full core dumps anyhow for this exact reason and therefore needs to do some preprocessing in any case.
  • shiroiuma 7mo ago
    >The next logical step would be to somehow inform users so they could take action to replace the bad memory. This isn't really feasible: have you looked at memory prices lately? The users can't afford to replace bad memory now.
    • kmoser 7mo ago
      I have two identical computers; if the RAM on one is bad, I can swap out the RAM from another. But thank you for your concern.
    • hiddendoom45 7mo ago
      The memory issue may not necessarily be from bad ram, it can also be due to configuration issues. Or rather it may be fixable with configuration changes. I had memory issues with my PC build which I fixed by reducing the speed to 2800MHZ, which is much lower than its advertised speed of 5600MHZ. Actually looking back at this it might've configured its speed incorrectly in the first place, reducing it to 2800 just happened to hit a multiple of 2 of its base clock speed.
    • monadgonad 7mo ago
      The current situation really has zero bearing on the principle that it’s better to inform users of this.
  • 7mo ago
• kdklol 7mo ago
  I'm glad to see somebody is getting some data on this, I feel bad memory is one of the most underrated issues in computing generally. I'd like to see a more detailed writeup on this, like a short whitepaper.
• tredre3 7mo ago
  > In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! If I subtract crashes that are caused by resource exhaustion (such as out-of-memory crashes) this number goes up to around 15%. Crashes caused by resource exhaustion are still software bugs in Firefox. At least on sane operating systems where memory isn't over-comitted.
  • LorenPechtel 7mo ago
    Memory isn't the only resource.
  • rockdoe 7mo ago
    What's the expected behavior of a JavaScript program that allocates all memory on the machine?
    • gkbrk 7mo ago
      Browser killing the tab way before it happens
• vsgherzi 7mo ago
  is there a way to get the memory tester he mentioned? Is it open source? Once Ram goes bad is there a way or recovering it or is it toasted forever?
  • vizzier 7mo ago
    https://www.memtest86.com/ https://www.memtest86.com/ Errors may be caused by bad seating/contact in the slots or failing memory controllers (generally on the CPU nowadays) but if you have bad sticks they're generally done for.
  • foresto 7mo ago
    You can map known-bad memory regions to avoid using them. https://www.memtest86.com/blacklist-ram-badram-badmemorylist.html https://www.memtest86.com/blacklist-ram-badram-badmemorylist...
    • hinkley 7mo ago
      However if the third chip on your memory stick is properly broken, then the third bit out of every word of memory may get stuck high or low, and then the whole chip is absolutely worthless. The most expensive memory failure I had was of this sort, and frustratingly came from accidentally unplugging the wrong computer. After this I did buy some used memory from a recycling center that had the sorts of problems you described and was able to employ them by masking off the bad regions.
  • RachelF 7mo ago
    This is the best way of marking regions of RAM as bad in Windows: https://github.com/prsyahmi/BadMemory https://github.com/prsyahmi/BadMemory I've used it for many years. It only fixes physical hardware faults, not timing errors. For example if a RAM cell is damaged by radiation, not if you're overclocking your RAM.
• mrguyorama 7mo ago
  People I think are overindexing on this being about "Bad hardware". We have long known that single bit errors in RAM are basically "normal" in terms of modern computers. Google did this research in 2009 to quantify the number of error events in commodity DRAM https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf https://static.googleusercontent.com/media/research.google.c... They found 25,000 to 70,000 errors per billion device hours per Mbit and more than 8% of DIMMs affected by errors per year. At the time, they did not see an increase in this rate in "new" RAM technologies, which I think is DDR3 at that time. I wonder if there has been any change since then. A few years ago, I changed from putting my computer to sleep every night, to shutting it down every night. I boot it fresh every day, and the improvements are dramatic. RAM errors will accumulate if you simply put your computer to sleep regularly.
  • jmalicki 7mo ago
    There is DRAM which is mildly defective but got past QC. There are power suppliers that are mildly defective but got past QC. There are server designs where the memory is exposed to EMI and voltage differences that push it to violate ever more slightly that push it past QC. Hardware isn't "good" or "bad", almost all chips produced probably have undetected mild defects. There are a ton of causes for bitflips other than cosmic rays. For instance, that specific google paper you cited found a 3x increase in bitflips as datacenter temperature increased! How confident are you the average Firefox user's computer is as temperature-controlled as a google DC? It also found significantly higher rates as RAM ages! There are a ton of physical properties that can cause this, especially when running 24/7 at high temperatures.
  • shiroiuma 7mo ago
    It'd be interesting to see how your experience would differ if you put it to sleep at night after switching to ECC RAM. Unfortunately, not that many consumer platforms make this possible or affordable.
    • SoftTalker 7mo ago
      Most computers running Firefox won't have ECC RAM.
• NotGMan 7mo ago
  >> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! I find this impossible to believe. If this were so all devs for apps, games, etc... would be talking about this but since this is the first time I'm hearing about this I'm seriously doubting this. >> This is a bit skewed because users with flaky hardware will crash more often than users with functioning machines, but even then this dwarfs all the previous estimates I saw regarding this problem. Might be the case, but 10% is still huge. There imo has to be something else going on. Either their userbase/tracking is biased or something else...
  • netcoyote 7mo ago
    It is huge, but real (see https://news.ycombinator.com/item?id=47258500 https://news.ycombinator.com/item?id=47258500) Browsers, videogames, and Microsoft Excel push computers really hard compared to regular applications, so I expect they're more likely to cause these types of errors. The original Diablo 2 game servers for battle.net, which were Compaq 1U servers, failed at astonishing rates due to their extremely high utilization and consequent heat-generation. Compaq had never seen anything like it; most of their customers were, I guess, banking apps doing 3 TPS.
    • alpaca128 7mo ago
      In my case it doesn't seem to be related to system load. I have an issue where (mainly) using FF can trigger random system freezes on Linux, often with the browser going down first. But running CPU/memory stress tests, compiling things etc don't cause any errors and the cooler is downright bored.
      • alpaca128 7mo ago
        Update: it's starting to look like CPU C-states were the problem.
  • SoftTalker 7mo ago
    Computers today have many GB of RAM, and programs that use it. The more RAM you have, the higher the probabilty that there will be some bad bits. And the more RAM a program uses, the more likely it will be using some that is bad. Same phenomenon with huge hard drives.
• nubinetwork 7mo ago
  470k crashes in a week? Considering how low their market share is, that would suggest every install crashes several times a day... I gotta call bs.
  • titaniumtravel 7mo ago
    Based on what data? According to their reporting they have around 200 Million monthly users, which seems compatible with 470k crashes a week? See <https://data.firefox.com/dashboard/user-activity https://data.firefox.com/dashboard/user-activity>
    • nubinetwork 7mo ago
      2% worldwide? https://gs.statcounter.com/browser-market-share https://gs.statcounter.com/browser-market-share Granted, they're probably just as accurate as netcraft. /shrug
      • titaniumtravel 7mo ago
        The nuance here is of cause that there are a bunch of people using multiple browsers. Also I mean there are a lot of people using browsers on the world
        • hinkley 7mo ago
          If 10% of firefox users are also iOS users, which is not unlikely, then those people get double-counted. In my case I probably use my phone and tablet for at least 50% of my web traffic, not counting youtube, which also skews things.
  • vizzier 7mo ago
    For my part I'm not sure I recall a crash having daily driven firefox in quite some time. I'd suspect that the large number of bit errors might be driven by a small number of poor hardware clients.
  • deleted 7mo ago
• stnvh 7mo ago
  Try running two instances of Firefox in parallel with different profiles, then do a normal quit / close operation on one after any use. Demons exist here.
  • quesera 7mo ago
    Describe "demons"? I run four Firefox instances simultaneously, most of the time. No issues to report.
    • stnvh 7mo ago
      Long hangs / never closes, crash report screen triggers often. macOS. This occurs for me when launching instances from the about:profiles page and using each instance for what I'd describe as normal use
      • quesera 7mo ago
        I see. Yeah, I've seen this. It seems more likely to happen when the profile has been running for a long time (a couple weeks?) and/or using a large amount of RAM. There's a 60-secish timeout before it gives up and pops that crash report window. I don't think it's a crash per se, just an unresolved file lock or similar. I haven't noticed whether there's any relationship to running multiple profiles. I am almost always running several at a time, and the issue only occurs sometimes. It has no (other) negative side effects, as far as I can tell, but it was unsettling at first. I'm on macOS also, and I launch from the command line (effectively, I actually have separate launchers for each profile, but they just run a shell script with different arguments).
      • roryirvine 7mo ago
        Same, also on macOS. My "personal" firefox profile on my work Macbook Pro, which I use for occasional gmail, HN, wikipedia, and pretty much nothing else, has crashed twice in the last 6 weeks - both times when shutting down to update the OS. Honestly, I've been blaming MacOS for it since other apps also crashed at the same time (the first time it was Microsoft Intune, the second time it was Slack - I doubt either uses Firefox internally). I don't recall seeing a Firefox crash on my personal laptop running Linux at any point in the past few years.
• conartist6 7mo ago
  Also a polite reminder that most of those crashes will be concentrated on machines with faulty memory so the naive way of stating the statistic may overestimate its impact to the average user. For the average user this is the difference between 4/5 crashes are from software bugs and 5/5 crashes are from software bugs, and for a lot of people it will still be 5/5
• adonovan 7mo ago
  Very interesting. The Go toolchain has an (off by default) telemetry system. For Go 1.23, I added the runtime.SetCrashOutput function and used it to gather field reports containing stack traces for crashes in any running goroutine. Since we enabled it over a year ago in gopls, our LSP server, we have discovered hundreds of bugs. Even with only about 1 in 1000 users enabling telemetry, it has been an invaluable source of information about crashes. In most cases it is easy to reconstruct a test case that reproduces the problem, and the bug is fixed within an hour. We have fixed dozens of bugs this way. When the cause is not obvious, we "refine" the crash by adding if-statements and assertions so that after the next release we gain one additional bit of information from the stack trace about the state of execution. However there was always a stubborn tail of field reports that couldn't be explained: corrupt stack pointers, corrupt g registers (the thread-local pointer to the current goroutine), or panics dereferencing a pointer that had just passed a nil check. All of these point to memory corruption. In theory anything is possible if you abuse unsafe or have a data race, but I audited every use of unsafe in the executable and am convinced they are safe. Proving the absence of data races is harder, but nonetheless races usually exhibit some kind of locality in what variable gets clobbered, and that wasn't the case here. In some cases we have even seen crashes in non-memory instructions (e.g. MOV ZR, R1), which implicates misexecution: a fault in the CPU (or a bug in the telemetry bookkeeping, I suppose). As a programmer I've been burned too many times by prematurely blaming the compiler or runtime for mistakes in one's own code, so it took a long time to gain the confidence to suspect the foundations in this case. But I recently did some napkin math (see https://github.com/golang/go/issues/71425#issuecomment-3968582591 https://github.com/golang/go/issues/71425#issuecomment-39685...) and came to the conclusion that the surprising number of inexplicable field reports--about 10/week among our users--is well within the realm of faulty hardware, especially since our users are overwhelmingly using laptops, which don't have parity memory. I would love to get definitive confirmation though. I wonder what test the Firefox team runs on memory in their crash reporting software.
  • sieep 7mo ago
    Ive been trying to push my boss towards more analytics/telemetry in production that focus on crashes, thanks for sharing.
  • 7mo ago
• camkego 7mo ago
  It is rumored heavily on HN that when the first employee of Google, Craig Silverstein was asked about his biggest regret, he said: "Not pushing for ECC memory."
  • adonovan 7mo ago
    It's true that in the very early days Google used cheap computers without ECC memory, and this explains the desire for checksums in older storage formats such as RecordIO and SSTable, but our production machines have used ECC RAM for a long time now.
  • srean 7mo ago
    One of the nicest guys I have met. Was an intern at Google at that time, firing off mapreduces then (2003-2004) was quite a blast. The Peter Weinberger theme T-shirt too.
  • keyringlight 7mo ago
    One of the points Linus Torvalds made a few years back was that enthusiasts/PC gamers should be pissed that consumer product availability/support for ECC is spotty because as mentioned up-thread they're the kind of user that will push their system, and if memory is the cause of instability there will be a smoking gun (and they can then set the speed within its stable capacity). Diagnosing bad RAM is a pain in the rear even if you're actively looking for a cause, never mind trying to get a general user to go further than blaming software or gremlins in the system for weirdness on whatever frequency it's occurring at.
• netcoyote 7mo ago
  I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known values. Around 1 out of 1000 computers would fail this test! We'd save the test result to the registry and include the result in automated bug reports. The common causes we discovered for the problem were: - overclocked CPU - bad memory wait-state configuration - underpowered power supply - overheating due to under-specced cooling fans or dusty intakes These problems occurred because Guild Wars was rendering outdoor terrain, and so pushed a lot of polygons compared to many other 3d games of that era (which can clip extensively using binary-space partitioning, portals, etc. that don't work so well for outdoor stuff). So the game caused computers to run hot. Several years later I learned that Dell computers had larger-than-reasonable analog component problems because Dell sourced the absolute cheapest stuff for their computers; I expect that was also a cause. And then a few more years on I learned about RowHammer attacks on memory, which was likely another cause -- the math computations we used were designed to hit a memory row quite frequently. Sometimes I'm amazed that computers even work at all! Incidentally, my contribution to all this was to write code to launch the browser upon test-failure, and load up a web page telling players to clean out their dusty computer fan-intakes.
  • pndy 7mo ago
    I didn't expect to read bits of GW story here from one of the founders - thanks!
  • Analemma_ 7mo ago
    There's a famous Raymond Chen post about how a non-trivial percentage of the blue screen of death reports they were getting appeared to be caused by overclocking, sometimes from users who didn't realize they had been ripped off by the person who sold them the computer: https://devblogs.microsoft.com/oldnewthing/20050412-47/?p=35923 https://devblogs.microsoft.com/oldnewthing/20050412-47/?p=35.... Must've been really frustrating.
• brador 7mo ago
  How many are caused by cosmic radiation bitflips?
  • emmelaich 7mo ago
    An SO question indicates "10 GB of memory should show an ECC event every 1,000 to 10,000 hours," https://stackoverflow.com/questions/2580933/cosmic-rays-what-is-the-probability-they-will-affect-a-program https://stackoverflow.com/questions/2580933/cosmic-rays-what...
• bakugo 7mo ago
  I was running my PC with bad memory for a few weeks last year. Firefox crashed a LOT, way more than any other application I used during that time, so I've probably contributed a decent amount to these numbers...
  • shevy-java 7mo ago
    It could be that firefox is written inefficiently though.
    • black_knight 7mo ago
      Or so efficiently that every bit counts and plays a vital role! Even a single bit off and the thing derails…
• shevy-java 7mo ago
  > In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.
  • phyzome 7mo ago
    ...normally browsers don't crash at all. Something's wrong with your computer.
  • LM358 7mo ago
    10% of crashes does not imply 10% of your crashes.
    • michaelcampbell 7mo ago
      Doesn't mean 10% of any crashes either; the OP narrowed it down to roughly 1 in 20, then pulled a factor of 2 out of, well, thin air to get 10%.
  • WhatsTheBigIdea 7mo ago
    Your gut may be leading you astray? I also find that firefox crashes much more than chrome based browsers, but it is likely that chrome's superior stability is better handing of the other 90% of crashes. If 50% of chrome crashes were due to bit flips, and bit flips effect the two browsers at basically the same rate, that would indicate that chrome experiences 1/5th the total crashes of firefox... even though the bit flip crashes happen at the same rate on both browsers. It would have been better news for firefox if the number of crashes due to faulty hardware were actually much higher! These numbers indicate the vast majority of firefox crashes are actually from buggy software : (
    • chrismorgan 7mo ago
      I run Firefox Nightly, and occasionally a little Chromium stable. Both are running under Wayland, which I believe is still not considered stable in either. In the last year of Firefox, I had one full crash (the first in maybe three years), and about four tab crashes. Plus duplicates from deliberately reproducing issues. All but one (which I’m not certain about) were Nightly-only, fixed long before reaching stable. Were I running stable, I suspect I would not have had more than three crashes of any kind in the past five years. I can’t say the same for Chromium. Despite barely using it, I had at least one tab or iframe crash last year, and there’s a moderate chance (I’ll suggest 15%) on any given day of leaving it open that it will just spontaneously die while I’m not paying attention to it (my wild guess, based on observations about Inkscape if it’s executing something CPU-bound for too long: it’s not responding in a timely fashion to the compositor, and is either getting killed or killing itself, not sure which that would be). Frankly, from a crashing perspective, both are very reliable these days. Chromium is still far more prone to misrendering and other misbehaviour—they prefer to ship half-baked implementations and fix them later; Firefox, on the other hand, moves slower but has fewer issues in what they do ship.
• darkhorn 7mo ago
  What brands or types of memory cards are less likely to crash by bitflips?
  • estimator7292 7mo ago
    ECC
• bhelkey 7mo ago
  I would love to see DDR4 vs DDR5 bitflips. As I understand it DDR5 must come with some level of ECC [1]. [1] https://www.corsair.com/us/en/explorer/diy-builder/memory/is-ddr5-ecc-memory/?srsltid=AfmBOopZvzojEF1VC4RbhPAsxkowIZnMuiTJO37kTr1lNKWjGiTHtf1J https://www.corsair.com/us/en/explorer/diy-builder/memory/is...
  • kevin_thibedeau 7mo ago
    DDR5 comes with marginal DRAM that is patched up with ECC to boost yields. It's not the same as fully reliable RAM.
    • stinkbeetle 7mo ago
      Similar to CPUs, where many arrays have spare yield capacity, even whole cores can get disabled (and possibly sold in a different bin). DRAM stores redundant electrons in capacitors to patch it up and boost yields. Everything in reliability is a spectrum. "ECC" does not give you fully reliable RAM. UEs are still be observed. What's the chance of fail? If you have one device that achieves equal performance with less reliable cells and redundancy to another device that uses more reliable cells without redundancy, it's not really any different. NAND is horribly flaky, cell errors are a matter of course. You could buy boutique NOR or SLC NAND or something if you want really good cells. You wouldn't though, because it would be ruinously expensive, but also it would not really give you a result that an SSD with ECC can't achieve.
    • Aurornis 7mo ago
      The net error rate is lower with the internal ECC. DDR4 is not fully reliable memory either. This is common for many high speed electrical engineering challenges: Running a slightly higher error rate option with ECC on top can have an overall lower error rate at higher throughput than the alternative of running it slow enough to push the error rate down below some threshold. It makes some people nervous because they don’t like the idea of errors being corrected, but the system designers are looking at overall error rates. The ECC is included in the system’s operation so it isn’t something that is worthwhile to separate out.
• kev009 7mo ago
  It's high enough that I would wonder if some systems software issues are mixed in, like rare races in malloc or page table management.
• AndriyKunitsyn 7mo ago
  >That fancy ARM-based MacBook with RAM soldered on the CPU package? We've got plenty of crashes from those, good luck replacing that RAM without super-specialized equipment and an extraordinarily talented technician doing the job. CPU caches and registers - how exactly are they different from a RAM on a SoC in this regard?
  • phs2501 7mo ago
    For one thing, static vs dynamic RAM. Static RAM (which is what's used for your typical CPU cache) is implemented with flip-flops and doesn't need to be refreshed, reads aren't destructive like DRAM, etc.
  • benjaminl 7mo ago
    In just about every way. CPU caches are made from SRAM and live on the CPU itself. Main system RAM is made from DRAM and live on separate chips even if they are soldered into the same physical package (system in package or SiP). The RAM still isn't on the SoC.
    • brcmthrowaway 7mo ago
      Unless its gpu
  • wmf 7mo ago
    Caches and registers are also subject to bitflips. In many CPUs the caches use ECC so it's less of a problem. Intel did a study showing that many bits in registers are unused so flipping them doesn't cause problems.
  • stinkbeetle 7mo ago
    At that level, they are not different. They could suffer from UE due to defect, marginal system (voltage, temperature, frequency), or radiation upset, suffer electromigration/aging, etc. And you can't replace them either. CPUs tend to be built to tolerate upsets, like having ECC and parity in arrays and structures whereas the DRAM on a Macbook probably does not. But there is no objective standard for these things, and redundancy is not foolproof it is just another lever to move reliability equation with.
• 1over137 7mo ago
  Curious why this article is written into divided up chunks?
  • wmf 7mo ago
    They're tweets.
• eek2121 7mo ago
  Definitely going to hard disagree with Gabriele Svelto's take. I could point to the comments, however, let me bring up my own experiences across personal devices and organizational devices. In particular, note where he says this: "I can't answer that question directly because crash reports have been designed so that they can't be tracked down to a single user. I could crunch the data to find the ones that are likely coming from the same machine, but it would require a bit of effort and it would still only be a rough estimate." You can't claim any percentage if you don't know what you are measuring. Based on his hot take, I can run an overclocked machine have firefox crash a few hundred thousand times a day and he'll use my data to support his position. Further, see below: First: A pre-text: I use Firefox, even now, despite what I post below. I use it because it is generally reliable, outside of specific pain points I mention, free, open source, compatible with most sites, and for now, is more privacy oriented than chrome. Second: On both corporate and home devices, Firefox has shown to crash more often than Chrome/Chromium/Electron powered stuff. Only Safari on Windows beats it out in terms of crashes, and Safari on Windows is hot garbage. If bit flips were causing issues, why are chromium based browsers such as edge and Chrome so much more reliable? Third: Admittedly, I do not pay close enough attention to know when Firefox sends crash reports, however, what I do know is that it thinks it crashes far more often than it does. A `sudo reboot` on linux, for example, will often make firefox think it crashed on my machine. (it didn't, Linux just kills everything quickly, flushes IO buffers, and reboots...and Firefox often can't even recover the session after...) Fourth: some crashes ARE repeatable (see above), which means bit flips aren't the issue. Just my thoughts.
  • jesup 7mo ago
    force-kills like sudo reboot will show UI on restart indicating it didn't shut down cleanly, but that isn't reported as a crash. You can see how often you actually crash via about:crashes (and also see what happened)
  • hedora 7mo ago
    Do you have any evidence that Firefox crashes more? Also, the latest version of Safari for Windows was released in 2012. How old is your Firefox?
• dana321 7mo ago
  And.. how do they not know its their software being leaky and causing these bitflips? These are potential bitflips. I found an issue only yesterday in firefox that does not happen in other browsers on specific hardware. My guess is that the software is riddled with edge-case bugs.
• Animats 7mo ago
  ECC should have become standard around the time memories passed 1GB. It's seriously annoying that ECC memory is hard to get and expensive, but memory with useless LEDs attached is cheap.
  • loeg 7mo ago
    It's not even ECC price/availability that bothers me so much, it's that getting CPUs and motherboards that support ECC is non-trivial outside of the server space. The whole consumer class ecosystem is kind of shitty. At least AMD allows consumer class CPUs to kinda sorta use ECC, unlike Intel's approach where only the prosumer/workstation stuff gets ECC.
    • rpcope1 7mo ago
      I've been honestly amazed people actually buy stuff that's not "workstation" gear given IME how much more reliably and consistently it works, but I guess even a generation or two used can be expensive.
      • loeg 7mo ago
        I've had zero issues with AMD's consumer tier of non-WX Threadripper and Ryzen models, FWIW.
      • thousand_nights 7mo ago
        overblown? billions of users use consumer tier hardware just fine. i have servers at home with years of uptime without any ECC memory
        • conception 7mo ago
          But how much bit rot? You’ll never know.
          • Maxion 7mo ago
            If I don't know about it, then how does it affect me / why should I care? My home server does what it is supposed to do and has done so for a decade. If bit rot /bit flips in memory does not affect my day-to-day life I much prefer cheaper hardware. I do hope the nuclear powerplant next door uses more fault tolerant hardware, though.
• phendrenad2 7mo ago
  Guesstimation at its finest.
• aforwardslash 7mo ago
  Going to be downvoted, but I call bullshit on this. Bitflips are frequent (and yes ECC is an improvement but does not solve the problem), but not that frequent. One can either assume users that enabled telemetry are an odd bunch with flaky hardware, or the implementation isnt actually detecting bitflips (potentially, as the messages indicate), but a plathora of problems. Having a 1/10 probability a given struct is either processed wrong, parsed wrong or saved wrong would have pretty severe effects in many, many scenarios - from image editing to cad. Also, bitflips on flaky hardware dont choose protection rings - it would also affect the OS routines such as reading/writing to devices and everything else that touches memory. Yup, i've seen plenty of faulty ram systems (many WinME crashes were actually caused by defective ram sticks that would run fine with W98), it doesnt choose browsers or applications.
  • dheera 7mo ago
    It says 10% of crashes If Firefox itself has so few bugs that it crashes very infrequently, it is not contradictory to what you are saying. I wouldn't be surprised if 99% of crashes in my "hello world" script are caused by bit flips.
    • aforwardslash 7mo ago
      Just updated with a comment. I see firefox crash routinely, so apparently our experiences are quite different :)
      • jesup 7mo ago
        You should look at about:crashes and see if there's any commonality in the causes, or bugs associated with them (though often bugs won't be associated with the crash if it isn't filed from crash-stats or have the crash signature in the bug)
      • antonf 7mo ago
        Maybe you should check your memory? I recently started to get quite a lot of Firefox crashes, and definitely contributed to this statistic. In the end, the problem was indeed memory - crashes stopped after I tuned down some of the timings. And I used this RAM for a few years with my original settings (XMP profile) without issue.
• chazburger 7mo ago
  [flagged]
  • 190n 7mo ago
    Operating systems use less RAM than Firefox.
  • DangitBobby 7mo ago
    I would expect operating systems to be very fault tolerant programs.
  • dankons 7mo ago
    Not necessarily, have had my fair share of dodgy OS behavior fixed by replacing RAM
  • chlorion 7mo ago
    Does it though? People experience "blue screens" and kernel panics and such pretty often.
• CamouflagedKiwi 7mo ago
  This is a pretty big claim which seems to imply this is much more common than expected, but there's no real information here and the numbers don't even stack up: > That's one crash every twenty potentially caused by bad/flaky memory, it's huge! And because it's a conservative heuristic we're underestimating the real number, it's probably going to be at least twice as much. So the data actually only supports 5% being caused by bitflips, then there's a magic multiple of 2? Come on. Let alone this conservative heuristic that is never explained - what is it doing that makes him so certain that it can never be wrong, and yet also detects these at this rate?
• fooker 7mo ago
  This seems like the kind of metric that 3 users with 15 year old machines can skew significantly. Has to be normalized, and outliers eliminated in some consistent manner.
  • rockdoe 7mo ago
    I'm pretty sure I saw them present on exactly this at FOSDEM?
• stinkbeetle 7mo ago
  This matches what I have long said, which is that adding ECC memory to consumer devices will not result in any incredible stability improvement. It will barely be a blip really. As we know from Google and other papers, most of these 10% of flips will be caused by broken or marginal hardware, of which a good proportion of which could be weeded out by running a memory tester for a while. So if you do that you're probably looking a couple out of every hundred crashes being caused by bitflips in RAM. A couple more might be due to other marginal hardware. The vast majority software. How often does your computer or browser crash? How many times per year? About 2-3 for me that I can remember. So in 50 years I might save myself one or two crashes if I had ECC. ECC itself takes about 12.5% overhead/cost. I have also had a couple of occasions where things have been OOM-killed or ground to a halt (probably because of memory shortage). Could be my money would be better spent with 10% more memory than ECC. People like to rave and rant at the greedy fatcats in the memory-industrial complex screwing consumers out of ECC, but the reality is it's not free and it's not a magical fix. Not when software causes the crashes. Software developers like Linus get incredibly annoyed about bug reports caused by bit flips. Which is understandable. I have been involved in more than one crazy Linux kernel bug that pulled in hardware teams bringing up new CPU that irritated the bug. And my experience would be far from unique. So there's a bit of throwing stones in glass houses there too. Software might be in a better position to demand improvement if they weren't responsible for most crashes by an order of magnitude...
• spiffy2025 7mo ago
  Travis Long had done something similar in 2022 at Mozilla. https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-what-flips-your-bit/ https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-...
• wakawaka28 7mo ago
  Ugh just write a real blog post dude.
• dbolgheroni 7mo ago
  When debugging something, I often remember the the quote, often misattributed to Einstein: "Insanity is doing the same thing over and over again and expecting different results". Then I remember about bitflips, and run a second, maybe a third time, just expecting the next bit to flip to not be in the routine I'm trying to debug.
• ptek 7mo ago
  So does this mean bool true = 3 or should bool true = 5? This will bloat the code a bit.
  • alok-g 7mo ago
    Interesting. Seems like software could be made a notch more robust by encoding true and false with a larger number of bit differences.
    • jdpage 7mo ago
      The canonical Boolean values in FORTH are 0 and -1 (that is, all bits set). IIRC the point of that is to unify the bitwise and logical operators, though, not detect bitflips. Also, at the machine code level, a Boolean controlling a branch or a while loop often doesn't ever make it out of the flags register, where it'll only be a single bit anyway because that's how the hardware works. Not really changeable in software.
• newscracker 7mo ago
  This is quite surprising to me, since I thought the percentage would be a lot lesser. But I don’t really know what the Firefox team does with crash reports and in making Firefox almost crash proof. I have been using it at work on Windows and for the last several years it always crashes on exit. I have religiously submitted every crash report. I even visit the “about:crashes” page to see if there are any unsubmitted ones and submit them. Occasionally I’ll click on the bugzilla link for a crash, only to see hardly any action or updates on those for months (or longer). Granted that I have a small bunch of extensions (all WebExtensions), but this crash-on-exit happens due to many different causes, as seen in the crash reports. I’m too loathe to troubleshoot with disabling all extensions and then trying it one by one. Why should an extension even cause a crash, especially when its a WebExtension (unlike the older XUL extensions that had a deeper integration into the browser)? It seems like there are fundamental issues within Firefox that make it crash prone. I can make Firefox not crash if I have a single window with a few tabs. That use case is anyway served by Edge and Chrome. The main reasons I use Firefox, apart from some ideological ones, are that it’s always been much better at handling multiple windows and tons of tabs and its extensibility (Manifest V2 FTW). I would sincerely appreciate Firefox not crashing as often for me.
  • ordu 7mo ago
    It is hard to judge, but a crash on exit seems to me a possible consequence of a damaged memory. Firefox frees all the resources and collects the garbage. I expect it to touch a lot of memory locations, and do something with values retrieved. > this crash-on-exit happens due to many different causes, as seen in the crash reports It points to the same direction: all these different causes are just symptoms, the root cause is hiding deeper, and it is triggered by the firefox stopping. It is all is not a guarantee that the root cause is bitflips, but you can rule it out by testing your memory.
    • asimovDev 7mo ago
      Surely hardware issues would manifest in other software or overall OS as well?
  • rebelwebmaster 7mo ago
• est 7mo ago
  so could software engineering sommehow catch those crashes?
• _0xdd 7mo ago
  So, why aren't we all using ECC in 2026?
  • lunar_rover 7mo ago
    Intel intentionally ripped ECC out of the sweet spot products to charge premium and unfortunately they succeeded. Pentium G4560 supports ECC, Core i7 10700 doesn't.
    • bpye 7mo ago
      They did improve this in more recent generations, but you need a W series chipset to use it.
  • haspok 7mo ago
    Because 99% of laptops don't have it, and can't be memory upgraded?
  • matja 7mo ago
    Doesn't sell as well as a beautiful citrus blush milled aluminium case.
• KenoFischer 7mo ago
  I'll submit my bit flip story for consideration also :) https://julialang.org/blog/2020/09/rr-memory-magic/ https://julialang.org/blog/2020/09/rr-memory-magic/
• soletta 7mo ago
  I’ve also found that compiling large packages in GCC or similar tends to surface problems with the system’s RAM. Which probably means most typical software is resilient to a bit-flip; makes you wonder how many typos in actual documents might have been caused by bad R@M.
  • sfink 7mo ago
    That's exactly how my bad RAM manifested itself. In fact, I was compiling Firefox, and gcc would get a segmentation fault at some random point during compilation. I'd have to clobber and restart the hour-long build. It was only when gcc started crashing while compiling other things that I even started considering the possibility of hardware failure. I'm a software developer, and based on what I produce myself, I just assume that all software is horribly buggy. ;-)
• Habgdnv 7mo ago
  I bought my PC like 2 weeks ago and ran my ram at 5800 to test its limits and forgot to lower it. After few strange crashes of my fedora desktop - super strange behavior, apps refuse start/stop, can't even escape to the console... I ran memtest today and it lit all red in the first 2 minutes! Then I log in to my stable desktop at 5200 MT and I see this in the front HN page! What are the chances?!!
• deleted 7mo ago
  [deleted]
• petterroea 7mo ago
  As someone who has a strong background from hobby projects with five-digit users before going into work, I think one of the most interesting differences I experienced was that the problems you see at scale simply don't exist on small scale projects. Bit flips/bad memory is one of them.
• SeanSullivan86 7mo ago
  Hmm, can someone educate me here? Why don't bit flips ever seem to impact the results of calculations in settings like big-data analytics on AWS? Is it a difference between server hardware managed by knowledgeable people and random hardware thrown together by home PC builders?
  • zadikian 7mo ago
    Servers and pro workstations normally have ECC RAM.
  • OkGoDoIt 7mo ago
    Presumably professional hardware uses ECC memory, which automatically corrects these kinds of errors.
  • huhhuh 7mo ago
    In Belgium elections, a party received 4096 unaccounted votes likely due to a bit flip: https://en.wikipedia.org/wiki/Electronic_voting_in_Belgium#Reported_problems https://en.wikipedia.org/wiki/Electronic_voting_in_Belgium#R....
  • matja 7mo ago
    You can only detect what you measure. Are these big-data analytics processes running multiple times to detect differences?
• d--b 7mo ago
  Does anyone know how they can detect hardware defects like this? This sounds like an incredibly hard problem. And I don’t see how they can do this without impacting performance significantly.
  • rockdoe 7mo ago
    If the crash is isolated (no other reports) and flipping one bit in the crashing pointer value would make the pointer valid, it's assumed to be a bitflip. This obviously will only catch a minor portion of bitflips, i.e. any image or video data with bitflips wouldn't crash. From what he's saying they run an actual memory test after a crash, too.
• jurakovic 7mo ago
  There is this app https://github.com/Smerity/bitflipped https://github.com/Smerity/bitflipped _Your computer is a cosmic ray detector. Literally._
• kleiba 7mo ago
  Firefox is about the only piece of software in my setup that occasionally crashes. I say "occasionally" for lack of a better word, it's not "all the time", but it is definitely more than I would want to. If that was caused by bad memory, I would expect other software to be similarly affected and hence crash with about comparable frequency. However, it looks like I'm falling more into the other 90% of cases (unsurprisingly) because I do not observe other software crashing as much as firefox does. Also, this whole crashing business is a fairly recent effect - I've been running firefox for forever and I cannot remember when it last was as much of an issue as it has become recently for me.
  • Agingcoder 7mo ago
    It depends on what you bitflip. I once had a bitflip pattern causing lowercase ascii to turn into uppercase ascii in a case insensitive system. Everything was fine until it tried to uppercase numbers and things went wrong The first time I had to deal with faulty ram ( more than 20y ago ), the bug would never trigger unless I used pretty much the whole dimm stick and put meaningful stuff in it etc in my case linking large executables , or untargzipping large source archives. Flipping a pixel had no impact though
  • lqet 7mo ago
    > Firefox is about the only piece of software in my setup that occasionally crashes. I would add Thunderbird to that list.
  • tuetuopay 7mo ago
    Just check your memory with memtest. Two years ago, I've had Factorio crash once on a null pointer exception. I reported the crash to the devs and, likely because the crash place had a null check, they told me my memory was bad. Same as you I said "wait no, no other software ever crashed weirdly on this machine!", but they were adamant. Lo and behold, I indeed had one of my four ram sticks with a few bad addresses. Not much, something like 10-15 addresses tops. You need bad luck to hit one of those addresses when the total memory is 64GB. It's likely the null pointer check got flipped. Browsers are good candidates to find bad memory: they eat a lot of ram, they scatter data around, they have a large chunk, and have JITs where a lot of machine code gets loaded left and right.
• pulkas 7mo ago
  what happens if bitflip occurs while you are detecting bitflip? bitflippin...
  • matja 7mo ago
    I have a machine with a 6 year uptime that was slowly accumulating single bit error corrections. The EDAC counter mysteriously stopped at 308 last year, and hasn't changed since, so I wonder if a bitflip in the counter circuit made it stop...
• INTPenis 7mo ago
  That's super interesting because I remember Linus Torvalds saying he requires ECC RAM in his computers, because he got tired of weird issues that were resolved by a reboot. But non-ECC is fine for most of us mortals gaming and streaming. I would expect pro gamers to opt for ECC though.
• moconnor 7mo ago
  Bit flips aren’t always bad hardware. I remember an anecdote from Sandia from my HPC days - they found they were getting more bit flips on some machines than others on their cluster and sometimes correlated. Turned out at their altitude cosmic rays were flipping bits in the top-most machines in the racks, sometimes then penetrating lower and flipping bits in more machines too.
• bob1029 7mo ago
  I've written genetic programming experiments that do not require an explicit mutation operator because the machine would tend to flip bits in the candidate genomes under the heavy system load. It took me a solid week to determine that I didn't actually have a bug in my code. It happens so fast on my machine (when it's properly loaded) that I can depend on it to some extent.
  • rcbdev 7mo ago
    Hyrum's law in action. https://xkcd.com/1172/ https://xkcd.com/1172/
• charcircuit 7mo ago
  When I had bad memory, Firefox was the only program which would crash because of it. I think there is also something to say about how Firefox's design could be improved to handle them better.
• bArray 7mo ago
  > In the last week we received ~470000 crash reports, these do not represent all crashes because it's an opt-in system, the real number of crashes will be several times larger. 470k crashes in a single week, and this is under-reported! I bet the number of crashes is far higher. My snap Firefox on Ubuntu would lock-up, forcing me to kill it from the system monitor, and this was never reported as a crash. Once upon a time I wrote software for safety critical systems in C/C++, where the code was deployed and expected to work for 10 years (or more) and interact with systems not built yet. Our system could lose power at any time (no battery) and we would have at best 1ms warning. Even if Firefox moves to Rust, it will not resolve these issues. 5% of their crashes could be coming from resource exhaustion, likely mostly RAM - why is this not being checked prior to allocation? 5% of their crashes could be resolved tomorrow if they just checked how much RAM was available prior to trying to allocate it. That accounts for ~23k crashes a week. Madness. With the RAM shortages and 8GB looking like it will remain the entry laptop norm, we need to start thinking more carefully about how software is developed.
• wosined 7mo ago
  The title should start with "Up to 10%"
• Neil44 7mo ago
  I guess the percentage of crashes due to hardware is high because people with faulty hardware are experiencing the vast majority of crashes. It sounds kind of dumb when put like that, I'm actually surprised it's that low a percentage.
  • danbruc 7mo ago
    I guess the percentage of crashes due to hardware is high because people with faulty hardware are experiencing the vast majority of crashes. It is not that simple, it does not only depend on the hardware but also the code. It is like a race, what happens first - you hit a bug in the code or your hardware glitches? If the code is bug free, then all crashes will be due to hardware issues, whether faulty hardware or stray particles from the sun. When the code is one giant bug and crashes immediately every time, then you will need really faulty hardware or have to place a uranium rod on top of your RAM and point a heat gun at your CPU to crash before you hit the first bug, i.e. almost all crashes will be due to bugs. So what you observe will depend on the prevalence of faulty hardware and how long it takes to hit an hardware issue vs how buggy the code is and how long it takes to hit a bug.
• fasteo 7mo ago
  >>> In the last week we received ~470000 crash reports, these do not represent all crashes because it's an opt-in system, the real number of crashes will be several times larger Having the number of unique machines would be great to see how skewed this estimate is.
  • sfink 7mo ago
    To be fully accurate, it would also require tracking unique machines when collecting crash reports.
• samus 7mo ago
  Maybe a partial solution would be to duplicate pointer data, compare pointers at every deference and panics if it doesn't match up. In essence a poor man's version of ECC. It's a considerable runtime overhead, but it might be possible to hide it behind a flag, only to be turned on to reproduce bugs. Also, anti-cheat measures already do something similar. Certain data is more sensitive as well and requires extra protection. Pointers and indexes obviously, which might send the whole application on a wild goose chase around memory. But also machine code, especially JIT-generated traces, is worth to be checksummed and verified before executing it.
• nickhodge 7mo ago
  Rust would fix this. Oh wait.
• bilekas 7mo ago
  Just out of interest is ECC memory supposed to me more resilient to these types of failure?
• wartywhoa23 7mo ago
  But muh memory-safe Rust!!! :'(
• lifeisstillgood 7mo ago
  I’m pretty sure Torvalds tells a story of spending days hunting down a compiler bug, only to find it was memory, and then simply never using anything other than EC memory again. 10+% is huge
  • seanalltogether 7mo ago
    He specifically mentions this story in the LTT video from a few months ago. https://youtu.be/mfv0V1SxbNA?si=hS4ZMRYqqLXMkxJW&t=526 https://youtu.be/mfv0V1SxbNA?si=hS4ZMRYqqLXMkxJW&t=526
• sinuhe69 7mo ago
  Oh, on my old PC, FF sometimes mysteriously crashed for apparently no reason. I sent bug reports and cleared the profile and it seemed to help for a while, then it crashed again. Much later, I suspected and tested the RAM and turned out, it had a faulty module!
• titzer 7mo ago
  I had a refurbished ThinkPad that had memory corruption. I only noticed because Firefox started to crash an unreasonable amount. Ran memcheck through BIOS and sure enough it was bad RAM. Have we considered that maybe Firefox is the cause of bad memory? /s
  • sfink 7mo ago
    It is. If a tree falls in the forest with nobody around to hear it, does it make a sound? If a computer flips bits while it's not doing anything with that memory, does it have bad RAM? A fair number of people pretty much only use their computers as web browsers. QED
• bergheim 7mo ago
  Strange. I have a tab hoarding problem, I often have over 1000 tabs open [1][2], and I cannot remember the last time Firefox crashed. I'm thinking it must have been years? I use ublock origin though, which might help since ads do their best to steal your computer and soul through any means possible of course. I also use a bunch of other extensions though, dark reader, vimium, sideberry... I'd expect me to be a bit more exposed than the average user. Yet it's just rock stable for me. Maybe it just works better on linux? 1: I know this because I installed https://addons.mozilla.org/en-US/firefox/addon/tab-counter-plus/ https://addons.mozilla.org/en-US/firefox/addon/tab-counter-p... to check :) 2: However after finding Karakeep I don't actually have 1000 tabs anymore!
  • andoando 7mo ago
    I dont get the people with 10+ tabs open drives me crazy, how do you even know whats what? Just bookmark shit you want to keep!
• ryukoposting 7mo ago
  It's worth noting that the thread says "up to 10%," not "10%" as the title suggests. So it's reasonable to believe the rate is as low as 5% based on the only real figure given (25000 / 470000) I think our education system should include a unit on "marketing bullshit" sometime early in elementary school. Maybe as part of math class, after they learn inequalities. "Ok kids, remind me, what does 'up to' mean?" "less than or equal to!"
• strongpigeon 7mo ago
  I might be too late to this thread to get an answer but I do wonder how much of those bitflips are due to rowhammer-style attacks. Firefox runs trillions of lines of untrusted code a day with a non-insignificant part that is of malicious intent. I wouldn’t be shocked if some of those “analog” crashes are due to that.
• m3047 7mo ago
  Stucke's talk about DNS being hazardous to your health is one of my all time favorites: https://www.youtube.com/watch?v=4PSc9BJDWhM https://www.youtube.com/watch?v=4PSc9BJDWhM
• fastaguy88 7mo ago
  It is perhaps worth noting that the 25,000 bit flips/out of 470,000 crashes (in a week) are probably not coming from all Firefox users. It would be useful to know how many of those crashes (and bit flips) are happening on the same machine. And whether the crashes/bit flips continue on the same machine continue from week to week. I can certainly imagine that a very small fraction of Firefox users are generating these results, so that bit flips are not a problem generally.
• Grisu_FTP 7mo ago
  IIRC Linus Torvalds said in the Linus Tech Tips video he was in that he thinks many of the bluescreens Windows gets a bad rep for are actually bitflips that happen due to most desktops not using ECC-Memory. Since i have seen this video this question has been in my mind from time to time.