3 ms·

A 5 part thread where they say they're "now 100% positive" the crashes are from bitflips, yet not a single word is spent on how they're supposedly detecting bit

A 5 part thread where they say they're "now 100% positive" the crashes are from bitflips, yet not a single word is spent on how they're supposedly detecting bitflips other than just "we analyze memory"?

tredre3 7mo ago
> last year we deployed an actual memory tester that runs on user machines after the browser crashes. He doesn't explain anything indeed but presumably that code is available somewhere.
- thatguy27 7mo ago
  [flagged]
- hedora 7mo ago
  That, and 50% of the machines where their heuristics say it is a hardware error fail basic memory tests. I've seen a lot of confirmed bitflips with ECC systems. The vast majority of machines that are impacted are impacted by single event upsets (not reproducible). (I worded that precisely but strangely because if one machine has a reproducible problem, it might hit it a billion times a second. That means you can't count by "number of corruptions".) My take is that their 10% estimate is a lower bound.
rincebrain 7mo ago
The simplest way to do this, what I believe memtest86 and friends do, is to write a fixed pattern over a region of memory and then read it back later and see if it changed; then you write patterns that require flipping the bits that you wrote before, and so on. Things like [1] will also tell you that something corrupted your memory, and if you see a nontrivial (e.g. lots of bits high and low) magic number that has only a single bit wrong, it's probably not a random overwrite - see the examples in [2]. There's also a fun prior example of experiments in this at [3], when someone camped on single-bit differences of a bunch of popular domains and examined how often people hit them. edit: Finally, digging through the Mozilla source, I would imagine [4] is what they're using as a tester when it crashes. [1] - https://github.com/mozilla-firefox/firefox/commit/917c4a6bfa66853cbf0df08973d48ea7cba6547f https://github.com/mozilla-firefox/firefox/commit/917c4a6bfa... [2] - https://bugzilla.mozilla.org/show_bug.cgi?id=1762568 https://bugzilla.mozilla.org/show_bug.cgi?id=1762568 [3] - https://media.defcon.org/DEF%20CON%2019/DEF%20CON%2019%20presentations/DEF%20CON%2019%20-%20Dinaburg-Bit-Squatting.pdf https://media.defcon.org/DEF%20CON%2019/DEF%20CON%2019%20pre... [4] - https://github.com/mozilla-firefox/firefox/blob/main/toolkit/crashreporter/client/app/src/memory_test.rs https://github.com/mozilla-firefox/firefox/blob/main/toolkit...
- rendaw 7mo ago
  That would tell you if there's a bitflip in your test, but not if there's a bitflip in normal program code causing a crash, no? IIUC GP's questions was how do they actually tell after a crash that that crash was caused by a bitflip.
  - rincebrain 7mo ago
    The example I gave in there is of adding sentinel values in your data, so you can check the constants in your data structures later and go "oh, this is overwritten with garbage" versus "oh, this is one or two bits off". I would imagine plumbing things like that through most common structures is what was done there, though I haven't done the archaeology to find out, because Firefox is an enormous codebase to try and find one person's commits from several years ago in.
    - patrulek 7mo ago
      But it would be also possible that sentinel value used for comparison changed because of bitflip, not data structure used by program.
      - rincebrain 7mo ago
        Detecting that is the point, yes.
    - kevincox 7mo ago
      This doesn't always protect against out-of-bounds writes. Although if these sentinel values are in read only memory mappings it probably gets pretty close. (Especially if you consider kernel memory corruption a "bitflip".)
      - rincebrain 7mo ago
        Well, yes, the author described it as "a heuristic", which checking sentinel values would seem to match.
hexyl_C_gut 7mo ago
It sounds like they don't know that the crashes are from bitflips but those crashes are from people with flaky memory which probably caused the crash?
wmf 7mo ago
A common case is a pointer that points to unallocated address space triggers a segfault and when you look at the pointer you can see that it's valid except for one bit.
- dboreham 7mo ago
  That tells you one bit was changed. It doesn't prove that single bit changed due to a hardware failure. It could have been changed by broken software.
  - LeifCarrotson 7mo ago
    Broken software causes null pointer references and similar logic errors. It would be extremely unusual to have an inadvertent ptr ^= (1 << rand_between(0,64)); that got inserted in the code by accident. That's just not the way that we write software.
    - vlovich123 7mo ago
      Except no one is claiming the bit flip is the pointer vs the data being pointed to or a non pointer value. Given how we write software there’s a lot more bits not in pointer values that still end up “contributing “ to a pointer value. Eg some offset field that’s added to a pointer has a bit flip, the resulting pointer also has a bit flip. But the offset field could have accidentally had a mask applied or a bit set accidentally due to the closeness of & and && or | and ||.
      - rockdoe 7mo ago
        I think that if you hit the crash in the same line of code many times, you can safely assume it's your own bug and not a memory issue. If it's only hit once by a random person, memory starts being more likely. (Unless that LOC is scanning memory or smth)
        vlovich123 7mo ago
hrmtst93837 7mo ago
[flagged]
- gcp 7mo ago
  I don't think Firefox has the access permissions needed to read MCE status, and the vast majority of our users don't have ECC, let alone they're going to run memtest86(+) after a Firefox crash. If they did, we wouldn't be having this discussion to begin with!

tredre3 7mo ago

> last year we deployed an actual memory tester that runs on user machines after the browser crashes. He doesn't explain anything indeed but presumably that code is available somewhere.

thatguy27 7mo ago
[flagged]
hedora 7mo ago
That, and 50% of the machines where their heuristics say it is a hardware error fail basic memory tests. I've seen a lot of confirmed bitflips with ECC systems. The vast majority of machines that are impacted are impacted by single event upsets (not reproducible). (I worded that precisely but strangely because if one machine has a reproducible problem, it might hit it a billion times a second. That means you can't count by "number of corruptions".) My take is that their 10% estimate is a lower bound.

rincebrain 7mo ago

The simplest way to do this, what I believe memtest86 and friends do, is to write a fixed pattern over a region of memory and then read it back later and see if it changed; then you write patterns that require flipping the bits that you wrote before, and so on. Things like [1] will also tell you that something corrupted your memory, and if you see a nontrivial (e.g. lots of bits high and low) magic number that has only a single bit wrong, it's probably not a random overwrite - see the examples in [2]. There's also a fun prior example of experiments in this at [3], when someone camped on single-bit differences of a bunch of popular domains and examined how often people hit them. edit: Finally, digging through the Mozilla source, I would imagine [4] is what they're using as a tester when it crashes. [1] - https://github.com/mozilla-firefox/firefox/commit/917c4a6bfa66853cbf0df08973d48ea7cba6547f https://github.com/mozilla-firefox/firefox/commit/917c4a6bfa... [2] - https://bugzilla.mozilla.org/show_bug.cgi?id=1762568 https://bugzilla.mozilla.org/show_bug.cgi?id=1762568 [3] - https://media.defcon.org/DEF%20CON%2019/DEF%20CON%2019%20presentations/DEF%20CON%2019%20-%20Dinaburg-Bit-Squatting.pdf https://media.defcon.org/DEF%20CON%2019/DEF%20CON%2019%20pre... [4] - https://github.com/mozilla-firefox/firefox/blob/main/toolkit/crashreporter/client/app/src/memory_test.rs https://github.com/mozilla-firefox/firefox/blob/main/toolkit...

rendaw 7mo ago
That would tell you if there's a bitflip in your test, but not if there's a bitflip in normal program code causing a crash, no? IIUC GP's questions was how do they actually tell after a crash that that crash was caused by a bitflip.
- rincebrain 7mo ago
  The example I gave in there is of adding sentinel values in your data, so you can check the constants in your data structures later and go "oh, this is overwritten with garbage" versus "oh, this is one or two bits off". I would imagine plumbing things like that through most common structures is what was done there, though I haven't done the archaeology to find out, because Firefox is an enormous codebase to try and find one person's commits from several years ago in.
  - patrulek 7mo ago
    But it would be also possible that sentinel value used for comparison changed because of bitflip, not data structure used by program.
    - rincebrain 7mo ago
      Detecting that is the point, yes.
  - kevincox 7mo ago
    This doesn't always protect against out-of-bounds writes. Although if these sentinel values are in read only memory mappings it probably gets pretty close. (Especially if you consider kernel memory corruption a "bitflip".)
    - rincebrain 7mo ago
      Well, yes, the author described it as "a heuristic", which checking sentinel values would seem to match.

hexyl_C_gut 7mo ago

It sounds like they don't know that the crashes are from bitflips but those crashes are from people with flaky memory which probably caused the crash?

wmf 7mo ago

A common case is a pointer that points to unallocated address space triggers a segfault and when you look at the pointer you can see that it's valid except for one bit.

dboreham 7mo ago
That tells you one bit was changed. It doesn't prove that single bit changed due to a hardware failure. It could have been changed by broken software.
- LeifCarrotson 7mo ago
  Broken software causes null pointer references and similar logic errors. It would be extremely unusual to have an inadvertent ptr ^= (1 << rand_between(0,64)); that got inserted in the code by accident. That's just not the way that we write software.
  - vlovich123 7mo ago
    Except no one is claiming the bit flip is the pointer vs the data being pointed to or a non pointer value. Given how we write software there’s a lot more bits not in pointer values that still end up “contributing “ to a pointer value. Eg some offset field that’s added to a pointer has a bit flip, the resulting pointer also has a bit flip. But the offset field could have accidentally had a mask applied or a bit set accidentally due to the closeness of & and && or | and ||.
    - rockdoe 7mo ago
      I think that if you hit the crash in the same line of code many times, you can safely assume it's your own bug and not a memory issue. If it's only hit once by a random person, memory starts being more likely. (Unless that LOC is scanning memory or smth)
      - vlovich123 7mo ago

hrmtst93837 7mo ago

[flagged]

gcp 7mo ago
I don't think Firefox has the access permissions needed to read MCE status, and the vast majority of our users don't have ECC, let alone they're going to run memtest86(+) after a Firefox crash. If they did, we wouldn't be having this discussion to begin with!