4 ms·

Bit flips are totally real, at scale you will definitely see them on large queries. There was a fun talk at DEFCON on bitsquatting, the process of buying 1 bit

Bit flips are totally real, at scale you will definitely see them on large queries. There was a fun talk at DEFCON on bitsquatting, the process of buying 1 bit off domain names and then accepting all incoming connections. Attacks like rowhammer similarly abuse erroneous bit flips. Supposedly microsoft can detect solar activity based on the number of windows crash logs they receive. DEFCON Talk: https://www.youtube.com/watch?v=aT7mnSstKGs https://www.youtube.com/watch?v=aT7mnSstKGs https://en.wikipedia.org/wiki/Bitsquatting https://en.wikipedia.org/wiki/Bitsquatting https://en.wikipedia.org/wiki/Row_hammer https://en.wikipedia.org/wiki/Row_hammer

BearOso 4y ago
I remember reading somewhere about that talk being debunked. Maybe someone more resourceful than me can find it. It was something about being more likely to be a human typo or a config change that rolled out to a bunch of machines. The statistics didn't add up, and it wasn't plausible that bit flips caused it.
- woodruffw 4y ago
  I don't know of a debunking, but more numbers can be found in the associated blog post[1]. [1]: http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html
- silisili 4y ago
  Unsure about the talk, but I read a research report from someone at I think it was Cisco who purchased a second level tld of a bitsquatted US state( think statenXX.us instead of state.XX.us ). The amount of email they received was staggering, and I'd have trouble believing that many people would make that mistake in typing.
  - chinaman425 4y ago
    [dead]
  - runnerup 4y ago
    that's more "typo-squatting" than "bit squatting" but its definitely the same broad concept.
- hayst4ck 4y ago
  There are many places that memory exists. Your processor cache is memory. Registers are memor-ish. Hard Drives have memory. If a bit flip happens in your hard drives memory before being written to disk, then it's not unreasonable to think that the bitflip would persist, even through reboots. I have run queries at large companies and found mistakes most easily explained as bitflips in domain names written to disk. Imagine an environmental variable configuring the use of a proxy without proper whitelists and it's not unimaginable to me that a production machine would be able to speak to machines on the internet at large. I am open to the idea that what I think is happening might not be the mechanics of what is happening, but I find the talk believable, not based on theory, but actually seeing persisted (and non-persisted) bit flips in domain names queried from data warehoused logs at world scale companies.
  - BearOso 4y ago
    That was the gist of the response, as far as I remember. It was repeated access from the same set of machines, so it was more likely that one bit flip persisted in a config and was subsequently rolled out to others. So the study grossly overestimated the number of actual bit flips occurring by like 20000x or something.
noman-land 4y ago
One of my favorite DEFCON talks of all time.
rozab 4y ago
There was an interesting post from Mozilla about this. There's a certain class of telemetry errors which seem to only be caused by cosmic rays https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-what-flips-your-bit/ https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-...
1letterunixname 4y ago
"Supposedly" is false. The sun doesn't produce cosmic ray-power particles and soft errors aren't affected by sunspot activity. It is affected by altitude and space weather, but not solar activity. https://en.wikipedia.org/wiki/Soft_error https://en.wikipedia.org/wiki/Soft_error I was in the audience of the talk. All devices should be required to use ECC because it's a security risk. Not as much as http:// http:// era, but silent corruption across networks and systems is a thing.
- colechristensen 4y ago
  > space weather, but not solar activity Space weather is solar activity.
  - dtgriscom 4y ago
    Well, mostly...
  - 1letterunixname 4y ago
    Not entirely. Extrasolar activity is the most likely cause for energetic cosmic rays at the Earth's surface.
- davidcuddeback 4y ago
  The link you provided contradicts your comment. From the link in your comment: "The average rate of cosmic-ray soft errors is inversely proportional to sunspot activity. That is, the average number of cosmic-ray soft errors decreases during the active portion of the sunspot cycle and increases during the quiet portion."

BearOso 4y ago

I remember reading somewhere about that talk being debunked. Maybe someone more resourceful than me can find it. It was something about being more likely to be a human typo or a config change that rolled out to a bunch of machines. The statistics didn't add up, and it wasn't plausible that bit flips caused it.

woodruffw 4y ago
I don't know of a debunking, but more numbers can be found in the associated blog post[1]. [1]: http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html
silisili 4y ago
Unsure about the talk, but I read a research report from someone at I think it was Cisco who purchased a second level tld of a bitsquatted US state( think statenXX.us instead of state.XX.us ). The amount of email they received was staggering, and I'd have trouble believing that many people would make that mistake in typing.
- chinaman425 4y ago
  [dead]
- runnerup 4y ago
  that's more "typo-squatting" than "bit squatting" but its definitely the same broad concept.
hayst4ck 4y ago
There are many places that memory exists. Your processor cache is memory. Registers are memor-ish. Hard Drives have memory. If a bit flip happens in your hard drives memory before being written to disk, then it's not unreasonable to think that the bitflip would persist, even through reboots. I have run queries at large companies and found mistakes most easily explained as bitflips in domain names written to disk. Imagine an environmental variable configuring the use of a proxy without proper whitelists and it's not unimaginable to me that a production machine would be able to speak to machines on the internet at large. I am open to the idea that what I think is happening might not be the mechanics of what is happening, but I find the talk believable, not based on theory, but actually seeing persisted (and non-persisted) bit flips in domain names queried from data warehoused logs at world scale companies.
- BearOso 4y ago
  That was the gist of the response, as far as I remember. It was repeated access from the same set of machines, so it was more likely that one bit flip persisted in a config and was subsequently rolled out to others. So the study grossly overestimated the number of actual bit flips occurring by like 20000x or something.

noman-land 4y ago

One of my favorite DEFCON talks of all time.

rozab 4y ago

There was an interesting post from Mozilla about this. There's a certain class of telemetry errors which seem to only be caused by cosmic rays https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-what-flips-your-bit/ https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-...

1letterunixname 4y ago

"Supposedly" is false. The sun doesn't produce cosmic ray-power particles and soft errors aren't affected by sunspot activity. It is affected by altitude and space weather, but not solar activity. https://en.wikipedia.org/wiki/Soft_error https://en.wikipedia.org/wiki/Soft_error I was in the audience of the talk. All devices should be required to use ECC because it's a security risk. Not as much as http:// http:// era, but silent corruption across networks and systems is a thing.

colechristensen 4y ago
> space weather, but not solar activity Space weather is solar activity.
- dtgriscom 4y ago
  Well, mostly...
- 1letterunixname 4y ago
  Not entirely. Extrasolar activity is the most likely cause for energetic cosmic rays at the Earth's surface.
davidcuddeback 4y ago
The link you provided contradicts your comment. From the link in your comment: "The average rate of cosmic-ray soft errors is inversely proportional to sunspot activity. That is, the average number of cosmic-ray soft errors decreases during the active portion of the sunspot cycle and increases during the quiet portion."