7 ms·

If you don't believe in bit flips, try this! http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html I did that for a bit on cloudfront.ne

If you don't believe in bit flips, try this! http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html I did that for a bit on cloudfront.net and got dozens of them in a short amount of time.

amichal 8y ago
bitflips are also single char typos in ascii. I'm guessing that is what you saw
- Kenji 8y ago
  Read the article before commenting, please. He addressed that.
- detaro 8y ago
  I think the article gives good reasons to believe they are not typos, why do you think otherwise?
  - amichal 8y ago
    While i like the sibling comment's network errors answer better then mine I didn't see any reasoning other then a unfounded statement[1] in the article or on a quick skim for "typo" in the referenced whitepaper: [1] "These requests were not typos or other manually entered URLs"
    - bgirard 8y ago
      Here's a better quote: "There are many frequently resolved domains that few people type or know." There's a lot more details in the article.
    - eli 8y ago
      "All of these requests used only four domains in the HTTP Host header, as shown in Table 4. Three out of four domains contain more than one bit error, ruling out a simple mistype of fbbdn.net for fbcdn.net. "
      - seandougall 8y ago
        Not to mention, the number of people typing "fbcdn.net" into a browser has to be vanishingly small to begin with.
- nestorD 8y ago
  An interesting control would have been to flip a single bit but also flip a single char so that it flips several bits and compare the results. edit: Reading the linked article, they describe one of their validation method: > The Host header contains the domain the HTTP client resolved to connect to the bitsquat server. If the Host header matches the original domain, the corruption occurred on the red path (DNS path). If the Host header matches a bitsquat domain, the corruption occurred on the blue path (content path).
- deleted 8y ago
  [deleted]
- asveikau 8y ago
  Is that why some Microsoft download URL with a GUID in it made it to his server? Typo in Windows Update? I don't think that sounds right.
- 3pt14159 8y ago
  This is easily testable because: 1. Not all single char typos are bitflips and when you test it statistically you get more traffic for bitflips than typos. It's a bit more complicated, some typos are really common because they're close to the key. 2. You can look at the raw traffic that comes in and tell what is a simple get request with a fat finger (GET https://en.7ikipedia.org/ https://en.7ikipedia.org/ vs GET https://en.wikipedia.org/w/index.php?search=america&title=Special%3ASearch&go=Go https://en.wikipedia.org/w/index.php?search=america&title=Sp...). Source: I have some domains that are bitsquats of some high traffic domains. I get access token headers when I do a raw dump of the traffic. (I turned off the servers after satisfying my curiosity.) Want to stop randos from getting some tiny portion of your traffic's access tokens? Use client side keys and send signed / encrypted access tokens or even full requests. Yeah I know. It's so far down the list it doesn't matter. But every combination of github.com, microsoft.com, etc is bitsquatted, so this attack can be considered ongoing. This is doubly true if you're on a domain like .co.uk which is a stupid subdomain of .uk and now that .uk addresses are available someone can buy ko.uk and bitsquat the entire fucking country.
- spullara 8y ago
  None of them were typed in. I was bitsquatting a CDN that no one would type into the browser bar. Everything was some resource referenced from a page. If you checked the original page, the correct URL was present in that page.
tjoff 8y ago
Bit flips in network surely is many orders of magnitude worse than in cache/memory bit flips? And I doubt it has to do with gamma rays. Those with poor or long distance DSL do see a big increase in errors. And while hopefully getting rarer now home routers could easily overheat and introduce errors as soon as they saw more than usual traffic. We recently had issues with the power supply for our router and it also caused issued.
- mattnewton 8y ago
  The end of gp’s Article points to some work by a verisign engineer that put the network probability very low. So my interpretation was these but flips were happening on device before the network stack was dispatched at a url.
- groestl 8y ago
  > Bit flips in network surely is many orders of magnitude worse Yes, but because it's so prevalent that people expect it, they've added checksums on multiple levels, so the network actually performs better. In this particular instance (DNS queries), it's very unlikely that the data was corrupted in transit: "We believe that UDP checksums are effective at preventing 'bitsquat' attacks and other types of errors that occur after a DNS query leaves a DNS resolver and enters the network."
  - tjoff 8y ago
    That doesn't take into consideration of flaky home-routers (note that they do recalculate the checksum). So the checksum might be processed on bad data. For a bad connection it will also be quite common that the checksum will happen to be valid. TCP/UDP checksums are quite bad.
    - deleted 8y ago
      [deleted]
  - dfox 8y ago
    One thing to keep in mind is that the network protocols usually use quite weak and similar checksums. To the extent that there are layer combinations that reliably produce errors that are not detectable by TCP checksum.
andikleen3 8y ago
I maintain a tool that reports errors on Linux systems with ECC memory, and also a website describing the tool (http://mcelog.org http://mcelog.org). I wrote about patterns in the access logs correlated to time some time ago in my blog: http://halobates.de/blog/p/181 http://halobates.de/blog/p/181
- sannee 8y ago
  > I wonder if it’s possible to detect solar flares in these logs. Need to find a good data source for them. Are there any other events generating lots of radiation that may affect servers? Cosmic ray showers [1] potentially. My prediction is that there should be "clusters" of bit errors every few minutes or so (given sufficient amount of servers...). I was talking to a sysadmin at a local company running a few thousand servers about getting their ECC error logs in order to look for these, but scraping them apparently wasn't trivially managable for them. [1] - https://en.wikipedia.org/wiki/Air_shower_(physics) https://en.wikipedia.org/wiki/Air_shower_(physics)
sevensor 8y ago
I used to work in DRAM manufacturing. I believe 100% in bitflips -- we saw them in testing all the time. I also believe the rate at which bitflips are caused by gamma rays is absolutely dwarfed by the rate at which bitflips are caused by manufacturing defects. It's really easy to manufacture a weak cell when you're making billions of tiny capacitors at the same time. SRAM might also see lots of bitflips, but that seems even more likely than DRAM to be driven by manufacturing defects, given the greater complexity of the SRAM cell.
- trhway 8y ago
  >the rate at which bitflips are caused by gamma rays is absolutely dwarfed by the rate at which bitflips are caused by manufacturing defects. back in USSR/Russia gamma rays and aliens were not-an-issue compare to the extremely low reliability of USSR/Russia hardware. The military hardware back then (and some telco hardware built in Russia in 199x (some even for export into Western countries!)) was built as triplicate systems - i.e. primitive quorum/consensus computing. Similarly, as far as i heard back then, due to low reliability of Itanium especially in the beginning, the Itanium based Tandems were also available as "Tridems".
  - toomanybeersies 8y ago
    Except for the case in the USSR when gamma rays were an issue at a train depot computer system due to radioactive cows from near Chernobyl: http://www.jakepoz.com/debugging-behind-the-iron-curtain/ http://www.jakepoz.com/debugging-behind-the-iron-curtain/
richardlblair 8y ago
I was talking to a guy at Github who had to debug an enterprise deployment that just stopped working one day. There was a bit that flipped and apparently took the whole thing to its knees. This is the stuff nightmares are made of.
Faaak 8y ago
Did also that with facebook.fr/google.fr/akamai.net domains. Got a _lot_ of hits, with mostly mobile user agents. Was fun; never published my results though..
- amenghra 8y ago
  Same here. If you want a lot of hits, pick their cdn domains. Get a tls cert and you might even be able to serve the original file with some additional js of your own :)
theclaw 8y ago
The youtube video[1] of his defcon talk is super interesting -although some of his concerns about serving malicious scripts from bitsquatted domains are thankfully mitigated by widespread adoption of TLS. [1] http://www.youtube.com/watch?v=lZ8s1JwtNas http://www.youtube.com/watch?v=lZ8s1JwtNas
Xcelerate 8y ago
From your link, > Bitsquat traffic represents a slice of normal traffic Wow. That's pretty amazing. With enough analysis, you could (possibly) recover the traffic pattern for major websites. That alone seems to have a lot of interesting implications.

amichal 8y ago

bitflips are also single char typos in ascii. I'm guessing that is what you saw

Kenji 8y ago
Read the article before commenting, please. He addressed that.
detaro 8y ago
I think the article gives good reasons to believe they are not typos, why do you think otherwise?
- amichal 8y ago
  While i like the sibling comment's network errors answer better then mine I didn't see any reasoning other then a unfounded statement[1] in the article or on a quick skim for "typo" in the referenced whitepaper: [1] "These requests were not typos or other manually entered URLs"
  - bgirard 8y ago
    Here's a better quote: "There are many frequently resolved domains that few people type or know." There's a lot more details in the article.
  - eli 8y ago
    "All of these requests used only four domains in the HTTP Host header, as shown in Table 4. Three out of four domains contain more than one bit error, ruling out a simple mistype of fbbdn.net for fbcdn.net. "
    - seandougall 8y ago
      Not to mention, the number of people typing "fbcdn.net" into a browser has to be vanishingly small to begin with.
nestorD 8y ago
An interesting control would have been to flip a single bit but also flip a single char so that it flips several bits and compare the results. edit: Reading the linked article, they describe one of their validation method: > The Host header contains the domain the HTTP client resolved to connect to the bitsquat server. If the Host header matches the original domain, the corruption occurred on the red path (DNS path). If the Host header matches a bitsquat domain, the corruption occurred on the blue path (content path).
deleted 8y ago
[deleted]
asveikau 8y ago
Is that why some Microsoft download URL with a GUID in it made it to his server? Typo in Windows Update? I don't think that sounds right.
3pt14159 8y ago
This is easily testable because: 1. Not all single char typos are bitflips and when you test it statistically you get more traffic for bitflips than typos. It's a bit more complicated, some typos are really common because they're close to the key. 2. You can look at the raw traffic that comes in and tell what is a simple get request with a fat finger (GET https://en.7ikipedia.org/ https://en.7ikipedia.org/ vs GET https://en.wikipedia.org/w/index.php?search=america&title=Special%3ASearch&go=Go https://en.wikipedia.org/w/index.php?search=america&title=Sp...). Source: I have some domains that are bitsquats of some high traffic domains. I get access token headers when I do a raw dump of the traffic. (I turned off the servers after satisfying my curiosity.) Want to stop randos from getting some tiny portion of your traffic's access tokens? Use client side keys and send signed / encrypted access tokens or even full requests. Yeah I know. It's so far down the list it doesn't matter. But every combination of github.com, microsoft.com, etc is bitsquatted, so this attack can be considered ongoing. This is doubly true if you're on a domain like .co.uk which is a stupid subdomain of .uk and now that .uk addresses are available someone can buy ko.uk and bitsquat the entire fucking country.
spullara 8y ago
None of them were typed in. I was bitsquatting a CDN that no one would type into the browser bar. Everything was some resource referenced from a page. If you checked the original page, the correct URL was present in that page.

tjoff 8y ago

Bit flips in network surely is many orders of magnitude worse than in cache/memory bit flips? And I doubt it has to do with gamma rays. Those with poor or long distance DSL do see a big increase in errors. And while hopefully getting rarer now home routers could easily overheat and introduce errors as soon as they saw more than usual traffic. We recently had issues with the power supply for our router and it also caused issued.

mattnewton 8y ago
The end of gp’s Article points to some work by a verisign engineer that put the network probability very low. So my interpretation was these but flips were happening on device before the network stack was dispatched at a url.
groestl 8y ago
> Bit flips in network surely is many orders of magnitude worse Yes, but because it's so prevalent that people expect it, they've added checksums on multiple levels, so the network actually performs better. In this particular instance (DNS queries), it's very unlikely that the data was corrupted in transit: "We believe that UDP checksums are effective at preventing 'bitsquat' attacks and other types of errors that occur after a DNS query leaves a DNS resolver and enters the network."
- tjoff 8y ago
  That doesn't take into consideration of flaky home-routers (note that they do recalculate the checksum). So the checksum might be processed on bad data. For a bad connection it will also be quite common that the checksum will happen to be valid. TCP/UDP checksums are quite bad.
  - deleted 8y ago
    [deleted]
- dfox 8y ago
  One thing to keep in mind is that the network protocols usually use quite weak and similar checksums. To the extent that there are layer combinations that reliably produce errors that are not detectable by TCP checksum.

andikleen3 8y ago

I maintain a tool that reports errors on Linux systems with ECC memory, and also a website describing the tool (http://mcelog.org http://mcelog.org). I wrote about patterns in the access logs correlated to time some time ago in my blog: http://halobates.de/blog/p/181 http://halobates.de/blog/p/181

sannee 8y ago
> I wonder if it’s possible to detect solar flares in these logs. Need to find a good data source for them. Are there any other events generating lots of radiation that may affect servers? Cosmic ray showers [1] potentially. My prediction is that there should be "clusters" of bit errors every few minutes or so (given sufficient amount of servers...). I was talking to a sysadmin at a local company running a few thousand servers about getting their ECC error logs in order to look for these, but scraping them apparently wasn't trivially managable for them. [1] - https://en.wikipedia.org/wiki/Air_shower_(physics) https://en.wikipedia.org/wiki/Air_shower_(physics)

sevensor 8y ago

I used to work in DRAM manufacturing. I believe 100% in bitflips -- we saw them in testing all the time. I also believe the rate at which bitflips are caused by gamma rays is absolutely dwarfed by the rate at which bitflips are caused by manufacturing defects. It's really easy to manufacture a weak cell when you're making billions of tiny capacitors at the same time. SRAM might also see lots of bitflips, but that seems even more likely than DRAM to be driven by manufacturing defects, given the greater complexity of the SRAM cell.

trhway 8y ago
>the rate at which bitflips are caused by gamma rays is absolutely dwarfed by the rate at which bitflips are caused by manufacturing defects. back in USSR/Russia gamma rays and aliens were not-an-issue compare to the extremely low reliability of USSR/Russia hardware. The military hardware back then (and some telco hardware built in Russia in 199x (some even for export into Western countries!)) was built as triplicate systems - i.e. primitive quorum/consensus computing. Similarly, as far as i heard back then, due to low reliability of Itanium especially in the beginning, the Itanium based Tandems were also available as "Tridems".
- toomanybeersies 8y ago
  Except for the case in the USSR when gamma rays were an issue at a train depot computer system due to radioactive cows from near Chernobyl: http://www.jakepoz.com/debugging-behind-the-iron-curtain/ http://www.jakepoz.com/debugging-behind-the-iron-curtain/

richardlblair 8y ago

I was talking to a guy at Github who had to debug an enterprise deployment that just stopped working one day. There was a bit that flipped and apparently took the whole thing to its knees. This is the stuff nightmares are made of.

Faaak 8y ago

Did also that with facebook.fr/google.fr/akamai.net domains. Got a _lot_ of hits, with mostly mobile user agents. Was fun; never published my results though..

amenghra 8y ago
Same here. If you want a lot of hits, pick their cdn domains. Get a tls cert and you might even be able to serve the original file with some additional js of your own :)

theclaw 8y ago

The youtube video[1] of his defcon talk is super interesting -although some of his concerns about serving malicious scripts from bitsquatted domains are thankfully mitigated by widespread adoption of TLS. [1] http://www.youtube.com/watch?v=lZ8s1JwtNas http://www.youtube.com/watch?v=lZ8s1JwtNas

Xcelerate 8y ago

From your link, > Bitsquat traffic represents a slice of normal traffic Wow. That's pretty amazing. With enough analysis, you could (possibly) recover the traffic pattern for major websites. That alone seems to have a lot of interesting implications.