7 ms·
Bit flips are scary even on Earth. At a previous job, we had a customer who suddenly couldn’t send us email anymore. When their IT sent us the server logs to “
by codeflo 4y ago
Bit flips are scary even on Earth.
At a previous job, we had a customer who suddenly couldn’t send us email anymore. When their IT sent us the server logs to “prove” it’s our fault, we saw that the one letter in the cached MX record was wrong. This was puzzling, until I looked at the ASCII table to verify that the difference was exactly one bit.
We never found out where in the name resolution process the bit got flipped. The problem healed itself a few days later when the DNS cache expired, so it wasn’t worth further investigation.
That really gave me pause how often random bits are wrong in other data.
- captn3m0 4y agoHappens even on end user browsers resulting in bit-flipped domains being looked up: https://securitee.org/files/bitsquatting_www2013.pdf https://securitee.org/files/bitsquatting_www2013.pdf
- lostlogin 4y agoBitsquatting is a great name. I’m not 100% that this isn’t just typosquatting though.
- LanternLight83 4y agoIt might be subset of typosquatting, but it's distinct from what the term usual refers to because the typos that are statistically likely to result from user input (swapping neighboring keys, duplicating or omitting letters, etc) occur manually on the level of keys and characters, where as from those that result from bitflips ("adjacent" bit sequences) occur invisibly / without a manual mistake on the level of bit-/byte-streams, forming two mostly exclusive sets of domains that could be targeted. A determined attacker would likely target both.
- duskwuff 4y agoIt isn't simple typosquatting because it can involve errors that aren't plausible typos (like "mic2osoft.com" for "microsoft.com"), or domains that a user would never enter manually (like "fbcdn.com" or "ytimg.com", which are used for content delivery at Facebook and YouTube, respectively).
- daniel-cussen 4y agoI have super bivalent opinions about Intel. This is the opposite of ambivalent, it means heavily charged in both directions, but cancellation is not allowed. So that's why they should have let all their chips do ECC instead of making it a premium feature, it would have been better for their brand as "Chipzilla" and had no real cost. And it's dangerous! In fact a soft-error at sea level killed an operating system update on me, lost about tens of thousands of data and money. I have standing to sue Intel, until this sentence clause in which I hereby forfeit the suit, together with requesting them to reconsider ECC (error correction codes) in all their chips as a safeguard needed due to Moore's Law, which was their business plan. Just give it a thought, Intel.
- nix23 4y agoIt's your error, having a system with important data no actual/realtime backup no second system and no plan to recover from a failed update and no ecc is YOUR error alone. However, intel should have made ecc the standard and not just for 1000$+ Xeons.
- gjvc 4y ago>However, intel should have made ecc the standard and not just for 1000$+ Xeons. Agree 100%. IMHO, the choice between "domestic" and "industrial-strength" should not mean choosing between different degrees of risks of failure.
- mrlonglong 4y agoI gather DDR5 will have ECC as standard du to the extreme density of memory that will bitflip a lot more than usual. Yay fo r consumers.
- kmeisthax 4y agoDDR5 supports on-chip ECC but the extra parity bits we typically associate as "ECC" are still as optional as ever, motherboard manufacturers will still not bother to route those signals anyway, and Intel will still demand you give up overclocking and pay more for Xeon in order to use ECC sticks.
- jdsampayo 4y agoVeritasium has a video explaining cases of bit flip that maybe you find interesting: https://youtu.be/AaZ_RSt0KP8 https://youtu.be/AaZ_RSt0KP8
- baruchel 4y agoBit flips are quite useful for sorting huge arrays of data: https://news.ycombinator.com/item?id=28766154 https://news.ycombinator.com/item?id=28766154
- smegma2 4y agoThat code has a race condition :^)
- xyzzyz 4y agoDid you mean ray's condition?
- snoshy 4y agoOddly apropos anecdote to your pun: About 15 years ago, while working on Windows at Microsoft, a test machine sitting at my desk hit a kernel panic (BSOD). As was standard working on the test team, the machine was already setup for kernel debugging, and so I set out to debug it a bit in order to file a decent bug report. Hours later, I couldn't make sense of it (I wasn't super experienced at this point). A few of the nearby devs couldn't either, and a small troop of us curious enough about the puzzler eventually escalated to the resident wizard, Raymond Chen[1]. Within 15 minutes of checking our work and poking at the machine, he traced the root cause down to a bit flip. Ray's condition indeed. :) [1] https://devblogs.microsoft.com/oldnewthing/ https://devblogs.microsoft.com/oldnewthing/
- elevaet 4y agoGreat story, a true random glitch. Interestingly, a blog post of his was up on HN front page the other day: "The x86 architecture is the weirdo, part 2" https://news.ycombinator.com/item?id=31077912 https://news.ycombinator.com/item?id=31077912
- sbierwagen 4y agoRay complained in 2005 that Microsoft sees lots of weirdo crash reports from computers with overclocked CPUs https://devblogs.microsoft.com/oldnewthing/20050412-47/?p=35923 https://devblogs.microsoft.com/oldnewthing/20050412-47/?p=35...
- trinovantes 4y agoIt's scary to think an election can be flipped due to cosmic rays https://en.wikipedia.org/wiki/Electronic_voting_in_Belgium https://en.wikipedia.org/wiki/Electronic_voting_in_Belgium
- dudeinjapan 4y agoCan't wait for the History Channel documentary on this... "Aliens"
- asasidh 4y agoBit flips sound like glitch in the matrix
- Helmut10001 4y agoThat is why I migrated from Raid 1 to ZFS on my home server and do weekly scrubs. Not sure if this would have prevented your case though.