18 ms·
AT&T Fiber in the SF Bay Area is flipping bits
- km3r 6y agoWhy doesnt TCP reject the flip bits when the checksum fails?
- jedberg 6y agoApparently MacOS doesn't check the checksums.
- tjalfi 6y agoThe TCP checksum catches some issues but it won't detect everything. Other newer algorithms like CRC32C[0] are much more effective at detecting corrupt data. The following is an excerpt from Performance of Checksums and CRCs over Real Data[1]. The TCP checksum is a 16-bit ones-complement sum of the data. This sum will catch any burst error of 15bits or less[8], and all 16-bit burst errors except for those which replace one 1’s complement zero with another (i.e., 16 adjacent 1 bits replaced by 16 zero bits, or vice-versa). Over uniformly distributed data, it is expected to detect other types of errors at a rate proportional to 1 in 2^16. If you're deeply interested in this topic then I would recommend Jonathan Stone's Phd thesis. [0] https://en.wikipedia.org/wiki/Cyclic_redundancy_check#Polynomial_representations_of_cyclic_redundancy_checks https://en.wikipedia.org/wiki/Cyclic_redundancy_check#Polyno... [1] https://www.researchgate.net/publication/3334567_Performance_of_checksums_and_CRCs_over_real_data https://www.researchgate.net/publication/3334567_Performance...
- dboreham 6y agoTCP checksum is notorious for not catching all corruption, to the point that all protocols run over TCP should have their own integrity checking. TLS is good enough AFAIK, which lets most traffic off the hook these days.
- megameter 6y agoThis has been affecting us intermittently for weeks.
- jonny_eh 6y agoI'm so happy to have had the opportunity to switch to Sonic's Fiber last month.
- gricardo99 6y agoThat may not make any difference, depending on your locality. Unfortunately AT&T infrastructure is still relied on by others such as Sonic, which leases lines from the likes of AT&T fiber, at least in my locality.
- greensoap 6y agoYup --- in most of the south bay Sonic is leading AT&T for its fiber.
- Bud 6y agoSonic's fiber service uses AT&T fiber in Alameda as well. Just another data point. I have that service. It's been working just fine, though. I haven't noticed any symptoms of flipped bits or anything similar to what others are reporting here.
- ribosometronome 6y agoOh man, I didn't even realize they were available outside of SF with fiber.
- jonny_eh 6y ago
- jedberg 6y agoOh man, I thought it was just my crappy old router causing the problems! I've just been too lazy to call them to replace it. I'm going to tweet at them right now. Edit: Reading the tweet thread, classic interaction: ATT says "please click this to check your connection", guy replies back and says "I think we know more about networks that you do, please get a network admin in here".
- secalex 6y agoWhat would you know about production networking, Jeremy? I'm in the same boat. If the dozens of IT pros who are complaining about this can't get AT&T to swap out a single router card, what hope do most folks have?
- segfaultbuserr 6y agoFortunately, this malfunction occurred in the SF Bay Area, getting AT&T to fix its network is only a matter of time, sooner or later, the correct person in the Valley would be alerted. If it happened to the rest of us at somewhere else, we are probably out of luck...
- x0x0 6y agoIt's happening to me too! I bought a new router a couple days ago because I couldn't figure out what else could be causing random, sporadic slowdowns. I can't even download the page 3 times in a row w/o corruption 4c4 < <titde>Example Domain</title> --- > <title>Example Domain</title>
- greensoap 6y agoCan anyone tell me what the symptoms of this would be?
- jedberg 6y agoComputer hanging when connecting to random websites some of the time. Apps on your phone just randomly hanging. Streaming services randomly not working. If you use Twitch you'll randomly have the video break with a connection failure in the middle of stream.
- avarun 6y agoInteresting, I've been having a lot of the same symptoms for months now with my Comcast service on the other end of the country. Comcast refuses to acknowledge there's any problem on their end, but I'm reasonably confident that my home infrastructure is rock solid after double checking every aspect of my network setup.
- proverbialbunny 6y agoDefault timeout time on firefox is 30 seconds before retrying and a ctrl+shift+r refresh does not refresh the load, so random websites pause while opening and you have no choice but to wait 30 seconds before the page starts loading. (Psst, the default can be changed.)
- jmount 6y agoWe are AT&T in SF Bay Area, and our service has been flaking out for weeks. Inability to get DNS, machine claims invalid security certificates, download/upload drop below 1 MbS. All we could get is our modem reset.
- floatingatoll 6y agoWhat’s the hostname on the invalid security certificates?
- hangonhn 6y agoI'm on Sonic.net fiber and they're a reseller of AT&T and I've experienced this exact scenario.
- deeviant 6y agoDefinitely have been having this issue, for weeks.
- sigmonsays 6y agoI am a new AT&T fiber customer and have noticed that connections to random places seems slow. New connections in general. There seems to be no pattern, even across udp or tcp. I assumed there was just bad connectivity issues. Any more details available on what is happening? Like, what bits are getting flipped? I might have to go back to the local cable company if this continues. Anyone experiencing this in SoCal?
- benlivengood 6y agoIt's ironic that for the most part Silicon Valley only has terrible ISP infrastructure. Cellular service isn't all that great either.
- cbhl 6y agoThe ISPs have tried, but they get pushback from local residents whenever they try to install the necessary network boxes at intervals down the street. They had to go out of their way to hide cellular towers as streetlamps to get the 5G rollout to happen; I remember getting two or three notices for this (each from different telcos, so the one street corner has three streetlamps and two traffic lights on one corner). I suspect if the droughts didn't make people get rid of their green grassy lawns homeowners would be more amenable to seeing green network boxes every few houses. It looks awful in the context of concrete sidewalks, though.
- freeone3000 6y agoSolution: Paint the boxes grey.
- NullPrefix 6y agoSolution 2: Lift the box up, attach it to the head of the street lamp.
- okigan 6y agoservicing at the top of the street lamp would me so much easier, right?
- toast0 6y agoNaw man, woodgrain stickers.
- gwittel 6y agoOther options: 1) Sprout the streetlamp from the box, add a car charging port. 2) Hire an artist to paint the box with artwork (property owner gets to pick from option set). Consider it to be city owned art installations.
- renewiltord 6y agoSonic has their own fibre in San Francisco. Thank god for that. Life's too short for Comcast or AT&T.
- deleted 6y ago[deleted]
- aaroninsf 6y agoSource? We have Sonic DSL and I see differences using the example.com test; and at installation time I was told I could only get fiber at my location (backside of Bernal) when AT&T put it in and they could lease it. That was in 2019. Do they own their own fiber in some areas?
- ribosometronome 6y agoAt the very least, Sonic did their own roll out back in 2015 in the Sunset. Saw the trucks and was a very happy customer. https://twitter.com/sonic/status/748371538265649153?lang=en https://twitter.com/sonic/status/748371538265649153?lang=en
- tuzakey 6y agoSonic has their own fiber in some parts of SF/Santa Rosa and you would know if you were on it, all Sonic DSL products are essentially resold AT&T uverse.
- snotrockets 6y agoIn some parts of the East Bay as well, but I can't find exact maps. If they sell you "Fusion IP Broadband", it's rebranded AT&T. If it's "Fusion Fiber", it's Sonic's own.
- Lammy 6y agohttps://thatdan.github.io/sonic_fiber/ https://thatdan.github.io/sonic_fiber/
- cbhl 6y agoWhen AT&T first did their Fiber rollout in SF, one of the things I remember was they charged you $10 extra if you didn't want them to MITM all your connections to insert JavaScript pointing to their own ads. They rolled this back when folks complained, but I wonder if the relevant infrastructure is still sitting around and mangling packets. I would probably expect this to be some network card or cable or connector is failing though...
- floatingatoll 6y agoYes, as of six months ago I had to look up how to opt out of MITMing for a coworker on AT&T Fiber, so that they could use the Internet again when the remote-activated MITM feature inside the modem broke.
- StillBored 6y agoThe tweet indicates it is at a specific bit position. That isn't symptomatic of a bad cable/sfp/etc. Analog problems like that tend to be more random. Random bit flipping in a fixed bit position is symptomatic of bad ram somewhere, or a router asic gone bad, or various software/config issues.
- jasomill 6y agoWhat I always wondered about this is, unless AT&T has permission from the copyright holder of each and every Web page so modified to distribute the resulting derivative work, how is this practice not criminal copyright infringement?
- kmeisthax 6y agoFor the same reason why ISPs are generally immunized from contributory liability that they would otherwise be completely buried in. Now, if you had written JavaScript to detect and remove these ads, and they went around that, then you might be able to construct a DMCA 1201 claim and sue the ISP for circumventing what is legally considered DRM. Yes, JavaScript can be legally protected DRM. The law doesn't say it has to be good DRM, it just has to have the effect of controlling access to a copyrighted work. And the safe harbors the DMCA provides ISPs wouldn't protect them in this case.
- EvanAnderson 6y agoI once had a combination of "dueling" VPN clients on a Windows PC that caused random bit flips. It was awful. SSL/TLS sessions randomly failing. SSH sessions closing randomly. Downloaded files failing integrity checks. It's an interesting juxtaposition of how tolerant TCP/IP is of random bit flips, yet how intolerant application layer protocols are.
- coddle-hark 6y agoSurely bit flips are invisible to application layer protocols?
- oluckyman 6y agoThis was used to flip votes in Dominion machines from our greatest president Trump to China Joe, and the stupid Dems forgot to turn it off. Otherwise Trump wins CA easy. Wait till Sidney Powell hears about this.
- francoisLabonte 6y agoFrom my professional experience of programming and debugging networking equipment, this could be a switch/router with a buffer with bad memory (stuck bit maybe). The better chips have CRC/Parity/ECC to cover such issues but there are always those magical choke points where the past CRC is tossed and the new one is generated that can leave a gaping hole. The tricky part is how often is this bad memory buffer used... I would use traceroute to find a common bad point for everyone. It is also possible that the networking point where the problem occurs is invisible to traceroute as it could be part of a provider network probably MPLS but at least the common ends of the tunnel would be visible. The fact that it is a a specific interval indicates a stuck bit in memory. Some good previous public stories about such incidents https://www.verizondigitalmedia.com/blog/being-good-stewards-of-the-internet/ https://www.verizondigitalmedia.com/blog/being-good-stewards... https://twitter.com/cperciva/status/1309568337408454658 https://twitter.com/cperciva/status/1309568337408454658
- cpach 6y ago”I would use traceroute to find a common bad point for everyone“ How do you mean? I use traceroute from time to time but I’m not sure how it would apply in a case like this. Feel free to elaborate :)
- kevinherron 6y agoI think he means if everyone on AT&T experiencing the issue ran trace route to example.com some common hops would emerge, which would be a place to start investigating.
- simcop2387 6y agoTake a traceroute from everyone experiencing the problem, look for the common hops among them all. Then compare that list against people not experiancing the problem to find the differences. The finsl set there are good places to start looking. Sweitches and routers along that way could be the cause
- toast0 6y ago
- gwittel 6y agoI've been having similar issues matching this as well (SF south bay), with ATT Uverse (fiber to neighborhood, 100mbit to home). I really wish we'd just have municipal ISPs here. I'm only a few miles from Google HQ, but the internet options are crap.
- lotsofpulp 6y agoI find it amusing that all these multi hundred thousand dollar per year income people working at the forefront of consumer tech don’t have access to a quality symmetric fiber internet connection.
- throwaway1777 6y agoThey all used to work from the office which had great connections. Another fallout of wfh
- lotsofpulp 6y agoIf it musters up the energy and political will to get municipal fiber going as a utility, I’ll call it a win.
- gwittel 6y agoYeah, amusing in a perverse way. Years ago I had gotten an "ATT fiber coming soon" notice, but still not here. In the end its simply the usual issue with internet access in the US -- ISPs have zero incentive to invest/improve, there's no meaningful competition. With everyone WFH its even worse -- oversubscribed networks are causing trouble all over (esp if you use a cable ISP). When a city (or someone like Google) brings fiber in a meaningful way, it only then lights a fire under the incumbents' ass. In theory cities with non-buried lines (like Mountain View) may be easier to upgrade; but it still depends on ATT granting pole access (which they promptly slowwalk).
- jrockway 6y agoWhat layer of the stack do we think this is happening at? I seem to recall that GPON has forward error correction, so the same bit flipping in every packet seems unlikely to be caused there. But, I suppose that core routers are complicated enough these days to cause problems like this (custom ASICs doing all the bit moving), and if it affects every customer without common pieces of network equipment in between, it sure sounds like a software problem. More programs, more problems!
- tshaddox 6y agoIt’s certainly a relief to have an apparent explanation for the weirdness I’ve been experiencing! I thought it was my home router or WiFi or something, but then I experienced the same problems at my partner’s house, so I thought maybe it’s just a problem with some websites I frequent or some common internet infrastructure. But we both have AT&T internet, so this must be it! For me, the problem has mostly manifested as web pages failing to load or appearing to be loading forever. Generally when I refresh the page would load quickly as expected.
- Cerium 6y agoI've been having problems like that with ATT fiber too.
- richardowright 6y agoI've also been having this issue with ATT Fiber, but I'm out of Atlanta.
- lukeman 6y agoI saw the headline and immediately realized I've noticed similar oddness in the Atlanta area recently. Glad to hear I'm not alone. So far I've been blaming it on weirdness from my devices roaming between mesh router nodes, but now I'm going to run the test script overnight and see if anything turns up.
- cameldrv 6y agoIf you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 https://twitter.com/bmastenbrook/status/1335400747794530304 It loads http://example.com http://example.com and https://example.com https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS handshake is an error and the connection dies. Tech support, even when Twitter accounts with a lot of followers message them, is completely useless, they just say that they don't show any outages at your location. They need to be flooded with complaints for someone to look at this, or maybe someone from AT&T is on here that can get it looked at...
- lowdose 6y ago> or maybe someone from AT&T is on here that can get it looked at... They are on Facebook and Linkedin, you have seem them.
- kmonsen 6y agoI ran the script and it is finding a difference after a while. I have not noticed anything wrong with the network recently, but that of course does not mean there has not been a problem. Edit: Actually I take that back, I have seen one of the issues where sometimes web sites will not load at random, and a reload fixes the issue.
- rconti 6y agoWhy is it always the same positions that differ? The files differ after anywhere from 1sec to 5sec for me, and it's always the same character positions, and it always seems to be the same lines, and the same number of lines.
- mcpherrinm 6y agoMost likely it’s the same position in packets; and there’s some bad RAM in a device along the route.
- 6y ago
- maverick2007 6y agoSame issue here, thought I was going crazy with speed being WAY down and random websites giving me errors. Also found a Reddit thread (https://www.reddit.com/r/bayarea/comments/k87h2f/psa_att_fiber_is_having_network_issue_connection/ https://www.reddit.com/r/bayarea/comments/k87h2f/psa_att_fib...) about the issue that advises to contact the executive service customer support (https://investors.att.com/resources/contacts#customer-service https://investors.att.com/resources/contacts#customer-servic...) if your calls to regular customer support don't fix the issue - I did, we'll see how it goes
- ProfessorLayton 6y agoI had AT&T via Sonic's FTTN in an effort to avoid Comcast/AT&T directly, especially since Sonic does not have data caps despite running on AT&T's network and actually respects their customer's privacy. Unfortunately my experience was so terrible and unreliable that I decided to give up and finally sign up with Comcast, and just have to mind that 1.2TB data cap. The Bay Area of all places should not have such terrible ISP infrastructure, but here we are.
- proverbialbunny 6y agoIt's because AT&T does not follow net neutrality. It has fast lanes and slow lanes, which cause a lot of problems for work VPNs even when down the street. This TLS handshake bit flip is just yet another issue AT&T has, including the 1.1.1.1 DNS issues, and everything else. I'm on Sonic.net fiber over AT&T too. To get the equivalent fiber from Comcast out here it's $270 a month. However, if you're lucky enough Sonic.net fiber that doesn't run over AT&T is amazing.
- city41 6y agoHuh, I had Sonic fiber over AT&T and loved it. In May I moved out of the bay area and now have Comcast cable. I'd do pretty much anything to get Sonic's level of quality out here.
- proverbialbunny 6y agoThat's because you're comparing fiber to cable. Get Comcast fiber if you want it again. I don't know if $270 a month is a statewide rate or not.
- e40 6y agoI was seriously considering signing up for Sonic service, and I have Comcast now. Guess I won't do that. I hesitated because I really really hate AT&T. Guess I should have trusted that feeling.
- 6y ago
- dekhn 6y agoI have AT&T fiber. I noticed Wikipedia started to have trouble loading around then, assumed Wikipedia just had capacity issues. Script definitely shows corruption.
- rconti 6y agoSonic-resold ATT fiber for the past 18 months, Redwood City. Just started noticing issues a few days ago. Script definitely shows some issues.
- bfirsh 6y agoUrgh. I wasted an hour or two trying to debug my local network due to this.
- yrral 6y agoFor this type of issue, I would recommend trying the nanog mailing list. You often see network admins ask for someone "with clue" at a different company when they get the runaround by tier 1/2 tech support. https://archive.nanog.org/list/join https://archive.nanog.org/list/join https://seclists.org/nanog/2019/Aug/103 https://seclists.org/nanog/2019/Aug/103
- 35fbe7d3d5b9 6y agoThis is absolutely the right way to "informally" escalate things to people that know what they're doing at big ISPs. I'd recommend most newbies to the list show their work and post what's broken and how you know it isn't your fault. The investigative work on that Twitter thread is top notch and would do it in a second. NANOG and outages@ are the two mailing lists that I've been subscribed to forever and are indispensable if you do operations.
- yadaeno 6y agoDoesnt tcp have a checksum field? Would this mean they are doing mitm business with non-encrypted traffic?
- ytch 6y agoI am thinking the same problem. TCP checksum includes data field. Is it means AT&T flip bits and re-calculate checksum?
- yadaeno 6y agoIt looks like someone on the twitter thread checked and the checksums are wrong. Im not sure why these packets aren't discarded by your computer. edit: apparently osx doesnt drop these packets according to the twitter thread.
- kjetijor 6y ago"It depends" - Your NIC could be doing some tcp offload for you, at which point any checksum you'd observe with tcpdump/wireshark would be flat out wrong. Generally - both the tcp checksum and the 802.11/Ethernet CRC isn't all that strong, so it's entirely within the realm of possibility of having bitflip patterns that fool both.
- secalex 6y agoYou can contribute to our attempts to find the bad router card here: https://twitter.com/alexstamos/status/1336099461622157312 https://twitter.com/alexstamos/status/1336099461622157312 Almost certainly 12.242.117.22
- boulos 6y agoIndeed. My mobile.twitter.com traceroute prefers going through that path, as does en.wikipedia.org (both of which have sucked for me) while a Google route (to 172.217.6.78) hops through 12.122.149.186.
- ummonk 6y agoYup, I'd been having issues with twitter, wikipedia, and sometimes duck duck go but never google.
- deleted 6y ago[deleted]
- cpach 6y agoYou might want to send an e-mail to the NANOG mailing list: https://mailman.nanog.org/mailman/listinfo/nanog https://mailman.nanog.org/mailman/listinfo/nanog It’s not impossible that people from AT&T read messages posted there.
- krasin 6y agoI confirm that I am seeing these bit flips, and 12.242.117.22 is in my traceroute. You people are amazing!
- iaabtpbtpnn 6y agoConfirming this IP is in my traceroute to example.com as well
- tjohns 6y agoI've been getting intermittent slow connections (30+ seconds to load Wikipedia) and SSL handshake failures for a couple weeks now. I thought it was just my old router. I had no idea it was more widespread. I was just about to start upgrading network equipment.
- fritex 6y agoI believe I felt that one in Croatia. Started on 6-7th November. My ISP Tele2, Croatia. But, I am on a 4G LTE router. Haven't had it for 2 years since I've been with that ISP nothing the same as described. Or at least we had some strange stuff for a few days, I thought it was my dedicated server SSH and MTU, bu not. When connected through the VPN, the speed is normal, fine. Also, FTP was working fine. But, without a VPN, the network seems to be "throttled" somehow, do not know why. Persists today. I have to use VPN for SSH connection to my server and to download on my "full speed" over FTP. And my router is D-LINK, since firmware upgrade cannot access HTTPS configuration. Bad certificate :/
- eznzt 6y agoMust be fun to have a game ban you for being a cheater because you've tampered with the byte stream :P
- AaronFriel 6y agoTrying to explain this issue to AT&T support is like trying to convince a doctor you're the only person on earth with a particular disease. Even explaining the issue is hard. It's not an outage, my internet isn't out, it's intermittently wrong. The phone support agents aren't prepared for this, and I can't find any way to escalate or speak to a network engineer. I feel like if I spoke to the right engineer, there'd be a ticket on this and they'd roll at truck to their facilities or the IPX within an hour. It's a major network issue to flip bits, it's costing them bandwidth with retransmits and could be breaking SLAs with their business customers. On the phone the most they could do was roll a truck to me.
- deleted 6y ago[deleted]
- dbt00 6y agoI talked to somebody and he told me he had talked to his "IT department" and they're working on it but who the fuck knows.
- warent 6y agoThe obnoxious thing to me is the hubris that must be behind this. Either they considered it and decided they would never encounter a system error like this and refused to implement an escalation route, or they never even considered it. Or perhaps even worse than that, maybe they considered it, decided it was possible, but just don't care because of their insane borderline monopoly. I don't understand how internet companies provide such consistently awful service. Slightly off topic story, I've recently changed to another provider called Starry, and they force you to have a second router in front of your own router which they claim "decodes" their stream between the modem. I don't know the real reason but I'm pretty sure that's not it. If you plug their modem directly into a non-Starry router, the router just doesn't detect a connection. One day, I tried to torrent something, and my internet would immediately get throttled to 0mbps. After investigating I found out that their router had a custom OS which hid a firewall and various security settings. Amusingly you could still access those settings if you just manually entered the page names into the address bar. Now all their stupid settings are disabled and I just feel badly for all the folks who use their service and don't have the savvy I do to actually get what they're paying for.
- danceparty 6y agoAT&T runs transparent proxies on their network that have been known to intercept and mangle tls, and directly manipulate tcp packets on the wire. Take a tcpdump capture on both server/client side and compare the handshake/tcp headers/negotiated window size. Often you see the server sending one thing, but the client sees another
- hevisko 6y agoquestion: could people confirm the ssl certificate chains being used by the server, and reported by the client?
- frunzales 6y agoObligatory read: http://mina.naguib.ca/blog/2012/10/22/the-little-ssh-that-sometimes-couldnt.html http://mina.naguib.ca/blog/2012/10/22/the-little-ssh-that-so...
- zacwest 6y agoSonic resells me AT&T and their best suggestion, since they cannot file issues with AT&T (what?), was to post on their forums and hope the Sonic CEO sees it and acts on it. Does seem like they have noticed enough to start escalating it though: https://twitter.com/sonic/status/1336013450909257730?s=21 https://twitter.com/sonic/status/1336013450909257730?s=21
- syntaxing 6y agoWow. I happen to setup my PiHole at the same time and did not expect it to be the ISP. I even bought the newest RPi 4 and disabling lists without any luck. The interesting part is that I turn off my wifi on my cellphone if the site load hangs and it always worked. I always thought it was because I was using my non-pihole DNS.
- osamagirl69 6y agoI see this on my end too (redwood city) using the check against example.com diff example-http example-tls 4c4 < <title6Example Domain</title> > <title>Example Domain</title> Been wondering why my tls connections kept failing...
- osamagirl69 6y agoUpdate - they seem to have fixed it! The same script successfully ran for 5 minutes for me without problems today
- jamilbk 6y agoThe script linked in the gist (https://gist.github.com/bmastenbrook/14c0e22fc02b95d4a48f82d3ec3123db https://gist.github.com/bmastenbrook/14c0e22fc02b95d4a48f82d...) is reporting diffs for me on Sonic fiber as well, which is leased from AT&T in much of the Bay Area. So seems to be a network layer or below issue. Interestingly, the incorrect ASCII chars are always 8 chars higher or lower than their counterpart, indicating that bit 3 is consistently being flipped: flip: f|0b1100110 orig: n|0b1101110 --- flip: i|0b1101001 orig: a|0b1100001 --- flip: 6|0b110110 orig: >|0b111110 --- flip: (|0b101000 orig: |0b100000 --- flip: p|0b1110000 orig: x|0b1111000 --- flip: m|0b1101101 orig: e|0b1100101 Edit: Can confirm connecting to VPN (ProtonVPN in this case) solves the issue.
- heleninboodler 6y agoI love this. In 1992, I bought a Kenwood stereo receiver, which I still have in my living room to this day, which you could program your radio station call letters into for your presets, and I noticed that after setting one of the presets to "KNBR" it switched to "KNBB". I assumed I made a mistake and corrected it, but a few days later I noticed it happened again. Assuming ASCII encoding, B is 0x42 and R is 0x52, so the difference is the single bit (and even assuming non-ASCII, the delta of 16 is likely still the same), so I experimented and determined that that particular memory location always eventually zeroed the 4th least significant bit, so all characters from P-Z would be shifted down to the corresponding character 16 positions less than it. Nowadays I don't really use the radio presets anymore, but I still like knowing the flaw is there nearly 30 years later.
- dboreham 6y agoIt has to be a good 30 years since I saw the first instance of this: an Ethernet bridge that randomly flipped packet payload bits and nicely regenerated a correct CRC on the mangled packet going out. Every few years we come across some variant of the same "evil box" problem.
- 29athrowaway 6y agoI thought that the link layer has built in mechanisms for detecting and correcting errors. Can someone ELI5 what's going on?
- maerF0x0 6y agoNot that I have any specific evidence this is the case, but it wouldnt surprise me to later learn this is an attack on encryption by the NSA. AT&T in San Francisco is ground 0 for some of their bulk collection. After The Times disclosed the Bush administration’s warrantless wiretapping program in December 2005, plaintiffs began trying to sue AT&T and the N.S.A. In a 2006 lawsuit, a retired AT&T technician named Mark Klein claimed that three years earlier, he had seen a secret room in a company building in San Francisco where the N.S.A. had installed equipment. [1] [1]: https://www.nytimes.com/2015/08/16/us/politics/att-helped-nsa-spy-on-an-array-of-internet-traffic.html https://www.nytimes.com/2015/08/16/us/politics/att-helped-ns...
- PieUser 6y agoLOL
- Lammy 6y agoThank you for voicing this despite the downvotes. I don't think it's the most likely scenario, but it hurts to see how harshly people on HN reject speculation about other paths of possibility than the most obvious, especially considering AT&T's history. Never forget: "NSA documents show that the relationship with AT&T has been considered unique and especially productive. One document described it as 'highly collaborative', while another lauded the company’s 'extreme willingness to help'."
- Bluecobra 6y agoI would think that if that room did/still exists they would most likely use passive optical taps, not interfere with transit traffic. Optical taps simply mirror a fiber optic signal one way to a tap aggregation switch/packet capture devices. They would also be mostly undetectable, you will just see a reduced light level.
- maerF0x0 6y agoI'm pointing out that flipping bits could be a form of cryptographic attack (potentially against an implementation bug, not necessarily against the math/algo) . For example flipping a bit possibly could cause a checksum failure and retransmit. I'm not saying something _is_ happening, mostly just musing on the possibilities.
- bfirsh 6y agoIf you've got AT&T fiber via Sonic, give them a call and ask them to escalate the issue. The people on the phone know about the issue, take you seriously, and the more people complaining the more pressure they'll put on AT&T, I expect.
- iaabtpbtpnn 6y agoI have done this earlier today and it is exactly as you say. All they can do is file another ticket with AT&T, but multiple tickets coming to one ISP from another ISP mean more than calls or tweets to the AT&T customer support line, so here's hoping! Some people on Twitter report this problem going back all the way to November. Glad it's finally getting attention tonight.
- sneak 6y agoRequisite link to the excellent paper by Artem Dinaburg about Bitsquatting, the practice of passively exploiting single bit flips (usually in ram, but on the wire works too): https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinaburg_Bitsquatting_WP.pdf https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabu...
- monkeybutton 6y agoI guess I'm one of the lucky 10,000 today; thanks for posting this!
- baggy_trough 6y agoPretty crappy that it's up to the users to debug this. Why aren't there tests / alarms for this kind of thing?
- bakul 6y agoSome new data.... I ran the same script on two different machines connected via the same AT&T provided modem. Machine A is NATted, B uses a public address. A always gets the error within 50-60 fetches. B doesn't (I ran the test a number of times for a few seconds each). I even ran them both concurrently (repeated the test on A since it keeps failing) but it was always A that got the error. I am wondering if this is modem related -- may be they silently "upgraded" it or changed some default such as TCP checksum offload? Just guessing, I haven't checked. Experiment 2: I switched machine A to use a public address (from my static IP address block) and the problem disappeared! I then switched it back to the way it was and still there are no errors. Not sure what to make of this...
- AaronFriel 6y agoHow are you bypassing the EAP authentication on your AT&T provided gateway? Are you using https://github.com/jaysoffian/eap_proxy https://github.com/jaysoffian/eap_proxy ? My router is also getting a public IP address, I've set my AT&T gateway to bridge mode, but the AT&T gateway is still a monster-in-the-middle, a tumor I can't excise because it has to be between your router and the ONT or else the fiber turns off.
- bakul 6y agoI recall hardly any of the details now but IIRC I talked to the AT&T support person and they changed some settings. Looking at the settings now, I see in "Home Networks->Subnets & DHCP" the [public subnet mode] is on. I haven't had to deal with any EAP authentication (but again I may be forgetting!). And IP Passthrough is off. Basically the modem is routing between the public and private subnets. BTW, I do have a static IP block from AT&T.
- toast0 6y agoWhen I was in the service area, I had written a EAP proxy and did it that way (I've still got the code around somewhere, ran on FreeBSD, email in my profile if you want it), but I've heard the easier way is to put an unmanaged switch between the ONT and the residential gateway, let the gateway do the 802.11x, then unplug it from the switch and use hardware of your choice. As long as your switch isn't too smart, it won't know to send an 802.11x signoff, and the ONT will leave the network port working. Yay, security theater. You may need to spoof mac for DHCP, but I don't know if that's strictly required.
- deleted 6y ago[deleted]
- noncoml 6y agohttps://en.wikipedia.org/wiki/Room_641A https://en.wikipedia.org/wiki/Room_641A gone wrong?
- byecomputer 6y agoFlipped to 641B
- spekcular 6y agoFrom Twitter: > I’m hearing AT&T got their shit together and things are working now. Big thanks to @vikxin and @bmastenbrook for doing the heavy lifting here. https://twitter.com/Catfish_Man/status/1336170784784412673 https://twitter.com/Catfish_Man/status/1336170784784412673
- shuckles 6y agoThe same day there is an enormous prognostication thread on the future of the Bay Area, a bunch of nerds on Twitter observe, test, diagnose, and root cause a local fiber network issue. I can only think of a few places in the world where this may happen: https://twitter.com/bd/status/1336110887145361410 https://twitter.com/bd/status/1336110887145361410
- dmode 6y agoI have tried AT&T Fiber twice and the experience have been underwhelming both times. Intermittent outages and dead connection on a frequent basis. Unfortunately, I switched to Comcast 1GB connection and it has been miles better than AT&T
- fpgeek 6y agoAccording to https://twitter.com/alexstamos/status/1336163726345441280 https://twitter.com/alexstamos/status/1336163726345441280 it appears to be fixed now.
- sumosudo 6y agoWell I flipped the bit right back at them! But seriously now, have they tried turning it off and on again?
- ummonk 6y agoThat would explain a lot. I've been having intermittent internet issues for a while now.
- _wldu 6y agoCould be some dust on a link connection somewhere messing up certain bit patterns. That has happened before. https://www.fomsn.com/fiber_optic_connector/angelina123/why-should-we-keep-fiber-optic-connectors-clean/ https://www.fomsn.com/fiber_optic_connector/angelina123/why-...
- Unklejoe 6y agoMy first thought was: how come the frame isn't just dropped due to a bad checksum? Then I remembered a piece of Ethernet test hardware I worked with. It had a bad SRAM chip or something which would result in all of the payloads having a stuck bit every 8 bytes. However, the FCS was calculated _after_ that, so it was still a valid frame. My guess is AT&T has some hardware that's recalculating the checksums after the frame has already been corrupted.
- satisfaction 6y agoI don't have fiber but I do spend slightly more on business class internet, they have tech support that is an order of magnitude more helpful and knowledgeable than normal residential internet. With business class you can get real static IPs and better support SLAs.
- bdd 6y agoDane Jasper, CEO of Sonic, which resells AT&T's fiber service in some areas where they don't have their own fiber, responded to a tweet asking if he could help with pushing this to a remedy with AT&T: > Yep, we are engaged. And they are clueless. Huge challenge to find someone who really has ability to troubleshoot." https://twitter.com/dane/status/1336111107430207488 https://twitter.com/dane/status/1336111107430207488
- todd_m 6y agoSimilar problem, on the peninsula, running ATT Fiber via Sonic. There is a known issue with 2 part numbers of ONT that are incompatible with ATT main station, resulting in packet loss, accrued errors, etc. An ATT service tech can come out to replace this in 3 minutes. Have had zero problems since replacement, after getting knocked off network many times per hour.