14 ms·
If you have AT&T fiber, run the script in the linked gist: https://twitter.com/bmastenbrook/status/1335400747794530304 https://twitter.com/bmastenbrook/status/
by cameldrv 6y ago
If you have AT&T fiber, run the script in the linked gist:
https://twitter.com/bmastenbrook/status/1335400747794530304 https://twitter.com/bmastenbrook/status/1335400747794530304
It loads http://example.com http://example.com and https://example.com https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS handshake is an error and the connection dies.
Tech support, even when Twitter accounts with a lot of followers message them, is completely useless, they just say that they don't show any outages at your location. They need to be flooded with complaints for someone to look at this, or maybe someone from AT&T is on here that can get it looked at...
- lowdose 6y ago> or maybe someone from AT&T is on here that can get it looked at... They are on Facebook and Linkedin, you have seem them.
- kmonsen 6y agoI ran the script and it is finding a difference after a while. I have not noticed anything wrong with the network recently, but that of course does not mean there has not been a problem. Edit: Actually I take that back, I have seen one of the issues where sometimes web sites will not load at random, and a reload fixes the issue.
- rconti 6y agoWhy is it always the same positions that differ? The files differ after anywhere from 1sec to 5sec for me, and it's always the same character positions, and it always seems to be the same lines, and the same number of lines.
- mcpherrinm 6y agoMost likely it’s the same position in packets; and there’s some bad RAM in a device along the route.
- p1necone 6y agoAll of these large companies seem to have (correctly) realized that 95% of tech support cases are trivial issues that can be resolved via automated responses. The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking. But if you're one of the 5% you're fucked. If there's anyone out there designing tech support procedures, you should add an "is this a 5% problem?" question to whatever checklist you give to support staff.
- cpach 6y agoObligatory XKCD reference: https://xkcd.com/806/ https://xkcd.com/806/
- Nextgrid 6y agoThe problem is that the idiots the hire to do their "technical" support have zero skill (nor motivation to learn) to assess whether it's a 5% problem or not, and the majority of end-users aren't capable to answer that question either, nor that they are incentivized to answer truthfully. The solution could be a priority support tier where you pay upfront for an hour of a real network engineer's time (decently compensated so that he actually cares about solving the problem) and the charge is refunded only if the problem indeed ends up being on the ISP's side. This should self-regulate as anyone wasting the engineers' time for a simple problem they could resolve themselves would pay for that time.
- GauntletWizard 6y agoWho decides if it's the ISP's problem? What if it's "Both"? About 20 years ago I got escalated to high-tier Comcast support for an issue that turned out to be a little of A, a little of B: Comcast (Might have been @Home, based on the timing) required that your MAC address be whitelisted as part of their onboarding process. Early home routers had a "MAC Address Clone" feature for precisely this reason. At some point, the leased network card got returned to Comcast. Our router continued to work just fine... until about a year later, when the local office put that network card into some other customer's home. We started getting random disconnects in the middle of the day, and it took forever to diagnose, as the other customer was not particularly active with their internet use. Whose fault was it? Ours, for ARP Spoofing? Theirs, for requiring the spoofing?
- baggy_trough 6y agoOddly, I can't even resolve example.com (using AT&T DNS).
- dismalpedigree 6y agoSounds like the fixed the “glitch” of excessive traffic against example.com. :)
- d23 6y agoMy AT&T bay area fiber DNS has been garbage since the day we got it. I've had to forcibly update every device we own to Google DNS as well as put another router in between our devices and the AT&T one since AT&T doesn't let you actually change those settings on your device.
- mjrpes 6y agoIt's not just Fiber. It's also V/DSL connections including resold VDSL through a company like Sonic.
- ericseppanen 6y agoI can confirm that on Sonic (resold AT&T) VDSL I am seeing this exact corruption. Nice find! I had been wondering why I had been seeing odd TLS failure messages recently.
- city41 6y agoSonic might be the best avenue to get this fixed. They care about their customers and presumably can talk with AT&T in a much more meaningful way.
- tlrobinson 6y agoData point: I’m on Sonic fiber (resold AT&T fiber) and this script has been running without errors for 5+ minutes.
- deleted 6y ago[deleted]
- Lammy 6y ago> resold AT&T fiber Only in certain areas! Within San Francisco on overhead-cabled blocks (for example) their lines are their own.
- tlrobinson 6y agoOh, good to know, I am indeed in SF.
- darig 6y agoWhy wouldn't the first conclusion be that the example.com service was flipping bits? Load balancing TLS requests to a different front-end pool is very common. Has anyone claimed to see these flipped bits on a domain other than example.com?
- cpach 6y ago”Has anyone claimed to see these flipped bits on a domain other than example.com?“ See e.g. https://gist.github.com/bmastenbrook/14c0e22fc02b95d4a48f82d3ec3123db#gistcomment-3553770 https://gist.github.com/bmastenbrook/14c0e22fc02b95d4a48f82d...
- toast0 6y agoThis is a pretty useful piece of debugging here. It might be worthwhile to try to get EdgeCast involved, as it could be a broken thing between AT&T and EdgeCast. Given that (it seems) only AT&T customers are complaining, and (it seems) it only affects servers on EdgeCast.
- bakul 6y agoI see no errors running the same script from another host that doesn't go through at&t.
- beervirus 6y agoWhy would a TCP connection allow flipped bits to make it through?
- cpach 6y agoHow do you mean? How would it know if a bit in the HTTP payload got flipped?
- sidewndr46 6y agoEach TCP packet has a checksum but it does not catch 100% of all possible bit flips
- garmaine 6y agoAnd some routing hardware have been known to ignore this--meaning they'll forward the packet data along and re-calculate the checksum.
- tyu2 6y agoHardware that forwards packets usually forwards IP packets, it doesn't care about TCP checksums and doesn't recalculate them. TCP stack in your OS is the one that does that. It has to be a special kind of hardware that does much deeper packet inspection (DPI) to recalculate TCP checksums, usually used for spying, throttling, censorship, injecting ads, injecting exploits, etc., but not merely routing/forwarding packets.
- pclmulqdq 6y agoSwitches (L2 devices) recalculate Ethernet CRCs (L2 error detection codes), and routers tend to calculate TCP/UDP checksums (L3 error detection codes) and everything below. I've seen exactly this issue with switches that have bad RAM before, and I assume that they just have a router with bad RAM (and without ECC RAM, which looks embarrassing).
- nodesocket 6y agoIn the above linked gist, how we we know it's a low level router bit flip instead of some code / programming error in their MITM / dns / JavaScript injection tomfoolery?
- cpach 6y agoAFAIU the same issue also affects HTTPS packets, just with different symptoms. E.g. the TLS handshake will fail and stuff like that.
- noncoml 6y agoMuch scarier when you actually see it with your own eyes: $ diff example-* 4c4 < <titde>Example Domain</title> --- > <title>Example Domain</title> 11c11 < backoround-color: #f0f0f2; --- > background-color: #f0f0f2; 23c23 < box-shadow: 2pp 3px 7px 2px rgba(0,0,0,0.02); --- > box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02); 41c41 < <p>This domain is for use in illustrative examples in documents. You may usm this --- > <p>This domain is for use in illustrative examples in documents. You may use this
- baggy_trough 6y agoI get the same. Pretty amazing that the only effect from such garbled data that I noticed were some annoying hangs and bad Twitter links!
- bakul 6y agoThe same here. It is always the 0x08 bit. (p-x, d-l, g-o etc.).
- Scoundreller 6y agoI wonder if there’s some 64 line I/O somewhere with a couple traces bridged together, where: 0 and 0 are still 0 and 0, 1 and 1 are still 1 and 1, but 0 and 1 become 1 and 1 (or 0&0) and 1 and 0 become 1 and 1 (or 0&0). I remember seeing this wackiness when I bridged two address lines on an EEPROM with the tiniest amount of solder. RoHS/tin whiskers strikes again?
- bakul 6y agoThis is unlikely, given how many people are seeing this. And now the problem has gone away for me!
- p1necone 6y agoI was under the impression there was enough layers of checksums once you get to the level of UDP/TCP that these kind of single bit flips should be detected and dropped before you get to read them. What's going on here? Is networking gear not calculating checksums like it should?
- deleted 6y ago[deleted]
- tialaramex 6y ago> presumably because a corrupted packet in a TLS handshake is an error and the connection dies Not just in the handshake, TLS moves these things called TLSPlaintext records (about 16kbytes each), not only in the handshake, but also for all the actual data - and they'll always have integrity protection to ensure bad guys can't change anything. TLS can't know the difference between a bad guy tampering with data and your crappy Internet mangling the data in transit, in either case the TLS protocol design says to "alert" bad_record_mac which your browser or similar software will probably treat as a failed connection, even if it happens mid-way through an HTTP transaction. Because TLS guarantees integrity even if your fiber is a complete shit show, any TLSPlaintext records which do get from one end to the other are guaranteed to be as intended.
- SV_BubbleTime 6y agoHelp me understand, are you saying only HTTP bits are being flipped? Because yea, if a HTTPS bit was flipped the whole packet dies. So is this issue blowing up all sorts of traffic everywhere?
- X-Istence 6y agoYes, all bits are being flipped. TLS connections drop because the message can't be authenticated, HTTP or other plaintext protocols will continue on with bad data. So yes, it is blowing up all sorts of traffic everywhere. You just don't notice when it is plaintext.
- superjan 6y agoThanks for sharing this! I have long suspected that TLS does this. It is great: for many applications preventing bitflips in transit is arguably more important than privacy. Is there an authorative source where this is documented?
- tialaramex 6y agoYou mean besides the RFCs (the most recent RFC 8446 describes TLS 1.3) ? https://tools.ietf.org/html/rfc8446 https://tools.ietf.org/html/rfc8446 If you aren't interested in why this works just the Introduction to the RFC explains the intent, specifically what we care about here is that TLS delivers: "Integrity: Data sent over the channel after establishment cannot be modified by attackers without detection." And further notes "These properties should be true even in the face of an attacker who has complete control of the network".
- Rapzid 6y agoI have ATT fiber in Texas and was having issues recently, probably for a few days, where DNS props would just fail(I use Google's DNS), huge pauses in page loads with the occasional just doesn't. Happened over the long weekend IIRC and was sporadic enough that I didn't look into it further. I thought it had largely cleared, but now I'm wondering about some ongoing page load pauses..
- jlgaddis 6y ago> ... where DNS props would just fail ... What's a "DNS prop"?
- ComputerGuru 6y agoPropagation, but I’m not entirely sure it makes sense in this context.
- Rapzid 6y agoSorry, I meant probes as in the Chrome errors.
- arnaudsm 6y agoImagine if such an issue appeared in a non-techy-area, it could go unnoticed for years.
- enturn 6y agoFor me, TLS errors would happen every so often after we had used all our data allowance and our connection was being shaped to a slow speed. Once the speed returned to normal it was fine. I always thought it was the slow speed but maybe there were bugs in the shaping software.