4 ms·
Substantial ATT Fiber Issues in SF Bay Area?
For the past 24 hours, my fiber connection in Redwood City has been virtually unusable. Gigabit one moment, 0Mbps upload the next.
I’m finding substantial anecdotal evidence of ATT Fiber home internet issues from San Jose to San Francisco on the Sonic forums (for those of us using resold ATT fiber), Nextdoor, heck even AT&T’s own website problem map shows outages all up the Peninsula, but then says my zip (94063) has no problems.
Is this getting coverage anywhere I can follow? Does anybody know what’s actually going on? Feels like substantial congestion breaking connections. Typical web page load takes easily 10 tries before it successfully completes.
- backprop1989 4y agoI’m nearby near Woodside. Same thing for me. Interestingly enough my issue has been upload speeds near 0.01 Mbs, downloads are generally fine. Rumor has it the lines were cut somewhere but I haven’t been able to confirm.
- rconti 4y agoSame symptom. Upload starts off fast and tails off to 0.01. I’m surprised I can’t find a test suite I can run on my system that will hit, say, 50 various websites and report on what’s happening- slow elements, failures on certain types of elements, etc. With the bad upload I suspect many requests on a web page load simply fail to arrive at the remote, but I really don’t know.
- toast0 4y agoI've got a test suite [1] for another issue that might be somewhat helpful. It does probing for path MTU issues; server -> client probing may all happen on a single connection if everything goes well, but client -> server probing is one connection per probe, so that might be enough to trigger some issues; although the probe size search uses some shortcuts to reduce the number of probes. Anyway, if you see anything unusual in the testing, maybe we can get some pcaps and see if anything stands out. Of course, getting that information up the chain to someone at ATT who can fix it, would be up to you :) (email in profile if you see anything useful) [1] http://pmtud.enslaves.us/ http://pmtud.enslaves.us/
- puntium 4y agoseeing the same thing here. Seems to have started sometime on Friday. Were you able to get any confirmation?
- teaearlgraycold 4y agoIn Redwood City on Sonic. fast.com is showing peaks of 5 Gb/s and sustained 2.7 Gb/s down and 1.5 Gb/s up. No issues in the last few days at all. So whatever the issue is it's very localized.
- rconti 4y agoYou’re on native sonic, though, not ATT fiber, correct?
- teaearlgraycold 4y agoNot sure
- ev1 4y agoAFAIK 10G is only Sonic native
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- pwncake 4y agoLikely ATT being resold as Sonic in that area
- syntaxing 4y agoHah reminds me of an issue with AT&T in the Bay Area in 2020 during peak Covid. Somehow someone with a strong background in this narrowed it to a bad RAM where it’s flipping one of the bits which breaks SSL (?) on one of the stations within the tracenet. Everyone tried complaining on twitter to AT&T since telling customer rep did not help at all (went right over their head). This issue persisted for a couple weeks but I moved before they fixed it.
- rconti 4y agoTHANK YOU!! This issue was in the back of my head. Someone even created a utility where you could curl and it would tell you what % of the time the header was getting flipped. I’ll search it out later. This subconscious memory caused me to look at SSL vs HTTP. It seems to be problematic on HTTP too. But oddly it’s not yet been an issue with curl http. But http in a browser is troublesome and inconsistent. EDIT: found it. https://news.ycombinator.com/item?id=25335936 https://news.ycombinator.com/item?id=25335936
- jklm 4y agoFor anyone still running into this - using a VPN seems to fix things but not sure why. Some kind of throttling / connectivity issues on ATT's side?
- toast0 4y agoA lot of issues like this end up being tied to one or two specific routers; in a previous issue, it was found to be ram with a stuck bit in a particular line card connecting ATT and a major CDN (or somebody) in Palo Alto [1]. Especially if it's an issue on a connection to CDNs, your VPN connection is likely to traverse a different path, and the path from the VPN to the destination will also likely traverse a different path. Most of the time, networking issues are at a single link on the path, and forward and reverse paths are different too, so it can be hard to track down. BGP does a great job of quickly routing around totally broken links, but there's not usually any sort of quality or utilization metric included, so a link where there's lots of problems but BGP still works is going to stay up and continue to be utilized. [1] See discussion here, but I didn't see the specific resolution https://news.ycombinator.com/item?id=25335936 https://news.ycombinator.com/item?id=25335936
- jklm 4y agoThat makes sense, thanks for explaining it. Got on the phone with support and they've been receptive and are escalating it.
- rconti 4y agoI have no direct info, but they've updated their outage maps to at least show the issue, and I've heard from numerous sources that they're actually working on something, so that's a plus.
- puntium 4y agoconfirming that using tailscale and an exit node in the google cloud seems to fix the problem for now
- tlefkowicz 4y agoMy Sonic resold AT&T fiber internet has been down for 8 days. For 8 days, Sonic has said that AT&T won’t dispatch a technician while there is an outage in the area. I’ve been a consistent squeaky wheel so Sonic has gotten AT&T to dispatch a repair technician tomorrow morning. We shall see if the tech shows up (typically 50-50), and if they do, if they bring the right equipment and can actually fix our connection. It has been a maddening week+ in SF.