19 ms·

AT&T Wireless traffic shaping apparently making some websites unusable

580 points by acaloiar 4y ago

ivalm 4y ago
How does traffic shaping interact with net neutrality regulations?
- downWidOutaFite 4y ago
  There are no net neutrality regulations. Republicans won that battle at the FCC.
- PuffinBlue 4y ago
  Like a neutrino interacting with regular matter, it would seem.
- CamperBob2 4y ago
  What net neutrality regulations? Yet another Trump triumph that I voted for Biden to fix, that he has shown absolutely no interest in fixing.
- knome 4y ago
  Net neutrality was removed under Ajit Pai's time leading the FCC. Did I miss some post-Pai restoration?
- exabrial 4y ago
  Easy. goto here: https://www.att.com/support https://www.att.com/support log in. cancel. tell them why. goto https://www.t-mobile.com https://www.t-mobile.com port your number and sign up. and you're done! You can't legislate good behavior. You can only hit them in the checkbook. Wireless fortunately doesn't have the cable providers do, where local governments have entrenched players into local monopolies. Switching away is the most powerful thing you can do.
  - pnw 4y ago
    I’d be happy to switch to T Mobile if they could stop leaking their customers data.
    - exabrial 4y ago
      ok, fair point.
  - whatshisface 4y ago
    This seems like a good idea if t-mobile doesn't have traffic shaping, but do they?
    - dublinben 4y ago
      T-Mobile was one of the earliest violators of net-neutrality: https://cyberlaw.stanford.edu/publications/t-mobile-likely-violating-net-neutrality https://cyberlaw.stanford.edu/publications/t-mobile-likely-v...
  - iSnow 4y ago
    >You can't legislate good behavior. Well, you absolutely can. It's just that the US voter regularly votes in politicians that do absolutely nothing to mandate at least fair behaviour.
  - downWidOutaFite 4y ago
    lol as if ATT is going to care if a tiny handful of techies switches because of this.
    - Bud 4y ago
      [dead]
- mullingitover 4y ago
  Wireless providers have never had the same net neutrality situation the landlines providers have.
snapcaster 4y ago
This explains some similar behavior I've had with various apps taking absurdly long to load when the internet connection itself feels fine. Assumed it was just iPhone's gradually getting shittier
- pimlottc 4y ago
  Now that you mention this, it does match my experience on AT&T with my iPhone - plenty of bars, no DNS issues but frustratingly long delays spent staring at layout templates (because progressive enhancement is long dead) waiting for any actual content to arrive.
explorer83 4y ago
I like the testing and documentation. But I also want to see more test results in more areas over a longer period of time.
- acaloiar 4y ago
  While I can't provide more results over a long period of time, yet -- I can provide a pcap file and some details about IPv4/6. I've added an "updates" section at the bottom of the post.
  - explorer83 4y ago
    Excellent! I've bookmarked the page and look forward to updates. Thank you for making this public.
justsomehnguy 4y ago
> Why AT&T wants bandwidth to appear lower than reality is a mystery to me, but I digress For you a speed test is just once in a life time event. For a wireless provider this is a thing what clogs the shared media of a radio channel for quite a lot (modern speed tests try to push 100 or even more MBytes?) and cause a disruption for everyone else on the same channel. I don't know why ATT throttles (if at all, not a customer) that site, but wire and wireless providers are different.
- DanAtC 4y ago
  fast.com was created by Netflix to show if your ISP was throttling Netflix traffic. A lot of mobile services (at least in the US) offer "unlimited streaming" which is really just bandwidth-limited to offer SD-quality streams and offer "HD" (read: less-limited) streaming as a paid add-on.
- dublinben 4y ago
  >For Netflix, part of the message is: If you’ve got a bandwidth problem, don’t blame us — blame your ISP.[0] This coverage from when Netflix launched Fast.com in 2016 alludes to the reason why the results on Fast.com may be slower than on something like Speedtest.net. ISPs like AT&T throttle connections to streaming video providers like Netflix, in order to force a lower quality stream. By offering a speed test from the same infrastructure that delivers your TV show, they are revealing the 'traffic shaping' that is degrading your connection. Sites like Speedtest.net are all excluded from this throttling, so it will always appear that you are getting 'full speed' when testing your connection. [0] https://variety.com/2016/digital/news/netflix-fast-internet-speed-test-1201777840/ https://variety.com/2016/digital/news/netflix-fast-internet-...
  - fjni 4y ago
    I never thought about this much, but what's interesting is that there's a missed opportunity for collaboration here. I think most ISPs throttle known streaming sites (netflix, youtube, etc.) by default for mobile plans. Some even advertise it [1]. And really what the ISPs have noticed is that if you throttle the throughput, the streaming services will switch to a lower quality stream (e.g. 720 or even 480 instead of 1080p) to allow for non-lagging streams. So in some weird way the streaming sites are the ones that provided the tool and the ISPs figured out how to exploit it. The two could have just worked together on a solution. Here they are pointing the finger at each other. ISPs don't want to have default 1080p streams for 4" device screens (which makes sense,) and streaming services don't want their network traffic throttled (which also makes sense.) [1] https://www.t-mobile.com/offers/binge-on-streaming-video https://www.t-mobile.com/offers/binge-on-streaming-video
lopkeny12ko 4y ago
While everyone knows that all wireless carriers are universally terrible, I've actually had a fairly noneventful (read: good) experience with T-Mobile over the last decade. I've never observed evidence of traffic shaping or any other shady business (like capturing NXDOMAIN DNS responses and directing you to a sponsored search page). I've also never observed significant performance degradation from congestion-based prioritization after exceeding the alloted 50 GB or whatever of monthly "normal-priority" data usage.
- jlund-molfese 4y ago
  I have too, mostly… But apart from all the data breaches, I was also able to verify T-Mobile doing this (arbitrary blocking of texts containing innocuous URLs) on my plan. Although they seem to have fixed it now. https://news.ycombinator.com/item?id=29744347 https://news.ycombinator.com/item?id=29744347
- canes123456 4y ago
  Many of their plans downscale video, or at least they used to
  - artogahr 4y ago
    This is interesting, I don't see how this is possible from a technical sense
    - dylan604 4y ago
      If it is an HLS or DASH type of video stream, they could just query the playlist for a smaller video size and throttle that connection so that you're only served the smaller encode vs the full frame highest quality. Pretty simple thing to do really
    - pjc50 4y ago
      Identify (by IP or traffic shape) video traffic, then throttle the stream to the desired bitrate. The viewer's player will usually silently adapt.
thorum 4y ago
I noticed this same issue with my AT&T internet. After some investigation, I found that using a VPN (Virtual Private Network) effectively resolved the problem. It seems that when internet traffic is encrypted and routed through a different server, the traffic shaping technique they employ isn’t able to identify and throttle specific types of data.
- m0dest 4y ago
  When you tunnel/encapsulate your traffic, many variables change at the same time, such as: - maximum transmission unit (MTU) - TCP maximum segment size (MSS) - different DNS responses leading to different edge servers - TCP reordering, which may now occur on the tunnel layer - lost packet retransmission, which may now occur on the tunnel layer - time to live (TTL) hop count on the packet - IPv4 may be used for some connections that were previously using IPv6, as some VPN services are IPv4-only - different peering between the VPN server and the edge server ISP traffic categorization is only one variable.
j1elo 4y ago
This might, in a very "take it with a pinch of salt" way, be actually good for stirring up the incentives of lazy web devs (read: greedy bosses who don't care about technical prowess) to make slimmer websites... if enough people in the developed world have issues accessing them, it might at last have the effect that couldn't be achieved by probably orders of magnitude more people having worse issues from developing countries.
- dsfyu404ed 4y ago
  >(read: greedy bosses who don't care about technical prowess) Oh come on. There's plenty of ICs out there who'll happily do slapdash work because nobody cares if you close tickets well as long as you close them good enough. So what if the fix only lasts 2yr before needing to be refactored for performance. Lord knows if the code will even be in use then.
  - j1elo 4y ago
    Yeah... I was just anticipating the opposite kind of responses: "we web debs would like to make things right but our bosses don't care and expect it to be made fast even if fat"
jskier 4y ago
Ditto, same in a rural area of Wisconsin I'm at a lot, for the past 3 years or so. Like, 3/4 of the Internet is slow through their shaping garbage. What worked for me was using WireGuard to my home network in an urban area (1 gig symmetrical fiber), or, using an ssh ~ socks proxy. Problems no more, until they decide to block those. Support has been awful, best of luck if you go that route. They took us off their unlimited plan without consent, took 6 months to get it back. Also, throttling doesn't actually seem to happen once over their limit on the advertised unlimited* data plan, it's just most of the Internet is awfully slow most of the time going directly through this, ah, ISP, which they claim to be. * Not actually an unlimited plan
- reaperman 4y ago
  I use a VPN to a cheap server in a lesser-known datacenter. Also works for bypassing government level blocks in middle eastern countries. But yes, AT&T throttles a wide diversity of content, even on their very highest tier plan (Unlimited Elite, now named Unlimited Premium). This is advertised as never throttling, no matter how much data you use (all other customers are supposed to get throttled first). Also the throttling happens with mobile hotspot, I have to use VPN on my laptop with many many sites as well, even when I'm inside the "40GB/mo of unthrottled hotspot data". The speeds instantly go way, way up once I hop on VPN.
lotsofpulp 4y ago
> This is an “unlimited” 100Mbit plan with 50GB for Business Fast Track (prioritized) data. Being that I was far below the 50GB of monthly Fast Track data, my data should have had top priority, so I became suspicious. And yet ATT will get away with this fraud or theft of selling something it is not delivering because no one in government wants to hold them accountable.
taf2 4y ago
What’s the best way to simulate this kind of network?
- pyrolistical 4y ago
  Throttle all traffic from cloundfront
maximilianroos 4y ago
> for some reason, AT&T traffic to fast.com is throttled. Why AT&T wants bandwidth to appear lower than reality is a mystery to me My guess is that Netflix is throttled, and given that fast.com is a Netflix site, fast.com is throttled too.
- dghlsakjg 4y ago
  Netflix intentionally hosts fast.com on the same IPs as their video sites. It is a very intentional way for consumers to see if their isp is throttling Netflix.
- oh_sigh 4y ago
  Since almost everyone usingfast.com on an AT&t network also use AT&t DNS, I'm surprised they don't do something funky with that, for example throttling IPs that did a recent Netflix.com lookup, but not IPs that did a recent fast.com lookup
  - jrockway 4y ago
    Because if this was a workaround to get better Netflix quality, then there would be 1000 blogspam articles about it. It's too good to be true ("visit fast.com to make Netflix load faster"), and the Internet loves that kind of thing.
anecdotal1 4y ago
Get Calyx. Unlimited data, no throttling service via T-Mobile. Grandfathered contract from Clearwire https://members.calyxinstitute.org/enroll/membership https://members.calyxinstitute.org/enroll/membership I've done nearly gigabit symmetrical over 5G with it before and several Terabytes in a month. No issues.
crazygringo 4y ago
But why would AT&T be throttling Cloudfront requests specifically? That's not traffic shaping, it's just dumb. Traffic shaping is slowing down types of traffic like torrents or video in order to prioritize regular sites. But this is degrading regular sites that are usually what benefit from traffic shaping if there's overall network congestion. I'm not questioning anything the author reports, it all seems extremely well documented. I am questioning what AT&T is trying to accomplish here though. It makes no sense, unless they're in some kind of negotiation with Cloudfront right now, punishing them until they pay up or something? But that doesn't explain why it would only happen on a data-only plan. I'm mystified.
- fjni 4y ago
  I think the era of traffic shaping as you describe it is no longer. Most connections are encrypted these days, a good thing. But that also means, me as an ISP in the middle, I only see the layer 4 packet. So I know where a packet is coming from and where it's going (and ttl and some other not really helpful data for this purpose.) I don't even know the port. So I'm left with doing (dumb) traffic shaping by destination and target. If I had to guess, I'd say that they incorrectly thought that some specific IP address (range) serves predominantly one type of data. So they throttle by the only data point they have, destination ip, and the collateral damage is everything else hosted on that ip address.
  - kaszanka 4y ago
    Wait, why wouldn't they know the port? I'm not aware of any cases where the TCP header itself is encrypted.
    - fjni 4y ago
      You are correct. The port is part of the tcp header and my statement above is incorrect!
    - kevincox 4y ago
      99% of consumer traffic is on 443 these days (citation needed)
fjni 4y ago
> I already knew from previous experience that for some reason, AT&T traffic to fast.com is throttled. Why AT&T wants bandwidth to appear lower than reality is a mystery to me, but I digress This I think is because they throttle ip addresses for known video streaming sites. That is one of (the only reliable) ways an ISP can get the streaming provider to drop the stream to a lower quality one by default. Since fast.com is a Netflix ip, and the isp can’t distinguish whether it’s video that is being transferred or a file to measure throughout, the speed test gets caught up in it. Said the other way around: fast.com is great to see actual throughput from Netflix as opposed to some fake throughout from a dedicated speed test site for the exact same reason.
- paulddraper 4y ago
  Indeed that's the point of fast.com.... To measure the speed from Netflix servers
- avianlyric 4y ago
  > Said the other way around: fast.com is great to see actual throughput from Netflix as opposed to some fake throughout from a dedicated speed test site for the exact same reason. Yup. This is pretty much the reason Netflix created fast.com. They wanted a speed test service that couldn’t be gamed by ISPs. Many ISPs will prioritise traffic to know speed test services (like Ookla’s speedtest.net), making their services appear faster than they’re under more normal usage. By placing fast.com on Netflix IPs, ISPs either have to prioritise all Netflix traffic (which they’re very unlikely to do), or accept that fast.com is going to provide a more realistic measurement of their performance.
  - neurostimulant 4y ago
    Now that my ISP bundles Netflix subscription into their internet plans, access to netflix and fast.com now practically saturate the fiber link, while before it was outright blocked. Hooray for no net neutrality I guess. Another fun part: when netflix IPs was blocked by this ISP, it's pretty much impossible to use netflix because the only way to get around the block was to use VPN, but netflix itself blocks VPN access.
esalman 4y ago
Fun story: I work remotely for an org located in the Bay area. Few months ago I get a call early morning from the IT security person. Apparently someone had been trying to RDP into my work laptop, over a thousand attempts per day for last couple of days. What changed before the last couple of days? My laptop was frequently dropping WiFi connection- a lot of times in the middle of zoom meetings. We use a router provided by AT&T. My wife complained of the same thing, so I thought I'll ping their customer support. The customer support person proceeded to tell us they wanted to reconfigure our router to enable "5G". After some bewilderment we realized they were talking about 2.4G vs. 5G WiFi. And they will do it remotely. Ok fine go ahead, we said, and forgot about it.. until I got the call from my office IT security. Apparently the AT&T support person left our router in passthrough mode. According to their K/B, "Placing a device in passthrough mode will remove firewall protection provided by the AT&T gateway." I reset my router to default settings, and got myself a 100ft Ethernet cable to fix the issues.
- LoganDark 4y ago
  > Placing a device in passthrough mode will remove firewall protection provided by the AT&T gateway Did... did they seriously turn the router into a DMZ*, without your consent? Where every port of your computer is just open to the internet? That's scary. *that's what it was called on my router, the "forward every single port to one device" mode. Not sure if that is the correct term for it.
  - esalman 4y ago
    Yeah. We checked with censys.io, port 3389 and 8080 were open as publicly available service on the router.
    - jandrese 3y ago
      Oh yeah, if you leave RDP open to the internet the scanbots will be all over that like white on rice. If only we would finish the switch to IPv6 we could pull the rug out from under those bots.
  - 4y ago
rsaxvc 4y ago
When I had AT&T VDSL they had weak peering. Some traffic that should have gone across town would routinely get routed several states away before coming back to town, because they hadn't peered with the local exchanges. Some traffic was ok. When I switched to an ISP with local peering, ping times between my home and my server downtown dropped to a few milliseconds. It's like being on my LAN. If this is still an issue for AT&T, traceroute+reverse DNS of each router may indicate it. This poor peering isn't directly a throughput problem, but really increases the chance of being routed through a congested link, which you could sometimes guestimate from ping times and peeringdb.
- Scoundreller 4y ago
  Same issue in Canada: our big ISPs historically refuse to peer freely at our domestic IXs like TORIX. So smaller ISP and DCs end up peering with them in the USA and your packets would make circuitous cross-border round trips to go a few km. Some national players do have ports at domestic IXs, but only as backup links or for negotiated (paid?) access. The hazards of ISPs also being backbone providers. (It looks like Rogers has gotten better with this behaviour)
  - ShroudedNight 4y ago
    As far as I can tell, Bell holds its Canadian customers hostage to force peers to pay them for what should be mutually beneficial peering. It does not participate in _ANY_ of the Canadian internet exchanges. Even Rogers connects to TORIX. https://bgp.he.net/AS577#_ix https://bgp.he.net/AS577#_ix
    - Scoundreller 4y ago
      I know Rogers for a while had 'conditional' peering at Torix, but doing some traceroutes now, more routes are going through Torix than I recall in the past. Seems like they've lightened up over the past couple years. Too bad the indie ISP market has been decimated. Ebox & Distributel --> Bell. Start --> Telus. Oxio --> Cogeco. VMedia --> Quebecor (Videotron). Mostly only Teksavvy left standing Indeed with Bell & Telus, you could only peer freely on US soil: https://www.peeringdb.com/net/1550 https://www.peeringdb.com/net/1550 https://www.peeringdb.com/net/76 https://www.peeringdb.com/net/76
chrsjxn 4y ago
You've got to love a website that makes text selection invisible by default. I'd probably double check assumptions on the cloudfront issues. Switching carriers or adding VPN might connect you to a different edge node. Some web assets being throttled for specific AT&T accounts seems a little too targeted to just be traffic shaping. I'd expect them to throttle traffic for all users, like they do with the speed tests.
- sgtfrankieboy 4y ago
  >You've got to love a website that makes text selection invisible by default. First time seeing it being invisible, mostly they just disable it. Terribly annoying practice. I often select text while reading articles.
  - chrsjxn 4y ago
    Same here! I think it helps me read faster online, but it's probably just a fidget-y habit.
    - zo1 4y ago
      You guys are not alone, I do the same. It is indeed a fidgety habit, but also, I do it as it helps me see text more clearly due to my astigmatism.
Tarball10 4y ago
AT&T uses different APNs for regular phone plans and data-only hotspot plans. It's very possible that the phone traffic is being routed completely differently than the hotspot traffic, with congestion at a peering point occurring on the hotspot. AT&T tends to be known for having poor/congested peering. You could try changing the DNS server on your hotspot to a different public resolver like 1.1.1.1 or 8.8.8.8. If CloudFront is using DNS based geolocation to route you to the nearest data center, a different DNS server may get you routed around the issue to a different data center.
- acaloiar 4y ago
  Good tip. I just tried with both and wasn't routed to a faster point of presence with either. I'll add a traceroute from the phone and one from a device connected to the LTE router to the updates section next. [update] Traceroutes added
  - Tarball10 4y ago
    It's definitely odd that you're getting routed across the country to an east coast pop. Ideally in your location you'd be routed to a west or central pop. Perhaps a bad combination of Cloudfront getting geolocation wrong for your IP range, and poor performance of at&t's network for that specific source-destination combination.
    - acaloiar 4y ago
      I found that odd as well, and my iPhone hotspot does get routed through a different list of hosts as you can see in the traceroute. I suppose it should be noted. The iphone antenna and router antennas are no more than a meter from each other. They're unlikely to be hitting different towers.
      - catminou 4y ago
        What are you using for DNS?
  - 4y ago
getpost 4y ago
"Modernizing its search engine has become an obsession at Google".... No doubt it can be modernized, but just making Google search as good as it used to be would be helpful. And why not better than it used to be with the traditional interface? That would still be worth doing even if Google adds on a superb AI. Search is the proverbial goose that laid the golden egg and Google is killing it, and I don't mean "killing" in the contemporary ironic usage.
- Flammy 4y ago
  Wrong thread?
- ikurei 4y ago
  I think you might have written this comment on the wrong post
- delecti 4y ago
  Google search can't be as good as it used to, because the web isn't as "clean" (pure? idealistic? honest? legitimate?) as it used to be. People making websites with the goal of appearing higher in Google's results means Google has to work on countermeasures. That sets off a game of cat-and-mouse, and means actual good results are caught in the crossfire.
- thatguy0900 4y ago
  Is it possible to return Google to pre-seo blackhat days? I think the golden egg has been poisoned, even if Google tried it's best
themagician 4y ago
A decade long fight about Net Neutrality in the public, and the ISPs just went and did what they were always going to do anyway in the background. No surprise. No news. No fanfare. Just, “Here’s your broken internet where every websites loads at a different speed. Get Unlimited* access with a 35GB limit before more limits kick in. Some websites are unusable. Enjoy. *Some limits apply. See T&C for details.”
- Scoundreller 4y ago
  But OP is paying for the first 50gb/month to be a "business" unthrottled connection on their unlimited connection!
- twoodfin 4y ago
  Net Neutrality was never going to apply to mobile networks under any FCC proposal.
ekosz 4y ago
We've been hearing complaints from AT&T users for months now that our website (https://vstream.com https://vstream.com) doesn't work for them. I've tried hunting down the reasons & tried getting in touch with AT&T to no avail. This seems like it could be the very thing that's causing the issues for our users. Now I just need to figure out what to tell our users that are having these issues...
- acaloiar 4y ago
  I believe I can confirm that vstream.com is affected by this. It fails to fully load in a reasonable amount of time unless I connect via VPN, Verizon, or my (phone) hotspot. Update: For what it's worth, the "above the fold" portion of the content loads down to "All Debuts". After that, there's a long blank space with a Loading indicator near the bottom. Eventually it loaded. I didn't keep track of how long, but it was almost certainly longer than 1 minute.
- Tarball10 4y ago
  This appears to load fine for me on both AT&T fiber and AT&T cellular. It looks like all content on that site is being loaded from Cloudflare, so I'm curious if those users would be having issues with other Cloudflare protected sites as well?
Animats 4y ago
Remember how 5G was supposed to support "the metaverse"?[1] I've been working on a high performance metaverse client. All Rust, all multi-threaded. Designed to max out a gamer PC. Supports Second Life and Open Simulator. Second Life had a reputation for being sluggish. Fixing that. I'm pulling content from the servers at 200Mb/s, sustained. 400Mb/s in tests, but don't need to go that fast. The servers (AWS front-ended by Akamai caches) can handle that just fine. Gigabit fiber can handle that just fine. The 3D world appears in high detail in seconds. Looks like an AAA game title. Not like low-rez Meta Horizon or Decentraland. 5G can handle that, right? Says so right here in the promotional materials.Verizon: [2] AT&T: [3] The carriers said "unlimited", right? So they can't complain if you're downloading 100GB per hour. [1] https://www.forbes.com/sites/michaelgale/2022/05/24/how-5-and-even-6g-will-make-the-metaverse-a-true-reality/?sh=5ed4d49c16f0 https://www.forbes.com/sites/michaelgale/2022/05/24/how-5-an... [2] https://www.ericsson.com/en/blog/2022/4/why-metaverse-needs-5g https://www.ericsson.com/en/blog/2022/4/why-metaverse-needs-... [3] https://www.verizon.com/about/news/5g-makes-metaverse-real https://www.verizon.com/about/news/5g-makes-metaverse-real [4] https://www.xrtoday.com/event-news/5g-networks-crucial-to-metaverse-future-att-says/ https://www.xrtoday.com/event-news/5g-networks-crucial-to-me...
- dboreham 4y ago
  Unlimited is a term of art that means "limited".
  - DangitBobby 4y ago
    Can we just start calling fraud by its own name?
    - doublerabbit 4y ago
      I'll order one criminality with extra politicians and one large diet-fraud-lite(tm) with extra ice. Just charge it to my offshore. Cheers
  - xeromal 4y ago
    Yup. Buffets have been that way forever. No one is actually allowed to eat unlimited.
0887437208577 4y ago
[flagged]
Bluecobra 4y ago
Makes me wonder if it’s just something as simple as suboptimal routing and the path it’s taking is through a hop experiencing packet loss or some other issue further upstream. I would think that there’s a higher chance of AT&T being incompetent then doing something nefarious.
lordgrenville 4y ago
> While wget is not my goto for command line HTTP fetching It is for me. Is there something better I'm not aware of?
- mijoharas 4y ago
  Curl tends to be the standard.
  - acaloiar 4y ago
    It's what I reach for first. `wget` made for a better gif, though :)
kevin_nisbet 4y ago
Taking a quick 5 minute look at the packet capture attached to the post, it looks to me like this is likely traffic shaping. While it's always difficult to be 100% sure from just the client side capture, the capture looks relatively clean of errors, and the amount of data in flight doesn't appear to even approach the advertised window. We're getting some merging of segments likely from a segment offload, but I doubt that's throwing off the results. So there's a good chance there's a shaper letting through about 320kbit/s as it's relatively even throughout the capture.
- acaloiar 4y ago
  Thanks for the feedback. I was hoping to see more of this sort of analysis from the HN crowd :) There's one other person from HN doing some analysis with me via email. Maybe something will come of it. You both came to the same conclusion based on the pcap data. They're also analyzing the iphone tether pcap data I just provided in an update.
  - kevin_nisbet 4y ago
    You're welcome. I took a very quick look at the iPhone capture. I don't see any notable issue (which is expected since this is "working" behavour), and the main noticeable item in the comparison is an apparent lower latency to the server. This appears to match the speculation I saw in the updates as well. One thing you might be able to try, if you're able to change the APN on the hotspot, is to see if you're able to connect with a different APN from the consumer side. This is pure speculation, but what some carriers do is use a different APN for their business services, that will route to enterprise only packet gateways. Some carriers may not necessarily disable access to the consumer APNs though that might go through a different P-GW that's closer to you. The iPhone looks like it's using a path about 120ms shorter than the hotspot. And just by chance, going through a different P-GW you might hit different traffic shaping equipment or configuration. Edit: although be aware that the carrier could also charge different APNs separately. I have no idea what AT&T does.
    - acaloiar 4y ago
varenc 4y ago
One nit of the author's analysis: The Strava javascript file used for speed tests is 1.68MB uncompressed. But in a browser and most all other situations it should be requested with `Accept-Encoding: ...`. In Chrome, Strava responds with a gzipped response that's 463kB in size. This doesn't really matter for the CLI speed tests, when its requested without compression, but it does mean that the speed test performance won't correspond with the actual in-browser performance and the traffic shaping may not be comparable when the request is made without compression. Adding `--compression=gzip` to the wget command will fix this. To quickly show the difference: $ curl -s 'https://web-assets.strava.com/assets/federated/find-and-invite-friends/827.js' --compressed -w '%{size_download}\n' -o /dev/null 462525 $ curl -s 'https://web-assets.strava.com/assets/federated/find-and-invite-friends/827.js' -w '%{size_download}\n' -o /dev/null 1759662
- acaloiar 4y ago
  Fair nit. Although this is why I chose to focus on transfer rates instead of time. But consider this. When I load Strava's dashboard, open the Network tab and search for "cloudfront", these are the metrics: > 374 requests 15.80 MB / 7.07 MB transferred Finish: 4.89 min This is not a good time, and no amount of compression is going to help the situation.
  - varenc 4y ago
    Oh I absolutely agree! That's some sketchy stuff. Didn't mean to sound like I was discounting the rest of your experience.
ehPReth 4y ago
Bell in Canada is doing the same with their new 'unlimited' (vs the previous generation 'unlimited') and their cheaper data plans. Even if you have any amount of 'fast data' left you're throttled to SD speeds for streaming video unless you pay for the higher tier of their plans. Data isn't equal any more. Their higher plans say "HD video is up to 1080p" so I suppose no 1440p/4K/60fps/etc either. It's honestly fucking infuriating but there's nothing that can be done except use a VPN I guess. I'm on the previous generation so "data is data" and they don't throttle me yet but if I ever change plans it's there. I'm not totally sure if the other carriers do (or their 'competition' in forms of MVNOs that they themselves own) but they tend to have a habit of copying each other, at least after a little delay. https://www.bell.ca/Mobility/Cell_phone_plans/Unlimited-plans https://www.bell.ca/Mobility/Cell_phone_plans/Unlimited-plan... (select your local area or just choose Ontario) bonus fun bs: every so often when you try to log into the web portal they try to trick you into letting them build a profile off your browsing/usage data so they can make even more money off of you. a popup with 'advertising is a reality in today's world' comes up with the nice attractive 'get this out of my face so I can do things' blue button being 'yes please opt me in'. shady dark pattern! back when, they used to opt people in by default and make you explicitly opt out... but then the regulators said hey that's illegal. so now they resort to stuff like that to get the numbers back up. here's some of the blurb if anyone is interested: "Advertising is a reality in today’s world, and people find that they receive ads that are irrelevant to them. With our tailored marketing program, Bell will work to ensure that the offers participants receive when using our services may be more relevant, rather than random marketing ads. In other words, participants won’t see more ads, just more relevant ads." full text: https://pastebin.com/ESskYEUy https://pastebin.com/ESskYEUy it's honestly super gross what they collect and how they try to trick people into agreeing.
extr0pian 4y ago
Wireless carriers have been pulling stunts like this over the last several years. For example, T-Mobile blocks all SMS messages with links that have a .xyz TLD. There's no indication that it was blocked by neither the sender or the receiver. I assume this was done to prevent spam and sketchy websites, but personally speaking I've seen way more sketchy .com sites and more spam coming from gmail accounts.
alexpotato 4y ago
Back in the early 2000's, I worked for a small company that was hired by the law firm representing one of the big cable companies (let's call them Company A). Company A was suing another big cable company (Company X) b/c A claimed that X was throttling customers without telling them and advertising what were in effect unthrottled bandwidth numbers. In order to prove this, the company I worked for went out and hired people to get cable modem subscriptions for Company X, install desktop that were locked down so the people couldn't use the desktops themselves and then collected ping and HTTP request data for months. My job was to take that data and analyze it for throttling patterns. This was my first job out of grad school and involved lots of: - data cleaning and organizing using Perl - analyzing statistics using Excel - creating charts that could then be shown to the lawyers and in turn could be used in court I didn't know much about databases or stats packages in Perl which, in hindsight, would have made my life easier. I still remember being on the call with the lawyers and walking them through the data that proved that Company X was indeed throttling bandwidth at peak usage times. The one lawyer on the conference call said "Wow! This data is amazing. You guys are now basically national experts in this." I was both: - flattered that someone would call me a national expert in something - scared shitless that I would end up being cross examined about how we processed this data Turns out that the court case never happened and they settled out of court. The reason I mention this story is that it's FASCINATING to me that this is still occurring today. I guess some things never do change.
- Scalene2 4y ago
  A big part of the reason that things didn't change is because that lawsuit and others like it ended in a settlement. If this behaviour and data was established as part of the judicial/public record, it would have been much easier to address.
- MacroChip 4y ago
  Did they wait to sue until after you had your data ready? Or were they gonna sue regardless?
- nine_k 4y ago
  Businesses, like teenagers, always keep probing the limits of what's allowed or tolerated. As long as it's an out-of-court settlement for a moderate sum, it's just a cost of doing business, a cost of R&D of sorts, only in the legal area. (Similar with taxes.)
kazinator 4y ago
> Strava’s javascript resources clocked in at 1.68MB That looks like the problem to me. Throttle the fuck out of that shit.
kup0 4y ago
I had this problem with Boost Mobile for phones. Apple Store and various podcast sources seemed ultra-throttled and often wouldn't even want to load at all. Eventually switched to Mint and problem went away. Maybe a fluke, but I wouldn't be surprised if it was some shaping BS
initramfs 4y ago
wow, 40KB/s is approx the speed I get after I use up my 4G LTE data on Unreal Mobile. I get unlimited 2G but it's also the cheapest plan in the U.S.
albert_e 4y ago
So not just traffic shaping but also pretty a heavy moderation of discussion on that topic? Interested to see what justification AT&T comes up with for suppressing these posts on their forum and marking them as 'private' without any explanation to the original poster at least.
rendall 4y ago
> "After switching back and re-authenticating with the broadband APN, I’m no longer being throttled." If the author is reading this, I recommend putting an update at the head of your article with this. It will be kinder to your readers.
- acaloiar 4y ago
  Update #6 has been at the end of the post since I posted that here on HN.
catminou 4y ago
Are you near Utah? The fact you are being routed to a cloudfront pop in IAD is kinda broken. What are you using for your DNS resolver?
- acaloiar 4y ago
  Durnago, CO is close to Utah, so that's not a surprise. I've used `1.1.1.1`, `8.8.8.8`, and others during this debacle, which I can't recall. Here are some query responses that I received during the incident, but I can't tell you which resolvers were in use during the queries. ``` host web-assets.strava.com web-assets.strava.com is an alias for dgpcy4fyk1eox.cloudfront.net. dgpcy4fyk1eox.cloudfront.net has address 99.84.208.46 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.56 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.10 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.115 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:8000:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:e600:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:c600:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:b600:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:d000:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:6400:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:f000:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:400:17:4613:2840:93a1 ``` ``` host web-assets.strava.com web-assets.strava.com is an alias for dgpcy4fyk1eox.cloudfront.net. dgpcy4fyk1eox.cloudfront.net has address 99.84.208.56 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.10 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.115 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.46 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:4c00:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:ae00:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:c000:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:7200:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:1000:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:b800:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:9400:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:9e00:17:4613:2840:93a1 ``` The only difference I've seen is order, which is expected. [edit] I actually spot some different IPv6 hosts above, after having taken a closer look.
punnerud 4y ago
Just found that Google Mesh Wifi does this at my home. 25Mbit/s and 900ms. Directly with cable 800/800Mbit and 4ms (full speed). Deactivate priority for Zoom, and now I have full speed.
vrglvrglvrgl 4y ago
[dead]
nashashmi 4y ago
I think it is the latency. 130 ms will cause websites with lots of little files across multiple https servers to load slowly. If the resources was bundled into a single file, it would load faster. (My corporate network used to have similarly exaggerated latency on a bad old wire.)
- Avamander 4y ago
  This shouldn't matter much with H2. "Long fat pipe" is a well-known issue with a bunch of solutions.
- acaloiar 4y ago
  What you say is true, but 130ms latency is quite normal for even very healthy LTE connections. It's simply a high latency environment.
naasking 4y ago
The net neutrality battle is ongoing.
tomhallett 4y ago
Adriano, In the article you mention "Cloudflare" a few times, but I think you mean "Cloudfront". For example: "web-assets.strava.com resolves to Cloudflare, see the writeup on my site for details" I'm assuming this was a typo based on the names sounding so similar and both being CDN's. Is that correct?

580 points by acaloiar 4y ago

ivalm 4y ago

How does traffic shaping interact with net neutrality regulations?

downWidOutaFite 4y ago
There are no net neutrality regulations. Republicans won that battle at the FCC.
PuffinBlue 4y ago
Like a neutrino interacting with regular matter, it would seem.
CamperBob2 4y ago
What net neutrality regulations? Yet another Trump triumph that I voted for Biden to fix, that he has shown absolutely no interest in fixing.
knome 4y ago
Net neutrality was removed under Ajit Pai's time leading the FCC. Did I miss some post-Pai restoration?
exabrial 4y ago
Easy. goto here: https://www.att.com/support https://www.att.com/support log in. cancel. tell them why. goto https://www.t-mobile.com https://www.t-mobile.com port your number and sign up. and you're done! You can't legislate good behavior. You can only hit them in the checkbook. Wireless fortunately doesn't have the cable providers do, where local governments have entrenched players into local monopolies. Switching away is the most powerful thing you can do.
- pnw 4y ago
  I’d be happy to switch to T Mobile if they could stop leaking their customers data.
  - exabrial 4y ago
    ok, fair point.
- whatshisface 4y ago
  This seems like a good idea if t-mobile doesn't have traffic shaping, but do they?
  - dublinben 4y ago
    T-Mobile was one of the earliest violators of net-neutrality: https://cyberlaw.stanford.edu/publications/t-mobile-likely-violating-net-neutrality https://cyberlaw.stanford.edu/publications/t-mobile-likely-v...
- iSnow 4y ago
  >You can't legislate good behavior. Well, you absolutely can. It's just that the US voter regularly votes in politicians that do absolutely nothing to mandate at least fair behaviour.
- downWidOutaFite 4y ago
  lol as if ATT is going to care if a tiny handful of techies switches because of this.
  - Bud 4y ago
    [dead]
mullingitover 4y ago
Wireless providers have never had the same net neutrality situation the landlines providers have.

snapcaster 4y ago

This explains some similar behavior I've had with various apps taking absurdly long to load when the internet connection itself feels fine. Assumed it was just iPhone's gradually getting shittier

pimlottc 4y ago
Now that you mention this, it does match my experience on AT&T with my iPhone - plenty of bars, no DNS issues but frustratingly long delays spent staring at layout templates (because progressive enhancement is long dead) waiting for any actual content to arrive.

explorer83 4y ago

I like the testing and documentation. But I also want to see more test results in more areas over a longer period of time.

acaloiar 4y ago
While I can't provide more results over a long period of time, yet -- I can provide a pcap file and some details about IPv4/6. I've added an "updates" section at the bottom of the post.
- explorer83 4y ago
  Excellent! I've bookmarked the page and look forward to updates. Thank you for making this public.

justsomehnguy 4y ago

> Why AT&T wants bandwidth to appear lower than reality is a mystery to me, but I digress For you a speed test is just once in a life time event. For a wireless provider this is a thing what clogs the shared media of a radio channel for quite a lot (modern speed tests try to push 100 or even more MBytes?) and cause a disruption for everyone else on the same channel. I don't know why ATT throttles (if at all, not a customer) that site, but wire and wireless providers are different.

DanAtC 4y ago
fast.com was created by Netflix to show if your ISP was throttling Netflix traffic. A lot of mobile services (at least in the US) offer "unlimited streaming" which is really just bandwidth-limited to offer SD-quality streams and offer "HD" (read: less-limited) streaming as a paid add-on.
dublinben 4y ago
>For Netflix, part of the message is: If you’ve got a bandwidth problem, don’t blame us — blame your ISP.[0] This coverage from when Netflix launched Fast.com in 2016 alludes to the reason why the results on Fast.com may be slower than on something like Speedtest.net. ISPs like AT&T throttle connections to streaming video providers like Netflix, in order to force a lower quality stream. By offering a speed test from the same infrastructure that delivers your TV show, they are revealing the 'traffic shaping' that is degrading your connection. Sites like Speedtest.net are all excluded from this throttling, so it will always appear that you are getting 'full speed' when testing your connection. [0] https://variety.com/2016/digital/news/netflix-fast-internet-speed-test-1201777840/ https://variety.com/2016/digital/news/netflix-fast-internet-...
- fjni 4y ago
  I never thought about this much, but what's interesting is that there's a missed opportunity for collaboration here. I think most ISPs throttle known streaming sites (netflix, youtube, etc.) by default for mobile plans. Some even advertise it [1]. And really what the ISPs have noticed is that if you throttle the throughput, the streaming services will switch to a lower quality stream (e.g. 720 or even 480 instead of 1080p) to allow for non-lagging streams. So in some weird way the streaming sites are the ones that provided the tool and the ISPs figured out how to exploit it. The two could have just worked together on a solution. Here they are pointing the finger at each other. ISPs don't want to have default 1080p streams for 4" device screens (which makes sense,) and streaming services don't want their network traffic throttled (which also makes sense.) [1] https://www.t-mobile.com/offers/binge-on-streaming-video https://www.t-mobile.com/offers/binge-on-streaming-video

lopkeny12ko 4y ago

While everyone knows that all wireless carriers are universally terrible, I've actually had a fairly noneventful (read: good) experience with T-Mobile over the last decade. I've never observed evidence of traffic shaping or any other shady business (like capturing NXDOMAIN DNS responses and directing you to a sponsored search page). I've also never observed significant performance degradation from congestion-based prioritization after exceeding the alloted 50 GB or whatever of monthly "normal-priority" data usage.

jlund-molfese 4y ago
I have too, mostly… But apart from all the data breaches, I was also able to verify T-Mobile doing this (arbitrary blocking of texts containing innocuous URLs) on my plan. Although they seem to have fixed it now. https://news.ycombinator.com/item?id=29744347 https://news.ycombinator.com/item?id=29744347
canes123456 4y ago
Many of their plans downscale video, or at least they used to
- artogahr 4y ago
  This is interesting, I don't see how this is possible from a technical sense
  - dylan604 4y ago
    If it is an HLS or DASH type of video stream, they could just query the playlist for a smaller video size and throttle that connection so that you're only served the smaller encode vs the full frame highest quality. Pretty simple thing to do really
  - pjc50 4y ago
    Identify (by IP or traffic shape) video traffic, then throttle the stream to the desired bitrate. The viewer's player will usually silently adapt.

thorum 4y ago

I noticed this same issue with my AT&T internet. After some investigation, I found that using a VPN (Virtual Private Network) effectively resolved the problem. It seems that when internet traffic is encrypted and routed through a different server, the traffic shaping technique they employ isn’t able to identify and throttle specific types of data.

m0dest 4y ago
When you tunnel/encapsulate your traffic, many variables change at the same time, such as: - maximum transmission unit (MTU) - TCP maximum segment size (MSS) - different DNS responses leading to different edge servers - TCP reordering, which may now occur on the tunnel layer - lost packet retransmission, which may now occur on the tunnel layer - time to live (TTL) hop count on the packet - IPv4 may be used for some connections that were previously using IPv6, as some VPN services are IPv4-only - different peering between the VPN server and the edge server ISP traffic categorization is only one variable.

j1elo 4y ago

This might, in a very "take it with a pinch of salt" way, be actually good for stirring up the incentives of lazy web devs (read: greedy bosses who don't care about technical prowess) to make slimmer websites... if enough people in the developed world have issues accessing them, it might at last have the effect that couldn't be achieved by probably orders of magnitude more people having worse issues from developing countries.

dsfyu404ed 4y ago
>(read: greedy bosses who don't care about technical prowess) Oh come on. There's plenty of ICs out there who'll happily do slapdash work because nobody cares if you close tickets well as long as you close them good enough. So what if the fix only lasts 2yr before needing to be refactored for performance. Lord knows if the code will even be in use then.
- j1elo 4y ago
  Yeah... I was just anticipating the opposite kind of responses: "we web debs would like to make things right but our bosses don't care and expect it to be made fast even if fat"

jskier 4y ago

Ditto, same in a rural area of Wisconsin I'm at a lot, for the past 3 years or so. Like, 3/4 of the Internet is slow through their shaping garbage. What worked for me was using WireGuard to my home network in an urban area (1 gig symmetrical fiber), or, using an ssh ~ socks proxy. Problems no more, until they decide to block those. Support has been awful, best of luck if you go that route. They took us off their unlimited plan without consent, took 6 months to get it back. Also, throttling doesn't actually seem to happen once over their limit on the advertised unlimited* data plan, it's just most of the Internet is awfully slow most of the time going directly through this, ah, ISP, which they claim to be. * Not actually an unlimited plan

reaperman 4y ago
I use a VPN to a cheap server in a lesser-known datacenter. Also works for bypassing government level blocks in middle eastern countries. But yes, AT&T throttles a wide diversity of content, even on their very highest tier plan (Unlimited Elite, now named Unlimited Premium). This is advertised as never throttling, no matter how much data you use (all other customers are supposed to get throttled first). Also the throttling happens with mobile hotspot, I have to use VPN on my laptop with many many sites as well, even when I'm inside the "40GB/mo of unthrottled hotspot data". The speeds instantly go way, way up once I hop on VPN.

lotsofpulp 4y ago

> This is an “unlimited” 100Mbit plan with 50GB for Business Fast Track (prioritized) data. Being that I was far below the 50GB of monthly Fast Track data, my data should have had top priority, so I became suspicious. And yet ATT will get away with this fraud or theft of selling something it is not delivering because no one in government wants to hold them accountable.

taf2 4y ago

What’s the best way to simulate this kind of network?

pyrolistical 4y ago
Throttle all traffic from cloundfront

maximilianroos 4y ago

> for some reason, AT&T traffic to fast.com is throttled. Why AT&T wants bandwidth to appear lower than reality is a mystery to me My guess is that Netflix is throttled, and given that fast.com is a Netflix site, fast.com is throttled too.

dghlsakjg 4y ago
Netflix intentionally hosts fast.com on the same IPs as their video sites. It is a very intentional way for consumers to see if their isp is throttling Netflix.
oh_sigh 4y ago
Since almost everyone usingfast.com on an AT&t network also use AT&t DNS, I'm surprised they don't do something funky with that, for example throttling IPs that did a recent Netflix.com lookup, but not IPs that did a recent fast.com lookup
- jrockway 4y ago
  Because if this was a workaround to get better Netflix quality, then there would be 1000 blogspam articles about it. It's too good to be true ("visit fast.com to make Netflix load faster"), and the Internet loves that kind of thing.

anecdotal1 4y ago

Get Calyx. Unlimited data, no throttling service via T-Mobile. Grandfathered contract from Clearwire https://members.calyxinstitute.org/enroll/membership https://members.calyxinstitute.org/enroll/membership I've done nearly gigabit symmetrical over 5G with it before and several Terabytes in a month. No issues.

crazygringo 4y ago

But why would AT&T be throttling Cloudfront requests specifically? That's not traffic shaping, it's just dumb. Traffic shaping is slowing down types of traffic like torrents or video in order to prioritize regular sites. But this is degrading regular sites that are usually what benefit from traffic shaping if there's overall network congestion. I'm not questioning anything the author reports, it all seems extremely well documented. I am questioning what AT&T is trying to accomplish here though. It makes no sense, unless they're in some kind of negotiation with Cloudfront right now, punishing them until they pay up or something? But that doesn't explain why it would only happen on a data-only plan. I'm mystified.

fjni 4y ago
I think the era of traffic shaping as you describe it is no longer. Most connections are encrypted these days, a good thing. But that also means, me as an ISP in the middle, I only see the layer 4 packet. So I know where a packet is coming from and where it's going (and ttl and some other not really helpful data for this purpose.) I don't even know the port. So I'm left with doing (dumb) traffic shaping by destination and target. If I had to guess, I'd say that they incorrectly thought that some specific IP address (range) serves predominantly one type of data. So they throttle by the only data point they have, destination ip, and the collateral damage is everything else hosted on that ip address.
- kaszanka 4y ago
  Wait, why wouldn't they know the port? I'm not aware of any cases where the TCP header itself is encrypted.
  - fjni 4y ago
    You are correct. The port is part of the tcp header and my statement above is incorrect!
  - kevincox 4y ago
    99% of consumer traffic is on 443 these days (citation needed)

fjni 4y ago

> I already knew from previous experience that for some reason, AT&T traffic to fast.com is throttled. Why AT&T wants bandwidth to appear lower than reality is a mystery to me, but I digress This I think is because they throttle ip addresses for known video streaming sites. That is one of (the only reliable) ways an ISP can get the streaming provider to drop the stream to a lower quality one by default. Since fast.com is a Netflix ip, and the isp can’t distinguish whether it’s video that is being transferred or a file to measure throughout, the speed test gets caught up in it. Said the other way around: fast.com is great to see actual throughput from Netflix as opposed to some fake throughout from a dedicated speed test site for the exact same reason.

paulddraper 4y ago
Indeed that's the point of fast.com.... To measure the speed from Netflix servers
avianlyric 4y ago
> Said the other way around: fast.com is great to see actual throughput from Netflix as opposed to some fake throughout from a dedicated speed test site for the exact same reason. Yup. This is pretty much the reason Netflix created fast.com. They wanted a speed test service that couldn’t be gamed by ISPs. Many ISPs will prioritise traffic to know speed test services (like Ookla’s speedtest.net), making their services appear faster than they’re under more normal usage. By placing fast.com on Netflix IPs, ISPs either have to prioritise all Netflix traffic (which they’re very unlikely to do), or accept that fast.com is going to provide a more realistic measurement of their performance.
- neurostimulant 4y ago
  Now that my ISP bundles Netflix subscription into their internet plans, access to netflix and fast.com now practically saturate the fiber link, while before it was outright blocked. Hooray for no net neutrality I guess. Another fun part: when netflix IPs was blocked by this ISP, it's pretty much impossible to use netflix because the only way to get around the block was to use VPN, but netflix itself blocks VPN access.

esalman 4y ago

Fun story: I work remotely for an org located in the Bay area. Few months ago I get a call early morning from the IT security person. Apparently someone had been trying to RDP into my work laptop, over a thousand attempts per day for last couple of days. What changed before the last couple of days? My laptop was frequently dropping WiFi connection- a lot of times in the middle of zoom meetings. We use a router provided by AT&T. My wife complained of the same thing, so I thought I'll ping their customer support. The customer support person proceeded to tell us they wanted to reconfigure our router to enable "5G". After some bewilderment we realized they were talking about 2.4G vs. 5G WiFi. And they will do it remotely. Ok fine go ahead, we said, and forgot about it.. until I got the call from my office IT security. Apparently the AT&T support person left our router in passthrough mode. According to their K/B, "Placing a device in passthrough mode will remove firewall protection provided by the AT&T gateway." I reset my router to default settings, and got myself a 100ft Ethernet cable to fix the issues.

LoganDark 4y ago
> Placing a device in passthrough mode will remove firewall protection provided by the AT&T gateway Did... did they seriously turn the router into a DMZ*, without your consent? Where every port of your computer is just open to the internet? That's scary. *that's what it was called on my router, the "forward every single port to one device" mode. Not sure if that is the correct term for it.
- esalman 4y ago
  Yeah. We checked with censys.io, port 3389 and 8080 were open as publicly available service on the router.
  - jandrese 3y ago
    Oh yeah, if you leave RDP open to the internet the scanbots will be all over that like white on rice. If only we would finish the switch to IPv6 we could pull the rug out from under those bots.
- 4y ago

rsaxvc 4y ago

When I had AT&T VDSL they had weak peering. Some traffic that should have gone across town would routinely get routed several states away before coming back to town, because they hadn't peered with the local exchanges. Some traffic was ok. When I switched to an ISP with local peering, ping times between my home and my server downtown dropped to a few milliseconds. It's like being on my LAN. If this is still an issue for AT&T, traceroute+reverse DNS of each router may indicate it. This poor peering isn't directly a throughput problem, but really increases the chance of being routed through a congested link, which you could sometimes guestimate from ping times and peeringdb.

Scoundreller 4y ago
Same issue in Canada: our big ISPs historically refuse to peer freely at our domestic IXs like TORIX. So smaller ISP and DCs end up peering with them in the USA and your packets would make circuitous cross-border round trips to go a few km. Some national players do have ports at domestic IXs, but only as backup links or for negotiated (paid?) access. The hazards of ISPs also being backbone providers. (It looks like Rogers has gotten better with this behaviour)
- ShroudedNight 4y ago
  As far as I can tell, Bell holds its Canadian customers hostage to force peers to pay them for what should be mutually beneficial peering. It does not participate in _ANY_ of the Canadian internet exchanges. Even Rogers connects to TORIX. https://bgp.he.net/AS577#_ix https://bgp.he.net/AS577#_ix
  - Scoundreller 4y ago
    I know Rogers for a while had 'conditional' peering at Torix, but doing some traceroutes now, more routes are going through Torix than I recall in the past. Seems like they've lightened up over the past couple years. Too bad the indie ISP market has been decimated. Ebox & Distributel --> Bell. Start --> Telus. Oxio --> Cogeco. VMedia --> Quebecor (Videotron). Mostly only Teksavvy left standing Indeed with Bell & Telus, you could only peer freely on US soil: https://www.peeringdb.com/net/1550 https://www.peeringdb.com/net/1550 https://www.peeringdb.com/net/76 https://www.peeringdb.com/net/76

chrsjxn 4y ago

You've got to love a website that makes text selection invisible by default. I'd probably double check assumptions on the cloudfront issues. Switching carriers or adding VPN might connect you to a different edge node. Some web assets being throttled for specific AT&T accounts seems a little too targeted to just be traffic shaping. I'd expect them to throttle traffic for all users, like they do with the speed tests.

sgtfrankieboy 4y ago
>You've got to love a website that makes text selection invisible by default. First time seeing it being invisible, mostly they just disable it. Terribly annoying practice. I often select text while reading articles.
- chrsjxn 4y ago
  Same here! I think it helps me read faster online, but it's probably just a fidget-y habit.
  - zo1 4y ago
    You guys are not alone, I do the same. It is indeed a fidgety habit, but also, I do it as it helps me see text more clearly due to my astigmatism.

Tarball10 4y ago

AT&T uses different APNs for regular phone plans and data-only hotspot plans. It's very possible that the phone traffic is being routed completely differently than the hotspot traffic, with congestion at a peering point occurring on the hotspot. AT&T tends to be known for having poor/congested peering. You could try changing the DNS server on your hotspot to a different public resolver like 1.1.1.1 or 8.8.8.8. If CloudFront is using DNS based geolocation to route you to the nearest data center, a different DNS server may get you routed around the issue to a different data center.

acaloiar 4y ago
Good tip. I just tried with both and wasn't routed to a faster point of presence with either. I'll add a traceroute from the phone and one from a device connected to the LTE router to the updates section next. [update] Traceroutes added
- Tarball10 4y ago
  It's definitely odd that you're getting routed across the country to an east coast pop. Ideally in your location you'd be routed to a west or central pop. Perhaps a bad combination of Cloudfront getting geolocation wrong for your IP range, and poor performance of at&t's network for that specific source-destination combination.
  - acaloiar 4y ago
    I found that odd as well, and my iPhone hotspot does get routed through a different list of hosts as you can see in the traceroute. I suppose it should be noted. The iphone antenna and router antennas are no more than a meter from each other. They're unlikely to be hitting different towers.
    - catminou 4y ago
      What are you using for DNS?
- 4y ago

getpost 4y ago

"Modernizing its search engine has become an obsession at Google".... No doubt it can be modernized, but just making Google search as good as it used to be would be helpful. And why not better than it used to be with the traditional interface? That would still be worth doing even if Google adds on a superb AI. Search is the proverbial goose that laid the golden egg and Google is killing it, and I don't mean "killing" in the contemporary ironic usage.

Flammy 4y ago
Wrong thread?
ikurei 4y ago
I think you might have written this comment on the wrong post
delecti 4y ago
Google search can't be as good as it used to, because the web isn't as "clean" (pure? idealistic? honest? legitimate?) as it used to be. People making websites with the goal of appearing higher in Google's results means Google has to work on countermeasures. That sets off a game of cat-and-mouse, and means actual good results are caught in the crossfire.
thatguy0900 4y ago
Is it possible to return Google to pre-seo blackhat days? I think the golden egg has been poisoned, even if Google tried it's best

themagician 4y ago

A decade long fight about Net Neutrality in the public, and the ISPs just went and did what they were always going to do anyway in the background. No surprise. No news. No fanfare. Just, “Here’s your broken internet where every websites loads at a different speed. Get Unlimited* access with a 35GB limit before more limits kick in. Some websites are unusable. Enjoy. *Some limits apply. See T&C for details.”

Scoundreller 4y ago
But OP is paying for the first 50gb/month to be a "business" unthrottled connection on their unlimited connection!
twoodfin 4y ago
Net Neutrality was never going to apply to mobile networks under any FCC proposal.

ekosz 4y ago

We've been hearing complaints from AT&T users for months now that our website (https://vstream.com https://vstream.com) doesn't work for them. I've tried hunting down the reasons & tried getting in touch with AT&T to no avail. This seems like it could be the very thing that's causing the issues for our users. Now I just need to figure out what to tell our users that are having these issues...

acaloiar 4y ago
I believe I can confirm that vstream.com is affected by this. It fails to fully load in a reasonable amount of time unless I connect via VPN, Verizon, or my (phone) hotspot. Update: For what it's worth, the "above the fold" portion of the content loads down to "All Debuts". After that, there's a long blank space with a Loading indicator near the bottom. Eventually it loaded. I didn't keep track of how long, but it was almost certainly longer than 1 minute.
Tarball10 4y ago
This appears to load fine for me on both AT&T fiber and AT&T cellular. It looks like all content on that site is being loaded from Cloudflare, so I'm curious if those users would be having issues with other Cloudflare protected sites as well?

Animats 4y ago

Remember how 5G was supposed to support "the metaverse"?[1] I've been working on a high performance metaverse client. All Rust, all multi-threaded. Designed to max out a gamer PC. Supports Second Life and Open Simulator. Second Life had a reputation for being sluggish. Fixing that. I'm pulling content from the servers at 200Mb/s, sustained. 400Mb/s in tests, but don't need to go that fast. The servers (AWS front-ended by Akamai caches) can handle that just fine. Gigabit fiber can handle that just fine. The 3D world appears in high detail in seconds. Looks like an AAA game title. Not like low-rez Meta Horizon or Decentraland. 5G can handle that, right? Says so right here in the promotional materials.Verizon: [2] AT&T: [3] The carriers said "unlimited", right? So they can't complain if you're downloading 100GB per hour. [1] https://www.forbes.com/sites/michaelgale/2022/05/24/how-5-and-even-6g-will-make-the-metaverse-a-true-reality/?sh=5ed4d49c16f0 https://www.forbes.com/sites/michaelgale/2022/05/24/how-5-an... [2] https://www.ericsson.com/en/blog/2022/4/why-metaverse-needs-5g https://www.ericsson.com/en/blog/2022/4/why-metaverse-needs-... [3] https://www.verizon.com/about/news/5g-makes-metaverse-real https://www.verizon.com/about/news/5g-makes-metaverse-real [4] https://www.xrtoday.com/event-news/5g-networks-crucial-to-metaverse-future-att-says/ https://www.xrtoday.com/event-news/5g-networks-crucial-to-me...

dboreham 4y ago
Unlimited is a term of art that means "limited".
- DangitBobby 4y ago
  Can we just start calling fraud by its own name?
  - doublerabbit 4y ago
    I'll order one criminality with extra politicians and one large diet-fraud-lite(tm) with extra ice. Just charge it to my offshore. Cheers
- xeromal 4y ago
  Yup. Buffets have been that way forever. No one is actually allowed to eat unlimited.

0887437208577 4y ago

[flagged]

Bluecobra 4y ago

Makes me wonder if it’s just something as simple as suboptimal routing and the path it’s taking is through a hop experiencing packet loss or some other issue further upstream. I would think that there’s a higher chance of AT&T being incompetent then doing something nefarious.

lordgrenville 4y ago

> While wget is not my goto for command line HTTP fetching It is for me. Is there something better I'm not aware of?

mijoharas 4y ago
Curl tends to be the standard.
- acaloiar 4y ago
  It's what I reach for first. `wget` made for a better gif, though :)

kevin_nisbet 4y ago

Taking a quick 5 minute look at the packet capture attached to the post, it looks to me like this is likely traffic shaping. While it's always difficult to be 100% sure from just the client side capture, the capture looks relatively clean of errors, and the amount of data in flight doesn't appear to even approach the advertised window. We're getting some merging of segments likely from a segment offload, but I doubt that's throwing off the results. So there's a good chance there's a shaper letting through about 320kbit/s as it's relatively even throughout the capture.

acaloiar 4y ago
Thanks for the feedback. I was hoping to see more of this sort of analysis from the HN crowd :) There's one other person from HN doing some analysis with me via email. Maybe something will come of it. You both came to the same conclusion based on the pcap data. They're also analyzing the iphone tether pcap data I just provided in an update.
- kevin_nisbet 4y ago
  You're welcome. I took a very quick look at the iPhone capture. I don't see any notable issue (which is expected since this is "working" behavour), and the main noticeable item in the comparison is an apparent lower latency to the server. This appears to match the speculation I saw in the updates as well. One thing you might be able to try, if you're able to change the APN on the hotspot, is to see if you're able to connect with a different APN from the consumer side. This is pure speculation, but what some carriers do is use a different APN for their business services, that will route to enterprise only packet gateways. Some carriers may not necessarily disable access to the consumer APNs though that might go through a different P-GW that's closer to you. The iPhone looks like it's using a path about 120ms shorter than the hotspot. And just by chance, going through a different P-GW you might hit different traffic shaping equipment or configuration. Edit: although be aware that the carrier could also charge different APNs separately. I have no idea what AT&T does.
  - acaloiar 4y ago

varenc 4y ago

One nit of the author's analysis: The Strava javascript file used for speed tests is 1.68MB uncompressed. But in a browser and most all other situations it should be requested with `Accept-Encoding: ...`. In Chrome, Strava responds with a gzipped response that's 463kB in size. This doesn't really matter for the CLI speed tests, when its requested without compression, but it does mean that the speed test performance won't correspond with the actual in-browser performance and the traffic shaping may not be comparable when the request is made without compression. Adding `--compression=gzip` to the wget command will fix this. To quickly show the difference: $ curl -s 'https://web-assets.strava.com/assets/federated/find-and-invite-friends/827.js' --compressed -w '%{size_download}\n' -o /dev/null 462525 $ curl -s 'https://web-assets.strava.com/assets/federated/find-and-invite-friends/827.js' -w '%{size_download}\n' -o /dev/null 1759662

acaloiar 4y ago
Fair nit. Although this is why I chose to focus on transfer rates instead of time. But consider this. When I load Strava's dashboard, open the Network tab and search for "cloudfront", these are the metrics: > 374 requests 15.80 MB / 7.07 MB transferred Finish: 4.89 min This is not a good time, and no amount of compression is going to help the situation.
- varenc 4y ago
  Oh I absolutely agree! That's some sketchy stuff. Didn't mean to sound like I was discounting the rest of your experience.

ehPReth 4y ago

Bell in Canada is doing the same with their new 'unlimited' (vs the previous generation 'unlimited') and their cheaper data plans. Even if you have any amount of 'fast data' left you're throttled to SD speeds for streaming video unless you pay for the higher tier of their plans. Data isn't equal any more. Their higher plans say "HD video is up to 1080p" so I suppose no 1440p/4K/60fps/etc either. It's honestly fucking infuriating but there's nothing that can be done except use a VPN I guess. I'm on the previous generation so "data is data" and they don't throttle me yet but if I ever change plans it's there. I'm not totally sure if the other carriers do (or their 'competition' in forms of MVNOs that they themselves own) but they tend to have a habit of copying each other, at least after a little delay. https://www.bell.ca/Mobility/Cell_phone_plans/Unlimited-plans https://www.bell.ca/Mobility/Cell_phone_plans/Unlimited-plan... (select your local area or just choose Ontario) bonus fun bs: every so often when you try to log into the web portal they try to trick you into letting them build a profile off your browsing/usage data so they can make even more money off of you. a popup with 'advertising is a reality in today's world' comes up with the nice attractive 'get this out of my face so I can do things' blue button being 'yes please opt me in'. shady dark pattern! back when, they used to opt people in by default and make you explicitly opt out... but then the regulators said hey that's illegal. so now they resort to stuff like that to get the numbers back up. here's some of the blurb if anyone is interested: "Advertising is a reality in today’s world, and people find that they receive ads that are irrelevant to them. With our tailored marketing program, Bell will work to ensure that the offers participants receive when using our services may be more relevant, rather than random marketing ads. In other words, participants won’t see more ads, just more relevant ads." full text: https://pastebin.com/ESskYEUy https://pastebin.com/ESskYEUy it's honestly super gross what they collect and how they try to trick people into agreeing.

extr0pian 4y ago

Wireless carriers have been pulling stunts like this over the last several years. For example, T-Mobile blocks all SMS messages with links that have a .xyz TLD. There's no indication that it was blocked by neither the sender or the receiver. I assume this was done to prevent spam and sketchy websites, but personally speaking I've seen way more sketchy .com sites and more spam coming from gmail accounts.

alexpotato 4y ago

Back in the early 2000's, I worked for a small company that was hired by the law firm representing one of the big cable companies (let's call them Company A). Company A was suing another big cable company (Company X) b/c A claimed that X was throttling customers without telling them and advertising what were in effect unthrottled bandwidth numbers. In order to prove this, the company I worked for went out and hired people to get cable modem subscriptions for Company X, install desktop that were locked down so the people couldn't use the desktops themselves and then collected ping and HTTP request data for months. My job was to take that data and analyze it for throttling patterns. This was my first job out of grad school and involved lots of: - data cleaning and organizing using Perl - analyzing statistics using Excel - creating charts that could then be shown to the lawyers and in turn could be used in court I didn't know much about databases or stats packages in Perl which, in hindsight, would have made my life easier. I still remember being on the call with the lawyers and walking them through the data that proved that Company X was indeed throttling bandwidth at peak usage times. The one lawyer on the conference call said "Wow! This data is amazing. You guys are now basically national experts in this." I was both: - flattered that someone would call me a national expert in something - scared shitless that I would end up being cross examined about how we processed this data Turns out that the court case never happened and they settled out of court. The reason I mention this story is that it's FASCINATING to me that this is still occurring today. I guess some things never do change.

Scalene2 4y ago
A big part of the reason that things didn't change is because that lawsuit and others like it ended in a settlement. If this behaviour and data was established as part of the judicial/public record, it would have been much easier to address.
MacroChip 4y ago
Did they wait to sue until after you had your data ready? Or were they gonna sue regardless?
nine_k 4y ago
Businesses, like teenagers, always keep probing the limits of what's allowed or tolerated. As long as it's an out-of-court settlement for a moderate sum, it's just a cost of doing business, a cost of R&D of sorts, only in the legal area. (Similar with taxes.)

kazinator 4y ago

> Strava’s javascript resources clocked in at 1.68MB That looks like the problem to me. Throttle the fuck out of that shit.

kup0 4y ago

I had this problem with Boost Mobile for phones. Apple Store and various podcast sources seemed ultra-throttled and often wouldn't even want to load at all. Eventually switched to Mint and problem went away. Maybe a fluke, but I wouldn't be surprised if it was some shaping BS

initramfs 4y ago

wow, 40KB/s is approx the speed I get after I use up my 4G LTE data on Unreal Mobile. I get unlimited 2G but it's also the cheapest plan in the U.S.

albert_e 4y ago

So not just traffic shaping but also pretty a heavy moderation of discussion on that topic? Interested to see what justification AT&T comes up with for suppressing these posts on their forum and marking them as 'private' without any explanation to the original poster at least.

rendall 4y ago

> "After switching back and re-authenticating with the broadband APN, I’m no longer being throttled." If the author is reading this, I recommend putting an update at the head of your article with this. It will be kinder to your readers.

acaloiar 4y ago
Update #6 has been at the end of the post since I posted that here on HN.

catminou 4y ago

Are you near Utah? The fact you are being routed to a cloudfront pop in IAD is kinda broken. What are you using for your DNS resolver?

acaloiar 4y ago
Durnago, CO is close to Utah, so that's not a surprise. I've used `1.1.1.1`, `8.8.8.8`, and others during this debacle, which I can't recall. Here are some query responses that I received during the incident, but I can't tell you which resolvers were in use during the queries. ``` host web-assets.strava.com web-assets.strava.com is an alias for dgpcy4fyk1eox.cloudfront.net. dgpcy4fyk1eox.cloudfront.net has address 99.84.208.46 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.56 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.10 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.115 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:8000:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:e600:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:c600:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:b600:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:d000:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:6400:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:f000:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:400:17:4613:2840:93a1 ``` ``` host web-assets.strava.com web-assets.strava.com is an alias for dgpcy4fyk1eox.cloudfront.net. dgpcy4fyk1eox.cloudfront.net has address 99.84.208.56 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.10 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.115 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.46 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:4c00:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:ae00:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:c000:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:7200:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:1000:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:b800:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:9400:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:9e00:17:4613:2840:93a1 ``` The only difference I've seen is order, which is expected. [edit] I actually spot some different IPv6 hosts above, after having taken a closer look.

punnerud 4y ago

Just found that Google Mesh Wifi does this at my home. 25Mbit/s and 900ms. Directly with cable 800/800Mbit and 4ms (full speed). Deactivate priority for Zoom, and now I have full speed.

vrglvrglvrgl 4y ago

[dead]

nashashmi 4y ago

I think it is the latency. 130 ms will cause websites with lots of little files across multiple https servers to load slowly. If the resources was bundled into a single file, it would load faster. (My corporate network used to have similarly exaggerated latency on a bad old wire.)

Avamander 4y ago
This shouldn't matter much with H2. "Long fat pipe" is a well-known issue with a bunch of solutions.
acaloiar 4y ago
What you say is true, but 130ms latency is quite normal for even very healthy LTE connections. It's simply a high latency environment.

naasking 4y ago

The net neutrality battle is ongoing.

tomhallett 4y ago

Adriano, In the article you mention "Cloudflare" a few times, but I think you mean "Cloudfront". For example: "web-assets.strava.com resolves to Cloudflare, see the writeup on my site for details" I'm assuming this was a typo based on the names sounding so similar and both being CDN's. Is that correct?