12 ms·
I discovered caching CDNs were throttling my everyday browsing
- collsni 3y agoSeems like this site could used a caching CDN lol
- K0nserv 3y agoThe author seems to be serving it from their home network at 7Mbps upstream, which probably isn't quite up to handling HN front page levels of traffic
- farmdve 3y agoI didn't know people still did that.
- mnw21cam 3y agoI still do that.
- avhception 3y agoI would love to, but I'd very much prefer a static IP for that (instead of reverse proxy / wireguard shenanigans) but getting one is prohibitively expensive where I live. Basically, I'd need to purchase the big business package from my ISP.
- superkuh 3y agoLots do. I have since 1998. It's only become easier and better with time. Join us.
- Dylan16807 3y agoHome hosting is neat but I'm not going to use the phrase "easier and better" unless I'm talking to someone with a much faster upload than single digit Mbps.
- superkuh 3y agoSingle digit mbps is fine for 99.9% of the time. The slashdot effect would take down shared hosting just as easily.
- Dylan16807 3y agoWith image-rich content, 5mbps will be visibly sluggish with only one visitor and even a few people you know poking it at the same time well have a bad experience. Judging by https://unixism.net/2020/05/what-kind-of-traffic-does-hacker-news-generate/ https://unixism.net/2020/05/what-kind-of-traffic-does-hacker... and https://news.ycombinator.com/item?id=30481230 https://news.ycombinator.com/item?id=30481230, surviving a couple loads per second up to 25 will get you through many slashdottings, and with a solid symmetrical home connection you have a very good chance. If you have video, you're not going to survive a slashdotting, but 5Mbps will let you have about one viewer with a smooth experience, while 20-30 viewers could watch the same content on 100Mbps. Or maybe you want to deliver 4k and it's zero versus several peak viewers.
- superkuh 3y agoThe best part about personal websites is that you don't have to survive 99.99% of the time. It's okay if people can't access it for a day. No big deal.
- Dylan16807 3y agoIf I want to tell my friends about a new post, I want several of them to be able to click the link at the same time! And not feel like they're walking through mud. This isn't about getting tons of nines of uptime, this is about people enjoying the page a strong majority of the time they're visiting. That needs a certain amount of speed unless it's a super lightweight page.
- superkuh 3y agoIt's not like when you post a link in a chat they all load it at the exact same time. It's spread out over a minute or few. I'm currently on a relatively slow Comcast connection with 5 megabits/s upload and it works just fine for hosting and posting links for several (or more) people to look at.
- arcza 3y agoCorrect, my crappy VDSL2 connection is not cut out for this level of traffic. I am grateful for the traffic from HN nonetheless :)
- arcza 3y agoA short story on how I diagnosed half the Internet being broken for me, but the rest would be perfectly fine. And no, it's not DNS for once.
- bobdvb 3y agoIt's probably not traffic shaping, its almost certainly a peering capacity issue between Zen and the major CDNs. Zen probably needs more/better peering with Akamai and the other major CDNs than they have. I've written more complete answers about this elsewhere in this thread. Hanlon's Razor applies in this case.
- farmdve 3y agoOddly enough Pinging abctaylor.com [82.71.78.1] with 32 bytes of data: Request timed out. Reply from 82.71.78.1: bytes=32 time=186ms TTL=55 Reply from 82.71.78.1: bytes=32 time=208ms TTL=55 Request timed out. Reply from 82.71.78.1: bytes=32 time=200ms TTL=55
- wccrawford 3y agohttps://web.archive.org/web/20231123121535/https://blog.abctaylor.com/how-i-discovered-caching-cdns-were-throttling-my-everyday-browsing/ https://web.archive.org/web/20231123121535/https://blog.abct... In case anyone else wants to read it while it's hugged to death.
- traceroute66 3y agoYou might want to consider switching from Zen to AAISP[1]. Zen used to be decent, now pretty much only the only residential ISP left that offers quality support is AAISP. They also have all sorts of stats monitoring[2] on all customer lines by default, which they expose to customers on the portal. AAISP will also, unlike many ISPs, not shy away from giving Openscreech a strong poke with a very sharp stick if the underlying issue with your line is due to Openreach. They know how to play the BT game. No affiliation with AAISP other than knowing a number of their customers, not a customer myself due to completely unrelated reasons which are entirely beyond their control. They are not the cheapest ISP in town, but it very much is a case of you get what you pay for. [1]https://www.aa.net.uk/ https://www.aa.net.uk/ [2]https://support.aa.net.uk/Category:Diagnostic_Tools https://support.aa.net.uk/Category:Diagnostic_Tools
- arcza 3y agoI already have an AAISP line coming on Monday! Very excited. AAISP = the new Zen.
- flir 3y agoThe biggest difference you'll find is that when you phone support you'll probably be having a conversation with someone who knows more about networks and routing than you do. Doing first line support. I figure the only way this is economic is because they have a technical customer base who phone support maybe twice a decade. Years back I had a conversation with someone who had done a couple of ISP mergers, and his opinion was that techie-focused ISPs look fantastic on paper because their support costs are really low. Then someone buys them, tries to expand their userbase to "regular" people, and their support costs fall in line with the rest of the industry (eg Demon -> Thus -> Vodafone). So far, A&A have avoided that fate.
- daveoc64 3y agoSo many people rave about the support A&A provides, but I don't really understand why support is so important to people. Isn't a stable connection more important?
- baz00 3y agoI’m with zen. This lines up with my experience. Time to move to AA. Except farnell.com which is shit everywhere because their entire platform is a turd.
- farmdve 3y agoI thought farnell was an electronics store.
- traceroute66 3y ago> I thought farnell was an electronics store. They are. The name only comes up here because the blog post used their website as a test target.
- traceroute66 3y ago> Except farnell.com which is shit everywhere because their entire platform is a turd. Farnell the company is also a turd, another example of a once great company that has gone to the dogs. You can't even trust the stock numbers on the Farnell wesbite anymore.
- baz00 3y agoAh yeah. Place an order, find out an hour later only 75% of it is actually in stock. The 75% comes in 5 boxes from various locations at random times over the next week. At least it's not CPC. They sent me an empty box once.
- ta1243 3y agoI'm with AA, had problems very early on, traffic was really slow. Hopped on IRC and nobody was talking slowness, so I was assuming it was my end still, but then somebody mentioned sluggishness, so I spoke up. Quick traceroute later and within 5 minutes I had a new pppoe user to try, which moved my routing to a different router in docklands, and all was good. 10 minutes later they've shifted everyone to that and taken the router out for investigation.
- mjpa86 3y ago
- ubutler 3y agohttps://web.archive.org/web/20231123121535/https://blog.abctaylor.com/how-i-discovered-caching-cdns-were-throttling-my-everyday-browsing/ https://web.archive.org/web/20231123121535/https://blog.abct...
- matsemann 3y agoPerhaps the blog could use a CDN? ;)
- arcza 3y agodon't even :) well, a second copper connection is getting installed on Monday with a less awful provider.
- arcza 3y ago* UPDATE * pride has been swallowed and Cloudflare is fronting the site. Yes, it's ironic. Still sort of self-hosted and not on EC2 :)
- luuurker 3y agoTo get more out of it, don't forget to create a "page rule" to cache all content (including the page). By default Cloudflare only caches things like images, js, css, etc.
- londons_explore 3y agoI think this is just what you see with typical ISP "traffic shapers". They try to limit bandwidth to video sites, but since most video traffic is transferred by HTTPS these days they end up just making a massive list of IP's which look like they might be sending video data and dropping some percentage of traffic to those IP's. Most CDN's are probably on the list. End result is most video sites drop back to SD rather than HD. If you do a speedtest, it will come out as fast. If you VPN, that will also be fast. The IP range has nothing todo with it - it is the route the packets traverse and what the packets look like when they pass the shaper device that matters. You could theoretically find out which device on the path is doing the dropping by manipulating the TTL of packets in a live TCP session and seeing when you get back TTL exceeded messages.
- supriyo-biswas 3y agoThere’s fast.com and speed.cloudflare.com to get realistic speed measurements since there’s no way* the ISP can tell traffic to these sites vs. video traffic to Netflix/Cloudflare. *Technically some ISPs can (and probably do) look at the ServerNameIndication to determine hostname and therefore whether they’re video flows or not, but I’ve not heard any real world examples of this being done.
- paulryanrogers 3y agoESNI to the rescue?
- cesarb 3y ago> ESNI to the rescue? IIRC, it's now called ECH, because AFAIK the focus changed from "encrypt the SNI" to "encrypt the whole Client Hello".
- crotchfire 3y agoUh, no, the focus changed to don't encrypt the outer SNI but pretend we're encrypting something that will prevent filtering, even though it won't.
- stuaxo 3y agoI'm with Zen and have a good experience over the last few years, one of the only companies where customer service has been decent - when I was on ADSL the fault finder on their router helped identify a nearby a problem, once BT openreach replaced the cable connectivity was really good, probably would have been given the runaround by another provider and had to live with a flakey connection.
- meindnoch 3y agoLooks like net neutrality is going down the shitter. Except it's not the way ISPs would have originally wanted, with CDNs taking stewardship of what's allowed and what's not.
- cryptonym 3y agoBusiness of premium CDN is providing the bits as fast as possible (to legit users). This is what customer pays for. If the experience is worse on a CDN, customer will quickly put their traffic and money elsewhere. Strategy might be different for a free-tier/cheap CDN.
- bobdvb 3y agoThe whole net neutrality argument was made by people who've never operated a large scale network. I've shifted Tbps for a big content provider for some years now and I also worked for two ISPs. The idea of paying for premium access and it negatively affecting the competition is looking at the challenge wrong. It also presumes that ISPs have one fat pipe that gets divided up, which is not usually the case unless you're a tiny ISP. What actually affects performance and is probably the case with this user is that ISPs and CDNs need to come to an agreement over what connectivity they peer with. At scale they don't do that over public peering, it's private peering either through a third party (like Equinix or Digital Realty) or directly patching fibres within the major data centres and linking their networks together. New and unusual services will likely use public peering instead of private peering, or they won't use an tier 1 CDN like Akamai who an ISP would peer with, instead using someone like Bunny CDN, a fine CDN but not peering on the same scale. The fairness risk comes not from the CDN or content provider 'paying' for priority, but comes from the ISP not investing in public peering. That's not the content providers fault, it's just bad operational practice. You could say it's the content providers fault for subsidising the route that gets their traffic through, but its really the ISPs poor infrastructure investments playing out. There's a small risk from content providers doing deals with ISPs to "zero rate" traffic where that ISP (or more usually cellular provider) charges users for bandwidth (or caps it), where the big content provider can use their leverage to make their service cheaper to the consumer. But the reality is that zero rating isn't particularly commercially popular, I've seen it once or twice in my career.
- ta1243 3y agoCache here https://web.archive.org/web/20231123121535/https://blog.abctaylor.com/how-i-discovered-caching-cdns-were-throttling-my-everyday-browsing/ https://web.archive.org/web/20231123121535/https://blog.abct...
- philjohn 3y agoHow odd, I'm also with Zen, albeit on 900/100 FTTP and have no such issues, but then again, I also have a /48 IPv6 prefix delegation and so whatever wants to use IPv6 uses that. BBC, Farnell, everything else - just works, and works fast.
- flarecoder 3y agoJust curious if MSS or PMTU blocking has anything to do with the problem. In the 2 different Wireshark dumps, a relevant difference is MSS=1460 and MSS=1380 in the second one. I'd recommend setting the local NIC MTU to a low value just to see if it has an impact. However, the Wireshark dump doesn't show packet fragmentation, so perhaps this isn't a problem at all?
- flarecoder 3y agoSome random links about PMTU issues: https://www.znep.com/~marcs/mtu/ https://www.znep.com/~marcs/mtu/ https://serverfault.com/questions/1050567/why-does-setting-mtu-1452-solve-internet-problems-and-how-to-fix-root-cause https://serverfault.com/questions/1050567/why-does-setting-m... https://serverfault.com/questions/126468/mtu-dsl-router-and-stalling-tcp https://serverfault.com/questions/126468/mtu-dsl-router-and-... https://serverfault.com/questions/162062/how-to-check-who-blocks-icmp-during-mtu-path-negotiation https://serverfault.com/questions/162062/how-to-check-who-bl...
- t0mas88 3y agoThis is quite a common issue with PPPoE connections like the one OP seems to use with his own router. You need to increase the MTU of the physical underlying ethernet connection to 1508 to allow a 1500 MTU for the encapsulated packets inside the PPPoE tunnel. Otherwise you'll run into weird issues and unreachable websites.
- lxgr 3y agoIncreasing the MTU at the sender side to something >1500 is not a great idea. It’s unlikely that the path will support 1508 byte end-to-end. A better idea would be to reduce the MSS inside the tunnel.
- Nextgrid 3y agoThe MTU is only increased between the router and DSL modem to account for PPPoE overhead, so that the MTU inside the PPPoE tunnel (and thus to the internet) can be a standard 1500 (otherwise it would be 1492).
- MaximilianEmel 3y agoThe good ol' Hug of Death.
- eqvinox 3y agoThis behavior would be fully explained if the Akamai <-> Zen interconnect is simply overloaded. Internet connectivity is not transitive, throwing a VPN into the mix changes the A <-> B scenario to A <-> C <-> B, which can have very different properties, since the paths may have very little in common. For multihomed A and B, the paths may in fact have nothing in common at all. Same applies to IPv4 vs. IPv6, the routing may be entirely different, especially with a CDN you might even straight up get a different CDN instance.
- t0mas88 3y agoExactly what I was thinking reading this. A CDN has no interest throttling things, it will hurt their performance metrics. And if they think your IP is "bad" you'll get a straight up error or captcha, not just some packet loss. But a link between an ISP and a CDP provider being overloaded is quite common. The ISP is trying to get away with the minimal infrastructure investments possible, and good interconnect is expensive.
- nikanj 3y agoTime for a soapbox rant about interconnect prices, which are based on a "How much ya got" pricing instead of "How much it actually costs to hook up".
- eqvinox 3y agoTo be fair this is mostly a problem between ISPs, or with ISPs fleecing their (business) customers. A restrictive peering policy for a CDN is just batshit stupid, and the CDNs know this and will peer with anything that doesn't run away fast enough. You do have to meet the CDNs at some PoP though, and that getting-there part is generally the real issue. Anything last mile is just… ugh.
- bobdvb 3y agoMany ISPs have been lobbying for a tax on "content generators" which they say they would use to pay for infrastructure upgrades (or line the pockets of their shareholders who knows). The EU has a consultation on it, although I think it'll fail to get traction.
- jackweirdy 3y agoI had a problem with Zen recently-ish too. Ultimately was an Openreach thing at the local exchange apparently. The good Zen support was still ultimately there, but it took a little time for things to fall into place. Standard L1 checklist inflation. Thankfully though Zen are one of the few ISPs where I felt like it was worth it to send packet traces because a decent chunk of folks there would know what they are. On the other hand, I think any ISP at the mercy of openreach is doomed to have limited support. I have fibre to the property, and was having periods of 1hr-2hr day of my gigabit speeds dropping to 4-5MB. openreach themselves were blindly sending engineers to look for an issue that couldn’t physically be at my house. Not much you can do there either as an ISP or as a customer besides wait for openreach to figure out they’re wasting their own time
- dncornholio 3y agoWhere is the discovery though? I don't follow how you got to the point where you think it's CDN's that are throttling you. For all I know it could be something like a faulty router, right?
- arcza 3y agoIts not, because the same infrastructure doesn't crap out when through a tunnel (same routers and same ISP network, same MTU on my VDSL router etc)
- justsomehnguy 3y ago>> Stable 6ms ping to 1.1.1.1 Please note, pinging public DNS servers is a useless metric, because you would never know if your provider hijacks your DNS packets or even all traffic to those public servers.
- arcza 3y agogood point. what's the alternative? I like 1.1.1.1/8.8.8.8 etc as its memorable.
- justsomehnguy 3y ago> I like 1.1.1.1/8.8.8.8 etc as its memorable. Yes. > what's the alternative a) some ISP targets, eg mailcluster.zen.co.uk b) lg.he.net and bgp.he.net
- arcza 3y agothanks
- arcza 3y agoIf this didn't work for you earlier, the blog is now behind a CDN. Any good technologist would put practicality before pride :)
- deleted 3y ago[deleted]
- pvtmert 3y agoHN's hug of death https://web.archive.org/web/20231123142332/https://blog.abctaylor.com/how-i-discovered-caching-cdns-were-throttling-my-everyday-browsing/ https://web.archive.org/web/20231123142332/https://blog.abct... Funny thing is the author apparently doesn't use the caching CDN, thus users are not getting throttled but having 503...
- fkarg 3y agothe author actually _is_ using a CDN now :D
- miyuru 3y agoWhy don't you setup IPv6 if that solves the problem for IPv6 enabled sites? You also seem to know your way around networking as well, genuinely curious.
- tomcam 3y ago503 error Oh the irony
- NelsonMinar 3y agoI got blacklisted by Akamai once for some very lightweight automation of web page screenshots (once every ~5 seconds, different sites). Do not recommend the experience. The ironic thing was I was blacklisted from loading Akamai's help pages about what to do if you are blacklisted. I never did find their tool, I wonder if it would have been blocked too. https://www.akamai.com/us/en/clientrep-lookup/ https://www.akamai.com/us/en/clientrep-lookup/ The ban expired after about 3 days.
- eightysixfour 3y agoI had a weird one on my network I never managed to solve before I moved: I had symmetrical 1gbps up and down. When wired, I could get nearly the full amount on the WAN. When wireless, I could only get 300mbps to the WAN. However, when wireless, I could get ~800mbps to another device on the LAN. I could also get 800mbps to the internet if I proxied from my wireless devices to my wired device before going to the WAN. My router company sent me two additional routers, one with a similar chipset and one with a chipset from a different vendor and this persisted. I checked it with a competing router and it persisted. It did not matter what the wireless device was, Mac, windows, phones, or tablets, and it persisted. Moved somewhere else with a different ISP and it immediately stopped. I still don’t know how an ISP would identify and throttle a wireless device, but that was pretty much the only explanation I could come up with.
- Nextgrid 3y agoI wonder if this was some weird coupling/interference between wireless and the actual modem/NIC in the path to the internet? Bouncing off a separate machine would've desynced the two streams just enough that the wireless interference doesn't clash with the outbound NIC (or the other way around - the outbound NIC of the modem was causing interference which was slowing down your wireless).
- eightysixfour 3y agoThe router was plugged into a Cat6 cable which ran to a switch in a closet in the building’s networking room, which was connected by fiber to the network. No modem within 15 floors. Interesting guess though, wouldn’t have thought of that.
- 3np 3y ago- Could it be that there was a difference in IPv6/IPv4 presence/preference between the two? E.g. your wired connection acquired an IPv6 address while the wireless was IPv4 and had to go through NAT. - Did you make sure to compare results for non-concurrent speedtest? Ie maybe those ~800 were actually 4x~200 - many speedtests open parallel connections by default.
- jonathanlydall 3y agoI had an issue recently where my Spotify playback kept pausing due to being unable to download the songs quickly enough on my 75Mbps fiber connection. My ISP has a strong presence on a local forum where I posted my issue. Long story short, despite my ISP actually having an Akamai cluster on their own network, Akamai’s DNS was resolving my ISP’s customers to a cluster on a different ISP’s network. That different ISP either had terrible peering, or the theory is they were throttling their Akamai cluster’s IPs to other ISPs. Fortunately my ISP managed to convince Akamai to fix the DNS resolution. Needless to say, I’m super impressed I can actually get the attention of the right people at my ISP to resolve this kind of issue.
- deleted 3y ago[deleted]
- butz 3y agoYou should look into image optimization, especially when you are self hosting. Use thumbnails for big images, webp looks decent enough and files are smaller than png. Prefer system fonts - why should you serve those too, when each visitor usually have dozens of them available on their device already? Oh, and favicon really can be smaller.
- geek_at 3y agoYes what's up with that 2048x1536 image in the header that gets loaded with the page without a thumbnail?
- garganzol 3y agoI doubt it is a global throttling by an IP address. The most common reason is ISP traffic shaping and ISP ingress/egress deals with the networks it is connected to.
- lormayna 3y agoI guess that there is a congestion somewhere in the path, maybe between your ISP and CDN. I have been worked in an ISP for a while and this was the root cause of problems like yours.
- killingtime74 3y agoI presume you have a cell phone? Could you run all these tests hotspotting and contrast them?
- tardoe 3y agoSo I have seen this before - a lot of ISPs now days are using "optimiser" boxes that are designed to throttle Elephant Flows (https://en.wikipedia.org/wiki/Elephant_flow https://en.wikipedia.org/wiki/Elephant_flow) to reduce overall consumption. Usually they add a little bit of buffering or the occasional TCP congestion notification to cause a client to back-off and (for example), reduce the streaming video bitrate. But I've also seen bad configuration that can cause this sort of issue - e.g. an mis-configuration that limits you to 2kbps vs 2mbps. The reason the Wireguard tunnel works fine is because it's UDP-based and you can't trigger the same congestion notification behaviour over UDP. These boxes are usually inline to your traffic and are often referred to as "middle-boxes" - more commonly they're used in mobile (4G/5G) RAN aggregation networks where bandwidth is more scarce but they're now being sold into fix-line network providers as a cost-cutting measure.