7 ms·
When “Dumb Pipes” Get Too Smart
- z3t4 10y agoI have 10ms ping to news.ycombinator.com, and 100ms ping to www.amazon.com. Yet time to first byte is 20% faster to www.amazon.com. What actually happens is my PC connects to Cloudflare, witch in turn connects to HN. This is an unnecessary step, and is highly over-rated.
- Alupis 10y ago> This is an unnecessary step, and is highly over-rated Given the volume of traffic HN sees, and given their desire to not be DDoS'ed, it seems to be a necessary step, that is not over-rated. To be clear, you'd see similar results with any CDN sitting in front of HN... The benefits really start to show (from an end-user's perspective) when you're on poor connections or are far away from their (HN's, in this case) server/cluster. HN sees immediate benefits for the things above, plus it saves them a lot of bandwidth.
- mjevans 10y agoIt's needed because CDNs are presently 'protection rackets' for the Internet. Instead of having a mechanism by which a website under attack (or simply un-monitored heavy load) from a host or number of hosts can direct the ISPs of those hosts to not send it traffic the CDNs instead use the present lopsided nature of peering agreements to simply sink the hits. In the above mentioned solution either the direct ISP would filter out the requests before they hit a higher tier ISP/backbone, or an ISP that does not provide said filtering would (possibly blacklisting the entire client ISP for sufficient bad behavior).
- Alupis 10y ago> Instead of having a mechanism by which a website under attack ... from a host or number of hosts can direct the ISPs of those hosts to not send it traffic How is this any different? You're putting the responsibility of being internet police on the ISP's, which have shown their either do not want that responsibility, or will abuse that responsibility. Not to mention, this would go squarely against the notion of ISP's as "dumb pipes".
- JoshTriplett 10y agoI'd like to see ISPs doing better policing of their customers: disconnecting (and prosecuting) spammers, blocking users infected with malware until they get clean, and so on. I'd like a world in which I can reasonably expect to react to spam by tracking down where it came from and getting the spammer removed from the Internet. abuse@ ought to function and produce a rapid response. (That doesn't mean we shouldn't have anonymous, untrackable services as well, but using such services means you have to put up with spam too. When you have a service like email, with all the tracking information readily available in the headers, it shouldn't be as near-impossible as it currently is to get something done with that information.)
- AnthonyMouse 10y agoThe problem is there are two sources of spam. The first is large networks that tolerate spam, and those are irrelevant because their IP blocks are already on every blacklist in the world. The second is compromised machines, which are the real problem because they're ephemeral. Spammers can compromise a hundred new machines a week. You can't fix them or block them faster than they compromise new ones. The only solution to that is to improve computer security so they don't get compromised to begin with, which has nothing to do with ISPs. Removing people from the internet is never the answer, both because it's too broad (should the spammers not be able to go to the government's website to pay their taxes?), and because the nominal current source of the spam isn't actually where the spammer connects to the internet anyway. The better solution if you can actually find a real life spammer is to impose a fine on them that exceeds the profits of spamming.
- jfoutz 10y agoThere is also the great firewall attack that hit GitHub a while ago.
- kazagistar 10y agoWait, why can't you block spammers as fast as they are created? If your computer is compromised, it seems fine to disconnect it, or at least severely rate limit uploading until you fix the problem. Its the responsibility of the internet user to keep their computer in working order. Really, if your computer is spamming someone, even if you aren't aware of it, you are harming them, and ignorance shouldn't be protection. Maybe fines and jail are a bit too severe, but rate limiting and possible disconnection are more then fair.
- toast0 10y agoWhen you do have cooperative ISPs filtering DDoS traffic, it's usually a null route, so the IP under attack goes offline, but without taking the rest of the customers on your ISP offline.
- rasz_pl 10y agoHe described system that would work the other way around, null-route the ATTACKERS, not the victim.
- makerbraker 10y agoThat's not quite how routing works -- null routing the attackers affects RETURN traffic to the attackers. By then, the damage is done, the target host is already overwhelmed with attack traffic.
- krackers 10y agoIsn't that hard to do based on the distributed nature of DDOS attacks though?
- rdl 10y agoHN (especially when logged-in) is particularly uncacheable. If you tried the same thing with an image-heavy site, you would probably see the opposite result.
- kijin 10y agoCloudflare does not cache HTML pages. Images might load faster, but the time to first byte for the main HTML document is still much higher with Cloudflare.
- MichaelGG 10y agoYou can make CF cache HTML.
- kijin 10y agoSure you can, but it's not the default behavior, and you often don't want to cache HTML.
- rdl 10y agoThere is actually weird magic around keeping open tcp sessions open so you could be faster if close to a cloudflare node and distant to the server, too, even if not cached.
- saurik 10y agoThe issue with CloudFlare here is that they never really cared to be a good "normal CDN", which would make this performance boost possible by using a combination of intelligent caching and an extremely large number of POPs using good trunks; see Akamai or CDNetworks for typical examples of trying to actually play the game of a normal CDN. Instead, CloudFlare relies on crazy content transforms that started as essentially "what if we run mod_pagespeed for you in the cloud as a service" and then expanded in scope from there. I mean, even when they cache things, as I understand it their cache hit ratio is extremely poor (whether due to restrictive cache sizes or limited retention windows, I don't even venture to guess). This means that if your content isn't inherently slow (using poorly compressed images and tons of fragmented JavaScript and CSS files with broken ETags being consumed by HTML with script tags in "the wrong places", none of which is minified), CloudFlare is going to make your site slower. Really: the two things they ended up getting stuck in the ecosystem for were 1) being free or at a fixed low price for supposedly infinite bandwidth (though people report that when you actually use a ton, CloudFlare claims you are being attacked and need to upgrade to their protection racket, which leads us to), and 2) performing DoS attack protection by using a ton of annoying heuristics and IP filters and JavaScript delays and even captchas that you should notice no "real" site behind a normal CDN ever seems to need, and yet somehow is a hallmark of the experience of "this site is using CloudFlare".
- vmarsy 10y agoPicking amazon isn't really a fair comparison, It's Amazon, the people behind AWS, the same people who say that "every 100ms of latency costs them 1% of profit." Of course amazon.com and its servers are ultra-optimized to make everything as fast as possible. A fair comparison would be hacker news with vs hacker news without cloudfare?
- cortesoft 10y agoWait, are you arguing that using a CDN is overrated? For everything, or just for HN? A CDN is extremely valuable for a website that has cacheable content, since it allows a site to scale dynamically with increased load. This is the same reason people use AWS or any cloud provider; a website has to have the capacity to serve their highest peak traffic, but it is needlessly expensive to maintain that much infrastructure around the clock.
- fidget 10y agoHelps a ton if you're terminating SSL at the edge, due to the number of roundtrips you need before the server can even start generating that first byte.
- dpc_pw 10y agoWe need to switch dump pipes to be dumb content-addressable p2p pipes with maidsafe, ipfs, dat, or anything like that, and most problems that CDNs are trying to solve would disapear.
- kevinr 10y agoAt least until someone finds a collision in the hashing algorithm used for the content addresses...
- hobofan 10y agoAnd that's why you have a swappable hashing algorithm.
- ThisIs_MyName 10y agoCrypto agility is a thing :)
- kevinr 10y agoSomebody needs to inform Linus. :)
- syncsynchalt 10y agoLovely bit of debugging in this article, I really enjoyed it! Somehow a task that would be grueling to do myself is so much more enjoyable when read about.
- lanius 10y agoHas the WebCore scheduler improved since?
- edoceo 10y agoThe bug is in bad js authored by CF hammering ticks/event-loop. WebCore is doing what it's told, just just being told very stupid things (very, very quickly)
- chainsaw10 10y ago> The bug is in bad js authored by CF No, the bug is definitely in the browser. Web code is untrusted and should not be able to adversely affect the browser.
- saurik 10y agoSo, there was (I hope not "is"... I am having a difficult time remembering the sequence of events involving this bug and what I might have done on the WebKit side) definitely a bug in the browser, but just to be explicit: there was also a bug in the JavaScript (the array returned as object with no length, causing the code to loop), and managing to deploy a change that managed to DoS tons of deployed browsers and somehow not noticing and then not really caring afterwards was extremely careless.
- fapjacks 10y ago> Cloudflare likes to look triumphant. Yeah absolutely this. They spin everything they do as some kind of heroic "for the people!" decision even when it's just about cutting costs or not having to solve "hard" problems. One example are DNS "any" queries. Cloudflare just decided to toss standards out because they aren't up to conforming to them. As far as I'm concerned, this Cloudbleed thing is karma, and nobody should believe anything Cloudflare says about itself.
- daenney 10y agoTo be honest though, ANY is mostly used for reflection/amplification attacks or to scrape domain records. Sure there are legitimate uses for them but I can't think of many that need to happen over the internet vs. only allowing that from specific trusted parties. And yes I'm aware that it can be useful as a debugging tool. There are also ISP recursors that don't allow ANY queries through for similar reasons so relying on it will cause trouble for some and there are other broken implementations in the wild. Though ANY can be useful it shouldn't be assumed that it'll work. What I'm thinking they could've done in the case of ANY is to respond with truncated and switch to TCP mode at which point it becomes harder to do the spoofing dance and (ab)use ANY as a DNS amplification attack to DDoS a target. Unfortunately that does put an extra cost on the DNS server which might also be undesirable. However that would've probably been a worthwhile tradeoff. There's an IETF draft that attempts to provide some guidance in the area of the ANY query: https://tools.ietf.org/html/draft-ietf-dnsop-refuse-any-04 https://tools.ietf.org/html/draft-ietf-dnsop-refuse-any-04
- fapjacks 10y agoWell, two out of the three people proposing that document work for Cloudflare. The other works for Dyn. It's clear what their motivation is. And this is not the first time they've filed a document just like that, and it will expire this year just like it has in the past. That document is bullshit. It's trying to now change the standard after they've already broken their DNS implementations.
- chrisbolt 10y ago