7 ms·
The novel HTTP/2 'Rapid Reset' DDoS attack
- fefe23 3y agoI would like to remind everyone that Google invented HTTP/2. Now they are telling us a yarn about how they are heroically saving us from the problem they created, but without mentioning the part that they created it. The nerve of these tech companies! Microsoft has been doing this for decades, too.
- gsich 3y agoThey tried to solve problems that weren't existant.
- windows10code 3y ago[flagged]
- klabb3 3y ago“Cancelation” should really be added to the “hard CS problems” list. Like the others on that list (off by one, cache invalidation etc) it isn’t actually hard-hard, but rather underestimated and overlooked. I think if we took half the time we spend on creation, constructors, initialization, and spent that design time thinking about destruction, cleanup, teardown, cancelation etc, we’d have a lot fewer bugs, in particular resource exhaustion bugs.
- jart 3y agoI know that's true of C libraries. POSIX thread cancelation is one of those things where its mere existence pervades everything in its implications.
- pornel 3y agoI really like Rust's async for its ability to immediately cancel Futures, the entire call stack together, at any await point, without needing cooperation from individual calls.
- winternewt 3y agoHow is that possible if e.g. an external SQL server needs to be told that the operation should be canceled?
- vlovich123 3y agoWouldn’t this same attack apply to QUIC (and HTTP/3)?
- stonogo 3y agoIt doesn't apply to HTTP/3 because the receiver has to extend the stream concurrency maximum before the sender can open a new stream. This attack works because the sender doesn't have to wait for that after sending a reset in HTTP/2.
- londons_explore 3y agoBut the max is still ~100 streams... And you can open 100 streams all with one UDP packet using zero-rtt connections. I can send ~1 Million UDP packets per second from one machine. So thats 100 million HTTP requests per second you have to deal with. And when I bring in my 20,000 friends, you need to deal with 2 trillion requests per second. I'd say that's still a problem.
- stonogo 3y agoOk, but it's not the same problem, which was the question asked.
- vlovich123 3y agoI’m still missing something. Can’t you close a Quic stream and open another one within the same UDP packet?
- jsnell 3y agoYou can do it a few times, but you can't do it 500 times. For HTTP/3, the highest permitted stream ID is an explicit state variable communicated by the server to the client, eventually forcing a round-trip. That's different from HTTP/2 where the client is entitled to assume that new "stream id window space" (for the lack of a better term) opens up immediately after a stream is closed. (I'm fudging things a bit. You can probably build attacks that look kind of similar, but we don't think you you could build anything that is actually scalable. But we could be wrong about that! Hence the recommendation to apply similar mitigations to HTTP/3 as well, even if it isn't immediately vulnerable.)
- arisudesu 3y agoCan anyone can explain what's novel about this attack that isn't plain old requests flood?
- jsnell 3y agoIt depends on what you think a "request flood" attack is. With HTTP/1.1 you could send one request per RTT [0]. With HTTP/2 multiplexing you could send 100 requests per RTT. With this attack you can send an indefinite number of requests per RTT. I'd hope the diagram in this article (disclaimer: I'm a co-author) shows the difference, but maybe you mean yet another form of attack than the above? [0] Modulo HTTP/1.1 pipelining which can cut out one RTT component, but basically no real clients use HTTP/1.1 pipelining, so its use would be a very crisp signal that it's abusive traffic.
- tptacek 3y agoI think for this audience a good clarification is: * HTTP/1.1: 1 request per RTT per connection * HTTP/2 multiplexing: 100 requests per RTT per connection * HTTP/2 rapid reset: indefinite requests per connection In each case attackers are grinding down a performance limitation they had with previous generations of the attack over HTTP. It is a request flood; the thing people need to keep in mind is that HTTP made these floods annoying to generate.
- ynik 3y agoI wonder why exactly this attack can't be pulled off with HTTP/1.1 and TCP RST for cancellation. It seems that (even with SYN cookies involved) an attacker could create new connections, send HTTP request, then quickly after send a RST. Is it just that the kernel doesn't really communicate TCP RST all that well to the application, so the HTTP server continues to count the connection against the "open connection limit" even though it isn't open anymore?
- wbl 3y agoThe problem for the attacker is they then run into resource limits on the TCP connections. The resets are essential to get the consumption not counting.
- scrpl 3y agoAnother reason to keep foundational protocols small. HTTP/2 has been around for more than a decade (including SPDY), and this is a first time this attack type surfaced. I wonder what surprises HTTP/3 and QUIC hide...
- liveoneggs 3y agoQUIC didn't account for amplification attacks in its design and the people complaining about it were initially dismissed.
- cmeacham98 3y agoDNS is a small protocol and is abused by DDoS actors worldwide for relay attacks.
- scrpl 3y agoDNS is from 1983, give it some slack
- cmeacham98 3y agoThe point I'm trying to make is that "small" protocols aren't less likely to be DDoS vectors. Avoiding designing in DDoS relay/amplication vectors requires luck or intention, not just making the protocol small.
- scrpl 3y agoSmall, less complex protocols are inherently less likely to be insecure all things being equal, simply due to reduced attack surface. DNS was created for a different environment, at a time when security wasn't at forefront so it's not a good example of the opposite.
- Avamander 3y agoThis is such a strong claim I'd really appreciate something other than "smaller is better" Abuse and abuse vectors vary wildly in complexity, some complexity is certainly required exactly to avoid dumb bottlenecks if not vulnerabilities. So based on what are you saying something simple will inherently resist abuse better?
- ta1243 3y agoSee also https://news.ycombinator.com/item?id=37830998 https://news.ycombinator.com/item?id=37830998 https://news.ycombinator.com/item?id=37830987 https://news.ycombinator.com/item?id=37830987
- js2 3y ago> In a typical HTTP/2 server implementation, the server will still have to do significant amounts of work for canceled requests, such as allocating new stream data structures, parsing the query and doing header decompression, and mapping the URL to a resource. For reverse proxy implementations, the request may be proxied to the backend server before the RST_STREAM frame is processed. The client on the other hand paid almost no costs for sending the requests. This creates an exploitable cost asymmetry between the server and the client. I'm surprised this wasn't foreseen when HTTP/2 was designed. Amplification attacks were already well known from other protocols. I'm similarly similarly surprised it took this long for this attack to surface, but maybe HTTP/2 wasn't widely enough deployed to be a worthwhile target till recently?
- tptacek 3y agoIt's not really an amplification attack. It's just drastically more efficiently using TCP connections.
- js2 3y agoYou're right. I hadn't had my coffee yet and the asymmetric cost reminded me of amplification attacks. I'm still surprised this attack wasn't foreseen though. It just doesn't seem all that clever or original.
- callalex 3y agoIsn’t any kind of attack where a little bit of effort from the attacker causes a lot of work for the victim an amplification attack? Or do you only consider it an amplification attack if it is exploiting layer 3? I tried looking it up and couldn’t find an authoritative answer. Can you recommend a resource that you like for this subject?
- rasz 3y agoNo, this is Resource exhaustion attacks https://en.wikipedia.org/wiki/Resource_exhaustion_attack https://en.wikipedia.org/wiki/Resource_exhaustion_attack And its not the first HTTP2 rodeo https://www.akamai.com/blog/security/http2-vulnerabilities https://www.akamai.com/blog/security/http2-vulnerabilities One of the 2019 vulns https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-9514 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-9514 even sounds extremely similar to current attack.
- the8472 3y agoSo we needed HTTP2 to deliver ads, trackers and bloated frontend frameworks faster. And now it delivers attacks faster too.
- Borg3 3y ago[flagged]
- shepherdjerred 3y agoAre you suggesting that we didn't need HTTP2? What's the real alternative here?
- Etheryte 3y agoNothing in their comment claims that, there's no need to bring absurd strawmen into the discussion.
- ramranch 3y agoIt does strongly imply it. HTTP2 is needed for more than just the bloatware of the modern internet.
- shepherdjerred 3y agoI'm just trying to figure out what the parent was trying to communicate, since the comment by itself didn't add much to the discussion. It seems that they're more upset by the state of web development today than they are by HTTP2 or anything else that this thread pertains to.
- tsimionescu 3y agoThe comment lists three negative things as "the reason we needed HTTP/2". I don't even see how you could read it other than implying that HTTP/2 was not actually necessary.
- the8472 3y agoIn some alternative history there would have been a push to make http 1.1 pipelining work, trim fat from bloated websites (loading cookie consent banners from a 3rd party domain is a travesty on several levels) and maybe use websockets for tiny API requests. And the prioritization attributes on various resources. Then shoveling everything over ~2 TCP connections would have done the job?
- n4te 3y agoThe blog header popping up constantly makes the page unreadable.
- cratermoon 3y agoReadability hack: https://github.com/t-mart/kill-sticky https://github.com/t-mart/kill-sticky
- n4te 3y agoHa, it works great! I like it, thank you!
- ticulatedspline 3y agoNot sure about chrome but in Firefox there's a button for "reader view" on many sites which works great for cutting out UI crap like that.
- n4te 3y agoGood point, that also works. For some reason I never remember to use it.
- oars 3y agoThanks for sharing Kill Sticky!
- comice 3y agoNice to see that the haproxy people had spotted this kind of issue with http/2 and apparently mitigated it back in 2018: https://www.mail-archive.com/haproxy@formilux.org/msg44134.html https://www.mail-archive.com/haproxy@formilux.org/msg44134.h...
- jabart 3y agoNice, I was looking for this type of information for haproxy. Gives me a lot of confidence in their new QUIC feature.
- vdfs 3y agoIf anyone is curios, Nginx is vulnerable to this https://www.nginx.com/blog/http-2-rapid-reset-attack-impacting-f5-nginx-products https://www.nginx.com/blog/http-2-rapid-reset-attack-impacti...
- obituary_latte 3y agoIF configured away from the defaults: By relying on the default keepalive limit, NGINX prevents this type of attack. Creating additional connections to circumvent this limit exposes bad actors via standard layer 4 monitoring and alerting tools. However, if NGINX is configured with a keepalive that is substantially higher than the default and recommended setting, the attack may deplete system resources.
- deleted 3y ago[deleted]
- unethical_ban 3y agoI got out of web proxy management a while back and haven't had to delve into HTTP2 or HTTP3. It seems HTTP2 is TCP on TCP for HTTP messages specifically. This must be why HTTP3 is over a UDP based protocol.
- rollcat 3y agoHTTP2 is not TCP on TCP (that's a very basic recipe for a complete disaster, the moment any congestion kicks in); it's mostly just multiplexing concurrent HTTP requests over a single TCP connection. HTTP3 is using UDP for different reasons, although it effectively re-implements TCP from the application POV (it's still HTTP under the hood after all). Basically with plain old TCP your bandwidth is limited by latency, because every transmitted frame has to be acknowledged - sequentially. Some industries/applications (like transferring raw video files over the pond) have been using specialized, UDP-based transfer protocols for a while for this reason. You only need to re-transmit those frames you know didn't make it, in any order it suits you.
- devit 3y agoIsn't this trivially mitigated by throttling? And the throttling seems even simple: give each IP address an initial allowance of A requests, then increase the allowance every T time up to a maximum of B. Perhaps A=B=10, T=150ms.
- o11c 3y agoThe whole point of a 'D'DoS is that there are numerous compromised IP addresses, which only need to make maybe one connection each. You can't simply blacklist weird connections entirely, since legitimate clients can use those features.
- devit 3y agoThe whole point of this attack is to be able to make a lot of requests for each IP address. If you are making one or few requests per IP you don't need this attack, and also aren't likely to have any effect on a Google-sized entity.
- pests 3y agoIt is a little more complicated because a request is few layers deep. In HTTP2 you open a connection, start a stream, then send a request over that stream. Are you tracking per connection? Per stream? Isn't it normal for multiple requests to happen quite quickly? I load a single page with 50 external assets, those get multiplexed over the current stream - is that okay? Is that abusive? The other stream is handling a video player and its requesting (http2) frames of video data - too much? Too fast?
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- dang 3y agoRelated ongoing threads: The largest DDoS attack to date, peaking above 398M rps - https://news.ycombinator.com/item?id=37831062 https://news.ycombinator.com/item?id=37831062 HTTP/2 Zero-Day Vulnerability Results in Record-Breaking DDoS Attacks - https://news.ycombinator.com/item?id=37830998 https://news.ycombinator.com/item?id=37830998
- cloudyq 3y ago[dead]
- 1vuio0pswjnm7 3y agohttps://nvd.nist.gov/vuln/detail/CVE-2023-44487 https://nvd.nist.gov/vuln/detail/CVE-2023-44487 Dont forget about nghttp2
- winternewt 3y agoThis is a protocol bug hence affects pretty much any implementation.
- qingcharles 3y agoMicrosoft dropped its patch details here: https://github.com/dotnet/runtime/issues/93303 https://github.com/dotnet/runtime/issues/93303