4 ms·

tl;dr HTTP/2 allows clients to DDoS backends much more effectively by using the multiple-stream feature of HTTP/2 to amplify their attack directly inside the re

tl;dr HTTP/2 allows clients to DDoS backends much more effectively by using the multiple-stream feature of HTTP/2 to amplify their attack directly inside the reverse proxy (which typically translates HTTP/2 to HTTP/1). > When Cloudflare's reverse proxies process incoming HTTP/2 client traffic, they copy the data from the connection’s socket into a buffer and process that buffered data in order. As each request is read (HEADERS and DATA frames) it is dispatched to an upstream service. When RST_STREAM frames are read, the local state for the request is torn down and the upstream is notified that the request has been canceled. Rinse and repeat until the entire buffer is consumed. However this logic can be abused: when a malicious client started sending an enormous chain of requests and resets at the start of a connection, our servers would eagerly read them all and create stress on the upstream servers to the point of being unable to process any new incoming request.

bflesch 3y ago
how on earth did nobody anticipate this kind of attack when designing the protocol? it's very obvious it can be abused like this
- sebstefan 3y ago
  Hindsight is 20/20
- jsnell 3y ago
  It took 8 years for somebody to discover this. It can't have been that obvious.
  - wtarreau 3y ago
    > It took 8 years for somebody to discover this. It can't have been that obvious. Actually that's not true, it was already suggested here as a way to circumvent the max_concurrent_streams setting an it seemed particularly obvious: https://lists.w3.org/Archives/Public/ietf-http-wg/2019JanMar/0132.html https://lists.w3.org/Archives/Public/ietf-http-wg/2019JanMar... As soon as you start to implement a proxy that supports H2 on both sides, that's something you immediately spot, because setting too low timeouts on your first stage easily fills the second stage so you have to cover that case. I think that the reality is in fact that some big corp had several outages due to these attacks and it makes them look better to their customers to say "it's not our fault we had to fight zero-days" than "your service was running on half-baked stacks", so let's just go make a lot of noise about it to announce yet-another-end-of-the-net.
  - mgaunard 3y ago
    Not everyone cares about Cloudflare, or even HTTP/2. The exploit has more to do with their implementation than the protocol.
    - deleted 3y ago
      [deleted]
    - formerly_proven 3y ago
      It's pretty similar to HTTP/1 pipelining, though no reverse proxy I'm aware of supports it.
    - insanitybit 3y ago
      > The exploit has more to do with their implementation than the protocol. Is it? I imagine that implementations can do things like make creating/dropping a stream faster but how would an implementation flat out mitigate this?
      - deleted 3y ago
        [deleted]
      - mgaunard 3y ago
        There is a maximum bandwidth at which data can arrive. Simply make sure you can always process it faster than the next packet can arrive, or implement proper mitigation in cases where you cannot. It's called programming under soft real-time constraints.
        immibis 3y ago
        > Simply make sure you can always process it faster than the next packet can arrive This is pretty much impossible unless you make the client do a proof-of-work so they can't send requests very quickly. Okay, you could use a slow connection so that requests can't arrive very quickly, but then the DoS is upstream.
        insanitybit 3y ago
        Well yeah, that's just how DoS kinda works with these sorts of vulns. "Be faster" is obviously a good strategy, but is it viable ? Is setting up and canceling a stream something that can be done at GB/s speeds? Maybe, idk.
  - carapace 3y ago
    It took eight years for somebody to use this. We don't know when it was discovered (nor how many times by how many different people.)
    - bombela 3y ago
      I remember noticing this from the HTTP/2 RFC, maybe 8y ago. I was studying the head of line blocking issue on a custom protocol atop TCP and was curious to compare with HTTP/2. I think I might even have chatted with a coworker about it at the time, as he was implementing grpc (which uses HTTP/2) in Rust. It never occured to me that it could be used nefariously!
- is_true 3y ago
  This is the reason you need a security researcher that is actively exploiting things.
  - jacquesm 3y ago
    It is but most companies see that as a cost without upside until they get compromised.
- da_chicken 3y ago
  From what I can tell, people were talking about reset flooding that was dated back in July. Not as a novel thing, either. It's just one of several known vulnerabilities, suggesting they've known about this kind of thing for awhile. https://pentestmag.com/good-bad-and-the-ugly-of-http-2/ https://pentestmag.com/good-bad-and-the-ugly-of-http-2/ I genuinely don't know if this is real a zero day, or if it's a known protocol vulnerability that nobody was mitigating.
- fanf2 3y ago
  At least haproxy anticipated it https://www.mail-archive.com/haproxy@formilux.org/msg44134.html https://www.mail-archive.com/haproxy@formilux.org/msg44134.h...
fragmede 3y ago
> which typically translates HTTP/2 to HTTP/1 Sticking with HTTP/2, or going with grpc/similar is also possible. It depends on which corner of the Internet you inhabit. (Cloudflare isn't the whole Internet, yet)
deleted 3y ago
[deleted]

bflesch 3y ago

how on earth did nobody anticipate this kind of attack when designing the protocol? it's very obvious it can be abused like this

sebstefan 3y ago
Hindsight is 20/20
jsnell 3y ago
It took 8 years for somebody to discover this. It can't have been that obvious.
- wtarreau 3y ago
  > It took 8 years for somebody to discover this. It can't have been that obvious. Actually that's not true, it was already suggested here as a way to circumvent the max_concurrent_streams setting an it seemed particularly obvious: https://lists.w3.org/Archives/Public/ietf-http-wg/2019JanMar/0132.html https://lists.w3.org/Archives/Public/ietf-http-wg/2019JanMar... As soon as you start to implement a proxy that supports H2 on both sides, that's something you immediately spot, because setting too low timeouts on your first stage easily fills the second stage so you have to cover that case. I think that the reality is in fact that some big corp had several outages due to these attacks and it makes them look better to their customers to say "it's not our fault we had to fight zero-days" than "your service was running on half-baked stacks", so let's just go make a lot of noise about it to announce yet-another-end-of-the-net.
- mgaunard 3y ago
  Not everyone cares about Cloudflare, or even HTTP/2. The exploit has more to do with their implementation than the protocol.
  - deleted 3y ago
    [deleted]
  - formerly_proven 3y ago
    It's pretty similar to HTTP/1 pipelining, though no reverse proxy I'm aware of supports it.
  - insanitybit 3y ago
    > The exploit has more to do with their implementation than the protocol. Is it? I imagine that implementations can do things like make creating/dropping a stream faster but how would an implementation flat out mitigate this?
    - deleted 3y ago
      [deleted]
    - mgaunard 3y ago
      There is a maximum bandwidth at which data can arrive. Simply make sure you can always process it faster than the next packet can arrive, or implement proper mitigation in cases where you cannot. It's called programming under soft real-time constraints.
      - immibis 3y ago
        > Simply make sure you can always process it faster than the next packet can arrive This is pretty much impossible unless you make the client do a proof-of-work so they can't send requests very quickly. Okay, you could use a slow connection so that requests can't arrive very quickly, but then the DoS is upstream.
      - insanitybit 3y ago
        Well yeah, that's just how DoS kinda works with these sorts of vulns. "Be faster" is obviously a good strategy, but is it viable ? Is setting up and canceling a stream something that can be done at GB/s speeds? Maybe, idk.
- carapace 3y ago
  It took eight years for somebody to use this. We don't know when it was discovered (nor how many times by how many different people.)
  - bombela 3y ago
    I remember noticing this from the HTTP/2 RFC, maybe 8y ago. I was studying the head of line blocking issue on a custom protocol atop TCP and was curious to compare with HTTP/2. I think I might even have chatted with a coworker about it at the time, as he was implementing grpc (which uses HTTP/2) in Rust. It never occured to me that it could be used nefariously!
is_true 3y ago
This is the reason you need a security researcher that is actively exploiting things.
- jacquesm 3y ago
  It is but most companies see that as a cost without upside until they get compromised.
da_chicken 3y ago
From what I can tell, people were talking about reset flooding that was dated back in July. Not as a novel thing, either. It's just one of several known vulnerabilities, suggesting they've known about this kind of thing for awhile. https://pentestmag.com/good-bad-and-the-ugly-of-http-2/ https://pentestmag.com/good-bad-and-the-ugly-of-http-2/ I genuinely don't know if this is real a zero day, or if it's a known protocol vulnerability that nobody was mitigating.
fanf2 3y ago
At least haproxy anticipated it https://www.mail-archive.com/haproxy@formilux.org/msg44134.html https://www.mail-archive.com/haproxy@formilux.org/msg44134.h...

fragmede 3y ago

> which typically translates HTTP/2 to HTTP/1 Sticking with HTTP/2, or going with grpc/similar is also possible. It depends on which corner of the Internet you inhabit. (Cloudflare isn't the whole Internet, yet)

deleted 3y ago

[deleted]