3 ms·

It took 8 years for somebody to discover this. It can't have been that obvious.

by jsnell 3y ago

wtarreau 3y ago
> It took 8 years for somebody to discover this. It can't have been that obvious. Actually that's not true, it was already suggested here as a way to circumvent the max_concurrent_streams setting an it seemed particularly obvious: https://lists.w3.org/Archives/Public/ietf-http-wg/2019JanMar/0132.html https://lists.w3.org/Archives/Public/ietf-http-wg/2019JanMar... As soon as you start to implement a proxy that supports H2 on both sides, that's something you immediately spot, because setting too low timeouts on your first stage easily fills the second stage so you have to cover that case. I think that the reality is in fact that some big corp had several outages due to these attacks and it makes them look better to their customers to say "it's not our fault we had to fight zero-days" than "your service was running on half-baked stacks", so let's just go make a lot of noise about it to announce yet-another-end-of-the-net.
mgaunard 3y ago
Not everyone cares about Cloudflare, or even HTTP/2. The exploit has more to do with their implementation than the protocol.
- deleted 3y ago
  [deleted]
- formerly_proven 3y ago
  It's pretty similar to HTTP/1 pipelining, though no reverse proxy I'm aware of supports it.
- insanitybit 3y ago
  > The exploit has more to do with their implementation than the protocol. Is it? I imagine that implementations can do things like make creating/dropping a stream faster but how would an implementation flat out mitigate this?
  - deleted 3y ago
    [deleted]
  - mgaunard 3y ago
    There is a maximum bandwidth at which data can arrive. Simply make sure you can always process it faster than the next packet can arrive, or implement proper mitigation in cases where you cannot. It's called programming under soft real-time constraints.
    - immibis 3y ago
      > Simply make sure you can always process it faster than the next packet can arrive This is pretty much impossible unless you make the client do a proof-of-work so they can't send requests very quickly. Okay, you could use a slow connection so that requests can't arrive very quickly, but then the DoS is upstream.
    - insanitybit 3y ago
      Well yeah, that's just how DoS kinda works with these sorts of vulns. "Be faster" is obviously a good strategy, but is it viable ? Is setting up and canceling a stream something that can be done at GB/s speeds? Maybe, idk.
      - mgaunard 3y ago
        If you push arbitrary amount of pressure through a pipe that can only handle 1000 psi, you need a valve to release the excess pressure, or it will blow up. In the real world, pipes cannot put arbitrary pressure, so your constraint is more bounded than this. So if you receive 2000 psi but your pipes can only handle 1000, you just need a small component that can handle the 2000 to split the pressure in two, and you can handle it all without releasing any. The same applies to digital logic; it's always possible to build something such that you can guarantee processing within a bounded amount of time by optimizing and sizing the resources correctly. As the word "digital logic" suggests, these sorts of guarantees are more often applied when designing hardware than software, but they can apply to either.
    - 3y ago
- sophacles 3y ago
  Meanwhile several different HTTP/2 implementations are dropping fixes for this today.
- drewg123 3y ago
  Google was apparently DOSed by the same sort of attack: https://news.ycombinator.com/item?id=37831062 https://news.ycombinator.com/item?id=37831062
carapace 3y ago
It took eight years for somebody to use this. We don't know when it was discovered (nor how many times by how many different people.)
- bombela 3y ago
  I remember noticing this from the HTTP/2 RFC, maybe 8y ago. I was studying the head of line blocking issue on a custom protocol atop TCP and was curious to compare with HTTP/2. I think I might even have chatted with a coworker about it at the time, as he was implementing grpc (which uses HTTP/2) in Rust. It never occured to me that it could be used nefariously!

It took 8 years for somebody to discover this. It can't have been that obvious.

by jsnell 3y ago

It took 8 years for somebody to discover this. It can't have been that obvious.

> It took 8 years for somebody to discover this. It can't have been that obvious. Actually that's not true, it was already suggested here as a way to circumvent the max_concurrent_streams setting an it seemed particularly obvious: https://lists.w3.org/Archives/Public/ietf-http-wg/2019JanMar/0132.html https://lists.w3.org/Archives/Public/ietf-http-wg/2019JanMar... As soon as you start to implement a proxy that supports H2 on both sides, that's something you immediately spot, because setting too low timeouts on your first stage easily fills the second stage so you have to cover that case. I think that the reality is in fact that some big corp had several outages due to these attacks and it makes them look better to their customers to say "it's not our fault we had to fight zero-days" than "your service was running on half-baked stacks", so let's just go make a lot of noise about it to announce yet-another-end-of-the-net.

mgaunard 3y ago

Not everyone cares about Cloudflare, or even HTTP/2. The exploit has more to do with their implementation than the protocol.

deleted 3y ago
[deleted]
formerly_proven 3y ago
It's pretty similar to HTTP/1 pipelining, though no reverse proxy I'm aware of supports it.
insanitybit 3y ago
> The exploit has more to do with their implementation than the protocol. Is it? I imagine that implementations can do things like make creating/dropping a stream faster but how would an implementation flat out mitigate this?
- deleted 3y ago
  [deleted]
- mgaunard 3y ago
  There is a maximum bandwidth at which data can arrive. Simply make sure you can always process it faster than the next packet can arrive, or implement proper mitigation in cases where you cannot. It's called programming under soft real-time constraints.
  - immibis 3y ago
    > Simply make sure you can always process it faster than the next packet can arrive This is pretty much impossible unless you make the client do a proof-of-work so they can't send requests very quickly. Okay, you could use a slow connection so that requests can't arrive very quickly, but then the DoS is upstream.
  - insanitybit 3y ago
    Well yeah, that's just how DoS kinda works with these sorts of vulns. "Be faster" is obviously a good strategy, but is it viable ? Is setting up and canceling a stream something that can be done at GB/s speeds? Maybe, idk.
    - mgaunard 3y ago
      If you push arbitrary amount of pressure through a pipe that can only handle 1000 psi, you need a valve to release the excess pressure, or it will blow up. In the real world, pipes cannot put arbitrary pressure, so your constraint is more bounded than this. So if you receive 2000 psi but your pipes can only handle 1000, you just need a small component that can handle the 2000 to split the pressure in two, and you can handle it all without releasing any. The same applies to digital logic; it's always possible to build something such that you can guarantee processing within a bounded amount of time by optimizing and sizing the resources correctly. As the word "digital logic" suggests, these sorts of guarantees are more often applied when designing hardware than software, but they can apply to either.
  - 3y ago
sophacles 3y ago
Meanwhile several different HTTP/2 implementations are dropping fixes for this today.
drewg123 3y ago
Google was apparently DOSed by the same sort of attack: https://news.ycombinator.com/item?id=37831062 https://news.ycombinator.com/item?id=37831062

carapace 3y ago

It took eight years for somebody to use this. We don't know when it was discovered (nor how many times by how many different people.)

bombela 3y ago
I remember noticing this from the HTTP/2 RFC, maybe 8y ago. I was studying the head of line blocking issue on a custom protocol atop TCP and was curious to compare with HTTP/2. I think I might even have chatted with a coworker about it at the time, as he was implementing grpc (which uses HTTP/2) in Rust. It never occured to me that it could be used nefariously!