5 ms·

HTTP/2 Continuation Flood: Technical Details

296 points by campuscodi 3y ago

tzot 3y ago
I just love this typo: > After serveral retries
skywhopper 3y ago
Nice writeup and great find! Kudos to the author for taking such a broad approach and responsibly reporting their findings and finally for sharing the details in such a readable way.
nullindividual 3y ago
Previous article with impacted web servers/reverse proxies from the same author. https://nowotarski.info/http2-continuation-flood/ https://nowotarski.info/http2-continuation-flood/
pmlnr 3y ago
Now do this slowly, and you can call it slowloris v2 :(
mcmatterson 3y ago
I'd just mitigated this exact thing in Bandit last month! https://github.com/mtrudel/bandit/blob/main/lib/bandit/http2/connection.ex#L84 https://github.com/mtrudel/bandit/blob/main/lib/bandit/http2... TBH, from an implementors perspective this is a super obvious thing to cover off. It had long been on my radar and was something that I'd always figured other implementations had defended against as well.
- hinkley 3y ago
  Well you know what happens when we assume. You make a front page headline out of you and me.
  - eek2121 3y ago
    As someone who worked for a terrible startup that 'assumed' they would have scalability issues, engineered their entire software stack around solving said issues, and ended up with a worthless codebase that nobody could wrap their head around as a result, I feel this comment. Later they began a small refactor which easily handled the loads they were "assuming" could not be handled in the way that the refactor handled, and it was wildly successful and the code was much simpler to work on. To developers: don't over engineer. Most languages/frameworks/libraries can handle scale beyond what you'll ever get in your initial implementation. No, you entire website does NOT need to be asynchronous. It is very possible to have too many background jobs. I know this because I've seen the horror. I've also written an entire jobless/synchronous platform that serves millions of users without issue. If you run into scaling issues, that is a good problem to have. Tackle it as it happens. Bottom line is focus on secure, quality code above all else. Don't make assumptions.
    - snailscale 3y ago
      The default way we write applications is actually pretty scalable already. It always hurts to build something that “won’t scale” because it was framed as a negative. Realizing that something “scales” if it meets your current needs is pretty important. Framing scale in terms of how many people can work on it, how fast they can work on it, and how well it meets needs is often a better way of considering the “scale” of what your building. As you said, when request per second becomes a limiting factor you can adjust your scales but doing it from start rarely makes sense (largely because req / sec already scales pretty well)
      - hinkley 3y ago
        It’s often a fear or trauma response. Nobody wants to spend 6 months out of the year trying to keep the plates spinning, and they definitely don’t want to spend 60 hours a week when things get ahead of them. Everything takes twice as long as we think it will and we don’t trust that we can keep ahead of user demand. Many of us have experienced it or been adjacent, and for a long time after we overreact to that scenario. Because we don’t trust that we can keep the wheels on. Over time the memory fades, and the confidence improves, and we get more comfortable with things being okay instead of unassailable. But it can be a rough road until then.
jagger27 3y ago
> NOT affected: Nginx, Jetty, HAProxy, NetScaler, Varnish. [0] 0: https://nowotarski.info/http2-continuation-flood/ https://nowotarski.info/http2-continuation-flood/
- wtarreau 3y ago
  In other words the ones that have long been contesting the use of CONTINUATION due to the risk of DoS 10 years ago. Just read any of the long threads there to get an idea, it's always about how to avoid the nasty CONTINUATION: https://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep/thread.html https://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep... If at least it had been accepted to forbid it after a non-full HEADERS frame it would have been more robust but it was perceived that the encoding job itself could have been harder (byte boundaries in compressors etc). BTW I find it funny how we "rediscover" the same stuff every 10 years. Recently it was the well-known RESET_STREAM flood, now the CONTINUATION, soon it will probably be DATA frames of length zero, then single-byte WINDOW_UPDATES, then INITIAL_WINDOW SETTINGS that cost a lot of CPU, etc. The world is just circling in this security circus, provided it's possible to assign a name and possibly a logo to a known problem...
- bklyn11201 3y ago
  What about Caddy? It's a great project that deserves it's own line ;)
  - programd 3y ago
    On Ubuntu 22.04 LTS caddy from the Ubuntu apt repo is shown as on version 2.7.6 and built with Go 1.21.5. That version of Go does not have a fix for this issue. Caddy 2.7.6 is also the latest version released on GitHub. So no fix yet, but I think all that's needed is a recompile with the latest version of Go 1.22.2
    - nelse 3y ago
      I think that recompiling with upgraded Go will not solve the issue. It seems Caddy imports `golang.org/x/net/http2` and pins it to v0.22.0 which is vulnerable: https://github.com/caddyserver/caddy/issues/6219#issuecomment-2038265390 https://github.com/caddyserver/caddy/issues/6219#issuecommen....
shuntress 3y ago
HTTP/2 or How to Cram a Transport Layer "Upgrade" Into an Application Layer Protocol.
- MrStonedOne 3y ago
  [dead]
- deleted 3y ago
  [deleted]
unethical_ban 3y ago
This has been at the top all day. I wonder: For low-traffic websites, is it possible that running HTTP/1.1 is just safer?
- pornel 3y ago
  Not necessarily. Everyone saying HTTP/1.1 is simple has never implemented a full real-world-compatible parser for it. HTTP/1 has lots of unobvious edge cases, and legacy quirks. Text format has way more flexibility than it seems from valid headers. It has obscure features like multi-line headers and old MIME features, 100-continue race conditions, custom hop-by-hop headers, GET bodies. Fortunately new HTTP RFCs document many pitfalls. If you just implement what RFC 2616 said, you won't have a safe implementation. Actual size of a request or response can be specified in multiple ways (at the same time, with conflicting values), and depends on a combination of several features, and values of headers with weird parsing rules needed for backwards compat, so "simple" HTTP implementations can be tricked into request smuggling. Either way you need a robust, well-tested mature implementation.
- dexwiz 3y ago
  Probably. HTTP/2 is good for streaming, and even that is being replaced by newer protocols. For normal asset serving the only advantage is more assets can be loaded in parallel since HTTP/1 is limited on connections per domain. CDNs on different domains usually prevent this from being an issue. In theory you could serve unbundled JS assets via HTTP/2, but I have never seen it in production. Likely because you still need a compilation step most of the time.
- graemep 3y ago
  I was wondering about that. It is more mature an less complex so it seems probable it is safer.
- akira2501 3y ago
  Upgrading merely to upgrade is not good engineering practice. If you expect to receive no additional benefits from the upgrade then it is probably not justified.
userbinator 3y ago
In the last couple of months I checked dozens of implementations and, somehow, these protections were not implemented (or implemented incorrectly) even in major HTTP/2 servers I'll speak to the elephant in the room: this is what happens when you have an entire developer culture so used to automatically dynamically expanding everything and not caring how big it is, that they never think about how big something can be. This class of problems isn't necessarily restricted to HTTP/2, although its gross complexity probably contributes; it's just that in HTTP/1.x times, more developers would be used to languages like C where managing buffer lengths takes constant attention, and no one would bother to make header allocations expand limitlessly when they should be a few K in total at most for the whole request.
- 10000truths 3y ago
  The issue is that people constantly focus on and optimize for the happy path, but don't stop and think about what would happen if an adversary deliberately and repeatedly triggered the worst case scenario. So many denial-of-service attacks (slowloris, query parameter hash collisions, etc.) come into fruition because bounded resource usage is an afterthought.

296 points by campuscodi 3y ago

tzot 3y ago

I just love this typo: > After serveral retries

Nice writeup and great find! Kudos to the author for taking such a broad approach and responsibly reporting their findings and finally for sharing the details in such a readable way.

nullindividual 3y ago

Previous article with impacted web servers/reverse proxies from the same author. https://nowotarski.info/http2-continuation-flood/ https://nowotarski.info/http2-continuation-flood/

pmlnr 3y ago

Now do this slowly, and you can call it slowloris v2 :(

mcmatterson 3y ago

I'd just mitigated this exact thing in Bandit last month! https://github.com/mtrudel/bandit/blob/main/lib/bandit/http2/connection.ex#L84 https://github.com/mtrudel/bandit/blob/main/lib/bandit/http2... TBH, from an implementors perspective this is a super obvious thing to cover off. It had long been on my radar and was something that I'd always figured other implementations had defended against as well.

hinkley 3y ago
Well you know what happens when we assume. You make a front page headline out of you and me.
- eek2121 3y ago
  As someone who worked for a terrible startup that 'assumed' they would have scalability issues, engineered their entire software stack around solving said issues, and ended up with a worthless codebase that nobody could wrap their head around as a result, I feel this comment. Later they began a small refactor which easily handled the loads they were "assuming" could not be handled in the way that the refactor handled, and it was wildly successful and the code was much simpler to work on. To developers: don't over engineer. Most languages/frameworks/libraries can handle scale beyond what you'll ever get in your initial implementation. No, you entire website does NOT need to be asynchronous. It is very possible to have too many background jobs. I know this because I've seen the horror. I've also written an entire jobless/synchronous platform that serves millions of users without issue. If you run into scaling issues, that is a good problem to have. Tackle it as it happens. Bottom line is focus on secure, quality code above all else. Don't make assumptions.
  - snailscale 3y ago
    The default way we write applications is actually pretty scalable already. It always hurts to build something that “won’t scale” because it was framed as a negative. Realizing that something “scales” if it meets your current needs is pretty important. Framing scale in terms of how many people can work on it, how fast they can work on it, and how well it meets needs is often a better way of considering the “scale” of what your building. As you said, when request per second becomes a limiting factor you can adjust your scales but doing it from start rarely makes sense (largely because req / sec already scales pretty well)
    - hinkley 3y ago
      It’s often a fear or trauma response. Nobody wants to spend 6 months out of the year trying to keep the plates spinning, and they definitely don’t want to spend 60 hours a week when things get ahead of them. Everything takes twice as long as we think it will and we don’t trust that we can keep ahead of user demand. Many of us have experienced it or been adjacent, and for a long time after we overreact to that scenario. Because we don’t trust that we can keep the wheels on. Over time the memory fades, and the confidence improves, and we get more comfortable with things being okay instead of unassailable. But it can be a rough road until then.

jagger27 3y ago

> NOT affected: Nginx, Jetty, HAProxy, NetScaler, Varnish. [0] 0: https://nowotarski.info/http2-continuation-flood/ https://nowotarski.info/http2-continuation-flood/

wtarreau 3y ago
In other words the ones that have long been contesting the use of CONTINUATION due to the risk of DoS 10 years ago. Just read any of the long threads there to get an idea, it's always about how to avoid the nasty CONTINUATION: https://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep/thread.html https://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep... If at least it had been accepted to forbid it after a non-full HEADERS frame it would have been more robust but it was perceived that the encoding job itself could have been harder (byte boundaries in compressors etc). BTW I find it funny how we "rediscover" the same stuff every 10 years. Recently it was the well-known RESET_STREAM flood, now the CONTINUATION, soon it will probably be DATA frames of length zero, then single-byte WINDOW_UPDATES, then INITIAL_WINDOW SETTINGS that cost a lot of CPU, etc. The world is just circling in this security circus, provided it's possible to assign a name and possibly a logo to a known problem...
bklyn11201 3y ago
What about Caddy? It's a great project that deserves it's own line ;)
- programd 3y ago
  On Ubuntu 22.04 LTS caddy from the Ubuntu apt repo is shown as on version 2.7.6 and built with Go 1.21.5. That version of Go does not have a fix for this issue. Caddy 2.7.6 is also the latest version released on GitHub. So no fix yet, but I think all that's needed is a recompile with the latest version of Go 1.22.2
  - nelse 3y ago
    I think that recompiling with upgraded Go will not solve the issue. It seems Caddy imports `golang.org/x/net/http2` and pins it to v0.22.0 which is vulnerable: https://github.com/caddyserver/caddy/issues/6219#issuecomment-2038265390 https://github.com/caddyserver/caddy/issues/6219#issuecommen....

shuntress 3y ago

HTTP/2 or How to Cram a Transport Layer "Upgrade" Into an Application Layer Protocol.

MrStonedOne 3y ago
[dead]
deleted 3y ago
[deleted]

unethical_ban 3y ago

This has been at the top all day. I wonder: For low-traffic websites, is it possible that running HTTP/1.1 is just safer?

pornel 3y ago
Not necessarily. Everyone saying HTTP/1.1 is simple has never implemented a full real-world-compatible parser for it. HTTP/1 has lots of unobvious edge cases, and legacy quirks. Text format has way more flexibility than it seems from valid headers. It has obscure features like multi-line headers and old MIME features, 100-continue race conditions, custom hop-by-hop headers, GET bodies. Fortunately new HTTP RFCs document many pitfalls. If you just implement what RFC 2616 said, you won't have a safe implementation. Actual size of a request or response can be specified in multiple ways (at the same time, with conflicting values), and depends on a combination of several features, and values of headers with weird parsing rules needed for backwards compat, so "simple" HTTP implementations can be tricked into request smuggling. Either way you need a robust, well-tested mature implementation.
dexwiz 3y ago
Probably. HTTP/2 is good for streaming, and even that is being replaced by newer protocols. For normal asset serving the only advantage is more assets can be loaded in parallel since HTTP/1 is limited on connections per domain. CDNs on different domains usually prevent this from being an issue. In theory you could serve unbundled JS assets via HTTP/2, but I have never seen it in production. Likely because you still need a compilation step most of the time.
graemep 3y ago
I was wondering about that. It is more mature an less complex so it seems probable it is safer.
akira2501 3y ago
Upgrading merely to upgrade is not good engineering practice. If you expect to receive no additional benefits from the upgrade then it is probably not justified.

userbinator 3y ago

In the last couple of months I checked dozens of implementations and, somehow, these protections were not implemented (or implemented incorrectly) even in major HTTP/2 servers I'll speak to the elephant in the room: this is what happens when you have an entire developer culture so used to automatically dynamically expanding everything and not caring how big it is, that they never think about how big something can be. This class of problems isn't necessarily restricted to HTTP/2, although its gross complexity probably contributes; it's just that in HTTP/1.x times, more developers would be used to languages like C where managing buffer lengths takes constant attention, and no one would bother to make header allocations expand limitlessly when they should be a few K in total at most for the whole request.

10000truths 3y ago
The issue is that people constantly focus on and optimize for the happy path, but don't stop and think about what would happen if an adversary deliberately and repeatedly triggered the worst case scenario. So many denial-of-service attacks (slowloris, query parameter hash collisions, etc.) come into fruition because bounded resource usage is an afterthought.