12 ms·

HTTP/2-exclusive threats caused by implementation flaws and RFC imperfections

393 points by Berg0X00 5y ago

carey 5y ago
Given the long history of request parsing vulnerabilities in HTTP/1.1 servers and proxies, is HTTP/2 actually worse, or have most of the HTTP/1.1 bugs just been fixed already?
- rubiquity 5y ago
  When I implemented an HTTP2 server several years ago it was all of the "fun" of HTTP 1.1 parsing and semantics plus the extra challenges of the HTTP2 optimizations such as HPACK, mapping the abbreviated headers to cache in-memory representations, stream management, and if you supported Push Promises then that too.
- josefx 5y ago
  Is an new bucket leaking in a dozen places worse than an old one with all leaks fixed? I would say yes until those holes in the new one are also fixed.
- chrismorgan 5y ago
  These vulnerabilities are all from badly-written HTTP/2 → HTTP/1.1 translations. Most of them come from simple carelessness, rookie errors that should never have been made, dumping untrusted bytes from an HTTP/2 value into the HTTP/1.1 byte stream. This is security 101, straightforward injection attacks with absolutely nothing HTTP-specific in it. Some of them are a little more complex, requiring actual HTTP/2 and HTTP/1.1 knowledge (largely meaning HTTP/2 framing and the content-length and transfer-encoding headers), but not most of them. Is HTTP/2 actually worse? Not in the slightest; HTTP/1.1 is the problem here. This is growing pains from compatibility measures as part of removing the problems of an unstructured text protocol. If you have a pure-HTTP/2 system and don’t ever do the downgrade, you’re in a better position.
  - comex 5y ago
    I'd agree that HTTP/1 deserves a significant portion of the blame. On the other hand, one maxim I've learned from my time bug hunting is that nobody ever validates strings in binary protocols. As such, I'm utterly unsurprised there are so many implementations with these kinds of bugs, and I'd say they could have been predicted in advance. In fact… let's see… yep, they were predicted. Some of them, at least. In the HTTP/2 RFC, under Security Considerations, 10.3 'Intermediary Encapsulation Attacks' describes one of the attack classes from the blog post, the one involving stuffing newlines into header names. Does that mean something could have been done about it? Perhaps not. The ideal solution would be to somehow design the HTTP/2 protocol itself to be resistant to misimplementation, but that seems pretty much impossible. The spec already bans colons and newlines in header names, but there's no way to be sure implementations won't allow them anyway, short of actually making them a delimiter like HTTP/1 did – in other words, reverting to a text-based protocol. But a text-based protocol would come with its own share of misimplementation risks, the same ones that HTTP/1 has. On the other hand, perhaps the bug classes could have been mitigated if someone designed test cases to trigger them, and either included them in conformance tests (apparently there was an official HTTP/2 test suite [2] though it doesn't seem to have been very popular), or set up some kind of bot to try them on the entire web. In principle you could blame the authors of HTTP/2 collectively for the fact that nobody did this. But I admit that's pretty handwavey. [1] https://datatracker.ietf.org/doc/html/rfc7540#section-10.3 https://datatracker.ietf.org/doc/html/rfc7540#section-10.3 [2] https://github.com/http2/http2-test https://github.com/http2/http2-test
    - atoav 5y ago
      > On the other hand, one maxim I've learned from my time bug hunting is that nobody ever validates strings in binary protocols. I wonder how much this has to do with the way strings need to be handled in the programming languages these protocols are implemented in. If dealing with strings is something that seems to be even more of a danger (if done incorrectly) you might just not do it.
  - tptacek 5y ago
    It's tough to say that something is a "rookie error" when basically every serious professional team makes the same mistake. This broke apparently broke every AWS ALB, for instance.
RcouF1uZ4gsC 5y ago
> This means there's little room for ambiguity about the length of a message, and might leave you wondering how desync attacks using HTTP/2 are possible. The answer is HTTP/2 downgrading. I would guess a huge percentage of attacks are made possible because of protocol or algorithm downgrading. I wonder if built in downgrading abilities into protocols is a security smell.
- yyyk 5y ago
  The shown attacks aren't a true case of protocol downgrading, but of an overeager direct translation of request headers across protocols. The attacks weren't possible because the client is doing some backhanded protocol negotiation, but because the client/front/back are being very directly connected. If, instead of forwarding a direct translation of the headers, the front ends calculated the relevant requests and sent that, there wouldn't have been any HTTP header attacks (it would have been much slower though).
deleted 5y ago
[deleted]
jiggawatts 5y ago
I've been burnt one too many times by trying to "upgrade" to HTTP/2. It simply broke client certificate authentication and NTLM authentication, leaving only cookie-based authentication fully functional. Can you guess which two of the three popular authentication mechanisms Google doesn't use? It's not a protocol designed to advance the Internet, it's a protocol designed by Google to shave 1% off their network bill.
- quicklime 5y ago
  To be fair, NTLM authentication being broken has nothing to do with HTTP/2.
- georgyo 5y ago
  I always considere NTLM and Kerberos to be very similar, so I am surprised to hear that NTLM is broken. Kerberos and HTTP/2 work just fine together. What in particular is broken with NTLM?
  - edoceo 5y ago
    NTLM
  - carey 5y ago
    At least when it was released, IIS 10 supported HTTP/2, but did not support Kerberos, Negotiate or NTLM authentication over HTTP/2. https://docs.microsoft.com/en-us/iis/get-started/whats-new-in-iis-10/http2-on-iis https://docs.microsoft.com/en-us/iis/get-started/whats-new-i... I’m not sure that it’s a problem with HTTP/2 that Microsoft found no reason to implement Windows authentication for it.
    - jiggawatts 5y ago
      The HTTP/2 protocol specifically makes this kind of authentication difficult. It's not just IIS, in general it doesn't quite work right.
  - marcosdumay 5y ago
nerdponx 5y ago
I'm not a security person at all. Is there a quick explanation for how these attacks allow the attacker to mess with other clients' requests? I'm also confused at their usage of the word "prefix". Did they mean "suffix"? Or does "prefix" have a specific meaning in this case?
- remram 5y ago
  The next client's request gets sent after your request. If you get frontend and backend servers to desync, the backend gets to your first request, responds, and the response is sent to you; then the backend gets your second request + the next client's request, responds, and the response is sent to that next client. This means you are making other clients get responses for their request prefixed by your smuggled second request.
- deleted 5y ago
  [deleted]
d_meeze 5y ago
Burying the lede... Unfortunately, at the time of this whitepaper being published - 86 days after Apache was notified of the vulnerability - 2.4.49 not yet come out, so although there's a patch on master, this is effectively a zero-day.
- slivanes 5y ago
  I'm not seeing anything about this in nginx changelogs (or matching CVE's) either. Disabling http2 for now.
  - yxhuvud 5y ago
    Why would an Apache bug show up in Nginx changelogs?
    - masklinn 5y ago
      It would not, but given how widespread this issue is it seems likely nginx is vulnerable, and sensible to assume it is unless otherwise demonstrated. It’s not like the feature is super useful in general.
paco3346 5y ago
Really good explanations here. As a Cloudflare Enterprise customer I've opened a case with them to see if any of these vulnerabilities apply to their WAF. I was a bit surprised to see the article mention that Imperva's WAF was vulnerable. edit: Cloudflare has responded that "[they] are asking your question internally."
- albinowax_ 5y ago
  As far as I'm aware, none of the techniques referenced in this paper work on Cloudflare as-is. Of course I probably missed a bunch of variations, so it's still worth them doing some internal checks.
- El_RIDO 5y ago
  You can partially answer this question yourself. So the issue can occur, when HTTP/2 requests get translated into HTTP/1.1 calls. Your HTTP calls stack from client to web app presumably looks something like this: Client → (HTTP/2) → Cloudflare ingress → (???) → Cloudflare egress → (???) → Your endpoint → (???) → Your web app On your endpoint (LB or webserver) logs you can check if the requests coming in from Cloudbees are using HTTP/2 or not. If they do use HTTP/2, they internally might still do translation down to HTTP/1.1 and back. But more importantly, you should investigate how your LB or webserver talks to the web app. In many cases that part won't be TLS encrypted and therefore on an HTTP/1.1 channel. The exception is if your outward facing, TLS terminating, webserver interacts with the web app via CGI or FastCGI. While that part of the call stack would not be affected by this particular attack, (Fast)CGI comes with it's own set of risks. In the end, the only part that you can't easily validate yourself of being susceptible to this attack (aka having to translate between HTTP/2 and HTTP/1.1) is the bit that occurs internally to Cloudflare.
  - hmsimha 5y ago
    > But more importantly, you should investigate how your LB or webserver talks to the web app. In many cases that part won't be TLS encrypted and therefore on an HTTP/1.1 channel. Or even HTTP 1.0 . I found out recently while inspecting some requests in the upstream server that nginx was using HTTP 1.0 after terminating TLS. I was dumbfounded that this was still the default. http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_http_version http://nginx.org/en/docs/http/ngx_http_proxy_module.html#pro...
not2b 5y ago
It seems most of the vulnerabilities here (at least, the first several that I read in the article) come at a point where an HTTP/2 request is transformed into an HTTP 1.1 request. So the insecurity seems to come at the point where a translation occurs and there is redundant information that can be handled in several ways, meaning that there's lots of room for bugs.
mhitza 5y ago
So I was scratching my head for a good couple of minutes trying to figure out how this works, being familiar only with HTTP Response Splitting/HTTP Cache Poisoning. So it seemed that somewhere along the years while I haven't been paying any attention to websec, it became a common practice to send requests from different clients through the same TLS connection. And due to the non conforming way HTTP 2/1.1 interop was implemented by these webservers/load balancers request boundaries were not delimited correctly, making it possible to inject requests on behalf of followup clients. Fine I get the issue. Sounds like another "good enough" optimization that backfired. What is the solution aside from playing the patching wack-a-mole game? Should maybe a HTTP2.1 protocol work without a strict TLS requirement so that protocol downgrades aren't necessary to squeeze out extra performance (unless I misunderstand why the HTTP2/HTTP1.1 bridge was in place). Or is the problem that some application servers still don't support HTTP2 out of the box?
- tptacek 5y ago
  It's worth knowing that this is an extension of a very widespread attack on HTTP/1.1; many (maybe most?) 1.1 implementations were recently broken by desync just a couple years ago.
- ikiris 5y ago
  this kind of terrible implementation is a lot of why encrypted quic exists.
  - georgyo 5y ago
    Though I think these same problems would happen over http3/quic. Http3 is almost exactly http2 but over quic. This means the requests should be possible to force the same desyncs talked about in the article.
- oefrha 5y ago
  Out of the ecosystems I’m familiar with, Python application servers have terrible http2 support: neither gunicorn nor uwsgi supports it, and even new hotness like uvicorn is pretty far from it. I don’t think Ruby is doing much better? Correct me if I’m wrong.
paco3346 5y ago
In the reddit thread someone posted the recording: https://www.youtube.com/watch?v=rHxVVeM9R-M https://www.youtube.com/watch?v=rHxVVeM9R-M
bastawhiz 5y ago
I'd love to see a post from someone about their experiences using H2 between servers (i.e., instead of H1). Many big companies already use TLS internally, and I can imagine perf wins.
dagenix 5y ago
The title seem very much anti-HTTP/2. However, unless I missed something huge, the vast majority, if not all, of the issues found relate to how hard it is to handle HTTP/1. And it seems kinda odd to blame HTTP/2 for HTTP/1 being difficult to implement. I'm certainly not trying to downplay the seriousness of these issues. But it seems like a equally (it not more) valid title might be something like "HTTP/1: Continues to rear its ugly head" or something like that.
- stkdump 5y ago
  I guess this is more in line with the narrative that older protocols that people are more familiar with are somehow always better. That there is a hidden evil agenda for introducing something new. Or at least that we all need to pay a price for the sole benefit of a tech giant. It is a convenient narrative, which could be true, but it feels too easy a conclusion. Which makes it a good clickbait.
  - dathinab 5y ago
    > It is a convenient narrative It's also not he narrative the content of the article purses ;-)
- albinowax_ 5y ago
  Hi, I'm the author. This paper is about HTTP/2, and the dangerous things that happen behind the scenes with HTTP/1 when people enable it. It's arguing that enabling HTTP/2 on your front-end makes your security worse if it's downgrading behind the scenes. This is a deliberately cheeky title, selected as a presentation title, rather than being aimed at social media where many people will read the title but not the contents. For HN I'd go with something dry like "HTTP/2-exclusive attack vectors".
  - jpambrun 5y ago
    My employer's mandatory proxy downgrades all HTTP/2, presumably so they can better inspect traffic. This results in a bad/slow experience on many sites. I wish HTTP/2 would not have downgrade at all so they would have to deal with it..
get52 5y ago
I can't believe this, http and the internet as a whole is basically getting horsefucked
gouda-gouda 5y ago
That JIRA exploit is mind-boggling bad. Posts like this remind me that we work on a house cards, not a fortress.
mintyc 5y ago
Wow. A fantastic set of descriptions, service and detection tooling from portswigger and colleagues. The fact that http/1 downgrading is so prevalent and unlikely to change in the short term is the point.
albinowax_ 5y ago
Hi, I'm the author - please let me know if you have any questions!
- brabel 5y ago
  Just want to thank you for publishing this and taking the time to present it in such a clear manner. As someone who has implemented HTTP/1.1 message parsing myself, it blows my mind how downgrade implementers could've made such basic mistakes as to not validate the contents of headers. The companies selling security software with these mistakes should be held accountable IMHO, as the attacks are so simple and so many that they've probably been exploited extensively in the wild, and similar attacks almost certainly continue to be.
- mwcampbell 5y ago
  In light of these vulnerabilities, should we disable Internet-facing use of HTTP/2 wherever we can?
  - albinowax_ 5y ago
    I think HTTP/2 is fine when it's used end to end. So if you've got a single webserver setup, or a reverse proxy that speaks HTTP/2 to the back-end, it's great. However, if the only way you can use HTTP/2 is by having the front-end downgrade it to HTTP/1, I would recommend disabling it.
    - wtarreau 5y ago
      In fact it will depend on the implementations. As you've found, the root cause of the problem is that H1 and H2 use different delimiting techniques for protocol elements. Some implementations need to transcode H2 into an internal representation. In this case, and provided the protocol elements are checked in the intermediate representation, any protocol on the other side will be fine. But if the internal representation is H1-like with poor message delimitation, you could very well end up splitting requests internally and emitting 2 H2 requests on the backend for a single front H2 request for example. I'd say that the main cause of these issues is the directive language used in the H2 spec almost only saying "this must be done like this" without giving rationale for the rules, meaning that implementations that were unable to implement them exactly the same way were left with no hint about what the trouble was nor how to address it. The new spec in progress and the extraction of the core semantics makes the whole thing a lot cleaner. In addition, H2 uses much more resources per connection than H1, which encourages to coalesce them between the proxy and the server. But coalescing connections can easily cause some head-of-line blocking if coalesced front streams show different bandwidths. So even end-to-end H2 is not always a panacea.
nly 5y ago
I'm a bit confused how these attacks can be used. Inconsistencies between HTTP/1.1 and HTTP/2 allow for these behaviours but you still need to inject these malicious prefixes in to a user's request flow, right? Can you actually set the Content-Length header from a browser on a HTTP/2 request? And isn't the suffix just ignored by the final remote (or does this depend on HTTP/1.1 pipelining?)
- albinowax_ 5y ago
  Hi, I'm the author. This paper is building on techniques explained here: https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn https://portswigger.net/research/http-desync-attacks-request... This attack does not require a MITM - the attacker would use a tool like Burp Suite to issue the (technically RFC-violating) HTTP request. The prefix injection happens because front-end places the attacker's request and the victim's request on the same HTTP/1.1 connection to the back-end, as shown in this diagram: https://portswigger.net/cms/images/9c/c1/4c32-article-http2-desync.png https://portswigger.net/cms/images/9c/c1/4c32-article-http2-... > isn't the suffix just ignored by the final remote The back-end treats the suffix as the start of the next request, due to TCP buffering. The vast majority of servers have this kind of accidental pipelining support thanks to TCP.
- twic 5y ago
  The target's own front-end proxy acts as the man in the middle!
coffeetrackpad 5y ago
Thanks for this @albinowax_ - super interesting! Just a quick question about the H2.CL case study. Did Netflix have a separate vulnerability where their server wrongly trusted Host headers with netflix.com suffix and returned a 302 response which can redirected users to an arbitrary host? I'm just trying to see whether I interpret the case study correctly.
- albinowax_ 5y ago
  Yes that's right. If the back-end received a request that didn't contain X-Forwarded-SSL or suchlike, and had a host-header that ended in .netflix.com, it would redirect you to the host-header. I wouldn't exactly class this as a vulnerability - more of a useful gadget. The front-end would refuse to forward such a request so it's impossible to hit this code path without request smuggling. Even if you could hit it without request smuggling it would still be useless.
gnfargbl 5y ago
@albinowax_, can you say anything more about how widely you have investigated the prevalence of the "H2.X via Request Splitting" vulnerability? Although all of your attack vectors are fascinating, this seems to be the one which is particularly terrifying. It seems like one single request can basically mess up user authentication across an entire website, and for every active user, until the server becomes resynchronized somehow. If I were running a large public-facing site or application, checking for this vulnerability would be my #1 priority this morning.
- albinowax_ 5y ago
  Good question! So, my understanding is that the majority of servers that are vulnerable to regular cross-user HTTP Request Smuggling (IE, reuse connections to the back-end server) are exposed to this terrifying response queue poisoning attack. This applies to all desync types CL.TE, TE.CL, H2.CL, etc. The reason I discovered this in the H2.X case, is because it's particularly easy to trigger response queue poisoning by accident in this scenario. That said, I haven't actually tested this on very many live servers, for obvious reasons!
terom 5y ago
So if AWS ALB HTTP/2 listener -> HTTP/1.1 target downgrade was vulnerable, and the recommendation is to use HTTP/2 end to end... am I reading the AWS docs [1] correctly that ALB only supports HTTP/2 -> HTTP/1.1 downgrades, but not HTTP/1.1 -> HTTP/2 upgrades? In other words, using HTTP/2 as the target group protocol only works for HTTP/2 clients, and HTTP/1.1 clients will receive an error? [1] https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html#target-group-protocol-version https://docs.aws.amazon.com/elasticloadbalancing/latest/appl...

393 points by Berg0X00 5y ago

Given the long history of request parsing vulnerabilities in HTTP/1.1 servers and proxies, is HTTP/2 actually worse, or have most of the HTTP/1.1 bugs just been fixed already?

rubiquity 5y ago
When I implemented an HTTP2 server several years ago it was all of the "fun" of HTTP 1.1 parsing and semantics plus the extra challenges of the HTTP2 optimizations such as HPACK, mapping the abbreviated headers to cache in-memory representations, stream management, and if you supported Push Promises then that too.
josefx 5y ago
Is an new bucket leaking in a dozen places worse than an old one with all leaks fixed? I would say yes until those holes in the new one are also fixed.
chrismorgan 5y ago
These vulnerabilities are all from badly-written HTTP/2 → HTTP/1.1 translations. Most of them come from simple carelessness, rookie errors that should never have been made, dumping untrusted bytes from an HTTP/2 value into the HTTP/1.1 byte stream. This is security 101, straightforward injection attacks with absolutely nothing HTTP-specific in it. Some of them are a little more complex, requiring actual HTTP/2 and HTTP/1.1 knowledge (largely meaning HTTP/2 framing and the content-length and transfer-encoding headers), but not most of them. Is HTTP/2 actually worse? Not in the slightest; HTTP/1.1 is the problem here. This is growing pains from compatibility measures as part of removing the problems of an unstructured text protocol. If you have a pure-HTTP/2 system and don’t ever do the downgrade, you’re in a better position.
- comex 5y ago
  I'd agree that HTTP/1 deserves a significant portion of the blame. On the other hand, one maxim I've learned from my time bug hunting is that nobody ever validates strings in binary protocols. As such, I'm utterly unsurprised there are so many implementations with these kinds of bugs, and I'd say they could have been predicted in advance. In fact… let's see… yep, they were predicted. Some of them, at least. In the HTTP/2 RFC, under Security Considerations, 10.3 'Intermediary Encapsulation Attacks' describes one of the attack classes from the blog post, the one involving stuffing newlines into header names. Does that mean something could have been done about it? Perhaps not. The ideal solution would be to somehow design the HTTP/2 protocol itself to be resistant to misimplementation, but that seems pretty much impossible. The spec already bans colons and newlines in header names, but there's no way to be sure implementations won't allow them anyway, short of actually making them a delimiter like HTTP/1 did – in other words, reverting to a text-based protocol. But a text-based protocol would come with its own share of misimplementation risks, the same ones that HTTP/1 has. On the other hand, perhaps the bug classes could have been mitigated if someone designed test cases to trigger them, and either included them in conformance tests (apparently there was an official HTTP/2 test suite [2] though it doesn't seem to have been very popular), or set up some kind of bot to try them on the entire web. In principle you could blame the authors of HTTP/2 collectively for the fact that nobody did this. But I admit that's pretty handwavey. [1] https://datatracker.ietf.org/doc/html/rfc7540#section-10.3 https://datatracker.ietf.org/doc/html/rfc7540#section-10.3 [2] https://github.com/http2/http2-test https://github.com/http2/http2-test
  - atoav 5y ago
    > On the other hand, one maxim I've learned from my time bug hunting is that nobody ever validates strings in binary protocols. I wonder how much this has to do with the way strings need to be handled in the programming languages these protocols are implemented in. If dealing with strings is something that seems to be even more of a danger (if done incorrectly) you might just not do it.
- tptacek 5y ago
  It's tough to say that something is a "rookie error" when basically every serious professional team makes the same mistake. This broke apparently broke every AWS ALB, for instance.

RcouF1uZ4gsC 5y ago

> This means there's little room for ambiguity about the length of a message, and might leave you wondering how desync attacks using HTTP/2 are possible. The answer is HTTP/2 downgrading. I would guess a huge percentage of attacks are made possible because of protocol or algorithm downgrading. I wonder if built in downgrading abilities into protocols is a security smell.

yyyk 5y ago
The shown attacks aren't a true case of protocol downgrading, but of an overeager direct translation of request headers across protocols. The attacks weren't possible because the client is doing some backhanded protocol negotiation, but because the client/front/back are being very directly connected. If, instead of forwarding a direct translation of the headers, the front ends calculated the relevant requests and sent that, there wouldn't have been any HTTP header attacks (it would have been much slower though).

deleted 5y ago

[deleted]

jiggawatts 5y ago

I've been burnt one too many times by trying to "upgrade" to HTTP/2. It simply broke client certificate authentication and NTLM authentication, leaving only cookie-based authentication fully functional. Can you guess which two of the three popular authentication mechanisms Google doesn't use? It's not a protocol designed to advance the Internet, it's a protocol designed by Google to shave 1% off their network bill.

quicklime 5y ago
To be fair, NTLM authentication being broken has nothing to do with HTTP/2.
georgyo 5y ago
I always considere NTLM and Kerberos to be very similar, so I am surprised to hear that NTLM is broken. Kerberos and HTTP/2 work just fine together. What in particular is broken with NTLM?
- edoceo 5y ago
  NTLM
- carey 5y ago
  At least when it was released, IIS 10 supported HTTP/2, but did not support Kerberos, Negotiate or NTLM authentication over HTTP/2. https://docs.microsoft.com/en-us/iis/get-started/whats-new-in-iis-10/http2-on-iis https://docs.microsoft.com/en-us/iis/get-started/whats-new-i... I’m not sure that it’s a problem with HTTP/2 that Microsoft found no reason to implement Windows authentication for it.
  - jiggawatts 5y ago
    The HTTP/2 protocol specifically makes this kind of authentication difficult. It's not just IIS, in general it doesn't quite work right.
- marcosdumay 5y ago

nerdponx 5y ago

I'm not a security person at all. Is there a quick explanation for how these attacks allow the attacker to mess with other clients' requests? I'm also confused at their usage of the word "prefix". Did they mean "suffix"? Or does "prefix" have a specific meaning in this case?

remram 5y ago
The next client's request gets sent after your request. If you get frontend and backend servers to desync, the backend gets to your first request, responds, and the response is sent to you; then the backend gets your second request + the next client's request, responds, and the response is sent to that next client. This means you are making other clients get responses for their request prefixed by your smuggled second request.
deleted 5y ago
[deleted]

d_meeze 5y ago

Burying the lede... Unfortunately, at the time of this whitepaper being published - 86 days after Apache was notified of the vulnerability - 2.4.49 not yet come out, so although there's a patch on master, this is effectively a zero-day.

slivanes 5y ago
I'm not seeing anything about this in nginx changelogs (or matching CVE's) either. Disabling http2 for now.
- yxhuvud 5y ago
  Why would an Apache bug show up in Nginx changelogs?
  - masklinn 5y ago
    It would not, but given how widespread this issue is it seems likely nginx is vulnerable, and sensible to assume it is unless otherwise demonstrated. It’s not like the feature is super useful in general.

paco3346 5y ago

Really good explanations here. As a Cloudflare Enterprise customer I've opened a case with them to see if any of these vulnerabilities apply to their WAF. I was a bit surprised to see the article mention that Imperva's WAF was vulnerable. edit: Cloudflare has responded that "[they] are asking your question internally."

albinowax_ 5y ago
As far as I'm aware, none of the techniques referenced in this paper work on Cloudflare as-is. Of course I probably missed a bunch of variations, so it's still worth them doing some internal checks.
El_RIDO 5y ago
You can partially answer this question yourself. So the issue can occur, when HTTP/2 requests get translated into HTTP/1.1 calls. Your HTTP calls stack from client to web app presumably looks something like this: Client → (HTTP/2) → Cloudflare ingress → (???) → Cloudflare egress → (???) → Your endpoint → (???) → Your web app On your endpoint (LB or webserver) logs you can check if the requests coming in from Cloudbees are using HTTP/2 or not. If they do use HTTP/2, they internally might still do translation down to HTTP/1.1 and back. But more importantly, you should investigate how your LB or webserver talks to the web app. In many cases that part won't be TLS encrypted and therefore on an HTTP/1.1 channel. The exception is if your outward facing, TLS terminating, webserver interacts with the web app via CGI or FastCGI. While that part of the call stack would not be affected by this particular attack, (Fast)CGI comes with it's own set of risks. In the end, the only part that you can't easily validate yourself of being susceptible to this attack (aka having to translate between HTTP/2 and HTTP/1.1) is the bit that occurs internally to Cloudflare.
- hmsimha 5y ago
  > But more importantly, you should investigate how your LB or webserver talks to the web app. In many cases that part won't be TLS encrypted and therefore on an HTTP/1.1 channel. Or even HTTP 1.0 . I found out recently while inspecting some requests in the upstream server that nginx was using HTTP 1.0 after terminating TLS. I was dumbfounded that this was still the default. http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_http_version http://nginx.org/en/docs/http/ngx_http_proxy_module.html#pro...

not2b 5y ago

It seems most of the vulnerabilities here (at least, the first several that I read in the article) come at a point where an HTTP/2 request is transformed into an HTTP 1.1 request. So the insecurity seems to come at the point where a translation occurs and there is redundant information that can be handled in several ways, meaning that there's lots of room for bugs.

mhitza 5y ago

So I was scratching my head for a good couple of minutes trying to figure out how this works, being familiar only with HTTP Response Splitting/HTTP Cache Poisoning. So it seemed that somewhere along the years while I haven't been paying any attention to websec, it became a common practice to send requests from different clients through the same TLS connection. And due to the non conforming way HTTP 2/1.1 interop was implemented by these webservers/load balancers request boundaries were not delimited correctly, making it possible to inject requests on behalf of followup clients. Fine I get the issue. Sounds like another "good enough" optimization that backfired. What is the solution aside from playing the patching wack-a-mole game? Should maybe a HTTP2.1 protocol work without a strict TLS requirement so that protocol downgrades aren't necessary to squeeze out extra performance (unless I misunderstand why the HTTP2/HTTP1.1 bridge was in place). Or is the problem that some application servers still don't support HTTP2 out of the box?

tptacek 5y ago
It's worth knowing that this is an extension of a very widespread attack on HTTP/1.1; many (maybe most?) 1.1 implementations were recently broken by desync just a couple years ago.
ikiris 5y ago
this kind of terrible implementation is a lot of why encrypted quic exists.
- georgyo 5y ago
  Though I think these same problems would happen over http3/quic. Http3 is almost exactly http2 but over quic. This means the requests should be possible to force the same desyncs talked about in the article.
oefrha 5y ago
Out of the ecosystems I’m familiar with, Python application servers have terrible http2 support: neither gunicorn nor uwsgi supports it, and even new hotness like uvicorn is pretty far from it. I don’t think Ruby is doing much better? Correct me if I’m wrong.

paco3346 5y ago

In the reddit thread someone posted the recording: https://www.youtube.com/watch?v=rHxVVeM9R-M https://www.youtube.com/watch?v=rHxVVeM9R-M

bastawhiz 5y ago

I'd love to see a post from someone about their experiences using H2 between servers (i.e., instead of H1). Many big companies already use TLS internally, and I can imagine perf wins.

dagenix 5y ago

The title seem very much anti-HTTP/2. However, unless I missed something huge, the vast majority, if not all, of the issues found relate to how hard it is to handle HTTP/1. And it seems kinda odd to blame HTTP/2 for HTTP/1 being difficult to implement. I'm certainly not trying to downplay the seriousness of these issues. But it seems like a equally (it not more) valid title might be something like "HTTP/1: Continues to rear its ugly head" or something like that.

stkdump 5y ago
I guess this is more in line with the narrative that older protocols that people are more familiar with are somehow always better. That there is a hidden evil agenda for introducing something new. Or at least that we all need to pay a price for the sole benefit of a tech giant. It is a convenient narrative, which could be true, but it feels too easy a conclusion. Which makes it a good clickbait.
- dathinab 5y ago
  > It is a convenient narrative It's also not he narrative the content of the article purses ;-)
albinowax_ 5y ago
Hi, I'm the author. This paper is about HTTP/2, and the dangerous things that happen behind the scenes with HTTP/1 when people enable it. It's arguing that enabling HTTP/2 on your front-end makes your security worse if it's downgrading behind the scenes. This is a deliberately cheeky title, selected as a presentation title, rather than being aimed at social media where many people will read the title but not the contents. For HN I'd go with something dry like "HTTP/2-exclusive attack vectors".
- jpambrun 5y ago
  My employer's mandatory proxy downgrades all HTTP/2, presumably so they can better inspect traffic. This results in a bad/slow experience on many sites. I wish HTTP/2 would not have downgrade at all so they would have to deal with it..

get52 5y ago

I can't believe this, http and the internet as a whole is basically getting horsefucked

gouda-gouda 5y ago

That JIRA exploit is mind-boggling bad. Posts like this remind me that we work on a house cards, not a fortress.

mintyc 5y ago

Wow. A fantastic set of descriptions, service and detection tooling from portswigger and colleagues. The fact that http/1 downgrading is so prevalent and unlikely to change in the short term is the point.

albinowax_ 5y ago

Hi, I'm the author - please let me know if you have any questions!

brabel 5y ago
Just want to thank you for publishing this and taking the time to present it in such a clear manner. As someone who has implemented HTTP/1.1 message parsing myself, it blows my mind how downgrade implementers could've made such basic mistakes as to not validate the contents of headers. The companies selling security software with these mistakes should be held accountable IMHO, as the attacks are so simple and so many that they've probably been exploited extensively in the wild, and similar attacks almost certainly continue to be.
mwcampbell 5y ago
In light of these vulnerabilities, should we disable Internet-facing use of HTTP/2 wherever we can?
- albinowax_ 5y ago
  I think HTTP/2 is fine when it's used end to end. So if you've got a single webserver setup, or a reverse proxy that speaks HTTP/2 to the back-end, it's great. However, if the only way you can use HTTP/2 is by having the front-end downgrade it to HTTP/1, I would recommend disabling it.
  - wtarreau 5y ago
    In fact it will depend on the implementations. As you've found, the root cause of the problem is that H1 and H2 use different delimiting techniques for protocol elements. Some implementations need to transcode H2 into an internal representation. In this case, and provided the protocol elements are checked in the intermediate representation, any protocol on the other side will be fine. But if the internal representation is H1-like with poor message delimitation, you could very well end up splitting requests internally and emitting 2 H2 requests on the backend for a single front H2 request for example. I'd say that the main cause of these issues is the directive language used in the H2 spec almost only saying "this must be done like this" without giving rationale for the rules, meaning that implementations that were unable to implement them exactly the same way were left with no hint about what the trouble was nor how to address it. The new spec in progress and the extraction of the core semantics makes the whole thing a lot cleaner. In addition, H2 uses much more resources per connection than H1, which encourages to coalesce them between the proxy and the server. But coalescing connections can easily cause some head-of-line blocking if coalesced front streams show different bandwidths. So even end-to-end H2 is not always a panacea.

nly 5y ago

I'm a bit confused how these attacks can be used. Inconsistencies between HTTP/1.1 and HTTP/2 allow for these behaviours but you still need to inject these malicious prefixes in to a user's request flow, right? Can you actually set the Content-Length header from a browser on a HTTP/2 request? And isn't the suffix just ignored by the final remote (or does this depend on HTTP/1.1 pipelining?)

albinowax_ 5y ago
Hi, I'm the author. This paper is building on techniques explained here: https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn https://portswigger.net/research/http-desync-attacks-request... This attack does not require a MITM - the attacker would use a tool like Burp Suite to issue the (technically RFC-violating) HTTP request. The prefix injection happens because front-end places the attacker's request and the victim's request on the same HTTP/1.1 connection to the back-end, as shown in this diagram: https://portswigger.net/cms/images/9c/c1/4c32-article-http2-desync.png https://portswigger.net/cms/images/9c/c1/4c32-article-http2-... > isn't the suffix just ignored by the final remote The back-end treats the suffix as the start of the next request, due to TCP buffering. The vast majority of servers have this kind of accidental pipelining support thanks to TCP.
twic 5y ago
The target's own front-end proxy acts as the man in the middle!

coffeetrackpad 5y ago

Thanks for this @albinowax_ - super interesting! Just a quick question about the H2.CL case study. Did Netflix have a separate vulnerability where their server wrongly trusted Host headers with netflix.com suffix and returned a 302 response which can redirected users to an arbitrary host? I'm just trying to see whether I interpret the case study correctly.

albinowax_ 5y ago
Yes that's right. If the back-end received a request that didn't contain X-Forwarded-SSL or suchlike, and had a host-header that ended in .netflix.com, it would redirect you to the host-header. I wouldn't exactly class this as a vulnerability - more of a useful gadget. The front-end would refuse to forward such a request so it's impossible to hit this code path without request smuggling. Even if you could hit it without request smuggling it would still be useless.

gnfargbl 5y ago

@albinowax_, can you say anything more about how widely you have investigated the prevalence of the "H2.X via Request Splitting" vulnerability? Although all of your attack vectors are fascinating, this seems to be the one which is particularly terrifying. It seems like one single request can basically mess up user authentication across an entire website, and for every active user, until the server becomes resynchronized somehow. If I were running a large public-facing site or application, checking for this vulnerability would be my #1 priority this morning.

albinowax_ 5y ago
Good question! So, my understanding is that the majority of servers that are vulnerable to regular cross-user HTTP Request Smuggling (IE, reuse connections to the back-end server) are exposed to this terrifying response queue poisoning attack. This applies to all desync types CL.TE, TE.CL, H2.CL, etc. The reason I discovered this in the H2.X case, is because it's particularly easy to trigger response queue poisoning by accident in this scenario. That said, I haven't actually tested this on very many live servers, for obvious reasons!

terom 5y ago

So if AWS ALB HTTP/2 listener -> HTTP/1.1 target downgrade was vulnerable, and the recommendation is to use HTTP/2 end to end... am I reading the AWS docs [1] correctly that ALB only supports HTTP/2 -> HTTP/1.1 downgrades, but not HTTP/1.1 -> HTTP/2 upgrades? In other words, using HTTP/2 as the target group protocol only works for HTTP/2 clients, and HTTP/1.1 clients will receive an error? [1] https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html#target-group-protocol-version https://docs.aws.amazon.com/elasticloadbalancing/latest/appl...