8 ms·
Google Chrome 51 disables HTTP/2 on most Linux distros due to old OpenSSL
- chemodax 10y agoSupport for OpenSSL 1.0.1 will cease on 2016-12-31 [1], so distributions need to update anyway. [1] https://www.openssl.org/policies/releasestrat.html https://www.openssl.org/policies/releasestrat.html
- jnky 10y agoJust because something is no longer supported upstream does not mean that the distributions won't support it anymore. As far as I can tell it is more likely the package maintainers will backport fixes to their old version in those cases.
- chemodax 10y agoSure, package maintainers may backport fixes to their old versions. But they need to fully understand all upstream source code and follow all commits. Otherwise they can miss important fixes: following only security fixes for supported branches is not enough. Sometimes project developers fixes bug/security problem in the code, but doesn't flag it as CVE because current code usage doesn't trigger it. But code in old branch could.
- coldtea 10y ago>Sure, package maintainers may backport fixes to their old versions. But they need to fully understand all upstream source code and follow all commits. Only if they need to do a perfect job. But as history tells us they are just as content of making a ho-hum job.
- viraptor 10y agoThat's the current reality. That's how it was for years. Especially CentOS and other RH-based systems are more happy to patch than to upgrade. This caused the kernel 2.6.32-573 situation where lots of patches (over a hundred?) were applied by the distro.
- lucb1e 10y agoThat's still more than six months away though.
- nteon 10y agoThe Chromium team's blog post about this suggests that > 99% of HTTP/2 connections are _already_ negotiated with ALPN [1]. I imagine all of the CDNs and other large providers already use ALPN, so this mostly affects the long tail of small sites. It seems like the claims in the above blog post are overblown. 1 - http://blog.chromium.org/2016/02/transitioning-from-spdy-to-http2.html http://blog.chromium.org/2016/02/transitioning-from-spdy-to-...
- mcguire 10y agoI have this strange suspicion that they could push the change to Google's servers and get ~30%. Add reddit, CNN, and ESPN and they're pushing 90%.
- coldtea 10y agoThey might reach 30% with just Google -- but Reddit, and even more so CNN and ESPN are insignificant as percentages of web traffic, whether in the US or (even worse) globally. CNN is not even in the top-50 websites, and ESPN is not even in the top-100 (e.g. in the Alexa index). Heck, pornsites like xhamster have far better placements than either.
- mcguire 10y agoI'm old. Google, YouTube, and Facebook.
- mrsaint 10y ago> Upgrading OpenSSL packages isn't a trivial task, either. Since just about every other service links against the OpenSSL libraries, they too should be re-packaged (and tested!) to work against the latest OpenSSL release. Not necessarily. You could simply statically link the newer OpenSSL library with your web server and everybody is happy.
- bkeroack 10y agoSomebody tell me again about the supposed benefits of dynamic linking, if we have to recompile and redistribute all the dependent packages for every OpenSSL upgrade anyway?
- reidrac 10y agoI think it is because OpenSSL changes the API frequently and it'll break things in subtle ways. I may be wrong, but this is not a problem with dynamic linking but with OpenSSL as a project.
- mcguire 10y ago1.0.1 -> 1.0.2 would be invisible to API users, in a slightly better world, anyway.
- acdha 10y ago> if we have to recompile and redistribute all the dependent packages for every OpenSSL upgrade anyway? Because we don't – look at how many updates Debian, Red Hat, etc. have shipped for OpenSSL vs. the number of times you've had to recompile dependent packages. OpenSSL is also something an outlier because it's both critically exposed for security and has a somewhat unusual development model, so it's also important to remember that even if it _was_ true that we have to rebuild OpenSSL callers regularly, that's manifestly not true of the hundreds of other libraries where dynamic linking saves time and memory.
- fixermark 10y agoIt's honestly so mission-critical to interactions on the internet---and computers are so increasingly assumed to be on the internet---it's a bit of a wonder that no-one to my knowledge has tried to roll it into the kernel as a core and necessary function yet.
- predakanga 10y agoIt's definitely been discussed[1]. I last read about it when investigating the use of splice(2) in nginx; seems there are a few use-cases, but nothing really compelling. [1]: https://lwn.net/Articles/666509/ https://lwn.net/Articles/666509/
- alexchamberlain 10y agoPotentially, big sites won't be (shouldn't be) relying on system binaries for their web server etc. That would give them the freedom to remove unneeded modules, add custom modules and upgrade at their own speed.
- pfg 10y agoIf you're stuck on a OpenSSL version without ALPN, and don't want to deal with upgrading OpenSSL (or rather statically linking your web server to a newer release), consider using caddy[1]. It's a HTTP/2 web server written in Go, which comes with its own TLS library (crypto/tls) and thus doesn't have any dependency on OpenSSL. It even supports automatic deployment of a publicly-trusted certificate through Let's Encrypt. [1]: https://caddyserver.com/ https://caddyserver.com/
- jmiserez 10y agoSo instead of upgrading a library or the webserver you'd rather replace the webserver completely with a relatively new [1] webserver and redo all the configuration? (edit: more positive wording) [1] https://news.ycombinator.com/item?id=9452606 https://news.ycombinator.com/item?id=9452606
- chris_wot 10y agoAm I reading the same post? When I read it, he seemed to say "might consider"...
- deleted 10y ago[deleted]
- pfg 10y agoThat post is more than a year old. Caddy has since had more than 70 contributors and > 1K commits. Based on my experience, replacing nginx (in my case acting as a fairly simple reverse proxy for rails) with Caddy wasn't significantly harder than compiling a custom nginx. If you have a complex configuration, YMMV. I'm definitely not suggesting using Caddy as a drop-in replacement for your existing production web server without further testing. That being said, getting rid of a OpenSSL dependency isn't a bad thing.
- jmiserez 10y ago> Caddy has since had more than 70 contributors and > 1K commits. That is impressive, will definitely try it. Is there a list of companies using Caddy somewhere?
- pilif 10y agoIn our case, we were building our own nginx anyways, so the fix was relatively simple to just statically link it against libressl. This only affects the nginx we use on the frontend for SSL termination and the rest of the OS remains unaffected, linking against the OS OpenSSL.
- angrygoat 10y agoChrome was no longer accessing my site via http/2 – turns out that the docker nginx image has OpenSSL 1.0.1k. I switched to the Alpine image and that fixed it – a few nginx.conf changes needed, notably to change the user from www-data to 'nginx'. UID is 100 and GID is 101, which differs from the non-Alpine image.
- ryanobjc 10y agoAs usual, OpenSSL is screwing things up. Been delving into hairy problems at work and yup, it's openssl's fault. More specifically it's the misguided way it handles multithreaded, and memory management. Yay.
- fixermark 10y agoThis has definitely been a known issue for awhile, frustratingly. At one point, a team I was working with had to code their own RNG replacement for OpenSSL's; the one in the codebase was behaving badly in a multithreaded environment. I don't know if the change was ever accepted back upstream; if I recall correctly, the initial response was something to the effect of "you're using it wrong" (as if multithreaded applications are some new and fascinating beast, and not increasingly just the way you do it to leverage all the performance a modern computer offers in hardware).
- mwpmaybe 10y ago> At one point, a team I was working with had to code their own RNG replacement for OpenSSL's I recently released a RubyGem that does just that. It's unfortunate.
- Skunkleton 10y agoHTTP2 is a relatively bleeding edge technology at this point. Seems fine for it to depend on a recent version of Open SSL. It's not like HTTP 1.1 is going anywhere any time soon.
- HeadlessChild 10y ago> Debian 8 (Jessy)
- dang 10y agoCan someone suggest a more neutral title for this?
- billyhoffman 10y agoThe impact Chrome of switching from NPN to ALPN for HTTP/2
- kowdermeister 10y agoGoogle Chrome disabled HTTP/2 for nearly everyone today or Poor support for OpenSSL 1.0.2 forces Chrome team to disable HTTP/2
- yuhong 10y agoGoogle Chrome 51 disables HTTP/2 on most Linux distros due to old OpenSSL
- dang 10y agoOk, we'll use yours. Sorry for the delay—I forgot about this one over the weekend.
- jb613 10y ago> We knew, since the end of 2015, that this change was coming -- we were given 6 months time to get support for ALPN going, but by the current state of OpenSSL packages that was too little time. This hints at part of the problem - simply not enough resources. It's not that OpenSSL folks are slow or lazy (absolutely far from it) - but rather they've got tons on their plate already. One day (hopefully) corporations will recognize that everyone can benefit by helping open source (either providing resources or $funding).
- bluejekyll 10y agoThis is the package maintainers, not OpenSSL, right? Or does the OpenSSL group maintain all their packages on all these OSes too?
- krzyk 10y agoWhy disabled? More like "will disable", unless you are on some island on the pacific - for the majority of Earth May 15th will come in more than few hours.
- smithclay 10y agoI struggled with getting HTTP/2 working with ALPN on Ubuntu 14.04 back in February. After building nginx from source [1] I ended up switching to Docker base images that support OpenSSL 1.0.2 [2]. 1: https://www.clay.fail/posts/ubuntu-http2-in-mere-hours https://www.clay.fail/posts/ubuntu-http2-in-mere-hours 2: https://www.clay.fail/posts/hip-http2-using-docker/ https://www.clay.fail/posts/hip-http2-using-docker/
- leesalminen 10y agoI actually found your blog post [1] on Google earlier this morning after seeing this thread. It was hugely helpful. Thank you very much for writing it. Was there a bug or technical limitation that made you switch to a Docker image or just preferences/ease of updating?
- smithclay 10y agoThe main reason I switched Docker is there's an nginx image (on alpine) that supports ALPN. No compiling from source required, which simplified things (in my mind, at least).
- leesalminen 10y agoGotcha. As a heads up, there's a confirmed bug with HTTP/2 implementation on nginx 1.9.15 and 1.10.0. It seems to affect Safari and iOS (native and browser) markedly. For me, the first HTTP/2 POST request to a server running the affected version of nginx would never leave the browser and Safari left a less than descriptive "Failed to load resource: Could not connect to the server." console message. I'm having luck with 1.9.12 + OpenSSL 1.0.2h https://trac.nginx.org/nginx/ticket/979 https://trac.nginx.org/nginx/ticket/979 https://trac.nginx.org/nginx/ticket/959 https://trac.nginx.org/nginx/ticket/959
- mwpmaybe 10y agoAnd TLS False Start, which reduces the average time of an SSL handshake by 30%, and can be enabled independently of SPDY and HTTP/2. Fortunately, it was pretty easy to redeploy my HAproxy config to Xenial and enable ALPN.
- heavenlyhash 10y agoAnd right about here is where the claims that dynamic linking is better for everyone's security update schedule just falls apart, for this reader. Dynamic linking does not make it easier to update things. There, I said it. Updating things confidently and safely requires -- as the article wisely references -- testing those things, and the entire matrix of other things they interact with. Static vs dynamic linking simply doesn't register. In either case, the newer versions need to be built, and then tested together. The only thing dynamic linking does is make it possible for distros to hamstring themselves[^] by tossing everything into one big cauldron, so nothing can be upgraded because it requires testing too much at once. And here's the endgame. "Is OpenSSL getting updated on $distro?" "Nope." Sigh. --- [^] And this is a red herring, in turn: dynamic linking isn't the real enemy either; it's the global sloppile approach to sharing dynamic linked libraries that really causes the mess. NixOS and others like that still use dynlinking, and yet have successfully immunized themselves from these problems by using content-addressable library versioning. So let's stop with the "dynlinking is better for security updates" claims. It's not. It's either unrelated at best, or actively a ball-and-chain making things harder and slower, and I submit as evidence OpenSSL versions across these distros.
- hvs 10y agoI agree. Anyone who thinks that dynamic linking makes things easier to update never lived through the DLL-hell of Windows where installing one new product would break a completely unrelated one because it "upgraded" a library.
- pjmlp 10y agoTrue, specially when dealing with classical COM. However for plugins when performance is critical, is a good use case. For anything else, as the last decade as shown us, it is probably better to use OS IPC instead of exposing the application internals to third parties.
- stingraycharles 10y agoI would be the first to agree that dynamic linking is evil, but how is this related with the issue at hand, which is client-to-server communication? No amount of static linking of Chrome would magically upgrade the SSL version being used to package Debian, for example. What am I missing here?