11 ms·
Nginx doesn't suck at SSL after all
- thirsteh 15y agoSo Nginx got unwarranted hate for having the most secure defaults. That sucks. I hope the user nginxorg -- whom I assume is Igor Sysoev -- who dropped by the previous thread (http://news.ycombinator.com/item?id=2752136 http://news.ycombinator.com/item?id=2752136), sees this post too. Either way -- based on his attitude in the first post, I'm really surprised by how Matt owned up and did his homework for this one. (He should have done it from the beginning, of course, but none of the people bashing him in the previous thread actually provided anything to support what they were saying.)
- tptacek 15y agoIt's not the "best/most secure". All the other servers can trivially enable EDH as well; it's OpenSSL that implements it, not nginx. Reasonable people can disagree on what the right default is, but plenty of financial institutions have made the studied choice not to enable it.
- thirsteh 15y agoFair point. Clarifying...
- schleyfox 15y agoPlenty of financial institutions have also made the studied choice to limit the length and types of characters I can use in my password.
- tptacek 15y agoNo. I recognize this as snark, but it's inaccurate snark. Banks limit password lengths because the programmers who implement their apps are dumb. But programmers don't choose the SSL configurations for their app servers and load balancers; people who are paid to think about security do. Your attempt at snark here relies on an apples-oranges comparison.
- mapgrep 15y agoIf the institution chooses an insecure password policy, it heightens the likelihood it will fail to ensure good SSL settings. This tendency is independent of the fact that these functionalities are implemented by different teams, and that one team might happen to be competent enough to do the right thing despite the lack of institutional imperative. So the consumer might get lucky. So what? Your initial comment was that "plenty of financial institutions" do SSL a certain way, and the respondent correctly pointed out that this fact adds no information to the discussion about SSL techniques, because plenty of financial institutions do dumb things. It's "apples and oranges" only insofar as he's saying the orchard manager is a poison spreading dummy so you can't trust the apples or the oranges. Maybe you can elaborate on the "studied" part of your comment with specifics. That part was interesting.
- tptacek 15y agoYou use this word "the institution" as if companies were hive minds. Read the other comments on this thread. Again: the people who make decisions about password complexity are almost never the security people.
- mapgrep 15y agoExactly. You shouldn't trust a particular technique just because a financial institution uses it. They have very little institutional culture around security, as you yourself point out. So I'm not sure why you brought up the fact that financial institutions use a particular SSL technique - that tells us nothing.
- tptacek 15y agoYour argument is exasperating, because I already addressed this notion that password complexity requirements in banking apps have anything to do with what financial security people think are best practices for SSL/TLS.
- 15y ago
- masklinn 15y ago> All the other servers can trivially enable EDH as well Unless I'm reading OP wrong, that's not the case for the servers he uses for his tests: Stud can't enable it at all: > stud doesn't have at all. and in stunnel you have to compile it in for support, it's not just "not enabled by default", it's not compiled in: > stunnel has it as a compile time/certificate configurable option.
- tptacek 15y agonginx is a web server, like Apache. stud is a few hundred lines of trivial proxy code. And would you like to take a bet on how many lines of C code it would take to add support for configurable cipher suite modes in stud? Fair warning: I already know the answer to this (and I don't even know that stud doesn't allow it).
- burgerbrain 15y agoDo you happen to have a patch? I might be interested in that :)
- tptacek 15y agoFair warning: this comment is apparently all kinds of wrong, but I leave it here for posterity. Put: if(getenv("SSL_CIPHER_SUITES")) SSL_CTX_set_cipher_list(ctx, getenv("SSL_CIPHER_SUITES")); anywhere after SSL_CTX_new(). But don't bother doing it with stud, because (as I sort of predicted) stud already does this: stud -c <ciphersuites>. I don't understand what Matt is saying by "stud doesn't enable DH at all". Does stud build its own OpenSSL? The system OpenSSL will already support DHE.
- seiji 15y agoThat's the trick -- you can enable your DHE cipher all day long, but if the code doesn't set up DHparams, it will never work. Here's a quick (no error checking) way to set up DHparams if they are appended to a cert: BIO *bio = BIO_new_file(path_to_a_file_with_dhparams, "r"); DH *dh = PEM_read_bio_DHparams(bio, NULL, NULL, NULL); BIO_free(bio); SSL_CTX_set_tmp_dh(ctx, dh); DH_free(dh); Where do the DH parameters come from? You can generate it yourself (1024 bits here): openssl dhparam -rand - 1024 For a completely isolated implementation (requiring no user certificate changes), see function ngx_ssl_dhparam in nginx-1.0.4/src/event/ngx_event_openssl.c
- alnayyir 15y agoWell at least he followed up. Most people don't bother to correct their mistakes.
- cbetz 15y agoI can't tell if this is an apology or a non-apology. It seems to have elements of both. Clearly the moral of the story is: "Don't claim that X sucks unless you are are damn sure". Saying something sucks is fightin' words. Don't expect to people be nice if you are wrong.
- tptacek 15y agoWhy would someone apologize for writing an informative blog post? I'm glad he wrote both, even if he had to walk the first one back a bit.
- SnowLprd 15y agoI found both posts informative, and yet I'm with cbetz on this one. The issue isn't whether Matt's first post was informative but is instead whether it was fair|wise|necessary to say "Nginx sucks at SSL" instead of, say, "My initial testing, which needs to be investigated further, is showing Nginx SSL performance lower than other alternatives." The first headline is more likely to grab folks' attention, which is probably why Matt chose that headline. It is probably that choice that cbetz finds objectionable, and if so, I agree with him.
- pyre 15y ago> "My initial testing, which needs to be investigated > further, is showing Nginx SSL performance lower than > other alternatives." You keep using that word ('headline'), but I do not think it means what you think it means.
- SnowLprd 15y agoOkay, I'll bite. Here you go: "Initial Tests Show Slow Nginx SSL Performance" Wasn't that hard, was it?
- pyre 15y agoMy point was that you seemed to have purposely made that 'headline' that you thought he should have used needlessly verbose, to the point where it couldn't be considered a headline.
- beachaccount 15y agoI ran this against the slowest SSL website I know of. This site absolutely kills my phone web browser and I've been wondering about this problem for years. The site: manager.skype.com. The result? DHE-RSA-AES256-SHA. No wonder! Fix this guys! In regards to this post, if this is the default configuration of Nginx then I agree that Nginx sucks. This is not a good default configuration for the Internet.
- bdonlan 15y agoThere's nothing wrong with using DHE algorithms, particularly if you're going to be transferring financial secrets around. If your phone can't keep up, well, then it probably should have a better entropy generator.
- tptacek 15y agoAre you sure the problem is entropy generation and not just extra bignum math? Also: there are plenty of major financial apps that are not allowed to use DHE, because DHE makes it impossible for the provider to monitor its own connections ("conventional" SSL/TLS allows for middleboxes that monitor and archive sessions by holding a copy of the server's RSA key).
- burgerbrain 15y agoWhat is best for security in regulated fields and what is mandated or forbidden for security in regulated fields are often almost disjoint sets. Anyway, whether your phone has trouble coming up with the entropy, or preforming the math, you should probably be using something more substantial.
- tptacek 15y agoYou're suggesting that people should pick a different phone so they can get PFS with the small fraction of SSL servers that support it?
- 15y ago
- tptacek 15y agoIn case you're wondering what "Perfect Forward Secrecy" is: SSL/TLS, like most protocols, uses (expensive, dangerous) RSA to exchange (cheap, simple) AES or RC4 session keys; bulk data is encrypted with session key. In the normal protocol, if you lose the RSA key, an attacker can retroactively decrypt the session keys, which are protected only by that same RSA key. In ephemeral DH mode, instead of encrypting a session key with RSA, both sides run the Diffie Hellman protocol to exchange a key†. DH allows two unrelated parties who share no secrets to exchange a secret in public; it's kind of magical. But it's also trivial to man-in-the-middle. To get around that problem, ephemeral Diffie Hellman mode in SSL/TLS signs the DH exchange with the RSA key. The win here is that losing the RSA key now only allows you to MITM future SSL/TLS connections. This is still a disaster, but it does not allow you to retroactively unwind previous DH exchanges and decrypt earlier captured sessions. † DH is unbelievably simple; go read the Wikipedia page.
- thirsteh 15y ago"Ephemeral Diffie Hellman" It sounds magical too.
- shrikant 15y agoDiffie-Hellman and Boyer-Moore are two algos that completely blew me away with their simplicity when I first encountered some theory on them. http://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange http://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exch... http://en.wikipedia.org/wiki/Boyer%E2%80%93Moore_string_search_algorithm http://en.wikipedia.org/wiki/Boyer%E2%80%93Moore_string_sear...
- alecco 15y agoIt could be a really nice blog post. A lot of people get lost on wikipedia pages like that.
- deleted 15y ago[deleted]
- clintjhill 15y agoIt's great to follow up. Especially with such a particular detail. It's also good to have thick skin. HN can be aggressive. But for good reason. I'd be willing to bet this tiny sting will result in more rigor in the future. I know it has worked that way for me.
- ccollins 15y agoFrom the article, to find out what your website is doing: openssl s_client -host HOSTNAME -port 443 I ran this for my own website and a few bigger websites openssl s_client -host www.gusta.com -port 443 (My site, hosted on Heroku) Cipher : DHE-RSA-AES256-SHA openssl s_client -host www.google.com -port 443 Cipher : RC4-SHA openssl s_client -host www.airbnb.com -port 443 Cipher : AES256-SHA openssl s_client -host www.facebook.com -port 443 Cipher : RC4-MD5 openssl s_client -host www.paypal.com -port 443 Cipher : AES256-SHA openssl s_client -host www.amazon.com -port 443 Cipher : RC4-MD5
- rythie 15y agoPresumably Amazon, Facebook and Google are using RC4 for speed reasons, though it's not really thought to be secure anymore.
- paulitex 15y agoIs that so? openssl s_client -host online.citibank.com -port 443 Cipher : RC4-MD5 openssl s_client -host www.bankofamerica.com -port 443 Cipher : RC4-MD5
- benblack 15y agoBanks are unfortunately poster children for what not to do in this space, generally. The default cipher for google.com is RC4-SHA, and I can, if so inclined, force negotiation of AES-based ciphers by client config. Not so with Citibank (RC4-MD5, DES-CBC3-SHA, or DES-CBC-SHA) & BofA is only marginally better (RC4-MD5, RC4-SHA, AES128-SHA). To their credit, they are using 2048 bit RSA keys with short lifetimes and they have significantly improved their configurations from a couple of years ago when single DES defaults and 40-bit RC4 were all too common.
- tptacek 15y agoRC4 is fine in SSL/TLS. Nobody likes it, but until relatively recently, the AES ciphersuites were all CBC mode, which means they burned a couple bytes of padding for every record. RC4 is also faster than AES, which was, until very recently, an issue for server performance. We have AES-CTR ciphersuites now, but I'm not sure how widely deployed they are.
- WestCoastJustin 15y agochanges: slow ssl encryption ciphers on by default, keepalive before: nginx (ssl) -> haproxy: 90 requests per second after: nginx (AES256-SHA with keepalive 5 5;) -> haproxy: 4300 requests per second
- tlrobinson 15y ago"Final feeling: Twitter is better than HN in all social dimensions of engagement, kindness, and authenticity." Ouch.
- thirsteh 15y agoYeah, well, arrogantly complaining about people calling you out on unfounded speculation doesn't really work when you were originally quite arrogant in that speculation.
- rfugger 15y agoThis is interesting to me. Maybe it's because twitter places much more emphasis on identity than HN comments? Twitter users are also more likely to be using that service to market their personal brand, rather than just to share whatever they're thinking, so are probably much more careful about what they say.
- nandemo 15y agoTwitter is better than HN? That's like comparing apples to a fish market.
- michaelbuckbee 15y agoThere is some selection bias and context that the OP had via Twitter (people following already kind of know him and his work, etc.) whereas HN evaluates his work solely on this one post.
- AltIvan 15y agoYeah, the correct stament should be something like: Final feeling: My long time Twitter Followers & Friends are more engagement and kind than the anonymous programmers on HN.
- ynniv 15y agoAh, if only we had an occasional second upvote! This is far more useful than the original post, which was already well above average. If you are deploying nginx with SSL, you need to know about the configuration details in the article.
- grandalf 15y agoAny serious article ripping on the performance of something should at least link to the config file used.
- jeremyw 15y agoUnlike the above post, this fellow actually did some broad cipher testing (http://zombe.es/post/4078724716/openssl-cipher-selection http://zombe.es/post/4078724716/openssl-cipher-selection), particularly around AESNI instructions in recent Intel chips. With AESNI, use AES-128, AES-256, RC4-SHA, CAMELLIA-128. Without AESNI, use RC4-SHA, AES-128, AES-256, CAMELLIA-128. In nginx, this looks like: # (wo/AESNI): ssl_ciphers RC4:AES128-SHA:AES:CAMELLIA128-SHA:!MD5:!ADH:!DH:!ECDH:!PSK:!SSLv2 # (w/AESNI): ssl_ciphers AES128-SHA:AES:RC4:CAMELLIA128-SHA:!MD5:!ADH:!DH:!ECDH:!PSK:!SSLv2 You eliminate weak ciphers. You retain RC4 for compatibility and speed. You order by performance. (Note that AES-128 is still ranked as secure through 2030 [at least]. You don't need to prefer AES-256.)
- epi0Bauqu 15y agoSo what should ssl_ciphers be? Can't you just move that one to the end somehow?
- benblack 15y agoAn article about configuring SSL that doesn't 1) discuss trade-offs of security vs. resource consumption, 2) how to figure out your performance requirements, and 3) indicate the author really understands implications of decisions about crypto is an article you should probably disregard. Modern CPUs are so ridiculously good at crypto, and most sites have such ridiculously low connection rates, that optimizing for maximum performance at the expense of security is a fool's game in most cases. Instead, focus on measuring your real performance requirements first, and things like sane configuration of SSL, for example by explicitly listing ciphers instead of using the impenetrable +aNULL:-yourMom syntax. Here's my vintage code for scanning SSL configs: https://github.com/b/tlscollect https://github.com/b/tlscollect Here are a couple of must read posts from someone who really knows his SSL business: http://www.imperialviolet.org/2010/06/25/overclocking-ssl.html http://www.imperialviolet.org/2010/06/25/overclocking-ssl.ht... http://www.imperialviolet.org/2011/02/06/stillinexpensive.html http://www.imperialviolet.org/2011/02/06/stillinexpensive.ht... It's great to learn. Lil' B
- tptacek 15y agoDoes any of this have anything to do with Matt's post? Adam's first post says the same thing Matt's does: DHE is expensive. The "tradeoff" in security vs. performance you're referring to irrelevant to almost everyone building on nginx. If you've lost your RSA key, you are well and truly fucked. DHE is interesting, but sniping at people for not using it (in your case, implicitly) is unfair.
- benblack 15y agoAdam's post is rather more thorough and nuanced, which makes sense since he actually understands SSL and benchmarking. While you might summarize them both as "DHE is expensive", I don't know why you would. Here is each post on DHE: Adam - "However, with a pure RSA ciphersuite, an attacker can record traffic, crack (or steal) your private key at will and decrypt the traffic retrospectively, so consider your needs." Matt - "Unfortunately, it also includes a very computationally intensive cipher using an ephemeral Diffie-Hellman exchange for PFS. Sounds scary already, doesn't it? ... The problem cipher is DHE-RSA-AES256-SHA [b]." The first is factual and straightforward. The second is muddled and clearly skewed towards blindly disabling DHE. I believe we are in agreement that it is irrelevant to almost everyone building on nginx: their connection rates are so low they will not notice the overhead introduced by DHE. I am sniping at enthusiastic ignorance and encouraging others to behave similarly. I hope that is all quite clear now. Hugs and kisses, Lil' B
- lanstein 15y agokeywords in footer: nginx, openssl, ciphers, DHE-RSA-AES256-SHA bad, AES256-SHA good, hn, twitter, glee soundtrack
- getsat 15y agoPossible SEO play? Maybe not, considering his site is almost completely lacking on-page SEO. Perhaps an accidental copy/paste?
- sthatipamala 15y agoLook at his other posts. He just has joke keywords in all his posts.
- newman314 15y agoGuess I was right about the cipher used. http://news.ycombinator.com/item?id=2753903 http://news.ycombinator.com/item?id=2753903
- ldar15 15y agoSo, are these the new numbers? I copied the numbers from the original and the new post: haproxy direct: 6,000 requests per second stunnel -> haproxy: 430 requests per second (OLD) nginx (ssl) -> haproxy: 90 requests per second nginx (AES256-SHA) -> haproxy: 1300 requests per second nginx (AES256-SHA with keepalive 5 5;) -> haproxy: 4300 requests per second Did other things change or is nginx more than twice as fast as the next best solution?
- giberson 15y agoI use a similar directive in my apache2 configuration. Would I see an performance improvement in removing the DH option from the cipher suite? Or is this only directly related to ngix and how it implements its ssl protocol? Secondly, by removing the DH method do I restrict any browsers from connecting my site? Ie, are their any browsers, or security settings on browsers that prevent the site from being trusted if DH isn't available?
- blumentopf 15y agoIt's an openssl configuration option, so it does affect performance for Apache with mod_ssl. It's not specific to nginx. If you do not allow DH ciphers, you'll probably just lose users who deliberately configure their software to use only strong ciphers. What you can do is put DH ciphers at the end of the list. That way, weaker ciphers are preferred but you're still supporting strong ciphers.
- dfc 15y agoThis is why security is such a wierd/nice/confusing/irratating line of work to be in. Newsflash SSL is not a one size fits all secure you against anything technology. I did not see the original article so I won't pretend that I knew the answer ahead of time. I just hope that I did not accept SSL as being a onesize fits all completely uniform technical conmponent. There is a Dave Chapelle joke about cops sprinkling crack-cocaine over a crime scene in order to make the case quick and easy. Too many developers trest SSL like magic pixie dust for security. Or as ptacek says "thanks in advance for putting my kids through college."
- alexkon 15y agoIf you are exploring and testing SSL, the SSL Labs tools come in handy. For instance, see what Gmail and Github are doing: https://www.ssllabs.com/ssldb/analyze.html?d=mail.google.com https://www.ssllabs.com/ssldb/analyze.html?d=mail.google.com https://www.ssllabs.com/ssldb/analyze.html?d=github.com https://www.ssllabs.com/ssldb/analyze.html?d=github.com
- mmaunder 15y agoThanks to Matt and everyone who contributed to figuring this out - and then figuring it out again. Comments on the nginx mailing list (which I encourage you to subscribe to if you're a user): http://forum.nginx.org/read.php?2,212229 http://forum.nginx.org/read.php?2,212229