7 ms·
Nginx sucks at SSL, here's what to do instead
- randomstring 15y agoWithout any reference to the nginx options ssl_session_cache keepalive_timeout keepalice_requests and whether or not the other webservers mentioned support such options and if they are enabled in the tests. There is no way to know if this is alarming or just FUD. Also missing from this comparison is a vanilla Apache SSL benchmark (which will suck, but would serve as a reference point). Given nginx's track record, I prepared to give nginx the benefit of the doubt and assume that it's a invalid test. That said, I'm going to run some benchmarks of my own.
- greglindahl 15y agoMr. randomstring, your test results before blekko launched were much faster than this guy's number. Want me to quote from your email?
- stock_toaster 15y agoThe post didn't mention if ssl_session_cache was enabled in the nginx config or not. In fact, I didn't see any configs posted. :( Also, the article author apparently added support to stud (in his own fork) for x-forward-for. I don't think this is required any longer, due to this fairly recent stud commit: https://github.com/bumptech/stud/commit/9d9b52b7d3ce90fa84c67eca261b087336233775 https://github.com/bumptech/stud/commit/9d9b52b7d3ce90fa84c6...
- jamwt 15y agoI'm reading over Matt's work carefully, but my initial inclination is not to merge the bulk of this into stud mainline. I'd rather keep stud simple and protocol-naive and have HAProxy do the HTTP work.
- jamwt 15y agoWhich is to say indirectly that I think the right answer is for nginx (and daemons generally) to support the PROXY protocol, or some other agreed-upon standard for a naive upstream proxy to indicate host/port information.
- stock_toaster 15y agoI really agree with this. I think keeping stud as simple as possible is a great goal.
- sciurus 15y agoCould someone link to a description of the PROXY protocol?
- stock_toaster 15y agohttp://haproxy.1wt.eu/download/1.5/doc/proxy-protocol.txt http://haproxy.1wt.eu/download/1.5/doc/proxy-protocol.txt
- seiji 15y agoI'm reading over Matt's work carefully Thanks for the consideration! initial inclination is not to merge the bulk of this into stud mainline I agree. The HTTP stuff is still too integrated. ifdefs are ugly. The solution is to do what showed up when I was 99% done working on XFF -- the nice PROXY protocol addition. We just need to get PROXY support into nginx now to obviate my XFF machinations.
- seiji 15y agoI hear the same performance happens with our without the session cache enabled (for benchmarks). The http(s) benchmarking tools don't resume sessions. It's simulating a horde of new clients who never come back or request other resources. It would be interesting to see stud with a session cache too.
- pilif 15y agoIf I see such bad numbers, usually, before I write a post about it, I try to investigate and share the results of that with the post. I'll try to reproduce these results tomorrow, but if I had to guess, I'd say ssl_session_cache was left to its default (off) which means that every connection has to do the expensive SSL handshake.
- SaltwaterC 15y agoMost of the servers suck at SSL without some sort of caching. Also, properly configured nginx setups have a fair share of keep alives.
- yid 15y agoSigh. Sometimes posts like these make me want to go back to academia (it's not perfect, but they generally believe in this little thing called rigor). From TFA: I tested nginx as a proxy, serving static files, and serving nginx-generated redirects. I tried changing all the relevant ssl parameters I could find. All setups resulted in the same SSL performance from nginx. I even tried the setup on more than one server (the other server was quad-core nginx got up to 75 requests per second). So "all the relevant ssl parameters I could find", no details about what those involve, and the surprising result that it made no difference. In the same situation, I might think I was doing something wrong... And then this overarching statement: Never let nginx listen for SSL connections itself.
- palish 15y agoBut... that would take work! And he only works four hours a week. </tonguecheek>
- seiji 15y agoRigor is an interesting point. Do we prefer to have a flawed, but slightly useful, post now - or - do we prefer to wait a month or two for a squeaky clean post with all issues worked out?
- vog 15y ago
- tptacek 15y agoI don't know what the limits are in the nginx HTTP parser Matt's using, so this is probably moot, but code that does things like "realloc(ptr, size + newsz)" or "malloc(size + 1)" expecting things to be fine gives me the howling fantods.
- __rkaup__ 15y agoI'm quite new to C programming. What's wrong with doing those things?
- fleitz 15y ago"malloc(size+1)" is a sign you may have one off errors in your code. If you need to store a string of size s you need s+1 bytes allocated. The plus one is for null termination. If you want an array of t you can either pass the size around with the array or null terminate the array like strings, but then you can't store any null values in the array. Also, there's no bounds checking on size so in certain conditions such as a 2GB/4GB allocation you may allocate zero bytes or -2GB bytes.
- tptacek 15y agoNumbers can't grow indefinitely; they wrap (usually at the size of the register).
- seiji 15y agoI don't know what the limits are in the nginx HTTP parser You're correct in assuming the library enforces its own size limitations. It operates on length of received SSL data which is capped by the static receive buffer at 32k. Nice and tiny. (Also, you are, of course, painfully correct about lack of bounds checking and lack of return value checking on the malloc/realloc calls. If I ever graduate the branch to production status, the six malloc calls and three realloc calls will be wrapped in proper checks.)
- tptacek 15y agoFor what it's worth, don't check the retval of malloc/realloc/strdup; instead, rig them so they blow up if the allocation fails.
- fexl 15y agoI've seen good results with the "pound" front-end: http://www.apsis.ch/pound/ http://www.apsis.ch/pound/ I haven't done extensive benchmarking on it, but very knowledgeable people vouch for it.
- sigil 15y agoWhere's the benchmark code? I'd like to see how ucspi-ssl [1] performs. [1] http://www.superscript.com/ucspi-ssl/sslserver.html http://www.superscript.com/ucspi-ssl/sslserver.html
- greglindahl 15y agoFor a different point of view, check out: http://web.archive.org/web/20090619214443/http://www.o3magazine.com/4/a/0/2.html http://web.archive.org/web/20090619214443/http://www.o3magaz... The author benchmarks nginx at 26,590 TPS on a quad-core 2.5 ghz amd system.
- spydum 15y agoThat benchmark is using an SSL Accelerator card and nginx. Can honestly say I have seen very few people swing for SSL Accelerator cards.
- greglindahl 15y agoThe "open source SSL Accelerator" mentioned in the blog posting is a quad-core server running Linux and nginx.
- spydum 15y agowhoops! my apologies -- the capitalized "SSL Accelerator" set my mind into thinking dedicated hardware device.
- mleonhard 15y agoI suspect that they are re-using connections in that benchmark. SSL connection setup is CPU intensive. Once a session is set up, an SSL connection uses only slightly more CPU time than an unencrypted connection.
- cshesse 15y agoA friend of mine worked on some large scale SSL deployment, he wrote up the results of his tests here: http://zombe.es/post/5183420528/accelerated-ssl http://zombe.es/post/5183420528/accelerated-ssl He's concerned about the raw speed of the SSL calculations, not requests per second, but if you're actually concerned about SSL speed and you have enough requests per second to justify optimizing SSL speed, it could be pretty useful.
- piotrSikora 15y agoNo configs, no methodology, no graphs... Great "benchmark".
- wpietri 15y agoIf only there were some way you could create those things that you want and he didn't care about doing!
- damncabbage 15y agoCalling out a tool as having some terribly negative characteristic places the burden of proof on he or she doing the calling.
- auxbuss 15y agoI needed to use SSL on nginx and got great results from following a number of pieces of advice. I jotted down my noted here: http://auxbuss.com/blog/posts/2011_06_28_ssl_session_caching_on_nginx/ http://auxbuss.com/blog/posts/2011_06_28_ssl_session_caching... It made a significant performance difference to me.
- ams6110 15y agoQuoting: (on an 8 core server...) haproxy direct: 6,000 requests per second stunnel -> haproxy: 430 requests per second nginx (ssl) -> haproxy: 90 requests per second Yet Matt Cutts tells us that SSL is not computationally expensive anymore. Based on these results it's still an order of magnitude slower.
- ck2 15y agoIf anyone does benchmarking, please include litespeed as I am curious. I suspect it's much faster than nginx at ssl. Even with the connection limit on the free version I suspect it will still be feasible for testing.
- mmaunder 15y agoThanks! I didn't know nginx sucked at SSL. You may have increased our revenue. Many businesses like us have our conversion pages on SSL. Our front-end server is doing 2000 to 4000 http requests per second and we get over 3 million uniques on the main site where we sell stuff via SSL. If SSL is this slow, it probably impacts performance on our secure pages which affects revenue. Where do I send the beer? On a 4 core Xeon E5410 using ab -c 50 -n 5000 with 64 bit ubuntu 10.10 and kernel 2.6.35 I get: For a 43 byte transparent gif image on regular HTTP: Requests per second: 11703.19 [#/sec] (mean) Same file via HTTPS with various ssl_session_cache params set: ssl_session_cache shared:SSL:10m; Requests per second: 180.13 [#/sec] (mean) ssl_session_cache builtin:1000 shared:SSL:10m; Requests per second: 183.53 [#/sec] (mean) ssl_session_cache builtin:1000; Requests per second: 182.63 [#/sec] (mean) No ssl_session_cache: Requests per second: 184.67 [#/sec] (mean) The cache probably has no effect because each 'ab' request is a new visitor. But I'd guess the first https pageview for any visitor is the most critical pageview of most funnels.
- newman314 15y agoChoice of cipher as well as openssl version + features used make a difference too. See http://zombe.es/post/5183420528/accelerated-ssl http://zombe.es/post/5183420528/accelerated-ssl for some examples. Use "openssl speed -elapsed" to test performance on your system.
- schiptsov 15y agoLet me guess - it is not nginx code which is slow but one from openssl? ^_^
- schiptsov 15y agoI won - http://matt.io/technobabble/hivemind_devops_alert:_nginx_does_not_suck_at_ssl/ur http://matt.io/technobabble/hivemind_devops_alert:_nginx_doe...
- ominous_prime 15y agoI'm not sure why he would be getting numbers that low. The only setup I have at the moment which would give useful numbers for ssl req/sec is a small single core VM, running one nginx worker process, and that pumps out 135 new req/sec. Add a few cores, workers, put it on real hardware, and I don't see how this couldn't push well over 400 req/sec. This is using nginx strictly as an ssl termination, where I need to do some header manipulation that I couldn't do in stunnel/stud.
- ominous_prime 15y agoSelf reply, since I waited too long to edit: I remembered I had an older 8 core server sitting unused at the moment. I configured nginx with 8 workers, and ran `ab` against it. From a single (VM) host, I can get 680 connections per second (maxed the cpu on the host running the test). From 4 hosts, each host got > 290 connections per sec, so I got nginx up to over 1190 new connections per second, and can likely push it further. [EDIT] got it to peak at 1535 requests per second with 4 hosts testing.
- nginxorg 15y agoIt would be nice it Matt provided full details on the testbed, including the client. In a test scenario it is very important to understand what gets tested in the end. I liked that "Russia" tag too :)
- foobarbazetc 15y agoNot to rain on the parade here, but we handle several thousand connections per second on nginx + SSL per 8 core Westmere machine. The article needs way more detail.
- deleted 15y ago[deleted]
- nginxorg 15y agomore nginx ssl testing http://nginx.org/pipermail/nginx/2011-July/027960.html http://nginx.org/pipermail/nginx/2011-July/027960.html