13 ms·

35M Hot Dogs: Benchmarking Caddy vs. Nginx

399 points by EntICOnc 4y ago

anonymouse008 4y ago
> Wew lad! Now we’re cooking with gas. This is the new gold standard for benchmarks! OP / Author, stupendously tremendous job. The methodology is defensible and sensible. Thank you for doing this on behalf of the community.
- lelandfe 4y ago
  Seconding! I am also in love with the friendliness and tone of the article. I’m a complete dummy when it comes to stuff like this and still understood most of it. Feynman would be proud.
- mholt 4y ago
  Yeah, Tyler did an amazing job.
- tylerjl 4y ago
  That's very kind of you to say, thank you!
stefantalpalaru 4y ago
> I’ll build hosts with Terraform (because That’s What We Use These Days) in EC2 > [...] > Create two EC2 instances - their default size is c5.xlarge When you're benchmarking, you want a stable platform between runs. Virtual private servers don't offer that, because the host's resources are shared between multiple guests, in unpredictable ways.
- zmxz 4y ago
  Which platform would you suggest to use for this benchmark?
  - bdcravens 4y ago
    Ideally your own hardware with nothing else running on it. For convenience you could use a VM assuming they were setup identically.
  - 0x457 4y ago
    Well, AWS offers "metal" servers.
  - porker 4y ago
    Two dedicated servers from Hetzner or similar provider, rented for a month
- stevewatson301 4y ago
  The c5 instances get dedicated cores, and thus should be exempt from resource contention due to shared cores.
  - speedgoose 4y ago
    Do you get dedicated IOs on these too? AWS tends to throttle heavily most instances after some time.
    - stevewatson301 4y ago
      For dedicated disk IOPS you should take a look at the EBS provisioned IO volumes, or perhaps use the ephemeral stores that come with some of their more expensive instances.
- tylerjl 4y ago
  This is hard because while, yes, some platform with less potential for jitter and/or noisy neighbors would help eliminate outside influence on the metrics, I think it's also valuable to benchmark these in a scenario that I would assume _most_ operators would run them in, which is a VPS situation and not bare-metal. FWIW, I did try really hard to eliminate some of the contamination in the results that would arise from running in a VPS by doing things like using the _same_ host reconfigured to avoid potential shifts in the underlying hypervisor, etc. But I would certainly agree that, for the utmost accurate results, a bare-metal situation would probably be more accurate than what I have written.
petecooper 4y ago
I'm an Nginx guy, and I have been for some years, but I do love a little bit of Caddy jingoism[1] as the weekend approaches. This is a good write up. I was expecting Caddy to trounce Nginx, but that wasn't the case. I'll be back to re-read this with fresh eyes tomorrow. [1] For the avoidance of doubt, this is not meant as a snarky observation.
- mholt 4y ago
  You were expecting Caddy to "trounce" nginx? Most people expect the opposite. But Caddy certainly does in some cases, especially with the upcoming 2.6 release.
  - petecooper 4y ago
    > You were expecting Caddy to "trounce" nginx? Most people expect the opposite. I absolutely was, yes. As an observer I see a lot of people saying positive things about Caddy around here, and how it’s superior performance-wise to a variety of ‘classic’ httpd software. Lots of people love Caddy, and they’re quite vocal, so it’s not a stretch to assume there are reasons why they love it. Nginx development has slowed since the events in Ukraine, unsurprisingly, so again it’s not a leap to surmise Caddy is making good things happen in the meantime.
    - mholt 4y ago
      Ahh, right -- so there's a lot more to performance than just req/sec and HTTP errors. And that's probably the love/hype you're hearing about. (Though Caddy's req/sec performance is quite good too, as you can see!) Caddy scales better than NGINX especially with regards to TLS/HTTPS. Our certificate automation code is the best in the industry, and works nicely in clusters to coordinate and share, automatically. Caddy performs better in terms of security overall. Go has stronger memory safety guarantees than C, so your server is basically impervious to a whole class of vulnerabilities. And if you consider failure modes, there are pros and cons to each, but it can definitely be argued that Caddy dropping fewer requests than nginx (if any at all!) is "superior performance". I'm actually quite pleased that Caddy can now, in general, perform competitively with nginx, and hopefully most people can stop worrying about that. And if you operate at Cloudflare-"nginx-is-now-too-slow-for-us"-scale, let's talk. (I have some ideas.)
gordian_NOT 4y ago
I feel like we never see HAProxy in these reverse proxy comparisons. Lots of nginx, Apache, Caddy, Traefik, Envoy, etc. The HAProxy configuration is just as simple as Caddy for a reverse proxy setup. It's also written in C which is a comparison the author makes between nginx and Caddy. And it seems to be available on every *nix OS.
- RcouF1uZ4gsC 4y ago
  > The HAProxy configuration is just as simple as Caddy for a reverse proxy setup. Does HAProxy have built in support for Let’s Encrypt? That is one of my favorite features. Caddy just automatically manages the certificates for https.
  - gordian_NOT 4y ago
    It's not as turn-key as Caddy, that's for sure, but it's there: https://www.haproxy.com/blog/lets-encrypt-acme2-for-haproxy/ https://www.haproxy.com/blog/lets-encrypt-acme2-for-haproxy/
    - ei8ths 4y ago
      this is great, i'll implement this soon as my current cert is about to expire and have been wanting to get haproxy on lets encrypt.
  - fullstop 4y ago
    Built-in? Not exactly, but there is an acmev2 implemention from haproxytech: https://github.com/haproxytech/haproxy-lua-acme https://github.com/haproxytech/haproxy-lua-acme
  - abdusco 4y ago
    I use caddy mostly as a reverse proxy in front of an app. It's just one line in the caddy file: sub.domain.com { # transparent proxy + websocket support + letsencrypt TLS reverse_proxy 127.0.0.1:2345 } It's a fresh breath of air to have server with sensible defaults after dealing with apache and nginx (haproxy isn't much better in that regard).
frankjr 4y ago
I'm impressed with Caddy's performance. I was expecting it to fall behind mainly due to the fact it's written in Go but apparently not. It's a bit disappointing that it's slower in reverse proxying, as that's one of the most important use cases, but now that it's identified maybe they can make some improvements. Finally, there really should be a max memory / max connections setting (maybe there is?).
- pjmlp 4y ago
  I am not a big fan of Go's design, however that is exactly one reason I tend to argue for it. There is enough juice in compiled managed languages that expose value types and low level features, it is a matter to learn how to use the tools on the toolbox instead of taking always the hammer out.
- zekica 4y ago
  Goroutines are efficient enough, and Go compiles to native code. I'm sure that Rust/Tokio or handcrafted C can be faster, but I think Go is fast enough for 99% of use cases. I'm building a service manager à la systemd in Go as a side project, and I really like it - it's not as low level as Rust and has a huge runtime but it is impressively fast.
- shabbatt 4y ago
  The only reason for me to consider Caddy was reverse proxy. Now that reason is gone and I'm happy with nginx
mordornginx 4y ago
People still liking nginx making money on it. nginx awful use and only make easy accidentally shoot one’s feet’s.
fullstop 4y ago
The author's writing style reminds me of Andy Weir's Project Hail Mary or The Martian.
Havoc 4y ago
Close enough to not matter in most use cases. ie pick whatever is convenient
- zivkovicp 4y ago
  This is almost always the case, no matter the service we're talking about.
  - eddieroger 4y ago
    Also, "pick what you know" applies here, too. If you know NGINX, then all you get from switching to Caddy is experience, and likewise, vice versa.
    - mholt 4y ago
      *and memory safety* This cannot be understated. Caddy is not written in C! And it can even run your NGINX configs. :) https://github.com/caddyserver/nginx-adapter https://github.com/caddyserver/nginx-adapter
      - excitom 4y ago
        A solution in search of a problem.
        5d8767c68926 4y ago
        Nginx security page [0] lists a non-zero amount of exploitable issues rooted in manual memory management. [0] https://nginx.org/en/security_advisories.html https://nginx.org/en/security_advisories.html
  - no_time 4y ago
    Is this how we ended up with electron for desktop applications and Java for backend?
    - 4y ago
pdhborges 4y ago
The author linked to wrk2 but I think he ended up using a k6 executor that exhibits the problem wrk2 was designed to solve.
- hassy 4y ago
  Yep, k6 suffers from coordinated omission [1] with its default settings. A tool that can send a request at a constant rate i.e. wrk2 or Vegeta [2] is a much better fit for this type of a performance test. 1. https://www.scylladb.com/2021/04/22/on-coordinated-omission/ https://www.scylladb.com/2021/04/22/on-coordinated-omission/ 2. https://github.com/tsenart/vegeta https://github.com/tsenart/vegeta
  - imiric 4y ago
    With its default settings, yes, but k6 can be configured to use an executor that implements the open model[1]. See more discussion here[2]. [1]: https://k6.io/docs/using-k6/scenarios/arrival-rate/ https://k6.io/docs/using-k6/scenarios/arrival-rate/ [2]: https://community.k6.io/t/is-k6-safe-from-the-coordinated-omission-problem/1484 https://community.k6.io/t/is-k6-safe-from-the-coordinated-om...
    - hassy 4y ago
      Yes indeed but wrk2 or Vegeta is still better for this particular use case (unless k6 has support for setting a constant RPS rate, afaik it does not), as otherwise the overhead of establishing a new TCP connection for a single HTTP request will dominate the benchmark.
      - imiric 4y ago
        > unless k6 has support for setting a constant RPS rate, afaik it does not Yes it can, via the `constant-arrival-rate` executor[1]. > the overhead of establishing a new TCP connection for a single HTTP request will dominate the benchmark By default, k6 will reuse TCP connections, and you have to explicitly disable it[2]. I'm not saying that wrk2 or Vegeta wouldn't be a good fit for this test, but k6 is also capable of it, with some minor configuration changes. [1]: https://k6.io/docs/using-k6/scenarios/executors/constant-arrival-rate/ https://k6.io/docs/using-k6/scenarios/executors/constant-arr... [2]: https://k6.io/docs/using-k6/k6-options/reference#no-connection-reuse https://k6.io/docs/using-k6/k6-options/reference#no-connecti...
kijin 4y ago
If you tell nginx to limit itself to 4 workers x 1024 connections per worker = 4096 connections, and hurl 10k connections at it, of course it's going to throw errors. It's doing exactly what you told it to do. That's just one example of how OP's "optimized" nginx config is barely even optimized. There are lots of other variables that you can tweak to get even better performance and blow Caddy out the window, but those tweaks are going to depend on the specific workload you expect to handle. There isn't a single, perfectly optimized, set of values that's going to work for everyone. The beauty of Caddy is that you get most of that performance without having to tweak anything.
- teknopaul 4y ago
  Nginx scales to 1,000,0000 workers per vm in my tests, but bandwidth is silly. I got those results by seriously limiting the junk in http headers. Not with real browsers. If you have that demand for any commercial service, you have money to distribute your load globally across more than one nginx instance.
nickjj 4y ago
I'm surprised no benchmarks were done with logging turned on. I get wanting to isolate things but this is the problem with micro benchmarks, it doesn't test "real world" usage patterns. Chances are your real production server will be logging to at least syslog so logging performance is worth looking into. If one of them can write logs with 500 microseconds added to reach request but the other takes 5 milliseconds that could be a huge difference in the end.
- mholt 4y ago
  Caddy's logger (uber/zap) is zero-allocation. We've found that the writing of the logs is often much slower, e.g. printing to the terminal or writing to a file. And that's a system problem more than a Caddy one. But the actual emission of logs is quite fast last time I checked!
  - nickjj 4y ago
    I think your statement is exactly why logging should have been turned on, at least for 1 of the benchmarks. If it's a system problem then it's a problem that both tools need to deal with. If one of them can do 100,000 requests per second but the other can do 80,000 requests per second but you're both capped at 30,000 requests per second because of system level limitations then you could make a strong case that both products perform equally in the end.
- tylerjl 4y ago
  This is - along with some reverse proxy settings tweaks - one of the variables I'd be keen to test in the future, since it's probably _the_ most common delta against my tests versus real-world applications.
Kiro 4y ago
Why would any of those fail at a measly 10k clients? 10 billion clients maybe.
- EugeneOZ 4y ago
  We don't have enough humans for that test.
  - teknopaul 4y ago
    A test is limited to 1024 clients which is (despite aversion to the term webscale) not a lot, even on an Intranet. I would say if you are not testing 10kcc you are not pushing the difference between nginx and apache1.3 As soon as you do push 10kcc, kernel tcp buffers and the amount of junk in your browser http headers start to be more import than server perf. Just in the amount of data coming into the nic.
asb 4y ago
I wrote up a few notes on my Caddy setup here https://muxup.com/2022q3/muxup-implementation-notes#serving-using-caddy https://muxup.com/2022q3/muxup-implementation-notes#serving-... which may be a useful reference if you have a static site and wanted to tick off a few items likely on your list (brotli, http3, cache-control, more fine-grained control on redirects). I don't think performance is ever going to matter for my use case, but one thing I think is worth highlighting is the quality of the community and maintainership. In a thread I started asking for feedback on my Caddyfile (https://caddy.community/t/suggestions-for-simplifying-my-caddyfile/17135 https://caddy.community/t/suggestions-for-simplifying-my-cad...), mholt determined I'd found a bug and rapidly fixed it. I followed up with a PR (https://github.com/caddyserver/website/pull/264 https://github.com/caddyserver/website/pull/264) for the docs to clarify something related to this bug which was reviewed and merged within 30 minutes.
- fariszr 4y ago
  A very helpful post, thanks for sharing it!
- mholt 4y ago
  Thanks for the comments and participation! I'm still thinking about that `./`-pattern-matching problem. Will probably have to be addressed after 2.6...
mholt 4y ago
This is a great writeup overall. I was happy to see Tyler's initial outreach before conducting his tests [0]. However, please note that these tests are also being revised shortly after some brief feedback [1]: - The sendfile tests at the end actually didn't use sendfile, so expect much greater performance there. - All the Caddy tests had metrics enabled, which are known[2] to be quite slow currently. Nginx does not emit metrics in its configuration, so in that sense the tests are a bit uneven. From my own tests, when I remove metrics code, Caddy is 10-20% faster. (We're working on addressing that [3].) - The tests in this article did not tune reverse proxy buffers, which are 4KB by default. I was able to see moderate performance improvements (depending on the size of payload) by reducing the buffer size to 1KB and 2KB. I want to thank Tyler for his considerate and careful approach, and for all the effort put into this! [0]: https://caddy.community/t/seeking-performance-suggestions-for-optimized-caddyfile/17052?u=matt https://caddy.community/t/seeking-performance-suggestions-fo... [1]: https://twitter.com/mholt6/status/1570442275339239424 https://twitter.com/mholt6/status/1570442275339239424 (thread) [2]: https://github.com/caddyserver/caddy/issues/4644 https://github.com/caddyserver/caddy/issues/4644 [3]: https://github.com/caddyserver/caddy/pull/5042 https://github.com/caddyserver/caddy/pull/5042
- fariszr 4y ago
  > - All the tests had metrics enabled, which are known[1] to be quite slow. From my own tests, when I remove metrics code, Caddy is 10-20% faster. But disabling metrics is not supported in standard Caddy, you need to remove specirc code and recompile to disable it. So maybe benchmarking with it isn't fair to Nginx?
  - mholt 4y ago
    > But disabling metrics is not supported in standard Caddy, you need to remove specirc code and recompile to disable it. We're addressing that quite soon. Unfortunately the original contributor of the feature has been too busy lately to work on it, so we might just have to go the simple route and make it opt-in instead. Expect to see a way to toggle metrics soon! Update: PR opened: https://github.com/caddyserver/caddy/pull/5042 https://github.com/caddyserver/caddy/pull/5042
cies 4y ago
I like Caddy, but on prod I do not need SSL (off-loaded by the LB), so I stick to nginx after reading this. Guess I'm waiting for Cloudflare to FLOSS-release their proxy https://news.ycombinator.com/item?id=32864119 https://news.ycombinator.com/item?id=32864119 :)
samcrawford 4y ago
Great write-up! One question I had was around the use of keepalives. There's no mention in the article of whether keepalives were used between the client and reverse proxy, and no mention of whether it was used between the reverse proxy and backend. I know Nginx doesn't use keepalives to backends by default (and I see it wasn't setup in the optimised Nginx proxy config), but it looks like Caddy does have keepalives enabled by default. Perhaps that could explain the delta in failure rates, at least for one case?
- mholt 4y ago
  Are you talking about HTTP keepalive or TCP keepalive? Keepalives can actually reduce the performance of a server with many concurrent clients (i.e. a benchmark test), and have other weird effects on benchmarks: https://www.nginx.com/blog/http-keepalives-and-web-performance/ https://www.nginx.com/blog/http-keepalives-and-web-performan...
  - teknopaul 4y ago
    Same thing. Http has no keep alive feature, you don't send http keep alive requests, if http 1.1 asks for keepalives it's a tcp thing.
    - mholt 4y ago
      They are distinct in Go. The standard library uses "HTTP keep-alive" to mark connections as idle based on most recent HTTP request, whereas TCP keep-alive checks only ACKs.
    - skyde 4y ago
      http 2.0 has keep alive!
bagels 4y ago
I think I don't get the joke. What does the X-Hotdogs header do?
- Arnavion 4y ago
  The header does nothing. As the article says, the author sent the header in every request, made a total of 35M requests, and thus gained a reason to use 35M hot dogs in the article title.
  - tylerjl 4y ago
    Correct. Maybe it blows my credibility out of the water and I'll be shamed for life, who knows
- francislavoie 4y ago
  Have you heard of "hotdog eating contests"? Or any other kind of eating contest? The joke is that he's having the two servers compete in "eating hotdogs" because the requests have a hotdog header. I think, anyway.
la_fayette 4y ago
These are interesting tests.considering the energy cost of large software systems it would be also intersting, which of these two has a lower co2 footprint.
tylerjl 4y ago
Hey y'all, author here. Traffic/feedback/questions are coming in hot, as HN publicity tends to engender, but I'm happy to answer questions or discuss the findings generally here if you'd like (I'm looking through the comments, too, but I'm more likely to see it here).
- jacooper 4y ago
  The black color for caddy in the charts is very hard to read in dark mode it would be great if you can change it to other colors
CoolCold 4y ago
Would be nice to have SO_REUSEPORT on Nginx optimized - if I read configuration right, it was not used
metaltyphoon 4y ago
I wonder how this compares to YARP.
davidjfelix 4y ago
FYI to author (who is in the comments): you may want to prevent the graphs from allowing scroll to zoom, I was scrolling on the page and the graphs were zooming in and out.
- lenkite 4y ago
  Article made nearly unreadable by conflating scrolling and zooming. Too much fiddling needed. Could have just added a range slider for folks who wanted to expand graphs.
skyde 4y ago
TLDR: "Nginx will fail by refusing or dropping connections, Caddy will fail by slowing everything down" To me it seem that Caddy suffer from BufferBloat. Under heavy congestion the goodput (useful throughput) will drop to 0 because client will start timing-out before the server get a chance to respond. Caddy should use an algorithm similar to : https://github.com/Netflix/concurrency-limits https://github.com/Netflix/concurrency-limits Basically check what was the best request latency, and decrease concurrency limit until latency stop improving.
- mholt 4y ago
  Thanks, we'll consider that, maybe as an option. Want to open an issue so we don't forget? I'd probably learn toward CUBIC: https://en.wikipedia.org/wiki/CUBIC_TCP https://en.wikipedia.org/wiki/CUBIC_TCP (I implemented Reno in college, but times have changed) The nice thing about Caddy's failure mode is that the server won't give up; the server has no control over if or when a client will time out, so it I never felt it made much sense to optimize for that.
  - skyde 4y ago
    thanks for considering the feedback! What i was trying to explain is Server not giving up is never a good thing under congestion. This is why in ip switches they drop packet when the outbound queue is full. And letting the caller retry sending the packet. When there is no congestion it’s ok for the server to not timeout at all and wait for the queue to drain and let the client decide when to close the tcp connection. but when there is congestion you don’t want the queue inside the server to get too long. Because the server ( including caddy) might be behind a tcp load-balancer so it’s better for the client to queue it’s request inside another Caddy instance that is less busy. feel free to reach out to me at maxime.caron@gmail.com I would be happy to try to add it to Caddy if you are interested
teknopaul 4y ago
Worker_connections 1024; Hello? http://xtomp.tp23.org/book/100kcc.html http://xtomp.tp23.org/book/100kcc.html Try worker_connections 1000000;
boberoni 4y ago
The killer feature of Caddy for me is that it handles TLS/HTTPS certificates automatically for me. I only ever use Caddy as a reverse proxy for web apps (think Flask, Ruby on Rails, Phoenix Framework). My projects have never needed high performance, but if my projects ever take off, it's nice to see that Caddy is already competitive with Nginx on resilience, latency, and throughput.
Sugimot0 4y ago
I'd love to see a comparison like this for caddy and pingora once pingora is open sourced. Here's the HN link for the announcement. https://news.ycombinator.com/item?id=32836661 https://news.ycombinator.com/item?id=32836661
AtNightWeCode 4y ago
Interesting. Maybe I read too fast but what data was it benchmarked with? A common flaw is to not test with real world data. Most benchmarks are wrong. Would guess this is too. My gut feeling is that Nginx is significantly faster.
bhawks 4y ago
Interesting article but from a data viz perspective things would be a ton easier to read with a chart per load type comparing nginx, optimized nginx and caddy. A single megachart with a dozen different colored bars is not easily parsable.

399 points by EntICOnc 4y ago

> Wew lad! Now we’re cooking with gas. This is the new gold standard for benchmarks! OP / Author, stupendously tremendous job. The methodology is defensible and sensible. Thank you for doing this on behalf of the community.

lelandfe 4y ago
Seconding! I am also in love with the friendliness and tone of the article. I’m a complete dummy when it comes to stuff like this and still understood most of it. Feynman would be proud.
mholt 4y ago
Yeah, Tyler did an amazing job.
tylerjl 4y ago
That's very kind of you to say, thank you!

stefantalpalaru 4y ago

> I’ll build hosts with Terraform (because That’s What We Use These Days) in EC2 > [...] > Create two EC2 instances - their default size is c5.xlarge When you're benchmarking, you want a stable platform between runs. Virtual private servers don't offer that, because the host's resources are shared between multiple guests, in unpredictable ways.

zmxz 4y ago
Which platform would you suggest to use for this benchmark?
- bdcravens 4y ago
  Ideally your own hardware with nothing else running on it. For convenience you could use a VM assuming they were setup identically.
- 0x457 4y ago
  Well, AWS offers "metal" servers.
- porker 4y ago
  Two dedicated servers from Hetzner or similar provider, rented for a month
stevewatson301 4y ago
The c5 instances get dedicated cores, and thus should be exempt from resource contention due to shared cores.
- speedgoose 4y ago
  Do you get dedicated IOs on these too? AWS tends to throttle heavily most instances after some time.
  - stevewatson301 4y ago
    For dedicated disk IOPS you should take a look at the EBS provisioned IO volumes, or perhaps use the ephemeral stores that come with some of their more expensive instances.
tylerjl 4y ago
This is hard because while, yes, some platform with less potential for jitter and/or noisy neighbors would help eliminate outside influence on the metrics, I think it's also valuable to benchmark these in a scenario that I would assume _most_ operators would run them in, which is a VPS situation and not bare-metal. FWIW, I did try really hard to eliminate some of the contamination in the results that would arise from running in a VPS by doing things like using the _same_ host reconfigured to avoid potential shifts in the underlying hypervisor, etc. But I would certainly agree that, for the utmost accurate results, a bare-metal situation would probably be more accurate than what I have written.

petecooper 4y ago

I'm an Nginx guy, and I have been for some years, but I do love a little bit of Caddy jingoism[1] as the weekend approaches. This is a good write up. I was expecting Caddy to trounce Nginx, but that wasn't the case. I'll be back to re-read this with fresh eyes tomorrow. [1] For the avoidance of doubt, this is not meant as a snarky observation.

mholt 4y ago
You were expecting Caddy to "trounce" nginx? Most people expect the opposite. But Caddy certainly does in some cases, especially with the upcoming 2.6 release.
- petecooper 4y ago
  > You were expecting Caddy to "trounce" nginx? Most people expect the opposite. I absolutely was, yes. As an observer I see a lot of people saying positive things about Caddy around here, and how it’s superior performance-wise to a variety of ‘classic’ httpd software. Lots of people love Caddy, and they’re quite vocal, so it’s not a stretch to assume there are reasons why they love it. Nginx development has slowed since the events in Ukraine, unsurprisingly, so again it’s not a leap to surmise Caddy is making good things happen in the meantime.
  - mholt 4y ago
    Ahh, right -- so there's a lot more to performance than just req/sec and HTTP errors. And that's probably the love/hype you're hearing about. (Though Caddy's req/sec performance is quite good too, as you can see!) Caddy scales better than NGINX especially with regards to TLS/HTTPS. Our certificate automation code is the best in the industry, and works nicely in clusters to coordinate and share, automatically. Caddy performs better in terms of security overall. Go has stronger memory safety guarantees than C, so your server is basically impervious to a whole class of vulnerabilities. And if you consider failure modes, there are pros and cons to each, but it can definitely be argued that Caddy dropping fewer requests than nginx (if any at all!) is "superior performance". I'm actually quite pleased that Caddy can now, in general, perform competitively with nginx, and hopefully most people can stop worrying about that. And if you operate at Cloudflare-"nginx-is-now-too-slow-for-us"-scale, let's talk. (I have some ideas.)

gordian_NOT 4y ago

I feel like we never see HAProxy in these reverse proxy comparisons. Lots of nginx, Apache, Caddy, Traefik, Envoy, etc. The HAProxy configuration is just as simple as Caddy for a reverse proxy setup. It's also written in C which is a comparison the author makes between nginx and Caddy. And it seems to be available on every *nix OS.

RcouF1uZ4gsC 4y ago
> The HAProxy configuration is just as simple as Caddy for a reverse proxy setup. Does HAProxy have built in support for Let’s Encrypt? That is one of my favorite features. Caddy just automatically manages the certificates for https.
- gordian_NOT 4y ago
  It's not as turn-key as Caddy, that's for sure, but it's there: https://www.haproxy.com/blog/lets-encrypt-acme2-for-haproxy/ https://www.haproxy.com/blog/lets-encrypt-acme2-for-haproxy/
  - ei8ths 4y ago
    this is great, i'll implement this soon as my current cert is about to expire and have been wanting to get haproxy on lets encrypt.
- fullstop 4y ago
  Built-in? Not exactly, but there is an acmev2 implemention from haproxytech: https://github.com/haproxytech/haproxy-lua-acme https://github.com/haproxytech/haproxy-lua-acme
- abdusco 4y ago
  I use caddy mostly as a reverse proxy in front of an app. It's just one line in the caddy file: sub.domain.com { # transparent proxy + websocket support + letsencrypt TLS reverse_proxy 127.0.0.1:2345 } It's a fresh breath of air to have server with sensible defaults after dealing with apache and nginx (haproxy isn't much better in that regard).

frankjr 4y ago

I'm impressed with Caddy's performance. I was expecting it to fall behind mainly due to the fact it's written in Go but apparently not. It's a bit disappointing that it's slower in reverse proxying, as that's one of the most important use cases, but now that it's identified maybe they can make some improvements. Finally, there really should be a max memory / max connections setting (maybe there is?).

pjmlp 4y ago
I am not a big fan of Go's design, however that is exactly one reason I tend to argue for it. There is enough juice in compiled managed languages that expose value types and low level features, it is a matter to learn how to use the tools on the toolbox instead of taking always the hammer out.
zekica 4y ago
Goroutines are efficient enough, and Go compiles to native code. I'm sure that Rust/Tokio or handcrafted C can be faster, but I think Go is fast enough for 99% of use cases. I'm building a service manager à la systemd in Go as a side project, and I really like it - it's not as low level as Rust and has a huge runtime but it is impressively fast.
shabbatt 4y ago
The only reason for me to consider Caddy was reverse proxy. Now that reason is gone and I'm happy with nginx

mordornginx 4y ago

People still liking nginx making money on it. nginx awful use and only make easy accidentally shoot one’s feet’s.

fullstop 4y ago

The author's writing style reminds me of Andy Weir's Project Hail Mary or The Martian.

Havoc 4y ago

Close enough to not matter in most use cases. ie pick whatever is convenient

zivkovicp 4y ago
This is almost always the case, no matter the service we're talking about.
- eddieroger 4y ago
  Also, "pick what you know" applies here, too. If you know NGINX, then all you get from switching to Caddy is experience, and likewise, vice versa.
  - mholt 4y ago
    *and memory safety* This cannot be understated. Caddy is not written in C! And it can even run your NGINX configs. :) https://github.com/caddyserver/nginx-adapter https://github.com/caddyserver/nginx-adapter
    - excitom 4y ago
      A solution in search of a problem.
      - 5d8767c68926 4y ago
        Nginx security page [0] lists a non-zero amount of exploitable issues rooted in manual memory management. [0] https://nginx.org/en/security_advisories.html https://nginx.org/en/security_advisories.html
- no_time 4y ago
  Is this how we ended up with electron for desktop applications and Java for backend?
  - 4y ago

pdhborges 4y ago

The author linked to wrk2 but I think he ended up using a k6 executor that exhibits the problem wrk2 was designed to solve.

hassy 4y ago
Yep, k6 suffers from coordinated omission [1] with its default settings. A tool that can send a request at a constant rate i.e. wrk2 or Vegeta [2] is a much better fit for this type of a performance test. 1. https://www.scylladb.com/2021/04/22/on-coordinated-omission/ https://www.scylladb.com/2021/04/22/on-coordinated-omission/ 2. https://github.com/tsenart/vegeta https://github.com/tsenart/vegeta
- imiric 4y ago
  With its default settings, yes, but k6 can be configured to use an executor that implements the open model[1]. See more discussion here[2]. [1]: https://k6.io/docs/using-k6/scenarios/arrival-rate/ https://k6.io/docs/using-k6/scenarios/arrival-rate/ [2]: https://community.k6.io/t/is-k6-safe-from-the-coordinated-omission-problem/1484 https://community.k6.io/t/is-k6-safe-from-the-coordinated-om...
  - hassy 4y ago
    Yes indeed but wrk2 or Vegeta is still better for this particular use case (unless k6 has support for setting a constant RPS rate, afaik it does not), as otherwise the overhead of establishing a new TCP connection for a single HTTP request will dominate the benchmark.
    - imiric 4y ago
      > unless k6 has support for setting a constant RPS rate, afaik it does not Yes it can, via the `constant-arrival-rate` executor[1]. > the overhead of establishing a new TCP connection for a single HTTP request will dominate the benchmark By default, k6 will reuse TCP connections, and you have to explicitly disable it[2]. I'm not saying that wrk2 or Vegeta wouldn't be a good fit for this test, but k6 is also capable of it, with some minor configuration changes. [1]: https://k6.io/docs/using-k6/scenarios/executors/constant-arrival-rate/ https://k6.io/docs/using-k6/scenarios/executors/constant-arr... [2]: https://k6.io/docs/using-k6/k6-options/reference#no-connection-reuse https://k6.io/docs/using-k6/k6-options/reference#no-connecti...

kijin 4y ago

If you tell nginx to limit itself to 4 workers x 1024 connections per worker = 4096 connections, and hurl 10k connections at it, of course it's going to throw errors. It's doing exactly what you told it to do. That's just one example of how OP's "optimized" nginx config is barely even optimized. There are lots of other variables that you can tweak to get even better performance and blow Caddy out the window, but those tweaks are going to depend on the specific workload you expect to handle. There isn't a single, perfectly optimized, set of values that's going to work for everyone. The beauty of Caddy is that you get most of that performance without having to tweak anything.

teknopaul 4y ago
Nginx scales to 1,000,0000 workers per vm in my tests, but bandwidth is silly. I got those results by seriously limiting the junk in http headers. Not with real browsers. If you have that demand for any commercial service, you have money to distribute your load globally across more than one nginx instance.

nickjj 4y ago

I'm surprised no benchmarks were done with logging turned on. I get wanting to isolate things but this is the problem with micro benchmarks, it doesn't test "real world" usage patterns. Chances are your real production server will be logging to at least syslog so logging performance is worth looking into. If one of them can write logs with 500 microseconds added to reach request but the other takes 5 milliseconds that could be a huge difference in the end.

mholt 4y ago
Caddy's logger (uber/zap) is zero-allocation. We've found that the writing of the logs is often much slower, e.g. printing to the terminal or writing to a file. And that's a system problem more than a Caddy one. But the actual emission of logs is quite fast last time I checked!
- nickjj 4y ago
  I think your statement is exactly why logging should have been turned on, at least for 1 of the benchmarks. If it's a system problem then it's a problem that both tools need to deal with. If one of them can do 100,000 requests per second but the other can do 80,000 requests per second but you're both capped at 30,000 requests per second because of system level limitations then you could make a strong case that both products perform equally in the end.
tylerjl 4y ago
This is - along with some reverse proxy settings tweaks - one of the variables I'd be keen to test in the future, since it's probably _the_ most common delta against my tests versus real-world applications.

Kiro 4y ago

Why would any of those fail at a measly 10k clients? 10 billion clients maybe.

EugeneOZ 4y ago
We don't have enough humans for that test.
- teknopaul 4y ago
  A test is limited to 1024 clients which is (despite aversion to the term webscale) not a lot, even on an Intranet. I would say if you are not testing 10kcc you are not pushing the difference between nginx and apache1.3 As soon as you do push 10kcc, kernel tcp buffers and the amount of junk in your browser http headers start to be more import than server perf. Just in the amount of data coming into the nic.

asb 4y ago

I wrote up a few notes on my Caddy setup here https://muxup.com/2022q3/muxup-implementation-notes#serving-using-caddy https://muxup.com/2022q3/muxup-implementation-notes#serving-... which may be a useful reference if you have a static site and wanted to tick off a few items likely on your list (brotli, http3, cache-control, more fine-grained control on redirects). I don't think performance is ever going to matter for my use case, but one thing I think is worth highlighting is the quality of the community and maintainership. In a thread I started asking for feedback on my Caddyfile (https://caddy.community/t/suggestions-for-simplifying-my-caddyfile/17135 https://caddy.community/t/suggestions-for-simplifying-my-cad...), mholt determined I'd found a bug and rapidly fixed it. I followed up with a PR (https://github.com/caddyserver/website/pull/264 https://github.com/caddyserver/website/pull/264) for the docs to clarify something related to this bug which was reviewed and merged within 30 minutes.

fariszr 4y ago
A very helpful post, thanks for sharing it!
mholt 4y ago
Thanks for the comments and participation! I'm still thinking about that `./`-pattern-matching problem. Will probably have to be addressed after 2.6...

mholt 4y ago

This is a great writeup overall. I was happy to see Tyler's initial outreach before conducting his tests [0]. However, please note that these tests are also being revised shortly after some brief feedback [1]: - The sendfile tests at the end actually didn't use sendfile, so expect much greater performance there. - All the Caddy tests had metrics enabled, which are known[2] to be quite slow currently. Nginx does not emit metrics in its configuration, so in that sense the tests are a bit uneven. From my own tests, when I remove metrics code, Caddy is 10-20% faster. (We're working on addressing that [3].) - The tests in this article did not tune reverse proxy buffers, which are 4KB by default. I was able to see moderate performance improvements (depending on the size of payload) by reducing the buffer size to 1KB and 2KB. I want to thank Tyler for his considerate and careful approach, and for all the effort put into this! [0]: https://caddy.community/t/seeking-performance-suggestions-for-optimized-caddyfile/17052?u=matt https://caddy.community/t/seeking-performance-suggestions-fo... [1]: https://twitter.com/mholt6/status/1570442275339239424 https://twitter.com/mholt6/status/1570442275339239424 (thread) [2]: https://github.com/caddyserver/caddy/issues/4644 https://github.com/caddyserver/caddy/issues/4644 [3]: https://github.com/caddyserver/caddy/pull/5042 https://github.com/caddyserver/caddy/pull/5042

fariszr 4y ago
> - All the tests had metrics enabled, which are known[1] to be quite slow. From my own tests, when I remove metrics code, Caddy is 10-20% faster. But disabling metrics is not supported in standard Caddy, you need to remove specirc code and recompile to disable it. So maybe benchmarking with it isn't fair to Nginx?
- mholt 4y ago
  > But disabling metrics is not supported in standard Caddy, you need to remove specirc code and recompile to disable it. We're addressing that quite soon. Unfortunately the original contributor of the feature has been too busy lately to work on it, so we might just have to go the simple route and make it opt-in instead. Expect to see a way to toggle metrics soon! Update: PR opened: https://github.com/caddyserver/caddy/pull/5042 https://github.com/caddyserver/caddy/pull/5042

cies 4y ago

I like Caddy, but on prod I do not need SSL (off-loaded by the LB), so I stick to nginx after reading this. Guess I'm waiting for Cloudflare to FLOSS-release their proxy https://news.ycombinator.com/item?id=32864119 https://news.ycombinator.com/item?id=32864119 :)

samcrawford 4y ago

Great write-up! One question I had was around the use of keepalives. There's no mention in the article of whether keepalives were used between the client and reverse proxy, and no mention of whether it was used between the reverse proxy and backend. I know Nginx doesn't use keepalives to backends by default (and I see it wasn't setup in the optimised Nginx proxy config), but it looks like Caddy does have keepalives enabled by default. Perhaps that could explain the delta in failure rates, at least for one case?

mholt 4y ago
Are you talking about HTTP keepalive or TCP keepalive? Keepalives can actually reduce the performance of a server with many concurrent clients (i.e. a benchmark test), and have other weird effects on benchmarks: https://www.nginx.com/blog/http-keepalives-and-web-performance/ https://www.nginx.com/blog/http-keepalives-and-web-performan...
- teknopaul 4y ago
  Same thing. Http has no keep alive feature, you don't send http keep alive requests, if http 1.1 asks for keepalives it's a tcp thing.
  - mholt 4y ago
    They are distinct in Go. The standard library uses "HTTP keep-alive" to mark connections as idle based on most recent HTTP request, whereas TCP keep-alive checks only ACKs.
  - skyde 4y ago
    http 2.0 has keep alive!

bagels 4y ago

I think I don't get the joke. What does the X-Hotdogs header do?

Arnavion 4y ago
The header does nothing. As the article says, the author sent the header in every request, made a total of 35M requests, and thus gained a reason to use 35M hot dogs in the article title.
- tylerjl 4y ago
  Correct. Maybe it blows my credibility out of the water and I'll be shamed for life, who knows
francislavoie 4y ago
Have you heard of "hotdog eating contests"? Or any other kind of eating contest? The joke is that he's having the two servers compete in "eating hotdogs" because the requests have a hotdog header. I think, anyway.

la_fayette 4y ago

These are interesting tests.considering the energy cost of large software systems it would be also intersting, which of these two has a lower co2 footprint.

tylerjl 4y ago

Hey y'all, author here. Traffic/feedback/questions are coming in hot, as HN publicity tends to engender, but I'm happy to answer questions or discuss the findings generally here if you'd like (I'm looking through the comments, too, but I'm more likely to see it here).

jacooper 4y ago
The black color for caddy in the charts is very hard to read in dark mode it would be great if you can change it to other colors

CoolCold 4y ago

Would be nice to have SO_REUSEPORT on Nginx optimized - if I read configuration right, it was not used

metaltyphoon 4y ago

I wonder how this compares to YARP.

davidjfelix 4y ago

FYI to author (who is in the comments): you may want to prevent the graphs from allowing scroll to zoom, I was scrolling on the page and the graphs were zooming in and out.

lenkite 4y ago
Article made nearly unreadable by conflating scrolling and zooming. Too much fiddling needed. Could have just added a range slider for folks who wanted to expand graphs.

skyde 4y ago

TLDR: "Nginx will fail by refusing or dropping connections, Caddy will fail by slowing everything down" To me it seem that Caddy suffer from BufferBloat. Under heavy congestion the goodput (useful throughput) will drop to 0 because client will start timing-out before the server get a chance to respond. Caddy should use an algorithm similar to : https://github.com/Netflix/concurrency-limits https://github.com/Netflix/concurrency-limits Basically check what was the best request latency, and decrease concurrency limit until latency stop improving.

mholt 4y ago
Thanks, we'll consider that, maybe as an option. Want to open an issue so we don't forget? I'd probably learn toward CUBIC: https://en.wikipedia.org/wiki/CUBIC_TCP https://en.wikipedia.org/wiki/CUBIC_TCP (I implemented Reno in college, but times have changed) The nice thing about Caddy's failure mode is that the server won't give up; the server has no control over if or when a client will time out, so it I never felt it made much sense to optimize for that.
- skyde 4y ago
  thanks for considering the feedback! What i was trying to explain is Server not giving up is never a good thing under congestion. This is why in ip switches they drop packet when the outbound queue is full. And letting the caller retry sending the packet. When there is no congestion it’s ok for the server to not timeout at all and wait for the queue to drain and let the client decide when to close the tcp connection. but when there is congestion you don’t want the queue inside the server to get too long. Because the server ( including caddy) might be behind a tcp load-balancer so it’s better for the client to queue it’s request inside another Caddy instance that is less busy. feel free to reach out to me at maxime.caron@gmail.com I would be happy to try to add it to Caddy if you are interested

teknopaul 4y ago

Worker_connections 1024; Hello? http://xtomp.tp23.org/book/100kcc.html http://xtomp.tp23.org/book/100kcc.html Try worker_connections 1000000;

boberoni 4y ago

The killer feature of Caddy for me is that it handles TLS/HTTPS certificates automatically for me. I only ever use Caddy as a reverse proxy for web apps (think Flask, Ruby on Rails, Phoenix Framework). My projects have never needed high performance, but if my projects ever take off, it's nice to see that Caddy is already competitive with Nginx on resilience, latency, and throughput.

Sugimot0 4y ago

I'd love to see a comparison like this for caddy and pingora once pingora is open sourced. Here's the HN link for the announcement. https://news.ycombinator.com/item?id=32836661 https://news.ycombinator.com/item?id=32836661

AtNightWeCode 4y ago

Interesting. Maybe I read too fast but what data was it benchmarked with? A common flaw is to not test with real world data. Most benchmarks are wrong. Would guess this is too. My gut feeling is that Nginx is significantly faster.

bhawks 4y ago

Interesting article but from a data viz perspective things would be a ton easier to read with a chart per load type comparing nginx, optimized nginx and caddy. A single megachart with a dozen different colored bars is not easily parsable.