5 ms·

I love what the Caddy project is doing, but every time I look into it I can’t get a straight answer about performance. One of the top search results for Caddy

I love what the Caddy project is doing, but every time I look into it I can’t get a straight answer about performance. One of the top search results for Caddy performance is this Caddy forum thread with a response from the Caddy author: https://caddy.community/t/performance-compared-to-nginx/7993 https://caddy.community/t/performance-compared-to-nginx/7993 But it seems to be dodging the question and linking to the most misleading possible Tweets instead of providing any actual info: > Caddy is written in Go. It will perform just as well as any other Go web servers. > Google, Netflix, Cloudflare, Fastly, and other large network companies deploy Go on their edge. You can too. What a bizarre response. They then link to a Tweet from someone who claims that Caddy performed 20X better than nginx. An impressive claim! But further investigation shows that virtually every other benchmark I can find shows the opposite situation: Nginx beating Caddy by up to 8X. Performance isn’t everything and there are plenty of situations where Caddy’s ease of use might prevail, but I get nervous when project authors are giving cagey and misleading responses to straightforward (and important) questions.

mholt 5y ago
What are your performance requirements?
- PragmaticPulp 5y ago
  > What are your performance requirements? If the requirements were such that web server performance didn’t matter, I wouldn’t care and wouldn’t be asking about it. But the problem is that it doesn’t make sense to hide the performance statistics and then debate each user’s performance requirements instead of just letting the performance be a known quantity so we can all decide for ourselves.
  - mholt 5y ago
    We aren't "hiding" any performance stats. They just don't exist. It's super hard to generate them in a way that applies perfectly to everyone's use case. I'm still waiting to hear your performance requirements btw.
    - PragmaticPulp 5y ago
      > I'm still waiting to hear your performance requirements btw. I suppose this encapsulates why I get turned off of Caddy every time I look into it: Basic concerns (including questions about approximate performance) are met with suggestions that I don’t know what I want and therefore I’m not qualified to make such judgments.
      - throwaway894345 5y ago
        You're being unduly defensive. mholt is telling you that he can't provide meaningful general purpose performance numbers because the breadth of cases for Caddy vary too widely, so he's asking you for information about your specific use case so he can give you the very information you're complaining about not being able to access, but you're being cagey about sharing those requirements. You're certainly within your rights to not use Caddy, but it makes no sense to complain that you can't get the performance information you need when you won't share any information about your use case.
        PragmaticPulp 5y ago
        > but you're being cagey about sharing those requirements. I don’t have one single set of requirements for every location I use a web server. Nobody does. If Caddy is good for one thing but not so good for other things, why is it so hard to come out and say that? Also, the benchmark I was questioning in the opening comment came from mholt on the official community forum. It’s mind boggling that I’m getting attacked for questioning the benchmark result that they provided in the first place.
        joshmanders 5y ago
        Nobody is attacking you. Calm down, yeesh. You want performance metrics, but can't even tell us what metrics matter to you, but expect Matt and Caddy team to have them just available... You understand that your responses here are directly proving his point on not having them, right?
        5y ago
- yjftsjthsd-h 5y ago
  This is exactly what the original complaint was: Someone complains that Caddy is slower than the options, and the official response is not neither agreement nor giving data to show that Caddy is fast, but equivocating about whether it matters while avoiding commenting on whether the claim is true. If you want to ship a webserver that's slower but safer and easy to use, that's fine; I'm using Caddy to host my stuff because that's a reasonable trade off. It's just the dodging of the question that gets old fast.
  - vxNsr 5y ago
    just fyi, based on the bio of the person you're replying to, they're the author of caddy. not sure if this was known or not
    - mholt 5y ago
      Yeah, it's hard for us to hit a performance target that is being kept hidden from the devs.
      - qbasic_forever 5y ago
        This shouldn't be downvoted--it's very true, what performance are you trying to optimize? Throughput of long connections? Time from connection to first bytes? There are dozens of dimensions to web performance.
        yjftsjthsd-h 5y ago
        That's totally reasonable. Does Caddy actually win on any of those dimensions?
        withinboredom 5y ago
        There are millions of configurations for Caddy (which modules you compile into the binary) so this is literally impossible to know unless your benchmark your particular configuration for your particular use case.
earthboundkid 5y ago
I think the point is just that for most applications, the reverse proxy server is not a performance bottleneck, so it doesn't matter for any practical purpose. Caddy could be twice as fast or slow and it would not change the number of servers you need to deploy one way or another. If your reverse proxy is a bottleneck, you'll know and probably have the resources to build something custom instead of using Caddy or whatever.
- emteycz 5y ago
  Yeah but this means that you could wait 8x longer with NGINX before building a custom solution.
  - throwaway894345 5y ago
    Which itself is pretty meaningless considering almost no one bottlenecks on their reverse proxy.
- PragmaticPulp 5y ago
  > I think the point is just that for most applications… I guess it would make more sense if they came right out and said that it’s best used for these applications where it’s acting as a reverse proxy for a heavy backend and therefore performance isn’t a big issue. But I’m constantly confused by the vague responses combined with the allusions to claims that it’s super-fast because it uses Go like Google or Tweets about it being 20X more performant than nginx when it’s clearly not.
  - mholt 5y ago
    It's hard to quantify web server performance in a way that is both truly representative and truly generalizable, such that you can draw correct conclusions for specific setups and use cases. Ultimately we just recommend you do your own tuning and performance testing, especially since so many external factors are involved.
    - 5y ago
cunthorpe 5y ago
It’s funny that your complaint about opaque answers got responses that imply that performance doesn’t matter. What this tells me is that Caddy is slower than its competitors and they know it. If it wasn’t, then they wouldn’t hide behind meaningless responses. Basically they answered without answering.
- kreetx 5y ago
  Yup. They should just say "it's the ease of use" and be done with it. It's a pretty good reason to use a technology.
- francislavoie 5y ago
  You misunderstood then, because what was said was essentially "official benchmarks for servers are meaningless", not that "performance doesn't matter". Do your own testing, for your own usecase. But still, a server should rarely ever be your bottleneck. Your application's DB I/O will be.
  - PragmaticPulp 5y ago
    > You misunderstood then, because what was said was essentially "official benchmarks for servers are meaningless", not that "performance doesn't matter". Do your own testing, for your own usecase. The Caddy forum thread I linked to above highlighted a Tweet about Caddy being 20X more performant than nginx. This was from the Caddy team. The frustrating part is how benchmarks are championed as a selling point when they benefit the project, but the argument becomes “official benchmarks are meaningless” as soon as they don’t. That, and the weird opaque responses and insistence that we debate my performance requirements on the internet when I just wanted to know how Caddy compares to nginx in the most broad terms. Similar performance? Order of magnitude faster? Order of magnitude slower? Why does such a large debate have to erupt when such simple questions are asked? If Caddy is good for one use case but not for others, why not just say it? Why must it become a one-on-one debate? I don’t literally have one use case for a web server. I want to know when it’s appropriate to choose so I can make these decisions myself without engaging in an HN comment section back-and-forth to figure it out.
pachico 5y ago
Can't think of any situations I was involved where the bottleneck was the http server app rather than the application behind it. Am I the only one?
- PragmaticPulp 5y ago
  If you’re only serving up heavyweight web apps, then this is probably true. But Caddy also markets itself as an efficient static file webserver right on the home page: > Caddy is both a flexible, efficient static file server and a powerful, scalable reverse proxy. Perhaps the confusing part is the way that Caddy is marketed as “The Ultimate Server” when instead it’s designed to be a fast-enough reverse proxy for applications where performance doesn’t matter. There are many applications where performance really does matter. Static file serving is the most obvious example.
  - throwaway894345 5y ago
    > There are many applications where performance really does matter. Like what? Genuine question. I'm sure a cloud provider's layer 7 load balancer / Kubernetes Ingress would merit an optimized implementation but beyond those kinds of super-scale use cases I can't think of much.
- rampointerhater 5y ago
  No. But looking at the site Caddy appears to be a product for easy and quick deployment, so that zero-knowledge people can run a web server. Their point is unlikely to be performance, it is probably in the lines of "low code".
- Bayart 5y ago
  As the other person that answered to you pointed out, there's a point where your HTTP server becomes the bottleneck, ie your backend being more concurrent and faster at producing content than your server is at delivering it. But I feel the real bottleneck is developer pain. I switched to nginx from Apache like everyone else because nginx was nice to use and Apache wasn't. The performance aspect was only a rationalization.

mholt 5y ago

What are your performance requirements?

PragmaticPulp 5y ago
> What are your performance requirements? If the requirements were such that web server performance didn’t matter, I wouldn’t care and wouldn’t be asking about it. But the problem is that it doesn’t make sense to hide the performance statistics and then debate each user’s performance requirements instead of just letting the performance be a known quantity so we can all decide for ourselves.
- mholt 5y ago
  We aren't "hiding" any performance stats. They just don't exist. It's super hard to generate them in a way that applies perfectly to everyone's use case. I'm still waiting to hear your performance requirements btw.
  - PragmaticPulp 5y ago
    > I'm still waiting to hear your performance requirements btw. I suppose this encapsulates why I get turned off of Caddy every time I look into it: Basic concerns (including questions about approximate performance) are met with suggestions that I don’t know what I want and therefore I’m not qualified to make such judgments.
    - throwaway894345 5y ago
      You're being unduly defensive. mholt is telling you that he can't provide meaningful general purpose performance numbers because the breadth of cases for Caddy vary too widely, so he's asking you for information about your specific use case so he can give you the very information you're complaining about not being able to access, but you're being cagey about sharing those requirements. You're certainly within your rights to not use Caddy, but it makes no sense to complain that you can't get the performance information you need when you won't share any information about your use case.
      - PragmaticPulp 5y ago
        > but you're being cagey about sharing those requirements. I don’t have one single set of requirements for every location I use a web server. Nobody does. If Caddy is good for one thing but not so good for other things, why is it so hard to come out and say that? Also, the benchmark I was questioning in the opening comment came from mholt on the official community forum. It’s mind boggling that I’m getting attacked for questioning the benchmark result that they provided in the first place.
        joshmanders 5y ago
        Nobody is attacking you. Calm down, yeesh. You want performance metrics, but can't even tell us what metrics matter to you, but expect Matt and Caddy team to have them just available... You understand that your responses here are directly proving his point on not having them, right?
        5y ago
yjftsjthsd-h 5y ago
This is exactly what the original complaint was: Someone complains that Caddy is slower than the options, and the official response is not neither agreement nor giving data to show that Caddy is fast, but equivocating about whether it matters while avoiding commenting on whether the claim is true. If you want to ship a webserver that's slower but safer and easy to use, that's fine; I'm using Caddy to host my stuff because that's a reasonable trade off. It's just the dodging of the question that gets old fast.
- vxNsr 5y ago
  just fyi, based on the bio of the person you're replying to, they're the author of caddy. not sure if this was known or not
  - mholt 5y ago
    Yeah, it's hard for us to hit a performance target that is being kept hidden from the devs.
    - qbasic_forever 5y ago
      This shouldn't be downvoted--it's very true, what performance are you trying to optimize? Throughput of long connections? Time from connection to first bytes? There are dozens of dimensions to web performance.
      - yjftsjthsd-h 5y ago
        That's totally reasonable. Does Caddy actually win on any of those dimensions?
        withinboredom 5y ago
        There are millions of configurations for Caddy (which modules you compile into the binary) so this is literally impossible to know unless your benchmark your particular configuration for your particular use case.

earthboundkid 5y ago

I think the point is just that for most applications, the reverse proxy server is not a performance bottleneck, so it doesn't matter for any practical purpose. Caddy could be twice as fast or slow and it would not change the number of servers you need to deploy one way or another. If your reverse proxy is a bottleneck, you'll know and probably have the resources to build something custom instead of using Caddy or whatever.

emteycz 5y ago
Yeah but this means that you could wait 8x longer with NGINX before building a custom solution.
- throwaway894345 5y ago
  Which itself is pretty meaningless considering almost no one bottlenecks on their reverse proxy.
PragmaticPulp 5y ago
> I think the point is just that for most applications… I guess it would make more sense if they came right out and said that it’s best used for these applications where it’s acting as a reverse proxy for a heavy backend and therefore performance isn’t a big issue. But I’m constantly confused by the vague responses combined with the allusions to claims that it’s super-fast because it uses Go like Google or Tweets about it being 20X more performant than nginx when it’s clearly not.
- mholt 5y ago
  It's hard to quantify web server performance in a way that is both truly representative and truly generalizable, such that you can draw correct conclusions for specific setups and use cases. Ultimately we just recommend you do your own tuning and performance testing, especially since so many external factors are involved.
  - 5y ago

cunthorpe 5y ago

It’s funny that your complaint about opaque answers got responses that imply that performance doesn’t matter. What this tells me is that Caddy is slower than its competitors and they know it. If it wasn’t, then they wouldn’t hide behind meaningless responses. Basically they answered without answering.

kreetx 5y ago
Yup. They should just say "it's the ease of use" and be done with it. It's a pretty good reason to use a technology.
francislavoie 5y ago
You misunderstood then, because what was said was essentially "official benchmarks for servers are meaningless", not that "performance doesn't matter". Do your own testing, for your own usecase. But still, a server should rarely ever be your bottleneck. Your application's DB I/O will be.
- PragmaticPulp 5y ago
  > You misunderstood then, because what was said was essentially "official benchmarks for servers are meaningless", not that "performance doesn't matter". Do your own testing, for your own usecase. The Caddy forum thread I linked to above highlighted a Tweet about Caddy being 20X more performant than nginx. This was from the Caddy team. The frustrating part is how benchmarks are championed as a selling point when they benefit the project, but the argument becomes “official benchmarks are meaningless” as soon as they don’t. That, and the weird opaque responses and insistence that we debate my performance requirements on the internet when I just wanted to know how Caddy compares to nginx in the most broad terms. Similar performance? Order of magnitude faster? Order of magnitude slower? Why does such a large debate have to erupt when such simple questions are asked? If Caddy is good for one use case but not for others, why not just say it? Why must it become a one-on-one debate? I don’t literally have one use case for a web server. I want to know when it’s appropriate to choose so I can make these decisions myself without engaging in an HN comment section back-and-forth to figure it out.

pachico 5y ago

Can't think of any situations I was involved where the bottleneck was the http server app rather than the application behind it. Am I the only one?

PragmaticPulp 5y ago
If you’re only serving up heavyweight web apps, then this is probably true. But Caddy also markets itself as an efficient static file webserver right on the home page: > Caddy is both a flexible, efficient static file server and a powerful, scalable reverse proxy. Perhaps the confusing part is the way that Caddy is marketed as “The Ultimate Server” when instead it’s designed to be a fast-enough reverse proxy for applications where performance doesn’t matter. There are many applications where performance really does matter. Static file serving is the most obvious example.
- throwaway894345 5y ago
  > There are many applications where performance really does matter. Like what? Genuine question. I'm sure a cloud provider's layer 7 load balancer / Kubernetes Ingress would merit an optimized implementation but beyond those kinds of super-scale use cases I can't think of much.
rampointerhater 5y ago
No. But looking at the site Caddy appears to be a product for easy and quick deployment, so that zero-knowledge people can run a web server. Their point is unlikely to be performance, it is probably in the lines of "low code".
Bayart 5y ago
As the other person that answered to you pointed out, there's a point where your HTTP server becomes the bottleneck, ie your backend being more concurrent and faster at producing content than your server is at delivering it. But I feel the real bottleneck is developer pain. I switched to nginx from Apache like everyone else because nginx was nice to use and Apache wasn't. The performance aspect was only a rationalization.