4 ms·

> - All the tests had metrics enabled, which are known[1] to be quite slow. From my own tests, when I remove metrics code, Caddy is 10-20% faster. But disablin

> - All the tests had metrics enabled, which are known[1] to be quite slow. From my own tests, when I remove metrics code, Caddy is 10-20% faster. But disabling metrics is not supported in standard Caddy, you need to remove specirc code and recompile to disable it. So maybe benchmarking with it isn't fair to Nginx?

mholt 4y ago
> But disabling metrics is not supported in standard Caddy, you need to remove specirc code and recompile to disable it. We're addressing that quite soon. Unfortunately the original contributor of the feature has been too busy lately to work on it, so we might just have to go the simple route and make it opt-in instead. Expect to see a way to toggle metrics soon! Update: PR opened: https://github.com/caddyserver/caddy/pull/5042 https://github.com/caddyserver/caddy/pull/5042
- Bilal_io 4y ago
  That was fast. I love it!
- philipwhiuk 4y ago
  There's always going to be some cost to metrics, going forward you probably just want to document it and then update the figure as you tune it. Higher performance opt-in metrics are the sort of thing a company using it at scale ought to be able to help with/sponsor work on.
  - mholt 4y ago
    Absolutely. The plan is to make it opt-in for now, and then having a company sponsor the performance tuning would be very welcomed. Otherwise it'll probably sit until someone with the right skills/know-how and time comes along.
tialaramex 4y ago
Yeah, I think Fair comparisons are: * How do these things perform by default. This is how they're going to perform for many users, because if it's adequate nobody will tune them, why bother. * How do these things perform with performance configuration as often recommended online. This is how they'll perform for people who think they need performance but don't tune or don't know how to tune, this might be worse than default but that's actually useful information. * How do these things perform when their authors get to tune them for our test workload. This is how they'll perform for users who squeeze every drop and can afford to get somebody to do real work to facilitate, possibly even hiring the same authors to do it. In some cases I would also really want to see: * How do these things perform with recommended security. A benchmark mode with great scores but lousy security can promote a race to the bottom where everybody ships insecure garbage by default then has a mode which is never measured and has lousy performance yet is mandatory if you don't think Hunter2 is a great password.
- mholt 4y ago
  > How do these things perform by default. Agreed on this one -- today I'm looking at how to disable metrics by default and make them opt-in. At least until the performance regression can be addressed. Update: PR opened: https://github.com/caddyserver/caddy/pull/5042 https://github.com/caddyserver/caddy/pull/5042 - hoping to land that before 2.6.
  - dQw4w9WgXcQ 4y ago
    Good stuff dude. Listens to users, sees a problem, doesn't take it personally, makes a fix. Caddy's going places.
    - mholt 4y ago
      I'm also grateful that Dave, the original contributor of the metrics feature, isn't taking it personally. We love the functionality! Just gotta refine it...

mholt 4y ago

> But disabling metrics is not supported in standard Caddy, you need to remove specirc code and recompile to disable it. We're addressing that quite soon. Unfortunately the original contributor of the feature has been too busy lately to work on it, so we might just have to go the simple route and make it opt-in instead. Expect to see a way to toggle metrics soon! Update: PR opened: https://github.com/caddyserver/caddy/pull/5042 https://github.com/caddyserver/caddy/pull/5042

Bilal_io 4y ago
That was fast. I love it!
philipwhiuk 4y ago
There's always going to be some cost to metrics, going forward you probably just want to document it and then update the figure as you tune it. Higher performance opt-in metrics are the sort of thing a company using it at scale ought to be able to help with/sponsor work on.
- mholt 4y ago
  Absolutely. The plan is to make it opt-in for now, and then having a company sponsor the performance tuning would be very welcomed. Otherwise it'll probably sit until someone with the right skills/know-how and time comes along.

tialaramex 4y ago

Yeah, I think Fair comparisons are: * How do these things perform by default. This is how they're going to perform for many users, because if it's adequate nobody will tune them, why bother. * How do these things perform with performance configuration as often recommended online. This is how they'll perform for people who think they need performance but don't tune or don't know how to tune, this might be worse than default but that's actually useful information. * How do these things perform when their authors get to tune them for our test workload. This is how they'll perform for users who squeeze every drop and can afford to get somebody to do real work to facilitate, possibly even hiring the same authors to do it. In some cases I would also really want to see: * How do these things perform with recommended security. A benchmark mode with great scores but lousy security can promote a race to the bottom where everybody ships insecure garbage by default then has a mode which is never measured and has lousy performance yet is mandatory if you don't think Hunter2 is a great password.

mholt 4y ago
> How do these things perform by default. Agreed on this one -- today I'm looking at how to disable metrics by default and make them opt-in. At least until the performance regression can be addressed. Update: PR opened: https://github.com/caddyserver/caddy/pull/5042 https://github.com/caddyserver/caddy/pull/5042 - hoping to land that before 2.6.
- dQw4w9WgXcQ 4y ago
  Good stuff dude. Listens to users, sees a problem, doesn't take it personally, makes a fix. Caddy's going places.
  - mholt 4y ago
    I'm also grateful that Dave, the original contributor of the metrics feature, isn't taking it personally. We love the functionality! Just gotta refine it...