7 ms·

Weirdly enough, among the reasons to switch from nginx to Caddy I never find the lack of observability nginx and its almost useless Prometheus exporter have. I

Weirdly enough, among the reasons to switch from nginx to Caddy I never find the lack of observability nginx and its almost useless Prometheus exporter have. I see several posts discussing under which circumstances or use cases one might outperform the other but they never seem to care about having decent metrics. I might overrate the importance of this, who knows...

nginxlegacy 4y ago
- xet7 4y ago
  Aren't? But there is https://redbean.dev https://redbean.dev that uses Lua, and is faster than Go.
  - coder543 4y ago
    Redbean is a webserver. Go is not a webserver. Redbean cannot be "faster than Go". That's like saying that a car is faster than aluminum. Do you have benchmarks of Redbean versus Caddy that you can point to? I certainly haven't been able to find any. I also haven't been able to find any in-depth independent benchmarks of Redbean. About the only thing I could find was someone struggling to reproduce the benchmark results.[0] Personally, I've dealt with more than enough memory safety vulnerabilities caused by no programmer being good enough to write safe and correct C (or C++). Those vulnerabilities are easily prevented by choosing just about any other language people are likely to know about... so, I have no desire to deploy anything like Redbean when alternatives exist. It's cool to see the hacks they've been able to do with Cosmopolitan to make cross platform binaries, but hacks like that are not what I want for anything other than educational purposes, at least not until they've been more battle tested. [0]: https://github.com/jart/cosmopolitan/issues/72 https://github.com/jart/cosmopolitan/issues/72
    - mholt 4y ago
      Great points. I also think it's important to stop deploying C to the edge. PS. Go might someday compile for cosmopolitan if someone writes a port for it: https://github.com/golang/go/issues/51900 https://github.com/golang/go/issues/51900
    - xet7 4y ago
      I did see that benchmark result previously but did not find that page yet with quick look. From https://redbean.dev https://redbean.dev is this: "If you want Rust-like promises then redbean can be compiled with ASAN memory safety. Cosmopolitan Libc has the only open source implementation of the Address Sanitizer runtime that's intended for production use. It causes redbean worker processes to crash and log a report should a bug like a buffer overrun, use-after-free, or stack smash occur. This has a marginal impact on performance. It can be useful in environments where interruptions in a service are more desirable than risking the system being compromised." More at https://redbean.dev/#security https://redbean.dev/#security
      - coder543 4y ago
        ASAN by itself cannot make those guarantees, from what I understand. Cosmopolitan would also have to make sure never to reuse virtual memory addresses for a start, and there’s actually even more to it than that. So, no, I’m not impressed by arbitrary claims like that. It’s such a bold claim. If it were true, Apple and Google would be adopting this ASAN implementation ASAP since they have tons of C and C++ code that would benefit from better safety at marginal performance cost. In the mean time, if Redbean and Cosmopolitan want Rust-like promises of memory safety… they need to write it in Rust.
        jart 4y ago
        Uhm you realize Google invented ASAN? It's how they hunt for memory bugs in software like Chrome. If you love Rust, then we're actively working to port Rust to Cosmopolitan. https://github.com/ahgamut/rust-ape-example https://github.com/ahgamut/rust-ape-example Rust in practice has `unsafe` code which needs something like ASAN to make it safer. In order to do that, you need a C library that has first-class support for ASAN where it isn't just an afterthought. I believe that in the future, Cosmopolitan Libc is going to help Rust be even more safe going forward.
  - jaillmanner 4y ago
- excitom 4y ago
  Wow, that's some unsubstantiated mud slinging. On the paid support point: Isn't that the model for all open source projects that want to progress past the "If you like it, please donate" stage?
  - jaillmanner 4y ago
    - KronisLV 4y ago
      > I understand why it’s hard to convince the rabid nginx greybeards that it isn’t “the shit” anymore. I'm not sure about that. Nginx still feels like one of the more popular options and is generally a good fit for many of the problems that it tries to solve, at least in the deployments that I've seen it used in (reverse proxy, serving static assets, connected to PHP-FPM sometimes). It also is the de factor Kubernetes ingress controller and other options like Traefik still haven't quite taken off in that space, at least to that degree. For example, the K3s distro has Traefik as the default ingress, but it's not quite as well documented (e.g. setting up your own custom wildcard SSL certificate). Then again, much of the same criticism was geared towards Apache back in the day (with some of which I agree) and Nginx also has certain kinks to it: https://blog.kronis.dev/everything%20is%20broken/nginx-configuration-is-broken https://blog.kronis.dev/everything%20is%20broken/nginx-confi... Cannot really comment on the paid version, though. I can understand why it would be a thing, but it's also a bit concerning, as it usually is with open core and similar products.
- Seirdy 4y ago
  I generally recommend Caddy over Nginx, but Nginx does still have certain advantages: - Nginx supports OpenSSL commands that enable features like TLS record padding. - Performance: better latency and scalability to more connections. Not everyone uses a CDN for static/cached content - Kernel-accelerated TLS offload on Linux and FreeBSD - Many existing modules provide unique functionality. The many modules for live video streaming and image processing are good examples. - An ecosystem of patches with features like HPACK static dictionaries, dynamic TLS record sizing, etce > …has terrible language integration. Generally, "language integration" isn't really a use-case for vanilla Nginx; it's a use-case for Nginx Unit, an Nginx language-specific module, or OpenResty. I personally prefer the reverse-proxy route since it lets me use whatever language I want regardless of server support: Go, Rust, Python, C, etc. If none of these are that important then I absolutely would not recommend Nginx; Caddy would be the better tool. > People aren't writing internet scale software in lua for a reason. I'd include Itch.io, much of Taobao, and some of the most popular API gateways (including Kong) in the category of "Internet-scale software written by 'people'". POSSE (Publish on Own Site, Syndicate Elsewhere) note from https://seirdy.one/notes/2022/09/09/reasons-to-use-nginx/ https://seirdy.one/notes/2022/09/09/reasons-to-use-nginx/
  - mholt 4y ago
    While I'm glad you recommend Caddy in general, it's worth noting that those advantages come with costs though, too: - Padding: (I'm pretty sure Go already does this too: https://go.dev/src/crypto/tls/conn.go https://go.dev/src/crypto/tls/conn.go) - Performance: Requires nuanced tuning. Caddy performs competitively well for real world usage - kTLS: Sacrifices memory safety. - Existing modules: How do they perform compared to natively-compiled code? Caddy modules can do all that nginx modules can do, and more, but are natively compiled. I ran experiments with Caddy+Starlark that performed 2x as fast as Nginx+Lua.
    - Seirdy 4y ago
      I recommend users who link against OpenSSL to enable padding to multiples of at least 1024 bytes if they want to impede traffic analysis. The Nginx devs aren't interested in implementing random record padding or supporting the feature in BoringSSL/LibreSSL, unfortunately. Can Caddy leverage either form of padding? If so, I might need to give it another look! And regarding modules: most are written in C and dynamically loaded as shared objects or statically linked during compile-time. A bunch are listed at https://www.nginx.com/resources/wiki/modules/ https://www.nginx.com/resources/wiki/modules/. The ones for live streaming and VODs are the hardest to replace, IMO. IPScrub was my favorite but I haven't used it for a few years. Personally, I think live streaming and ffmpeg-based encoding are specialized enough to warrant a specialized server (like a custom Nginx build) and are a bit out of scope for a general-purpose user-friendly server like Caddy.
tmpz22 4y ago
What would decent metrics be to compare the performance of Nginx and Caddy?
- francislavoie 4y ago
  It's not really possible to compare the performance of webservers apples to apples. Really, you need to do your own benchmarking to determine which solution is best for you. But keep in mind that the web server is rarely the bottleneck, usually your app and database IO are where it takes the most time.
  - pachico 4y ago
    Indeed, this is why those hello world benchmarks are just noise.
  - password4321 4y ago
    What is this I'm reading here, the voice of reason?! The last frontier for these types of comparisons is unusual circumstances like embedded platforms with limited resources.
  - KronisLV 4y ago
    > It's not really possible to compare the performance of webservers apples to apples. I'd say that it's possible, just not always easy, due to how many different configurations there can be for any given workload - the same servers might be used to achieve the approximately same configuration in different ways. However, if you took a common enough use case or workload, such as reverse proxy with SSL and gzip compression for $FOO resource types, serving files for $BAR application and proxying $BAZ API, then it should definitely be possible. You would just need some meaningful real-world test instead of a purely synthetic benchmark, otherwise you'll probably test something slightly different than what your server will be doing in practice. The other option is just to settle on some bit of common enough functionality and try to increase your sample size as much as possible. For example, some attempts have been made by OpenBenchmarking and at least give a vague idea of the performance of some servers: Nginx: https://openbenchmarking.org/test/pts/nginx https://openbenchmarking.org/test/pts/nginx Apache:https://openbenchmarking.org/test/pts/apache https://openbenchmarking.org/test/pts/apache I'm mentioning this because while not everybody has the time to reproduce a given setup in any number of technologies, even similar enough tests in a controlled environment can produce meaningful information, at least to let you infer what orders of magnitude you're working with. For something concerning programming languages themselves and their frameworks, one just needs to look at what TechEmpower is trying to do: https://www.techempower.com/benchmarks/#section=data-r21 https://www.techempower.com/benchmarks/#section=data-r21 > Really, you need to do your own benchmarking to determine which solution is best for you. But keep in mind that the web server is rarely the bottleneck, usually your app and database IO are where it takes the most time. This is well said, though. In general, I'm inclined to agree: most of the time, the performance of most web servers can be described as "good enough". The exception to this might be using your application servers (e.g. Tomcat) as a web server and running into situations where serving static assets (that might be baked into your application) would slow down because of API calls being slower and processing them digging into comparatively more conservative HTTP thread limits for the whole thing. Then again, personally I'd argue that you should have one of the popular web servers in front of your applications (and typically serving static assets) to act as an ingress in most cases, but I've seen some interesting things over the years.

nginxlegacy 4y ago

xet7 4y ago
Aren't? But there is https://redbean.dev https://redbean.dev that uses Lua, and is faster than Go.
- coder543 4y ago
  Redbean is a webserver. Go is not a webserver. Redbean cannot be "faster than Go". That's like saying that a car is faster than aluminum. Do you have benchmarks of Redbean versus Caddy that you can point to? I certainly haven't been able to find any. I also haven't been able to find any in-depth independent benchmarks of Redbean. About the only thing I could find was someone struggling to reproduce the benchmark results.[0] Personally, I've dealt with more than enough memory safety vulnerabilities caused by no programmer being good enough to write safe and correct C (or C++). Those vulnerabilities are easily prevented by choosing just about any other language people are likely to know about... so, I have no desire to deploy anything like Redbean when alternatives exist. It's cool to see the hacks they've been able to do with Cosmopolitan to make cross platform binaries, but hacks like that are not what I want for anything other than educational purposes, at least not until they've been more battle tested. [0]: https://github.com/jart/cosmopolitan/issues/72 https://github.com/jart/cosmopolitan/issues/72
  - mholt 4y ago
    Great points. I also think it's important to stop deploying C to the edge. PS. Go might someday compile for cosmopolitan if someone writes a port for it: https://github.com/golang/go/issues/51900 https://github.com/golang/go/issues/51900
  - xet7 4y ago
    I did see that benchmark result previously but did not find that page yet with quick look. From https://redbean.dev https://redbean.dev is this: "If you want Rust-like promises then redbean can be compiled with ASAN memory safety. Cosmopolitan Libc has the only open source implementation of the Address Sanitizer runtime that's intended for production use. It causes redbean worker processes to crash and log a report should a bug like a buffer overrun, use-after-free, or stack smash occur. This has a marginal impact on performance. It can be useful in environments where interruptions in a service are more desirable than risking the system being compromised." More at https://redbean.dev/#security https://redbean.dev/#security
    - coder543 4y ago
      ASAN by itself cannot make those guarantees, from what I understand. Cosmopolitan would also have to make sure never to reuse virtual memory addresses for a start, and there’s actually even more to it than that. So, no, I’m not impressed by arbitrary claims like that. It’s such a bold claim. If it were true, Apple and Google would be adopting this ASAN implementation ASAP since they have tons of C and C++ code that would benefit from better safety at marginal performance cost. In the mean time, if Redbean and Cosmopolitan want Rust-like promises of memory safety… they need to write it in Rust.
      - jart 4y ago
        Uhm you realize Google invented ASAN? It's how they hunt for memory bugs in software like Chrome. If you love Rust, then we're actively working to port Rust to Cosmopolitan. https://github.com/ahgamut/rust-ape-example https://github.com/ahgamut/rust-ape-example Rust in practice has `unsafe` code which needs something like ASAN to make it safer. In order to do that, you need a C library that has first-class support for ASAN where it isn't just an afterthought. I believe that in the future, Cosmopolitan Libc is going to help Rust be even more safe going forward.
- jaillmanner 4y ago
excitom 4y ago
Wow, that's some unsubstantiated mud slinging. On the paid support point: Isn't that the model for all open source projects that want to progress past the "If you like it, please donate" stage?
- jaillmanner 4y ago
  - KronisLV 4y ago
    > I understand why it’s hard to convince the rabid nginx greybeards that it isn’t “the shit” anymore. I'm not sure about that. Nginx still feels like one of the more popular options and is generally a good fit for many of the problems that it tries to solve, at least in the deployments that I've seen it used in (reverse proxy, serving static assets, connected to PHP-FPM sometimes). It also is the de factor Kubernetes ingress controller and other options like Traefik still haven't quite taken off in that space, at least to that degree. For example, the K3s distro has Traefik as the default ingress, but it's not quite as well documented (e.g. setting up your own custom wildcard SSL certificate). Then again, much of the same criticism was geared towards Apache back in the day (with some of which I agree) and Nginx also has certain kinks to it: https://blog.kronis.dev/everything%20is%20broken/nginx-configuration-is-broken https://blog.kronis.dev/everything%20is%20broken/nginx-confi... Cannot really comment on the paid version, though. I can understand why it would be a thing, but it's also a bit concerning, as it usually is with open core and similar products.
Seirdy 4y ago
I generally recommend Caddy over Nginx, but Nginx does still have certain advantages: - Nginx supports OpenSSL commands that enable features like TLS record padding. - Performance: better latency and scalability to more connections. Not everyone uses a CDN for static/cached content - Kernel-accelerated TLS offload on Linux and FreeBSD - Many existing modules provide unique functionality. The many modules for live video streaming and image processing are good examples. - An ecosystem of patches with features like HPACK static dictionaries, dynamic TLS record sizing, etce > …has terrible language integration. Generally, "language integration" isn't really a use-case for vanilla Nginx; it's a use-case for Nginx Unit, an Nginx language-specific module, or OpenResty. I personally prefer the reverse-proxy route since it lets me use whatever language I want regardless of server support: Go, Rust, Python, C, etc. If none of these are that important then I absolutely would not recommend Nginx; Caddy would be the better tool. > People aren't writing internet scale software in lua for a reason. I'd include Itch.io, much of Taobao, and some of the most popular API gateways (including Kong) in the category of "Internet-scale software written by 'people'". POSSE (Publish on Own Site, Syndicate Elsewhere) note from https://seirdy.one/notes/2022/09/09/reasons-to-use-nginx/ https://seirdy.one/notes/2022/09/09/reasons-to-use-nginx/
- mholt 4y ago
  While I'm glad you recommend Caddy in general, it's worth noting that those advantages come with costs though, too: - Padding: (I'm pretty sure Go already does this too: https://go.dev/src/crypto/tls/conn.go https://go.dev/src/crypto/tls/conn.go) - Performance: Requires nuanced tuning. Caddy performs competitively well for real world usage - kTLS: Sacrifices memory safety. - Existing modules: How do they perform compared to natively-compiled code? Caddy modules can do all that nginx modules can do, and more, but are natively compiled. I ran experiments with Caddy+Starlark that performed 2x as fast as Nginx+Lua.
  - Seirdy 4y ago
    I recommend users who link against OpenSSL to enable padding to multiples of at least 1024 bytes if they want to impede traffic analysis. The Nginx devs aren't interested in implementing random record padding or supporting the feature in BoringSSL/LibreSSL, unfortunately. Can Caddy leverage either form of padding? If so, I might need to give it another look! And regarding modules: most are written in C and dynamically loaded as shared objects or statically linked during compile-time. A bunch are listed at https://www.nginx.com/resources/wiki/modules/ https://www.nginx.com/resources/wiki/modules/. The ones for live streaming and VODs are the hardest to replace, IMO. IPScrub was my favorite but I haven't used it for a few years. Personally, I think live streaming and ffmpeg-based encoding are specialized enough to warrant a specialized server (like a custom Nginx build) and are a bit out of scope for a general-purpose user-friendly server like Caddy.

tmpz22 4y ago

What would decent metrics be to compare the performance of Nginx and Caddy?

francislavoie 4y ago
It's not really possible to compare the performance of webservers apples to apples. Really, you need to do your own benchmarking to determine which solution is best for you. But keep in mind that the web server is rarely the bottleneck, usually your app and database IO are where it takes the most time.
- pachico 4y ago
  Indeed, this is why those hello world benchmarks are just noise.
- password4321 4y ago
  What is this I'm reading here, the voice of reason?! The last frontier for these types of comparisons is unusual circumstances like embedded platforms with limited resources.
- KronisLV 4y ago
  > It's not really possible to compare the performance of webservers apples to apples. I'd say that it's possible, just not always easy, due to how many different configurations there can be for any given workload - the same servers might be used to achieve the approximately same configuration in different ways. However, if you took a common enough use case or workload, such as reverse proxy with SSL and gzip compression for $FOO resource types, serving files for $BAR application and proxying $BAZ API, then it should definitely be possible. You would just need some meaningful real-world test instead of a purely synthetic benchmark, otherwise you'll probably test something slightly different than what your server will be doing in practice. The other option is just to settle on some bit of common enough functionality and try to increase your sample size as much as possible. For example, some attempts have been made by OpenBenchmarking and at least give a vague idea of the performance of some servers: Nginx: https://openbenchmarking.org/test/pts/nginx https://openbenchmarking.org/test/pts/nginx Apache:https://openbenchmarking.org/test/pts/apache https://openbenchmarking.org/test/pts/apache I'm mentioning this because while not everybody has the time to reproduce a given setup in any number of technologies, even similar enough tests in a controlled environment can produce meaningful information, at least to let you infer what orders of magnitude you're working with. For something concerning programming languages themselves and their frameworks, one just needs to look at what TechEmpower is trying to do: https://www.techempower.com/benchmarks/#section=data-r21 https://www.techempower.com/benchmarks/#section=data-r21 > Really, you need to do your own benchmarking to determine which solution is best for you. But keep in mind that the web server is rarely the bottleneck, usually your app and database IO are where it takes the most time. This is well said, though. In general, I'm inclined to agree: most of the time, the performance of most web servers can be described as "good enough". The exception to this might be using your application servers (e.g. Tomcat) as a web server and running into situations where serving static assets (that might be baked into your application) would slow down because of API calls being slower and processing them digging into comparatively more conservative HTTP thread limits for the whole thing. Then again, personally I'd argue that you should have one of the popular web servers in front of your applications (and typically serving static assets) to act as an ingress in most cases, but I've seen some interesting things over the years.