9 ms·

I used nginx for a very, very long time but abandoned it. I switched to Caddy because I just got so sick of the constant pain of certificates. Caddy does cert

I used nginx for a very, very long time but abandoned it. I switched to Caddy because I just got so sick of the constant pain of certificates. Caddy does certificates automatically. I also was sick of the complexity of nginx configuration. I could get the things done that I needed to do but it was always painful and time consuming. I've worked out how to do in Caddy everything that I did in nginx though that took alot of time and effort, but I tend to create quite sophisticated web server configs. At this stage there's nothing really that nginx could offer that would bring me back from Caddy.

steve_taylor 4y ago
I’m the same. I’m able to host unlimited subdomains using wildcard TLS and unlimited apex domains using TLS on-demand, all from one Caddy config. It’s amazing. No disrespect to the good people behind nginx. Caddy is simply part of a new generation of web servers.
- mrweasel 4y ago
  > Caddy is simply part of a new generation of web servers. I feel old now, as I continue to deploy Apache :-)
  - 1letterunixname 4y ago
    It's cool as long as you compile in mod_perl, use --with-mpm=prefork, set LC_ALL=C, and use /bin/csh. ;P
  - lsofzz 4y ago
    > I feel old now, as I continue to deploy Apache :-) we wrote bash scripts backends. don't worry about being old. keep going ;) some even had their cgi code running in kernel.. all the more reason to be merrier.
vbezhenar 4y ago
One pain point with caddy for me was cloudflare dns validation. I had to rebuild caddy from scratch, build my own container image. This process is slow as hell and very RAM-intensive, I had to add swap just for it. I wish they publish images with popular plugins built-in.
- deleted 4y ago
  [deleted]
- andrewstuart 4y ago
  They do provide pre built stuff. It's just really unclear on their website that this is exactly what they do. I had the same problem as you. Here is me whining about it on the Caddy frum: https://caddy.community/t/wildcard-certificates-building-from-scratch-and-what-caddy-is-all-about/15793 https://caddy.community/t/wildcard-certificates-building-fro... But in the end I managed to get the pre built stuff. Their download section is extremely unclear about whether or not you need to build Caddy. You don't.
  - krets 4y ago
    With this docker image you can just download the binary and add it as a volume! https://hotio.dev/containers/caddy/ https://hotio.dev/containers/caddy/
- hantusk 4y ago
  Hm, i found this to be pretty straight forward through Docker: e.g: FROM caddy:2-builder AS builder RUN xcaddy build \ --with github.com/greenpau/caddy-security FROM caddy:2 COPY --from=builder /usr/bin/caddy /usr/bin/caddy You can keep your container image, and just do the "COPY --from=builder" step in there. We do this in CI/CD on every build.
  - vbezhenar 4y ago
    I did something similar. It puts my server to knees and takes something like a hour to complete, requiring all services to stop before I can run it. This is extremely unergonomic for me.
muhehe 4y ago
Would you mind to elaborate a bit more on the sophisticated configs?
- andrewstuart 4y ago
  automatic certs multiple back end proxies authorization sub requests for multiple domains CORS and other stuff I can't recall offhand
1letterunixname 4y ago
Chef + reliable CAs other than Let's Encrypt that have an API = the end. Invest in: 0. Automation 1. Documentation 2. Take away SSH 3. Except for data stores, ephemeralize servers so they can be blown away and automatically reprovisioned 4. Have tested backups (untested aren't backups) 5. Save time 6. Solve other, bigger problems
- vladvasiliu 4y ago
  > reliable CAs other than Let's Encrypt that have an API Would you care to give an example or two?
  - TonyTrapp 4y ago
    Let's Encrypt's ACME protocol has been adopted by ZeroSSL, Buypass and ssl.com to offer free 90-day certificates.
    - 1letterunixname 4y ago
      Free CAs are prone to fraud and abuse because they have no innate funding model and there's no barrier to anyone issuing a cert to any particular domain while attempting to circumvent domain authentication.
      - TonyTrapp 4y ago
        > there's no barrier to anyone issuing a cert to any particular domain while attempting to circumvent domain authentication. Let's Encrypt has a much better track record at preventing people from issuing certs for random domains compared to many other commercial entities.
AceJohnny2 4y ago
> I also was sick of the complexity of nginx configuration I'm old enough to remember the same complaints about Apache, and nginx being the simpler alternative. Something about systems evolving to a level of complexity/sophistication until they collapse under its own weight...
- remus 4y ago
  I wonder if there's something about changing uses cases. If you design for x and y use cases then you end up with a nice, simple design for x and y. People then start doing a lot of z so you adapt to support z, but it adds some complexity because it's not exactly what you had in mind when you laid the foundations. A competitor comes along who's designed for y + z, and everyone loves how nice and simple it is. Then use case u comes along...
  - ideamotor 4y ago
    Nah, probably use case x. Rinse repeat.
  - iib 4y ago
    If we are to trust a certain philosophy based on simplicity, one of the _very hard_ things you have to do as a head maintainer of software is to have the ability to say no when that case arises.
    - edmundsauto 4y ago
      That’s the trick - sometimes, if you don’t add the features, you lose share because the market has shifted and now a different set of features are what is valued. So you can either patch and lose focus, or not patch and lose customers.
- resonious 4y ago
  This seems very much like the browser/frontend space, where Webpack was once cool and slim, but is now just as bloated and slow as the systems it set out to replace.
Goz3rr 4y ago
> I just got so sick of the constant pain of certificates. I've honestly never had any sort of "constant pain" by using nginx and acme.sh. Certbot is an actual abomination. I have a script just as a shortcut for these two commands: acme.sh --issue --dns dns_cf -d "$1" -d "*.$1" acme.sh --installcert -d "$1" -d "*.$1" --certpath /foo/$1/cert --keypath /foo/$1/key --fullchainpath /foo/$1/fullchain --reloadcmd "/usr/local/bin/docker-compose -f /foo/docker-compose.yml exec -T nginx nginx -s reload" You just have to run those commands once per domain and it'll keep that wildcard certificate valid forever, acme.sh sets up a cronjob to renew the cert when needed and will automatically reload my nginx container after.
- simongray 4y ago
  And if you use caddy you never need to know what any of that means, since caddy completely automates it. Maybe it's not a pain to you since you're used to it, but it's a major obstacle to most people looking to set up a website, particularly for the first time. Even though I've set up nginx and certbot before, I'm happy I don't need to think about that stuff with caddy. Total waste of mental resources. I just want to get stuff on the Web.
  - jamespo 4y ago
    I want more control over the cert request however, and to use the certs for SMTPS, IMAPS etc. But for simple HTTPS only caddy is great.
blueflow 4y ago
> At this stage there's nothing really that nginx could offer that would bring me back from Caddy. Performance?
- Snoddas 4y ago
  My guess is that a massive majority of websites will never notice any performance difference.
  - blueflow 4y ago
    Its noticeable whether you need 1 server or 3 servers for the same workload.
    - francislavoie 4y ago
      It's quite rare that your webserver will be the bottleneck at this point. Your app will be the bottleneck much more often. I don't think the difference in performance between Nginx and Caddy would mean you need 3 servers versus 1 of the same spec. But of course, you need to run your own benchmarks on your own config to determine which is best _for you_. Benchmarks cannot be done generally, because the config drives so much of what happens at runtime, and everyone has different needs.
      - CRConrad 4y ago
        Hint: *asterisks* in stead of _underscores_ --> italics.
        francislavoie 4y ago
        I guess I'm too used to markdown :) thanks
KronisLV 4y ago
> I used nginx for a very, very long time but abandoned it. I still think that Nginx is pretty good, however there are a few annoying things about it, certificates being just one of them. Personally, I also found that attempting to use it as a reverse proxy kills the entire instance when there is no DNS record for one of the sites (say, 1 out of 20 that are proxied). I ran into this when running containers that hadn't passed health checks and therefore didn't have any traffic routed to them, which meant that if any of them went down, all of them would be unreachable. Furthermore, the popular suggestion of using a variable for the proxy URL just broke redirects in some apps: https://blog.kronis.dev/everything%20is%20broken/nginx-configuration-is-broken https://blog.kronis.dev/everything%20is%20broken/nginx-confi... Caddy is pretty good, though I couldn't get the equivalents of all the options that are available in other web servers, for example allowing encoded slashes when hosting a Sonatype Nexus instance like what you can do with Apache. Things might have changed, but last I tried, it didn't quite seem to work: https://help.sonatype.com/repomanager3/planning-your-implementation/run-behind-a-reverse-proxy#RunBehindaReverseProxy-ApachehttpdwithnpmRepositories https://help.sonatype.com/repomanager3/planning-your-impleme... Furthermore, there were issues with certificates as well: if the configuration for one of the aforementioned 20 sites was bad and the certificate couldn't be renewed, then the entire instance went down. What does this mean? Well, if I run Caddy as a container, I'll get the "fail fast" approach, which will sadly then mean that I'll get a restart, that will still fail with the bad configuration and eventually will hit Let's Encrypt rate limits. Sure, I can have some alerting (extra work) or have restart backoff/delays (though this will be bad for restart times if the instance were to ever crash for other reasons, e.g. load), but neither seems like an actual solution. Very annoying, especially because everything will once again go down, instead of being built for resiliency. In the end, I kind of just went with Apache for the time being, because despite being a bit awkward at times, it's still an okay web server for the scales that I work at and is okay to deploy inside of containers. I wrote about it more on my blog, "How and why to use Apache httpd in 2022": https://blog.kronis.dev/tutorials/how-and-why-to-use-apache-httpd-in-2022 https://blog.kronis.dev/tutorials/how-and-why-to-use-apache-... It even has Let's Encrypt (well, ACME) support built in, thanks to mod_md nowadays: https://httpd.apache.org/docs/trunk/mod/mod_md.html https://httpd.apache.org/docs/trunk/mod/mod_md.html Of course, I still use Nginx + PHP-FPM with supervisord for containers where I need to run PHP, because of some other weirdness when you try to get it working with Apache, which probably has something to do with how I build container images. That said, I couldn't actually find what the problem was, so it was easier for me to just use Apache on the edge and Nginx for the apps (PHP, or maybe even serving static assets), about which I wrote on yet another blog post, "Containers are broken": https://blog.kronis.dev/everything%20is%20broken/containers-are-broken https://blog.kronis.dev/everything%20is%20broken/containers-... In short, it's nice to have a choice of web servers and being able to pick whatever is the best suited for your needs. It's just that there's a lot of weirdness going on with what should be simple configurations.
1vuio0pswjnm7 4y ago
Anyone else using Caddy as a forward proxy. https://github.com/caddyserver/forwardproxy https://github.com/caddyserver/forwardproxy
gsich 4y ago
Certs? Dehydrated
- jeltz 4y ago
  Yeah, I feel certbot is a pain to use for anything which is not a toy. Dehydrated on the other hand works well in real setups.
ravenstine 4y ago
The configuration format is easily the worst aspect of nginx. There was a stretch where I got heavily into using it because it really is capable of quite a lot, but I could never totally grasp the config, despite how I've managed to learn pretty much any programming language given to me. Depending on what you're doing, a certain other thing may be totally moot or out of place, but it's certainly not obvious, and good luck actually debugging your nginx config without resorting to The Google. Once the config is set, nginx can do its job quite well, but it's bad if you're frequently making architectural changes.
- minusf 4y ago
  every configuration format has it's quirks, and nginx certainly has a couple. but coming from apache it was the freshest breath of air possible. i still prefer it to any yaml/json alternatives newer products like envoy use. a pure ymmv point. i don't think that having a manual or reference open while doing more involved setups is a sign of bad design. some things are simply complicated...
- Avamander 4y ago
  That's the thing, it's not a programming language. It's a sequence of declarations. In-spirit very similar to iptables or nftables rules for example. Documenting things that took time for you to grasp is a great way to learn, in my experience.
  - ravenstine 4y ago
    I wasn't trying to equate it to a programming language, though. IMO, a config format shouldn't be more difficult to grasp than a Turing-complete language. To me, nginx conf is hard to gain proficiency in because the amount of experience to know whether a declaration is in the appropriate place or won't be ignored is very high. Obviously someone can indeed become proficient, but I've found it to be extraordinarily high for someone who just needs it occasionally; I really don't want to live in nginx land all the time, in part because I'm a software engineer and not really an ops person. For instance, should `upstream` be defined at the top-level or under `stream`? If you have another server doing a proxy_pass to that upstream, does it also need to be under `stream`? This is in no way obvious by either looking at nginx configs or by reading the docs. And then what if you have a web server? Does it need to be under `http`, or can it be top-level? It can be either one depending on who's config you're looking at. But if your server you're pointing to is already an HTTP server, why would you want nginx to use its own http serving on top of your HTTP server? That's just a couple of examples. And yes, there are absolutely logical answers to those, but I do not believe that the nginx config format speaks for itself.
jeltz 4y ago
Certificates are only annoying if you use certbot, which has terrible UX, virtually any other tool makes it trivial, e.g. Dehydrated or acme.sh.
boesboes 4y ago
heh, I'm on the exact opposite side of the fence. Tried caddy, found it confusing and incomplete so I went back to nginx. Buuut then again, I've been writing nginx config for 10-15 years :)
- francislavoie 4y ago
  Wouldn't mind hearing what you found confusing, so we can try to close any gaps
Avamander 4y ago
I'm using all three, Apache, Caddy and nginx almost daily. For the trivial, Caddy is amazing. When things get more complex alongside needs themselves, nginx is amazingly powerful. Apache at the same point becomes overly verbose, anomalous and painful. It does take some discipline though, to comment and structure your configuration files, but that applies to anything that isn't trivial. return 444; alone is a great feature I can't do without, not to mention all the modules one can use. The thing with certificates is that most tutorials are terrible. A very seamless way is to use the webroot method. Create a nginx configuration file that places LE's challenges to a specified folder on your system, include that config into your server blocks. Then you instruct certbot to use that folder. Really quite easy.
- TheRealDunkirk 4y ago
  > Really quite easy. “Your ideas are intriguing to me, and I wish to subscribe to your newsletter.” Is there a detailed writeup on this? This sounds great, but I don't want to faff about with it.
  - Avamander 4y ago
    In its most simple form, it works as follows. /etc/nginx/letsencrypt.conf: location /.well-known/acme-challenge { default_type "text/plain"; try_files $uri $uri/ =404; root /tmp/letsencrypt-auto; } /etc/nginx/sites-enabled/default: server { [...] include letsencrypt.conf; [...] } Reload nginx. Invoke certbot for the new domain once with: certbot certonly --webroot -w /tmp/letsencrypt-auto/ [--must-staple] -d example.com
    - braiam 4y ago
      Wait, how does that makes nginx use the certificates? For me it doesn't have any cerbot/letsencrytpt specific configuration as I use DNS-01. I just point ssl_certificate and ssl_certificate_key to the correct path where cerbot saves the certificates and be done with it.
heliodor 4y ago
Nginx bites you in the a**: you start your project, choose the most famous web server (nginx), then you build and build and start needing the pro features. But they don't give you an up-front price! You have to contact them for a custom quote. That's when you start pondering moving to Caddy. But you're too far down the rabbit hole. Luckily, Cloudflare does a decent job of providing the features you need, at a clear price.
excitom 4y ago
nginx config is only complicated if you want to take advantage of complex options. The config for a basic website is trivial. Seems like a poor tradeoff for a slower, less configurable web server.
- simon04 4y ago
  Unless you want a good HTTPS configuration. The output of https://ssl-config.mozilla.org/#server=nginx https://ssl-config.mozilla.org/#server=nginx is significantly longer than the one for caddy.
  - francislavoie 4y ago
    FYI that website is outdated and unmaintained. The person who worked on it left Mozilla AFAIK (would love to be proven wrong). See https://twitter.com/CubicleApril/status/1438940565916225545 https://twitter.com/CubicleApril/status/1438940565916225545 The correct thing with Caddy is to _not configure anything at all_ for TLS. The defaults are the correct thing to use. If you override the defaults, then you're more at risk of bitrot due to not remembering to update your own config. Let Caddy (and the Go stdlib, really) choose what's secure.
Abishek_Muthian 4y ago
It's been 5 years since I used any web server as I started using Go to build web applications. 1. HTTP server is production ready. 2. autocert fetches certificate automatically from LetsEncrypt. 3. Graceful restarts (zero downtime) using signals. I think the the biggest advantage of languages which enable portable applications like Go is not having to mess with web servers anymore, At least for most common use cases.
Sin2x 4y ago
Nginx is still faster and that's what matters for highload. If you are not using the server for highload, no need for Nginx.
the_third_wave 4y ago
> I switched to Caddy because I just got so sick of the constant pain of certificates. Caddy does certificates automatically That may be fine if you only use those certificates for port 443 but it falters where certificates are used for the remaining 65533 ports. Instead of leaving certificate request and renewal to individual tools I prefer to centralise it in one spot - a small container or VM is sufficient - which deals out ready-to-run certificates to those systems which need them. When dealt with this way nginx actually has no problems whatsoever with certificates, all it takes is those three (or more depending on your protocol needs) lines in the config file: listen 443 ssl http2; ssl_certificate /etc/letsencrypt/live/site.example.org/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/site.example.org/privkey.pem; The /etc/letsencrypt/live tree gets pushed to the relevant machines by the certificate handling container/VM with certificate files in the form and shape needed by whatever program is to use them. So, in short, the Caddy approach may be fine for those who only run web things but it 'fails' (as in 'adds additional complications') for those who venture beyond port 443.
- francislavoie 4y ago
  That's not true at all. Caddy can automate certificate issuance with any of the 3 challenge types, either HTTP-01 (requires port 80) TLS-ALPN-01 (requires port 443) or DNS-01 (requires building Caddy with a DNS plugin for your DNS provider). Also you can cluster Caddy by making sure the filesystems are synced, or using a storage module like Redis. Then any of the Caddy instances can do any step of the issuance phase (one can start it, another can complete it). You can use these certificates for sites on non-443 ports as well, once you have the cert. Nothing special at all to do there, you just configure Caddy to serve a site with a different port, e.g. `example.com:65533` You also get numerous other benefits, like OCSP stapling, issuer fallback (if LE is down, it'll try ZeroSSL), etc.
  - the_third_wave 4y ago
    You don't seem to understand what I mean when I say 'port 443' - i.e. HTTPS. I'm talking about the use of certificates in non-web applications, e.g. SMTPS, IMAPS, database servers, etc. I prefer not to run a web server on my database servers, mail servers and what have you.

steve_taylor 4y ago

I’m the same. I’m able to host unlimited subdomains using wildcard TLS and unlimited apex domains using TLS on-demand, all from one Caddy config. It’s amazing. No disrespect to the good people behind nginx. Caddy is simply part of a new generation of web servers.

mrweasel 4y ago
> Caddy is simply part of a new generation of web servers. I feel old now, as I continue to deploy Apache :-)
- 1letterunixname 4y ago
  It's cool as long as you compile in mod_perl, use --with-mpm=prefork, set LC_ALL=C, and use /bin/csh. ;P
- lsofzz 4y ago
  > I feel old now, as I continue to deploy Apache :-) we wrote bash scripts backends. don't worry about being old. keep going ;) some even had their cgi code running in kernel.. all the more reason to be merrier.

vbezhenar 4y ago

One pain point with caddy for me was cloudflare dns validation. I had to rebuild caddy from scratch, build my own container image. This process is slow as hell and very RAM-intensive, I had to add swap just for it. I wish they publish images with popular plugins built-in.

deleted 4y ago
[deleted]
andrewstuart 4y ago
They do provide pre built stuff. It's just really unclear on their website that this is exactly what they do. I had the same problem as you. Here is me whining about it on the Caddy frum: https://caddy.community/t/wildcard-certificates-building-from-scratch-and-what-caddy-is-all-about/15793 https://caddy.community/t/wildcard-certificates-building-fro... But in the end I managed to get the pre built stuff. Their download section is extremely unclear about whether or not you need to build Caddy. You don't.
- krets 4y ago
  With this docker image you can just download the binary and add it as a volume! https://hotio.dev/containers/caddy/ https://hotio.dev/containers/caddy/
hantusk 4y ago
Hm, i found this to be pretty straight forward through Docker: e.g: FROM caddy:2-builder AS builder RUN xcaddy build \ --with github.com/greenpau/caddy-security FROM caddy:2 COPY --from=builder /usr/bin/caddy /usr/bin/caddy You can keep your container image, and just do the "COPY --from=builder" step in there. We do this in CI/CD on every build.
- vbezhenar 4y ago
  I did something similar. It puts my server to knees and takes something like a hour to complete, requiring all services to stop before I can run it. This is extremely unergonomic for me.

muhehe 4y ago

Would you mind to elaborate a bit more on the sophisticated configs?

andrewstuart 4y ago
automatic certs multiple back end proxies authorization sub requests for multiple domains CORS and other stuff I can't recall offhand

1letterunixname 4y ago

Chef + reliable CAs other than Let's Encrypt that have an API = the end. Invest in: 0. Automation 1. Documentation 2. Take away SSH 3. Except for data stores, ephemeralize servers so they can be blown away and automatically reprovisioned 4. Have tested backups (untested aren't backups) 5. Save time 6. Solve other, bigger problems

vladvasiliu 4y ago
> reliable CAs other than Let's Encrypt that have an API Would you care to give an example or two?
- TonyTrapp 4y ago
  Let's Encrypt's ACME protocol has been adopted by ZeroSSL, Buypass and ssl.com to offer free 90-day certificates.
  - 1letterunixname 4y ago
    Free CAs are prone to fraud and abuse because they have no innate funding model and there's no barrier to anyone issuing a cert to any particular domain while attempting to circumvent domain authentication.
    - TonyTrapp 4y ago
      > there's no barrier to anyone issuing a cert to any particular domain while attempting to circumvent domain authentication. Let's Encrypt has a much better track record at preventing people from issuing certs for random domains compared to many other commercial entities.

AceJohnny2 4y ago

> I also was sick of the complexity of nginx configuration I'm old enough to remember the same complaints about Apache, and nginx being the simpler alternative. Something about systems evolving to a level of complexity/sophistication until they collapse under its own weight...

remus 4y ago
I wonder if there's something about changing uses cases. If you design for x and y use cases then you end up with a nice, simple design for x and y. People then start doing a lot of z so you adapt to support z, but it adds some complexity because it's not exactly what you had in mind when you laid the foundations. A competitor comes along who's designed for y + z, and everyone loves how nice and simple it is. Then use case u comes along...
- ideamotor 4y ago
  Nah, probably use case x. Rinse repeat.
- iib 4y ago
  If we are to trust a certain philosophy based on simplicity, one of the _very hard_ things you have to do as a head maintainer of software is to have the ability to say no when that case arises.
  - edmundsauto 4y ago
    That’s the trick - sometimes, if you don’t add the features, you lose share because the market has shifted and now a different set of features are what is valued. So you can either patch and lose focus, or not patch and lose customers.
resonious 4y ago
This seems very much like the browser/frontend space, where Webpack was once cool and slim, but is now just as bloated and slow as the systems it set out to replace.

Goz3rr 4y ago

> I just got so sick of the constant pain of certificates. I've honestly never had any sort of "constant pain" by using nginx and acme.sh. Certbot is an actual abomination. I have a script just as a shortcut for these two commands: acme.sh --issue --dns dns_cf -d "$1" -d "*.$1" acme.sh --installcert -d "$1" -d "*.$1" --certpath /foo/$1/cert --keypath /foo/$1/key --fullchainpath /foo/$1/fullchain --reloadcmd "/usr/local/bin/docker-compose -f /foo/docker-compose.yml exec -T nginx nginx -s reload" You just have to run those commands once per domain and it'll keep that wildcard certificate valid forever, acme.sh sets up a cronjob to renew the cert when needed and will automatically reload my nginx container after.

simongray 4y ago
And if you use caddy you never need to know what any of that means, since caddy completely automates it. Maybe it's not a pain to you since you're used to it, but it's a major obstacle to most people looking to set up a website, particularly for the first time. Even though I've set up nginx and certbot before, I'm happy I don't need to think about that stuff with caddy. Total waste of mental resources. I just want to get stuff on the Web.
- jamespo 4y ago
  I want more control over the cert request however, and to use the certs for SMTPS, IMAPS etc. But for simple HTTPS only caddy is great.

blueflow 4y ago

> At this stage there's nothing really that nginx could offer that would bring me back from Caddy. Performance?

Snoddas 4y ago
My guess is that a massive majority of websites will never notice any performance difference.
- blueflow 4y ago
  Its noticeable whether you need 1 server or 3 servers for the same workload.
  - francislavoie 4y ago
    It's quite rare that your webserver will be the bottleneck at this point. Your app will be the bottleneck much more often. I don't think the difference in performance between Nginx and Caddy would mean you need 3 servers versus 1 of the same spec. But of course, you need to run your own benchmarks on your own config to determine which is best _for you_. Benchmarks cannot be done generally, because the config drives so much of what happens at runtime, and everyone has different needs.
    - CRConrad 4y ago
      Hint: *asterisks* in stead of _underscores_ --> italics.
      - francislavoie 4y ago
        I guess I'm too used to markdown :) thanks

KronisLV 4y ago

> I used nginx for a very, very long time but abandoned it. I still think that Nginx is pretty good, however there are a few annoying things about it, certificates being just one of them. Personally, I also found that attempting to use it as a reverse proxy kills the entire instance when there is no DNS record for one of the sites (say, 1 out of 20 that are proxied). I ran into this when running containers that hadn't passed health checks and therefore didn't have any traffic routed to them, which meant that if any of them went down, all of them would be unreachable. Furthermore, the popular suggestion of using a variable for the proxy URL just broke redirects in some apps: https://blog.kronis.dev/everything%20is%20broken/nginx-configuration-is-broken https://blog.kronis.dev/everything%20is%20broken/nginx-confi... Caddy is pretty good, though I couldn't get the equivalents of all the options that are available in other web servers, for example allowing encoded slashes when hosting a Sonatype Nexus instance like what you can do with Apache. Things might have changed, but last I tried, it didn't quite seem to work: https://help.sonatype.com/repomanager3/planning-your-implementation/run-behind-a-reverse-proxy#RunBehindaReverseProxy-ApachehttpdwithnpmRepositories https://help.sonatype.com/repomanager3/planning-your-impleme... Furthermore, there were issues with certificates as well: if the configuration for one of the aforementioned 20 sites was bad and the certificate couldn't be renewed, then the entire instance went down. What does this mean? Well, if I run Caddy as a container, I'll get the "fail fast" approach, which will sadly then mean that I'll get a restart, that will still fail with the bad configuration and eventually will hit Let's Encrypt rate limits. Sure, I can have some alerting (extra work) or have restart backoff/delays (though this will be bad for restart times if the instance were to ever crash for other reasons, e.g. load), but neither seems like an actual solution. Very annoying, especially because everything will once again go down, instead of being built for resiliency. In the end, I kind of just went with Apache for the time being, because despite being a bit awkward at times, it's still an okay web server for the scales that I work at and is okay to deploy inside of containers. I wrote about it more on my blog, "How and why to use Apache httpd in 2022": https://blog.kronis.dev/tutorials/how-and-why-to-use-apache-httpd-in-2022 https://blog.kronis.dev/tutorials/how-and-why-to-use-apache-... It even has Let's Encrypt (well, ACME) support built in, thanks to mod_md nowadays: https://httpd.apache.org/docs/trunk/mod/mod_md.html https://httpd.apache.org/docs/trunk/mod/mod_md.html Of course, I still use Nginx + PHP-FPM with supervisord for containers where I need to run PHP, because of some other weirdness when you try to get it working with Apache, which probably has something to do with how I build container images. That said, I couldn't actually find what the problem was, so it was easier for me to just use Apache on the edge and Nginx for the apps (PHP, or maybe even serving static assets), about which I wrote on yet another blog post, "Containers are broken": https://blog.kronis.dev/everything%20is%20broken/containers-are-broken https://blog.kronis.dev/everything%20is%20broken/containers-... In short, it's nice to have a choice of web servers and being able to pick whatever is the best suited for your needs. It's just that there's a lot of weirdness going on with what should be simple configurations.

1vuio0pswjnm7 4y ago

Anyone else using Caddy as a forward proxy. https://github.com/caddyserver/forwardproxy https://github.com/caddyserver/forwardproxy

gsich 4y ago

Certs? Dehydrated

jeltz 4y ago
Yeah, I feel certbot is a pain to use for anything which is not a toy. Dehydrated on the other hand works well in real setups.

ravenstine 4y ago

The configuration format is easily the worst aspect of nginx. There was a stretch where I got heavily into using it because it really is capable of quite a lot, but I could never totally grasp the config, despite how I've managed to learn pretty much any programming language given to me. Depending on what you're doing, a certain other thing may be totally moot or out of place, but it's certainly not obvious, and good luck actually debugging your nginx config without resorting to The Google. Once the config is set, nginx can do its job quite well, but it's bad if you're frequently making architectural changes.

minusf 4y ago
every configuration format has it's quirks, and nginx certainly has a couple. but coming from apache it was the freshest breath of air possible. i still prefer it to any yaml/json alternatives newer products like envoy use. a pure ymmv point. i don't think that having a manual or reference open while doing more involved setups is a sign of bad design. some things are simply complicated...
Avamander 4y ago
That's the thing, it's not a programming language. It's a sequence of declarations. In-spirit very similar to iptables or nftables rules for example. Documenting things that took time for you to grasp is a great way to learn, in my experience.
- ravenstine 4y ago
  I wasn't trying to equate it to a programming language, though. IMO, a config format shouldn't be more difficult to grasp than a Turing-complete language. To me, nginx conf is hard to gain proficiency in because the amount of experience to know whether a declaration is in the appropriate place or won't be ignored is very high. Obviously someone can indeed become proficient, but I've found it to be extraordinarily high for someone who just needs it occasionally; I really don't want to live in nginx land all the time, in part because I'm a software engineer and not really an ops person. For instance, should `upstream` be defined at the top-level or under `stream`? If you have another server doing a proxy_pass to that upstream, does it also need to be under `stream`? This is in no way obvious by either looking at nginx configs or by reading the docs. And then what if you have a web server? Does it need to be under `http`, or can it be top-level? It can be either one depending on who's config you're looking at. But if your server you're pointing to is already an HTTP server, why would you want nginx to use its own http serving on top of your HTTP server? That's just a couple of examples. And yes, there are absolutely logical answers to those, but I do not believe that the nginx config format speaks for itself.

jeltz 4y ago

Certificates are only annoying if you use certbot, which has terrible UX, virtually any other tool makes it trivial, e.g. Dehydrated or acme.sh.

boesboes 4y ago

heh, I'm on the exact opposite side of the fence. Tried caddy, found it confusing and incomplete so I went back to nginx. Buuut then again, I've been writing nginx config for 10-15 years :)

francislavoie 4y ago
Wouldn't mind hearing what you found confusing, so we can try to close any gaps

Avamander 4y ago

I'm using all three, Apache, Caddy and nginx almost daily. For the trivial, Caddy is amazing. When things get more complex alongside needs themselves, nginx is amazingly powerful. Apache at the same point becomes overly verbose, anomalous and painful. It does take some discipline though, to comment and structure your configuration files, but that applies to anything that isn't trivial. return 444; alone is a great feature I can't do without, not to mention all the modules one can use. The thing with certificates is that most tutorials are terrible. A very seamless way is to use the webroot method. Create a nginx configuration file that places LE's challenges to a specified folder on your system, include that config into your server blocks. Then you instruct certbot to use that folder. Really quite easy.

TheRealDunkirk 4y ago
> Really quite easy. “Your ideas are intriguing to me, and I wish to subscribe to your newsletter.” Is there a detailed writeup on this? This sounds great, but I don't want to faff about with it.
- Avamander 4y ago
  In its most simple form, it works as follows. /etc/nginx/letsencrypt.conf: location /.well-known/acme-challenge { default_type "text/plain"; try_files $uri $uri/ =404; root /tmp/letsencrypt-auto; } /etc/nginx/sites-enabled/default: server { [...] include letsencrypt.conf; [...] } Reload nginx. Invoke certbot for the new domain once with: certbot certonly --webroot -w /tmp/letsencrypt-auto/ [--must-staple] -d example.com
  - braiam 4y ago
    Wait, how does that makes nginx use the certificates? For me it doesn't have any cerbot/letsencrytpt specific configuration as I use DNS-01. I just point ssl_certificate and ssl_certificate_key to the correct path where cerbot saves the certificates and be done with it.

heliodor 4y ago

Nginx bites you in the a**: you start your project, choose the most famous web server (nginx), then you build and build and start needing the pro features. But they don't give you an up-front price! You have to contact them for a custom quote. That's when you start pondering moving to Caddy. But you're too far down the rabbit hole. Luckily, Cloudflare does a decent job of providing the features you need, at a clear price.

excitom 4y ago

nginx config is only complicated if you want to take advantage of complex options. The config for a basic website is trivial. Seems like a poor tradeoff for a slower, less configurable web server.

simon04 4y ago
Unless you want a good HTTPS configuration. The output of https://ssl-config.mozilla.org/#server=nginx https://ssl-config.mozilla.org/#server=nginx is significantly longer than the one for caddy.
- francislavoie 4y ago
  FYI that website is outdated and unmaintained. The person who worked on it left Mozilla AFAIK (would love to be proven wrong). See https://twitter.com/CubicleApril/status/1438940565916225545 https://twitter.com/CubicleApril/status/1438940565916225545 The correct thing with Caddy is to _not configure anything at all_ for TLS. The defaults are the correct thing to use. If you override the defaults, then you're more at risk of bitrot due to not remembering to update your own config. Let Caddy (and the Go stdlib, really) choose what's secure.

Abishek_Muthian 4y ago

It's been 5 years since I used any web server as I started using Go to build web applications. 1. HTTP server is production ready. 2. autocert fetches certificate automatically from LetsEncrypt. 3. Graceful restarts (zero downtime) using signals. I think the the biggest advantage of languages which enable portable applications like Go is not having to mess with web servers anymore, At least for most common use cases.

Sin2x 4y ago

Nginx is still faster and that's what matters for highload. If you are not using the server for highload, no need for Nginx.

the_third_wave 4y ago

> I switched to Caddy because I just got so sick of the constant pain of certificates. Caddy does certificates automatically That may be fine if you only use those certificates for port 443 but it falters where certificates are used for the remaining 65533 ports. Instead of leaving certificate request and renewal to individual tools I prefer to centralise it in one spot - a small container or VM is sufficient - which deals out ready-to-run certificates to those systems which need them. When dealt with this way nginx actually has no problems whatsoever with certificates, all it takes is those three (or more depending on your protocol needs) lines in the config file: listen 443 ssl http2; ssl_certificate /etc/letsencrypt/live/site.example.org/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/site.example.org/privkey.pem; The /etc/letsencrypt/live tree gets pushed to the relevant machines by the certificate handling container/VM with certificate files in the form and shape needed by whatever program is to use them. So, in short, the Caddy approach may be fine for those who only run web things but it 'fails' (as in 'adds additional complications') for those who venture beyond port 443.

francislavoie 4y ago
That's not true at all. Caddy can automate certificate issuance with any of the 3 challenge types, either HTTP-01 (requires port 80) TLS-ALPN-01 (requires port 443) or DNS-01 (requires building Caddy with a DNS plugin for your DNS provider). Also you can cluster Caddy by making sure the filesystems are synced, or using a storage module like Redis. Then any of the Caddy instances can do any step of the issuance phase (one can start it, another can complete it). You can use these certificates for sites on non-443 ports as well, once you have the cert. Nothing special at all to do there, you just configure Caddy to serve a site with a different port, e.g. `example.com:65533` You also get numerous other benefits, like OCSP stapling, issuer fallback (if LE is down, it'll try ZeroSSL), etc.
- the_third_wave 4y ago
  You don't seem to understand what I mean when I say 'port 443' - i.e. HTTPS. I'm talking about the use of certificates in non-web applications, e.g. SMTPS, IMAPS, database servers, etc. I prefer not to run a web server on my database servers, mail servers and what have you.