6 ms·

Was fun finding this out during a random server cycle. Turns out, Caddy doesn't appreciate the ACME server being down, and refuses to start :) https://github.c

Was fun finding this out during a random server cycle. Turns out, Caddy doesn't appreciate the ACME server being down, and refuses to start :) https://github.com/mholt/caddy/issues/1680 https://github.com/mholt/caddy/issues/1680

discreditable 9y ago
Wow @ that close comment: > So, this is not a bug and all is working as intended. Caddy folks had better never restart the caddy service (or server) while LE happens to be down, even if you already have a valid cert!
- Vendan 9y ago
  I clicked the link, saw it was closed, and thought, "Wow, these guys are fast". Then I read it....
- tyingq 9y ago
  That's going to be a limiter for adoption. Hopefully @mholt reconsiders. Update: Mholt pushed a change where caddy only refuses to start if the cert is expiring in 7 days or less. https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e8fa41e9b4f90d7fb95 https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e...
  - scrollaway 9y ago
    Hm, yeah I hope so too :/ Been using Caddy in prod for a year now, this issue, rare as it may be, could single-handedly get me back on nginx. Having the server be unable to start through circumstances outside of the system's control is just such a huge no.
    - danielbln 9y ago
      Same here, was very pleased with Caddy so far, but it being tightly coupled to LE being up, despite having certs cached is a no-go for our production systems and if this stays like that, would make me go back to nginx for the services I've used caddy so far. edit: looks like the dev added a fix to only refuse the start if the cached certs are dangerously close to expire. that satisfies me and I'll be continue to be using caddy.
      - discreditable 9y ago
        Call me crazy but I think it's a little silly for an https server to refuse to start just because its certificate is invalid. I recognize that some folks might like that failure mode however.
    - enimodas 9y ago
      Why did you switch away from nginx?
      - scrollaway 9y ago
        Not having to deal with certificate renewal is a big deal.
        fb03 9y ago
        I am running nginx reverse-proxying to a python API right now. Dealing with certificate renewal is a matter of running a daily cronjob issuing 'certbot renew'. If it works it replaces the fullchain.pem certificate, and that's it, easy peasy. Am I missing something?
        scrollaway 9y ago
        And by running Caddy instead, I have one less piece to monitor and worry about. The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server.
        JshWright 9y ago
        > The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server. That's an odd thing to say in this particular conversation thread... Why would you want to tightly couple your webserver to the availability of another service provider?
        scrollaway 9y ago
        It's not tightly coupled any more than it would be with certbot. The issue I filed is an issue because it's unexpected behaviour (despite early closure, it has actually now been fixed).
        stephenr 9y ago
        > by running Caddy instead, I have one less piece to monitor and worry about No, with Caddy you have several vaguely related pieces glued together with superglue. > The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server. I think this thread is pretty much proof that that approach will bite you in the ass.
    - GiorgioG 9y ago
      I'm done with Caddy - this is security theater at its finest. Back to nginx we go. Why did I pick Caddy, because it was simple/easy/fast-to-setup.
      - e_d_e_v 9y ago
        I moved from Caddy to Traefik (https://traefik.io/ https://traefik.io/) several months ago. Granted, nginx has years (decades?) on some of these newer webserver/reverse-proxies, but I have been happy with all the built in niceties of traefik so far (single binary, etc), and haven't really experienced any negatives.
        mholt 9y ago
        You're not bit by this? https://github.com/containous/traefik/issues/791 https://github.com/containous/traefik/issues/791
        e_d_e_v 9y ago
        Nope.
  - merkaloid 9y ago
    Or someone forks it
- mholt 9y ago
  Caddy restarts gracefully with zero downtime. If you're killing the process and starting one anew, you're doing it wrong. Use signal USR1 to gracefully apply new configuration changes. Failed reloads fall back to the current working configuration without any changes or downtime. https://caddyserver.com/docs/cli#usr1 https://caddyserver.com/docs/cli#usr1
  - ptman 9y ago
    What if there's a reason to restart the server or spawn a new one?
    - imhoguy 9y ago
      Scaling, upgrading, disk issues, just migrating to new place... and foremost https://en.m.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.m.wikipedia.org/wiki/Fallacies_of_distributed_com... Also nowadays with Docker around I rarely see server/app bundles supporting graceful reload.
  - atonse 9y ago
    This misses the overall point. If I am hosting 5 sites on caddy, and add a 6th one, I restart the server. If the 6th site doesn't work (for example, if DNS didn't resolve for lets encrypt), the other 5 sites which were working before the restart, all fail to start as caddy completely crashes. This is basic resiliency you'd expect from your web server. Why should the other 5 sites fail to start if their configs are completely valid?
    - majewsky 9y ago
      > If I am hosting 5 sites on caddy, and add a 6th one, I restart the server. No, you don't. You reload the server, not restart it. Restarting a web server should only be required if you get an upgrade for it (or for OpenSSL etc.)
      - e_d_e_v 9y ago
        This goes wholly against most cattle-not-pets devops philosophy though, right?
- kuschku 9y ago
  Then wait until you see how caddy handles fully qualified domain names! Caddy will just refuse to even handle them. Every single other server on this planet handles them properly, but caddy doesn’t – and mholt considers that working as intended. Try out: https://www.google.co.uk./ https://www.google.co.uk./ https://www.microsoft.com./en-us/ https://www.microsoft.com./en-us/ https://www.amazon.com./ https://www.amazon.com./ serve the page directly; https://www.facebook.com./ https://www.facebook.com./ redirects to the relative domain and then https://caddyserver.com./ https://caddyserver.com./ (That said, traefik is equally dumb, as seen with https://traefik.io./ https://traefik.io./ )
  - yjftsjthsd-h 9y ago
    Is there a bug for this? That's out of spec
    - kuschku 9y ago
      Yes there is, @mholt said it’s a WONTFIX. You're supposed to script it yourself. https://github.com/mholt/caddy/issues/1632 https://github.com/mholt/caddy/issues/1632
    - Moter8 9y ago
      I submitted the issue for it (https://github.com/mholt/caddy/issues/1632 https://github.com/mholt/caddy/issues/1632) and it was immediately closed. Reason (after asking): we only keep issues open that are on our TODO list.
tokenizerrr 9y ago
And best part, according to the developer this is working as intended. A webserver with perfectly valid cached certificates refusing to start.
amq 9y ago
Same with traefik: https://github.com/containous/traefik/issues/791 https://github.com/containous/traefik/issues/791
stephenr 9y ago
This type of thing is why I (and I'm sure others) have literally zero intention of using tools like this. Separation of concerns means you are in control, and using separate layers means you can swap one out when a vulnerability/show-stopper bug is discovered. What exactly do you do when your look-ma-no-hands server won't even start? Edit: maybe "all-things-to-all-people" was the wrong term to use here.
- scrollaway 9y ago
  That's a bit of a strawman. Caddy is far more lightweight than apache and even nginx. It just happens to do something they don't do.
  - stephenr 9y ago
    > That's a bit of a straw man From https://caddyserver.com https://caddyserver.com: > The Most Beloved Server They started the hyperbolic claims, not me.
    - scrollaway 9y ago
      I have no idea how this is relevant to the conversation, nor who said anything about hyperbolic claims. You're claiming "caddy does everything". As opposed to what? If you're running apache or nginx, your server does far more than caddy, so you're quite simply mistaken.
      - stephenr 9y ago
        > You're claiming "caddy does everything". As opposed to what? Serving content over http(s), and obtaining TLS certificates are two very different tasks. > If you're running apache or nginx, your server does far more than caddy Far more, that is directly related to serving content over http/https.
        chronial 9y ago
        > Serving content over http(s), and obtaining TLS certificates are two very different tasks. Except that with let's encrypt one actually needs the other.
Ajedi32 9y ago
Updating to v0.10.3 should fix this problem and allow Caddy to start, provided your cert isn't less than 7 days from expiring: https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e8fa41e9b4f90d7fb95 https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e...
- tialaramex 9y ago
  Indeed, the 0.10.3 fix compared to the situation with OCSP stapling in Apache is illustrative This fix gets almost everybody where they should be, the next time the same thing happens (and it will) Caddy isn't a problem for three weeks, which is definitely enough time. Meanwhile we're going to see the same Apache crappiness for OCSP again each time until someone over there finally snaps out of it and asks someone who actually knows how OCSP stapling was supposed to work.

discreditable 9y ago

Wow @ that close comment: > So, this is not a bug and all is working as intended. Caddy folks had better never restart the caddy service (or server) while LE happens to be down, even if you already have a valid cert!

Vendan 9y ago
I clicked the link, saw it was closed, and thought, "Wow, these guys are fast". Then I read it....
tyingq 9y ago
That's going to be a limiter for adoption. Hopefully @mholt reconsiders. Update: Mholt pushed a change where caddy only refuses to start if the cert is expiring in 7 days or less. https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e8fa41e9b4f90d7fb95 https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e...
- scrollaway 9y ago
  Hm, yeah I hope so too :/ Been using Caddy in prod for a year now, this issue, rare as it may be, could single-handedly get me back on nginx. Having the server be unable to start through circumstances outside of the system's control is just such a huge no.
  - danielbln 9y ago
    Same here, was very pleased with Caddy so far, but it being tightly coupled to LE being up, despite having certs cached is a no-go for our production systems and if this stays like that, would make me go back to nginx for the services I've used caddy so far. edit: looks like the dev added a fix to only refuse the start if the cached certs are dangerously close to expire. that satisfies me and I'll be continue to be using caddy.
    - discreditable 9y ago
      Call me crazy but I think it's a little silly for an https server to refuse to start just because its certificate is invalid. I recognize that some folks might like that failure mode however.
  - enimodas 9y ago
    Why did you switch away from nginx?
    - scrollaway 9y ago
      Not having to deal with certificate renewal is a big deal.
      - fb03 9y ago
        I am running nginx reverse-proxying to a python API right now. Dealing with certificate renewal is a matter of running a daily cronjob issuing 'certbot renew'. If it works it replaces the fullchain.pem certificate, and that's it, easy peasy. Am I missing something?
        scrollaway 9y ago
        And by running Caddy instead, I have one less piece to monitor and worry about. The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server.
        JshWright 9y ago
        > The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server. That's an odd thing to say in this particular conversation thread... Why would you want to tightly couple your webserver to the availability of another service provider?
        scrollaway 9y ago
        It's not tightly coupled any more than it would be with certbot. The issue I filed is an issue because it's unexpected behaviour (despite early closure, it has actually now been fixed).
        stephenr 9y ago
        > by running Caddy instead, I have one less piece to monitor and worry about No, with Caddy you have several vaguely related pieces glued together with superglue. > The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server. I think this thread is pretty much proof that that approach will bite you in the ass.
  - GiorgioG 9y ago
    I'm done with Caddy - this is security theater at its finest. Back to nginx we go. Why did I pick Caddy, because it was simple/easy/fast-to-setup.
    - e_d_e_v 9y ago
      I moved from Caddy to Traefik (https://traefik.io/ https://traefik.io/) several months ago. Granted, nginx has years (decades?) on some of these newer webserver/reverse-proxies, but I have been happy with all the built in niceties of traefik so far (single binary, etc), and haven't really experienced any negatives.
      - mholt 9y ago
        You're not bit by this? https://github.com/containous/traefik/issues/791 https://github.com/containous/traefik/issues/791
        e_d_e_v 9y ago
        Nope.
- merkaloid 9y ago
  Or someone forks it
mholt 9y ago
Caddy restarts gracefully with zero downtime. If you're killing the process and starting one anew, you're doing it wrong. Use signal USR1 to gracefully apply new configuration changes. Failed reloads fall back to the current working configuration without any changes or downtime. https://caddyserver.com/docs/cli#usr1 https://caddyserver.com/docs/cli#usr1
- ptman 9y ago
  What if there's a reason to restart the server or spawn a new one?
  - imhoguy 9y ago
    Scaling, upgrading, disk issues, just migrating to new place... and foremost https://en.m.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.m.wikipedia.org/wiki/Fallacies_of_distributed_com... Also nowadays with Docker around I rarely see server/app bundles supporting graceful reload.
- atonse 9y ago
  This misses the overall point. If I am hosting 5 sites on caddy, and add a 6th one, I restart the server. If the 6th site doesn't work (for example, if DNS didn't resolve for lets encrypt), the other 5 sites which were working before the restart, all fail to start as caddy completely crashes. This is basic resiliency you'd expect from your web server. Why should the other 5 sites fail to start if their configs are completely valid?
  - majewsky 9y ago
    > If I am hosting 5 sites on caddy, and add a 6th one, I restart the server. No, you don't. You reload the server, not restart it. Restarting a web server should only be required if you get an upgrade for it (or for OpenSSL etc.)
    - e_d_e_v 9y ago
      This goes wholly against most cattle-not-pets devops philosophy though, right?
kuschku 9y ago
Then wait until you see how caddy handles fully qualified domain names! Caddy will just refuse to even handle them. Every single other server on this planet handles them properly, but caddy doesn’t – and mholt considers that working as intended. Try out: https://www.google.co.uk./ https://www.google.co.uk./ https://www.microsoft.com./en-us/ https://www.microsoft.com./en-us/ https://www.amazon.com./ https://www.amazon.com./ serve the page directly; https://www.facebook.com./ https://www.facebook.com./ redirects to the relative domain and then https://caddyserver.com./ https://caddyserver.com./ (That said, traefik is equally dumb, as seen with https://traefik.io./ https://traefik.io./ )
- yjftsjthsd-h 9y ago
  Is there a bug for this? That's out of spec
  - kuschku 9y ago
    Yes there is, @mholt said it’s a WONTFIX. You're supposed to script it yourself. https://github.com/mholt/caddy/issues/1632 https://github.com/mholt/caddy/issues/1632
  - Moter8 9y ago
    I submitted the issue for it (https://github.com/mholt/caddy/issues/1632 https://github.com/mholt/caddy/issues/1632) and it was immediately closed. Reason (after asking): we only keep issues open that are on our TODO list.

tokenizerrr 9y ago

And best part, according to the developer this is working as intended. A webserver with perfectly valid cached certificates refusing to start.

amq 9y ago

Same with traefik: https://github.com/containous/traefik/issues/791 https://github.com/containous/traefik/issues/791

stephenr 9y ago

This type of thing is why I (and I'm sure others) have literally zero intention of using tools like this. Separation of concerns means you are in control, and using separate layers means you can swap one out when a vulnerability/show-stopper bug is discovered. What exactly do you do when your look-ma-no-hands server won't even start? Edit: maybe "all-things-to-all-people" was the wrong term to use here.

scrollaway 9y ago
That's a bit of a strawman. Caddy is far more lightweight than apache and even nginx. It just happens to do something they don't do.
- stephenr 9y ago
  > That's a bit of a straw man From https://caddyserver.com https://caddyserver.com: > The Most Beloved Server They started the hyperbolic claims, not me.
  - scrollaway 9y ago
    I have no idea how this is relevant to the conversation, nor who said anything about hyperbolic claims. You're claiming "caddy does everything". As opposed to what? If you're running apache or nginx, your server does far more than caddy, so you're quite simply mistaken.
    - stephenr 9y ago
      > You're claiming "caddy does everything". As opposed to what? Serving content over http(s), and obtaining TLS certificates are two very different tasks. > If you're running apache or nginx, your server does far more than caddy Far more, that is directly related to serving content over http/https.
      - chronial 9y ago
        > Serving content over http(s), and obtaining TLS certificates are two very different tasks. Except that with let's encrypt one actually needs the other.

Ajedi32 9y ago

Updating to v0.10.3 should fix this problem and allow Caddy to start, provided your cert isn't less than 7 days from expiring: https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e8fa41e9b4f90d7fb95 https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e...

tialaramex 9y ago
Indeed, the 0.10.3 fix compared to the situation with OCSP stapling in Apache is illustrative This fix gets almost everybody where they should be, the next time the same thing happens (and it will) Caddy isn't a problem for three weeks, which is definitely enough time. Meanwhile we're going to see the same Apache crappiness for OCSP again each time until someone over there finally snaps out of it and asks someone who actually knows how OCSP stapling was supposed to work.