17 ms·

Let's Encrypt is down

312 points by bpierre 9y ago

scrollaway 9y ago
Was fun finding this out during a random server cycle. Turns out, Caddy doesn't appreciate the ACME server being down, and refuses to start :) https://github.com/mholt/caddy/issues/1680 https://github.com/mholt/caddy/issues/1680
- discreditable 9y ago
  Wow @ that close comment: > So, this is not a bug and all is working as intended. Caddy folks had better never restart the caddy service (or server) while LE happens to be down, even if you already have a valid cert!
  - Vendan 9y ago
    I clicked the link, saw it was closed, and thought, "Wow, these guys are fast". Then I read it....
  - tyingq 9y ago
    That's going to be a limiter for adoption. Hopefully @mholt reconsiders. Update: Mholt pushed a change where caddy only refuses to start if the cert is expiring in 7 days or less. https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e8fa41e9b4f90d7fb95 https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e...
    - scrollaway 9y ago
      Hm, yeah I hope so too :/ Been using Caddy in prod for a year now, this issue, rare as it may be, could single-handedly get me back on nginx. Having the server be unable to start through circumstances outside of the system's control is just such a huge no.
      - danielbln 9y ago
        Same here, was very pleased with Caddy so far, but it being tightly coupled to LE being up, despite having certs cached is a no-go for our production systems and if this stays like that, would make me go back to nginx for the services I've used caddy so far. edit: looks like the dev added a fix to only refuse the start if the cached certs are dangerously close to expire. that satisfies me and I'll be continue to be using caddy.
        discreditable 9y ago
        Call me crazy but I think it's a little silly for an https server to refuse to start just because its certificate is invalid. I recognize that some folks might like that failure mode however.
      - enimodas 9y ago
        Why did you switch away from nginx?
        scrollaway 9y ago
        Not having to deal with certificate renewal is a big deal.
        fb03 9y ago
        I am running nginx reverse-proxying to a python API right now. Dealing with certificate renewal is a matter of running a daily cronjob issuing 'certbot renew'. If it works it replaces the fullchain.pem certificate, and that's it, easy peasy. Am I missing something?
        scrollaway 9y ago
        And by running Caddy instead, I have one less piece to monitor and worry about. The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server.
    - merkaloid 9y ago
      Or someone forks it
  - mholt 9y ago
    Caddy restarts gracefully with zero downtime. If you're killing the process and starting one anew, you're doing it wrong. Use signal USR1 to gracefully apply new configuration changes. Failed reloads fall back to the current working configuration without any changes or downtime. https://caddyserver.com/docs/cli#usr1 https://caddyserver.com/docs/cli#usr1
    - ptman 9y ago
      What if there's a reason to restart the server or spawn a new one?
      - imhoguy 9y ago
        Scaling, upgrading, disk issues, just migrating to new place... and foremost https://en.m.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.m.wikipedia.org/wiki/Fallacies_of_distributed_com... Also nowadays with Docker around I rarely see server/app bundles supporting graceful reload.
    - atonse 9y ago
      This misses the overall point. If I am hosting 5 sites on caddy, and add a 6th one, I restart the server. If the 6th site doesn't work (for example, if DNS didn't resolve for lets encrypt), the other 5 sites which were working before the restart, all fail to start as caddy completely crashes. This is basic resiliency you'd expect from your web server. Why should the other 5 sites fail to start if their configs are completely valid?
      - majewsky 9y ago
        > If I am hosting 5 sites on caddy, and add a 6th one, I restart the server. No, you don't. You reload the server, not restart it. Restarting a web server should only be required if you get an upgrade for it (or for OpenSSL etc.)
        e_d_e_v 9y ago
        This goes wholly against most cattle-not-pets devops philosophy though, right?
  - kuschku 9y ago
    Then wait until you see how caddy handles fully qualified domain names! Caddy will just refuse to even handle them. Every single other server on this planet handles them properly, but caddy doesn’t – and mholt considers that working as intended. Try out: https://www.google.co.uk./ https://www.google.co.uk./ https://www.microsoft.com./en-us/ https://www.microsoft.com./en-us/ https://www.amazon.com./ https://www.amazon.com./ serve the page directly; https://www.facebook.com./ https://www.facebook.com./ redirects to the relative domain and then https://caddyserver.com./ https://caddyserver.com./ (That said, traefik is equally dumb, as seen with https://traefik.io./ https://traefik.io./ )
    - yjftsjthsd-h 9y ago
      Is there a bug for this? That's out of spec
      - kuschku 9y ago
        Yes there is, @mholt said it’s a WONTFIX. You're supposed to script it yourself. https://github.com/mholt/caddy/issues/1632 https://github.com/mholt/caddy/issues/1632
      - Moter8 9y ago
        I submitted the issue for it (https://github.com/mholt/caddy/issues/1632 https://github.com/mholt/caddy/issues/1632) and it was immediately closed. Reason (after asking): we only keep issues open that are on our TODO list.
- tokenizerrr 9y ago
  And best part, according to the developer this is working as intended. A webserver with perfectly valid cached certificates refusing to start.
- amq 9y ago
  Same with traefik: https://github.com/containous/traefik/issues/791 https://github.com/containous/traefik/issues/791
- stephenr 9y ago
  This type of thing is why I (and I'm sure others) have literally zero intention of using tools like this. Separation of concerns means you are in control, and using separate layers means you can swap one out when a vulnerability/show-stopper bug is discovered. What exactly do you do when your look-ma-no-hands server won't even start? Edit: maybe "all-things-to-all-people" was the wrong term to use here.
  - scrollaway 9y ago
    That's a bit of a strawman. Caddy is far more lightweight than apache and even nginx. It just happens to do something they don't do.
    - stephenr 9y ago
      > That's a bit of a straw man From https://caddyserver.com https://caddyserver.com: > The Most Beloved Server They started the hyperbolic claims, not me.
      - scrollaway 9y ago
        I have no idea how this is relevant to the conversation, nor who said anything about hyperbolic claims. You're claiming "caddy does everything". As opposed to what? If you're running apache or nginx, your server does far more than caddy, so you're quite simply mistaken.
        stephenr 9y ago
        > You're claiming "caddy does everything". As opposed to what? Serving content over http(s), and obtaining TLS certificates are two very different tasks. > If you're running apache or nginx, your server does far more than caddy Far more, that is directly related to serving content over http/https.
        chronial 9y ago
        > Serving content over http(s), and obtaining TLS certificates are two very different tasks. Except that with let's encrypt one actually needs the other.
- Ajedi32 9y ago
  Updating to v0.10.3 should fix this problem and allow Caddy to start, provided your cert isn't less than 7 days from expiring: https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e8fa41e9b4f90d7fb95 https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e...
  - tialaramex 9y ago
    Indeed, the 0.10.3 fix compared to the situation with OCSP stapling in Apache is illustrative This fix gets almost everybody where they should be, the next time the same thing happens (and it will) Caddy isn't a problem for three weeks, which is definitely enough time. Meanwhile we're going to see the same Apache crappiness for OCSP again each time until someone over there finally snaps out of it and asks someone who actually knows how OCSP stapling was supposed to work.
ramshanker 9y ago
On the plus side, as a side effect to this event, most of libraries shall start handling this case in better robust ways..
- nickpsecurity 9y ago
  Usually takes a serious failure first. Then, they start doing real robustness... on just that one thing. ;)
- deleted 9y ago
  [deleted]
Filligree 9y ago
This is also why you don't wait until the last day before renewing. (But no-one does that, right?)
- tscs37 9y ago
  >But no-one does that, right? Sadly, reality is not that nice and I can already feel a disturbance in the force in form of "Why Lets Encrypt is bad and you should by 1 year certs" blog posts all over the place.
- IgorPartola 9y ago
  If you use one of the myriad of automated tools for using LE, you will get your cert renewed as early as 30 days before it expires. So right now the issue should only be with new domains getting certs. If you renew LE certs manually, first what is wrong with you and don't you like yourself? Second, at that point it's no different than NameCheap going down and you getting your cert from 1and1 instead.
  - StevePerkins 9y ago
    Wish someone would write one of those automated tools for Google App Engine-hosted apps. That renewal process is exactly like regex. Once every three months I need it, and have to spend an hour re-learning it from scratch.
    - Ajedi32 9y ago
      Apparently they're working on that: https://issuetracker.google.com/issues/35900034 https://issuetracker.google.com/issues/35900034 (For some reason it seems you need to sign in to view this bug; not sure why.)
  - Bartweiss 9y ago
    > So right now the issue should only be with new domains getting certs. Certainly should have been, but it looks like a lot of libraries are choking on the outage even if the old cert is 100% valid.
kennu 9y ago
Just spent an hour debugging why development scripts work but production doesn't.. Good reminder to configure some kind of notifications from AWS Lambda also when the execution times out, not just on errors.
throw2016 9y ago
Nothing against letsencrypt but dependencies on services to be online is fragile and will break. Their 90 day limit makes it worse. Saying its for security is like saying 1 or 3 year certs are somehow insecure which is not the case. It's one more headache for an admin to think about even if automated. We really should reexamine the CA system. Self signed certs should have more value than they currently do, and identity can be verified by out of band methods. Surely it's worth exploring. What we have now effectively disempowers individuals and centralizes essential services which cannot be good in the long run.
- jhasse 9y ago
  You shouldn't renew the 90 day cert on day 89 anyway. As long as Let's Encrypt isn't down for more than a day I don't see the problem.
- mastax 9y ago
  I don't disagree about CAs, but LE suggests to renew certs every 60 days. So unless this outage lasts a month or some acme tool is poorly made it shouldn't affect anything except new registrations.
  - atonse 9y ago
    The only exception is OCSP data. If you're stapling, I think that lasts a week, but I'm not 100% sure. Anyway, in either case, your site should absolutely be able to weather CA downtime like this.
- tscs37 9y ago
  You can run your own ACME provider, the code is open source. Nothing stops you from running a CA that offers 1 year certs over ACME. Or just providing one that also offers 90 day certs. If people will trust that CA is another question. The automation of LE is not the problem either. Properly automated systems would extend/renew the cert well before they are invalid, almost every LE guide I know mentions this on grounds of "what if LE is down". The only libraries and services affected are those who do not properly code for an external service provider being temporarily offline. The problem with OOB verification of certs is that it's slow and inefficient for almost all methods this can be done with. And it doesn't scale either. Imagine if everyone wanted to OOB verify the Google certs on the same day.
IgorPartola 9y ago
I think LE is a huge boon to the internet. But I would really love for someone like Amazon, Google, Facebook, or Microsoft to set up a separate provider that implements the same thing. Redundancy is super important here and clearly just one organization can't guarantee 100% uptime.
- falcolas 9y ago
  Amazon does, though limited to its own services (which is, frankly, to be expected). AWS Certificate Manager
  - tyingq 9y ago
    ACM is nice, but it does require the manual step of clicking a link in a verification email.
    - atonse 9y ago
      Yes but it also issues certs for a year, which helps alleviate that email link issue.
      - alberts00 9y ago
        If you are running the AWS ACM certificate at the time when the old one is close to expiration it will automatically renew it without user intervention. http://docs.aws.amazon.com/acm/latest/userguide/configure-domain-for-automatic-validation.html http://docs.aws.amazon.com/acm/latest/userguide/configure-do...
  - yeukhon 9y ago
    I think the OP is asking / looking forward big tech like them to host the infrastructure for higher availability. Basically like a mirror. To end users they are just the same server. On the side note, is the LE infrastructure globally distributed or currently all reside in US/West and/or US/East? Is Mozilla currently the one hosting?
  - 9y ago
QUFB 9y ago
Not only is it a problem with certificate issuance - but their OCSP servers are also down. This caused an issue on one of my sites where I was using OCSP Stapling: normal browser connections were failing, but not tools like curl (which don't ask for the OCSP response over SSL).
- Ajedi32 9y ago
  What's the typical validity period for OCSP responses with Let's Encrypt? Shouldn't the stapled responses continue working for at least a couple hours even after Let's Encrypt goes down?
  - agrajag 9y ago
    1 week, so most servers likely won't be affected unless the outage goes on for a really long time.
    - avian 9y ago
      Not sure how this works. I have OCSP stapling turned on in Apache and Firefox wouldn't load my page when Let's Encrypt OCSP servers went down. My monitoring shows that last stapled response had 4 days of validity left. So it seems that Apache immediately threw away cached OCSP responses.
      - Ajedi32 9y ago
        Yeah, seems like Apache handles OCSP server outages pretty poorly. See: https://news.ycombinator.com/item?id=14375334 https://news.ycombinator.com/item?id=14375334
      - mholt 9y ago
        For what it's worth, Caddy is the only server that will locally cache the staples (and manage them) automatically. In other words, Caddy sites were not affected by this OCSP downtime.
graton 9y ago
Working for me. I just renewed some certs using the DNS method without issue.
nickpsecurity 9y ago
"High assurance datacenter" High assurance my ass. Those don't go down unless there's a DDOS or catastrophic failure (often several). Then, they're right back up. People need to stop misusing this label. Another is "high-assurance" certs from vendors that get compromised or subverted easily. Only one high-assurance CA that I know of. It's not around for business reasons, though. http://www.anthonyhall.org/c_by_c_secure_system.pdf http://www.anthonyhall.org/c_by_c_secure_system.pdf
- mort96 9y ago
  The issue here doesn't seem that a data center went down, but that there was a bug which caused downtime.
  - nickpsecurity 9y ago
    The issue I brought up is nothing about it is high-assurance except maybe tamper-resistance on a HSM involved. It's a term abused in the certificate market a lot. An easy hint to tell is if it's a product developed slowly in a safe subset of C, Java, or Ada. Those have the tooling needed for highly-robust implementations. Then look at the OS to see if it's something extremely hardened or unusual (eg separation kernel RTOS). The protocols will be ultra-simple with a lot of high-availability and easy recovery. Almost no modern tooling will be in the TCB for configuration or deployment unless it's simple. Most of it isn't. I'm not seeing any of this in the reporting that made it here. Definitely not high-assurance. Likely compromised by high-end attackers either for specific targets or in some general way. It will help protect in its intended way against the rest, though. Enormously positive development. Just not high-assurance security at any level.
bpierre 9y ago
Traefik is having a problem similar to Caddy, where having letsencrypt down prevents it to boot. https://github.com/containous/traefik/issues/791 https://github.com/containous/traefik/issues/791
hannob 9y ago
So in case this helps anyone, I had people complaining about strange OCSP errors all over the morning coming from my server (using apache httpd). It turns out apache does practically everything to behave as dumb as possible in case of OCSP downtimes. If the OCSP sends an error it will send the error as a stapled OCSP reply (instead of using an old, still valid OCSP reply). You can't make it behave sane here, but you can at least tell it to not return the error with SSLStaplingReturnResponderErrors set to off. However if the OCSP isn't available at all apache will fake its own OCSP error (sic!) and send it. This is controlled by the option SSLStaplingFakeTryLater, which defaults to on. So if your firefox users get strange OCSP errors, it's most likely this. The doc for SSLStaplingFakeTryLater claims that this option is only effective if SSLStaplingFakeTryLater is set to on, however that's wrong. tl;dr set both of these options to "off", then at least apache won't staple any garbage in your TLS connection, firefox will try to reach the ocsp on its own and fail and still accept the connection. Yes, that's all pretty fucked up.
- alternize 9y ago
  additionally, error responses from the OCSP servers are cached 600 seconds - use SSLStaplingErrorCacheTimeout to lower this to a more sane value: https://httpd.apache.org/docs/trunk/mod/mod_ssl.html#sslstaplingerrorcachetimeout https://httpd.apache.org/docs/trunk/mod/mod_ssl.html#sslstap...
- avian 9y ago
  Thanks for pointing this out! Going through the documentation, another thing that surprised me was the SSLStaplingStandardCacheTimeout setting. If I understand this correctly, by default Apache will only cache OCSP responses for 1 hour, even if they are still valid for days. I guess increasing this to 1 day or something would make sense as well.
  - hannob 9y ago
    > If I understand this correctly, by default Apache will only cache OCSP responses for 1 hour, even if they are still valid for days. I guess increasing this to 1 day or something would make sense as well. Yeah, that's another major problem. But increasing that doesn't really fix anything in a reasonable way. There shouldn't be any cache timeout, this option doesn't make any sense. It should cache as long as the reply is valid and replace it in time before it expires.
deleted 9y ago
[deleted]
101km 9y ago
Clearly it is high time for an EncryptWeShall nonprofit with a wholly separate implementation and team and all the tooling adjusted to randomly pick between the two.
fgrimes 9y ago
Holy frijoles, I was pondering exactly this scenario earlier today while messing around with Caddy and LE, as in: do I want to take a (mostly) static and offline process and directly inject it as another moving part into runtime world, for the sake of its cost, convenience, and overall worthwhileness? Is there a good alternative, without this sort of process? Or back to the sharks?
Tepix 9y ago
This is unrelated to this outage, however in the past when renewing, I've always had problems resolving acme-v01.api.letsencrypt.org - am I alone with this issue? I'm running a local dnscache instance (djbdns), perhaps that has something to do with it?
jaas 9y ago
Josh from Let's Encrypt here. First, my apologies for the trouble this has caused. I want to offer people here an early root cause analysis. I say early because we have not entirely completed our investigation or a post-mortem. OCSP requests that use the GET method use standard base64 encoding, which can contain two slashes one after another. While debugging why a small number of OCSP requests consistently failed our engineers observed a rather odd, but standard, web server behavior. When a server receives a request with multiple slashes one after another they will collapse them into a single slash. This caused our OCSP responder to consider requests that had this unusual encoding quirk invalid and would respond to with a '400 Bad Request' response. The fix seemed quite simple: disable the slash collapsing behavior. Unfortunately, stopping this behavior surfaced a more serious issue. The AIA extension that we include in certificates we issue contains a URI for our OCSP server. This URI contains a trailing slash. According to RFC 6960 Appendix 1 an OCSP request using the GET method is constructed as follows 'GET {url}/{url-encoding of base-64 encoding of the DER encoding of the OCSPRequest}' where the url 'may be derived from the value of the authority information access extension in the certificate being checked for revocation'. A number of user agents take this quite literally and will construct the URL without inspecting the contents of the AIA extension meaning that they ended up with a double slash between the host name and the base64 encoded OCSP request. Before we disabled slash collapsing this was fine as the web server was silently fixing this problem. Once we stopped collapsing slashes we started seeing problems. From our OCSP server's perspective a majority of the OCSP requests we were receiving were prepended with a slash and we were unable to decode them so we'd respond with a '400 Bad Request' response and move on. This coincided with a large number of previously cached responses on our CDN expiring, causing us to start getting hit with a large number of requests. Because we were responding with '400 Bad Request' responses we were setting explicit no-cache headers which meant we had a near 0% cache (CDN) offload rate and were hit with the full brunt of our OCSP request load at our origin servers. This caused our whole infrastructure to get bogged down.
- jpgvm 9y ago
  Thanks for the insight. Would love to read a full postmortem of both the OCSP and issuance issues.
- 9y ago
CaliforniaKarl 9y ago
It seems that Let's Encrypt is back up. I think, in the past, mods have put an extra down-weight on "X is down" stories, once 'X' is back up. Since this discussion now has interesting stuff related to Let's Encrypt—and products which use Let's Encrypt—I hope the mods are willing to forgo the down weight, and instead just change the post title to something like "Let's Encrypt Was Down".
- corford 9y ago
  I'm still getting 504s and timeouts from https://acme-v01.api.letsencrypt.org/acme/new-reg https://acme-v01.api.letsencrypt.org/acme/new-reg and https://acme-v01.api.letsencrypt.org/directory https://acme-v01.api.letsencrypt.org/directory (trying from Lisbon, PT) Edit: Akamai is issuing the 504 when hitting https://acme-v01.api.letsencrypt.org/acme/new-reg https://acme-v01.api.letsencrypt.org/acme/new-reg so I guess the origin servers are still overloaded...?
  - alixaxel 9y ago
    I'm experiencing the same from Kuala Lumpur.
    - corford 9y ago
      FYI, acme-staging.api.letsencrypt.org is working ok for me but the production endpoints are still timing out.
Romajashi 9y ago
oh my god, a website in the internet is currently down, let's discuss that.
theprop 9y ago
We've been using LetsEncrypt on dozens of our servers for several months now and it's worked flawlessly both in set-up and in operation!! Set up was quite easy thanks to A LOT of work by the team there. To make HTTPS simple and free to set up is a FANTASTIC mission & the team has overall built a SUPERB system. Congratulations & thanks for addressing this issue quickly, looks like it's already solved, good work!
deleted 9y ago
[deleted]
m3kw9 9y ago
Short version - they fixed an issue but the fix had dependencies
raarts 9y ago
Well, it's down again

312 points by bpierre 9y ago

Was fun finding this out during a random server cycle. Turns out, Caddy doesn't appreciate the ACME server being down, and refuses to start :) https://github.com/mholt/caddy/issues/1680 https://github.com/mholt/caddy/issues/1680

discreditable 9y ago
Wow @ that close comment: > So, this is not a bug and all is working as intended. Caddy folks had better never restart the caddy service (or server) while LE happens to be down, even if you already have a valid cert!
- Vendan 9y ago
  I clicked the link, saw it was closed, and thought, "Wow, these guys are fast". Then I read it....
- tyingq 9y ago
  That's going to be a limiter for adoption. Hopefully @mholt reconsiders. Update: Mholt pushed a change where caddy only refuses to start if the cert is expiring in 7 days or less. https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e8fa41e9b4f90d7fb95 https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e...
  - scrollaway 9y ago
    Hm, yeah I hope so too :/ Been using Caddy in prod for a year now, this issue, rare as it may be, could single-handedly get me back on nginx. Having the server be unable to start through circumstances outside of the system's control is just such a huge no.
    - danielbln 9y ago
      Same here, was very pleased with Caddy so far, but it being tightly coupled to LE being up, despite having certs cached is a no-go for our production systems and if this stays like that, would make me go back to nginx for the services I've used caddy so far. edit: looks like the dev added a fix to only refuse the start if the cached certs are dangerously close to expire. that satisfies me and I'll be continue to be using caddy.
      - discreditable 9y ago
        Call me crazy but I think it's a little silly for an https server to refuse to start just because its certificate is invalid. I recognize that some folks might like that failure mode however.
    - enimodas 9y ago
      Why did you switch away from nginx?
      - scrollaway 9y ago
        Not having to deal with certificate renewal is a big deal.
        fb03 9y ago
        I am running nginx reverse-proxying to a python API right now. Dealing with certificate renewal is a matter of running a daily cronjob issuing 'certbot renew'. If it works it replaces the fullchain.pem certificate, and that's it, easy peasy. Am I missing something?
        scrollaway 9y ago
        And by running Caddy instead, I have one less piece to monitor and worry about. The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server.
  - merkaloid 9y ago
    Or someone forks it
- mholt 9y ago
  Caddy restarts gracefully with zero downtime. If you're killing the process and starting one anew, you're doing it wrong. Use signal USR1 to gracefully apply new configuration changes. Failed reloads fall back to the current working configuration without any changes or downtime. https://caddyserver.com/docs/cli#usr1 https://caddyserver.com/docs/cli#usr1
  - ptman 9y ago
    What if there's a reason to restart the server or spawn a new one?
    - imhoguy 9y ago
      Scaling, upgrading, disk issues, just migrating to new place... and foremost https://en.m.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.m.wikipedia.org/wiki/Fallacies_of_distributed_com... Also nowadays with Docker around I rarely see server/app bundles supporting graceful reload.
  - atonse 9y ago
    This misses the overall point. If I am hosting 5 sites on caddy, and add a 6th one, I restart the server. If the 6th site doesn't work (for example, if DNS didn't resolve for lets encrypt), the other 5 sites which were working before the restart, all fail to start as caddy completely crashes. This is basic resiliency you'd expect from your web server. Why should the other 5 sites fail to start if their configs are completely valid?
    - majewsky 9y ago
      > If I am hosting 5 sites on caddy, and add a 6th one, I restart the server. No, you don't. You reload the server, not restart it. Restarting a web server should only be required if you get an upgrade for it (or for OpenSSL etc.)
      - e_d_e_v 9y ago
        This goes wholly against most cattle-not-pets devops philosophy though, right?
- kuschku 9y ago
  Then wait until you see how caddy handles fully qualified domain names! Caddy will just refuse to even handle them. Every single other server on this planet handles them properly, but caddy doesn’t – and mholt considers that working as intended. Try out: https://www.google.co.uk./ https://www.google.co.uk./ https://www.microsoft.com./en-us/ https://www.microsoft.com./en-us/ https://www.amazon.com./ https://www.amazon.com./ serve the page directly; https://www.facebook.com./ https://www.facebook.com./ redirects to the relative domain and then https://caddyserver.com./ https://caddyserver.com./ (That said, traefik is equally dumb, as seen with https://traefik.io./ https://traefik.io./ )
  - yjftsjthsd-h 9y ago
    Is there a bug for this? That's out of spec
    - kuschku 9y ago
      Yes there is, @mholt said it’s a WONTFIX. You're supposed to script it yourself. https://github.com/mholt/caddy/issues/1632 https://github.com/mholt/caddy/issues/1632
    - Moter8 9y ago
      I submitted the issue for it (https://github.com/mholt/caddy/issues/1632 https://github.com/mholt/caddy/issues/1632) and it was immediately closed. Reason (after asking): we only keep issues open that are on our TODO list.
tokenizerrr 9y ago
And best part, according to the developer this is working as intended. A webserver with perfectly valid cached certificates refusing to start.
amq 9y ago
Same with traefik: https://github.com/containous/traefik/issues/791 https://github.com/containous/traefik/issues/791
stephenr 9y ago
This type of thing is why I (and I'm sure others) have literally zero intention of using tools like this. Separation of concerns means you are in control, and using separate layers means you can swap one out when a vulnerability/show-stopper bug is discovered. What exactly do you do when your look-ma-no-hands server won't even start? Edit: maybe "all-things-to-all-people" was the wrong term to use here.
- scrollaway 9y ago
  That's a bit of a strawman. Caddy is far more lightweight than apache and even nginx. It just happens to do something they don't do.
  - stephenr 9y ago
    > That's a bit of a straw man From https://caddyserver.com https://caddyserver.com: > The Most Beloved Server They started the hyperbolic claims, not me.
    - scrollaway 9y ago
      I have no idea how this is relevant to the conversation, nor who said anything about hyperbolic claims. You're claiming "caddy does everything". As opposed to what? If you're running apache or nginx, your server does far more than caddy, so you're quite simply mistaken.
      - stephenr 9y ago
        > You're claiming "caddy does everything". As opposed to what? Serving content over http(s), and obtaining TLS certificates are two very different tasks. > If you're running apache or nginx, your server does far more than caddy Far more, that is directly related to serving content over http/https.
        chronial 9y ago
        > Serving content over http(s), and obtaining TLS certificates are two very different tasks. Except that with let's encrypt one actually needs the other.
Ajedi32 9y ago
Updating to v0.10.3 should fix this problem and allow Caddy to start, provided your cert isn't less than 7 days from expiring: https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e8fa41e9b4f90d7fb95 https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e...
- tialaramex 9y ago
  Indeed, the 0.10.3 fix compared to the situation with OCSP stapling in Apache is illustrative This fix gets almost everybody where they should be, the next time the same thing happens (and it will) Caddy isn't a problem for three weeks, which is definitely enough time. Meanwhile we're going to see the same Apache crappiness for OCSP again each time until someone over there finally snaps out of it and asks someone who actually knows how OCSP stapling was supposed to work.

ramshanker 9y ago

On the plus side, as a side effect to this event, most of libraries shall start handling this case in better robust ways..

nickpsecurity 9y ago
Usually takes a serious failure first. Then, they start doing real robustness... on just that one thing. ;)
deleted 9y ago
[deleted]

Filligree 9y ago

This is also why you don't wait until the last day before renewing. (But no-one does that, right?)

tscs37 9y ago
>But no-one does that, right? Sadly, reality is not that nice and I can already feel a disturbance in the force in form of "Why Lets Encrypt is bad and you should by 1 year certs" blog posts all over the place.
IgorPartola 9y ago
If you use one of the myriad of automated tools for using LE, you will get your cert renewed as early as 30 days before it expires. So right now the issue should only be with new domains getting certs. If you renew LE certs manually, first what is wrong with you and don't you like yourself? Second, at that point it's no different than NameCheap going down and you getting your cert from 1and1 instead.
- StevePerkins 9y ago
  Wish someone would write one of those automated tools for Google App Engine-hosted apps. That renewal process is exactly like regex. Once every three months I need it, and have to spend an hour re-learning it from scratch.
  - Ajedi32 9y ago
    Apparently they're working on that: https://issuetracker.google.com/issues/35900034 https://issuetracker.google.com/issues/35900034 (For some reason it seems you need to sign in to view this bug; not sure why.)
- Bartweiss 9y ago
  > So right now the issue should only be with new domains getting certs. Certainly should have been, but it looks like a lot of libraries are choking on the outage even if the old cert is 100% valid.

kennu 9y ago

Just spent an hour debugging why development scripts work but production doesn't.. Good reminder to configure some kind of notifications from AWS Lambda also when the execution times out, not just on errors.

throw2016 9y ago

Nothing against letsencrypt but dependencies on services to be online is fragile and will break. Their 90 day limit makes it worse. Saying its for security is like saying 1 or 3 year certs are somehow insecure which is not the case. It's one more headache for an admin to think about even if automated. We really should reexamine the CA system. Self signed certs should have more value than they currently do, and identity can be verified by out of band methods. Surely it's worth exploring. What we have now effectively disempowers individuals and centralizes essential services which cannot be good in the long run.

jhasse 9y ago
You shouldn't renew the 90 day cert on day 89 anyway. As long as Let's Encrypt isn't down for more than a day I don't see the problem.
mastax 9y ago
I don't disagree about CAs, but LE suggests to renew certs every 60 days. So unless this outage lasts a month or some acme tool is poorly made it shouldn't affect anything except new registrations.
- atonse 9y ago
  The only exception is OCSP data. If you're stapling, I think that lasts a week, but I'm not 100% sure. Anyway, in either case, your site should absolutely be able to weather CA downtime like this.
tscs37 9y ago
You can run your own ACME provider, the code is open source. Nothing stops you from running a CA that offers 1 year certs over ACME. Or just providing one that also offers 90 day certs. If people will trust that CA is another question. The automation of LE is not the problem either. Properly automated systems would extend/renew the cert well before they are invalid, almost every LE guide I know mentions this on grounds of "what if LE is down". The only libraries and services affected are those who do not properly code for an external service provider being temporarily offline. The problem with OOB verification of certs is that it's slow and inefficient for almost all methods this can be done with. And it doesn't scale either. Imagine if everyone wanted to OOB verify the Google certs on the same day.

IgorPartola 9y ago

I think LE is a huge boon to the internet. But I would really love for someone like Amazon, Google, Facebook, or Microsoft to set up a separate provider that implements the same thing. Redundancy is super important here and clearly just one organization can't guarantee 100% uptime.

falcolas 9y ago
Amazon does, though limited to its own services (which is, frankly, to be expected). AWS Certificate Manager
- tyingq 9y ago
  ACM is nice, but it does require the manual step of clicking a link in a verification email.
  - atonse 9y ago
    Yes but it also issues certs for a year, which helps alleviate that email link issue.
    - alberts00 9y ago
      If you are running the AWS ACM certificate at the time when the old one is close to expiration it will automatically renew it without user intervention. http://docs.aws.amazon.com/acm/latest/userguide/configure-domain-for-automatic-validation.html http://docs.aws.amazon.com/acm/latest/userguide/configure-do...
- yeukhon 9y ago
  I think the OP is asking / looking forward big tech like them to host the infrastructure for higher availability. Basically like a mirror. To end users they are just the same server. On the side note, is the LE infrastructure globally distributed or currently all reside in US/West and/or US/East? Is Mozilla currently the one hosting?
- 9y ago

QUFB 9y ago

Not only is it a problem with certificate issuance - but their OCSP servers are also down. This caused an issue on one of my sites where I was using OCSP Stapling: normal browser connections were failing, but not tools like curl (which don't ask for the OCSP response over SSL).

Ajedi32 9y ago
What's the typical validity period for OCSP responses with Let's Encrypt? Shouldn't the stapled responses continue working for at least a couple hours even after Let's Encrypt goes down?
- agrajag 9y ago
  1 week, so most servers likely won't be affected unless the outage goes on for a really long time.
  - avian 9y ago
    Not sure how this works. I have OCSP stapling turned on in Apache and Firefox wouldn't load my page when Let's Encrypt OCSP servers went down. My monitoring shows that last stapled response had 4 days of validity left. So it seems that Apache immediately threw away cached OCSP responses.
    - Ajedi32 9y ago
      Yeah, seems like Apache handles OCSP server outages pretty poorly. See: https://news.ycombinator.com/item?id=14375334 https://news.ycombinator.com/item?id=14375334
    - mholt 9y ago
      For what it's worth, Caddy is the only server that will locally cache the staples (and manage them) automatically. In other words, Caddy sites were not affected by this OCSP downtime.

graton 9y ago

Working for me. I just renewed some certs using the DNS method without issue.

nickpsecurity 9y ago

"High assurance datacenter" High assurance my ass. Those don't go down unless there's a DDOS or catastrophic failure (often several). Then, they're right back up. People need to stop misusing this label. Another is "high-assurance" certs from vendors that get compromised or subverted easily. Only one high-assurance CA that I know of. It's not around for business reasons, though. http://www.anthonyhall.org/c_by_c_secure_system.pdf http://www.anthonyhall.org/c_by_c_secure_system.pdf

mort96 9y ago
The issue here doesn't seem that a data center went down, but that there was a bug which caused downtime.
- nickpsecurity 9y ago
  The issue I brought up is nothing about it is high-assurance except maybe tamper-resistance on a HSM involved. It's a term abused in the certificate market a lot. An easy hint to tell is if it's a product developed slowly in a safe subset of C, Java, or Ada. Those have the tooling needed for highly-robust implementations. Then look at the OS to see if it's something extremely hardened or unusual (eg separation kernel RTOS). The protocols will be ultra-simple with a lot of high-availability and easy recovery. Almost no modern tooling will be in the TCB for configuration or deployment unless it's simple. Most of it isn't. I'm not seeing any of this in the reporting that made it here. Definitely not high-assurance. Likely compromised by high-end attackers either for specific targets or in some general way. It will help protect in its intended way against the rest, though. Enormously positive development. Just not high-assurance security at any level.

bpierre 9y ago

Traefik is having a problem similar to Caddy, where having letsencrypt down prevents it to boot. https://github.com/containous/traefik/issues/791 https://github.com/containous/traefik/issues/791

hannob 9y ago

So in case this helps anyone, I had people complaining about strange OCSP errors all over the morning coming from my server (using apache httpd). It turns out apache does practically everything to behave as dumb as possible in case of OCSP downtimes. If the OCSP sends an error it will send the error as a stapled OCSP reply (instead of using an old, still valid OCSP reply). You can't make it behave sane here, but you can at least tell it to not return the error with SSLStaplingReturnResponderErrors set to off. However if the OCSP isn't available at all apache will fake its own OCSP error (sic!) and send it. This is controlled by the option SSLStaplingFakeTryLater, which defaults to on. So if your firefox users get strange OCSP errors, it's most likely this. The doc for SSLStaplingFakeTryLater claims that this option is only effective if SSLStaplingFakeTryLater is set to on, however that's wrong. tl;dr set both of these options to "off", then at least apache won't staple any garbage in your TLS connection, firefox will try to reach the ocsp on its own and fail and still accept the connection. Yes, that's all pretty fucked up.

alternize 9y ago
additionally, error responses from the OCSP servers are cached 600 seconds - use SSLStaplingErrorCacheTimeout to lower this to a more sane value: https://httpd.apache.org/docs/trunk/mod/mod_ssl.html#sslstaplingerrorcachetimeout https://httpd.apache.org/docs/trunk/mod/mod_ssl.html#sslstap...
avian 9y ago
Thanks for pointing this out! Going through the documentation, another thing that surprised me was the SSLStaplingStandardCacheTimeout setting. If I understand this correctly, by default Apache will only cache OCSP responses for 1 hour, even if they are still valid for days. I guess increasing this to 1 day or something would make sense as well.
- hannob 9y ago
  > If I understand this correctly, by default Apache will only cache OCSP responses for 1 hour, even if they are still valid for days. I guess increasing this to 1 day or something would make sense as well. Yeah, that's another major problem. But increasing that doesn't really fix anything in a reasonable way. There shouldn't be any cache timeout, this option doesn't make any sense. It should cache as long as the reply is valid and replace it in time before it expires.

deleted 9y ago

[deleted]

101km 9y ago

Clearly it is high time for an EncryptWeShall nonprofit with a wholly separate implementation and team and all the tooling adjusted to randomly pick between the two.

fgrimes 9y ago

Holy frijoles, I was pondering exactly this scenario earlier today while messing around with Caddy and LE, as in: do I want to take a (mostly) static and offline process and directly inject it as another moving part into runtime world, for the sake of its cost, convenience, and overall worthwhileness? Is there a good alternative, without this sort of process? Or back to the sharks?

Tepix 9y ago

This is unrelated to this outage, however in the past when renewing, I've always had problems resolving acme-v01.api.letsencrypt.org - am I alone with this issue? I'm running a local dnscache instance (djbdns), perhaps that has something to do with it?

jaas 9y ago

Josh from Let's Encrypt here. First, my apologies for the trouble this has caused. I want to offer people here an early root cause analysis. I say early because we have not entirely completed our investigation or a post-mortem. OCSP requests that use the GET method use standard base64 encoding, which can contain two slashes one after another. While debugging why a small number of OCSP requests consistently failed our engineers observed a rather odd, but standard, web server behavior. When a server receives a request with multiple slashes one after another they will collapse them into a single slash. This caused our OCSP responder to consider requests that had this unusual encoding quirk invalid and would respond to with a '400 Bad Request' response. The fix seemed quite simple: disable the slash collapsing behavior. Unfortunately, stopping this behavior surfaced a more serious issue. The AIA extension that we include in certificates we issue contains a URI for our OCSP server. This URI contains a trailing slash. According to RFC 6960 Appendix 1 an OCSP request using the GET method is constructed as follows 'GET {url}/{url-encoding of base-64 encoding of the DER encoding of the OCSPRequest}' where the url 'may be derived from the value of the authority information access extension in the certificate being checked for revocation'. A number of user agents take this quite literally and will construct the URL without inspecting the contents of the AIA extension meaning that they ended up with a double slash between the host name and the base64 encoded OCSP request. Before we disabled slash collapsing this was fine as the web server was silently fixing this problem. Once we stopped collapsing slashes we started seeing problems. From our OCSP server's perspective a majority of the OCSP requests we were receiving were prepended with a slash and we were unable to decode them so we'd respond with a '400 Bad Request' response and move on. This coincided with a large number of previously cached responses on our CDN expiring, causing us to start getting hit with a large number of requests. Because we were responding with '400 Bad Request' responses we were setting explicit no-cache headers which meant we had a near 0% cache (CDN) offload rate and were hit with the full brunt of our OCSP request load at our origin servers. This caused our whole infrastructure to get bogged down.

jpgvm 9y ago
Thanks for the insight. Would love to read a full postmortem of both the OCSP and issuance issues.
9y ago

CaliforniaKarl 9y ago

It seems that Let's Encrypt is back up. I think, in the past, mods have put an extra down-weight on "X is down" stories, once 'X' is back up. Since this discussion now has interesting stuff related to Let's Encrypt—and products which use Let's Encrypt—I hope the mods are willing to forgo the down weight, and instead just change the post title to something like "Let's Encrypt Was Down".

corford 9y ago
I'm still getting 504s and timeouts from https://acme-v01.api.letsencrypt.org/acme/new-reg https://acme-v01.api.letsencrypt.org/acme/new-reg and https://acme-v01.api.letsencrypt.org/directory https://acme-v01.api.letsencrypt.org/directory (trying from Lisbon, PT) Edit: Akamai is issuing the 504 when hitting https://acme-v01.api.letsencrypt.org/acme/new-reg https://acme-v01.api.letsencrypt.org/acme/new-reg so I guess the origin servers are still overloaded...?
- alixaxel 9y ago
  I'm experiencing the same from Kuala Lumpur.
  - corford 9y ago
    FYI, acme-staging.api.letsencrypt.org is working ok for me but the production endpoints are still timing out.

Romajashi 9y ago

oh my god, a website in the internet is currently down, let's discuss that.

theprop 9y ago

We've been using LetsEncrypt on dozens of our servers for several months now and it's worked flawlessly both in set-up and in operation!! Set up was quite easy thanks to A LOT of work by the team there. To make HTTPS simple and free to set up is a FANTASTIC mission & the team has overall built a SUPERB system. Congratulations & thanks for addressing this issue quickly, looks like it's already solved, good work!

deleted 9y ago

[deleted]

m3kw9 9y ago

Short version - they fixed an issue but the fix had dependencies

raarts 9y ago

Well, it's down again