13 ms·

Let's Encrypt certificate issuance was down

141 points by romseb 7y ago

d0ugie 7y ago
Is there a detour here or is using letsencrypt reliant on these apis? https://medium.com/enigma-shards/lets-encrypt-uptime-and-operation-811cc1e018c0 https://medium.com/enigma-shards/lets-encrypt-uptime-and-ope...
- ownagefool 7y ago
  You need the API up; that's how it works. However, should you fail to renew a cert, your script should just keep retrying until the API is back and you should allow enough time for it to fail. I believe most clients renew certs ~10 days in advance. Obviously if you're relying on getting a new cert to bring up something like a preview environment or to hand out your own subdomains, this will result in a downtime/delay in provisioning, but most people would be fine with a single wildcard and never really experience a problem, as long as their script runs and they keep it reasonably up-to-date.
  - TimWolla 7y ago
    > I believe most clients renew certs ~10 days in advance. The recommendation and the time certbot uses is 30 days.
    - tehbeard 7y ago
      I've heard 10 days before, possibly older documentation?
      - Arnavion 7y ago
        Based on the reminder emails they've sent me, they recommend renewing when one-third of the current cert's validity is remaining. That is, 1 month for their 3-month certs.
  - BrandoElFollito 7y ago
    What is the problem with attempting the renewal every day? It will predictibly fail until D-30,and then there will be 30 attempts to renew (in case something went wrong firth the 1st,2nd etc attempt)
kissgyorgy 7y ago
A small outage should not matter at all in case of renewal, because clients are configured to renew days or weeks before cert expiration.
- GetOutOfBed 7y ago
  Yeah... my own setup for a hobby project of mine renew 30 days in advance but I don't think the way it is set up will try again tomorrow if it fails today.
  - Filligree 7y ago
    I've got a systemd timer that tries to renew it once per day. It just aborts early if the existing cert has more than 45 days left in it.
    - copperx 7y ago
      Isn't that placing a lot of load on a free service?
      - shawnz 7y ago
        With certbot, the check to see if certificates are close to expiry is done offline. So the service is only hit if a renewal is needed. The developers of certbot actually recommend that you schedule the cron twice a day. EDIT: Actually I'm wrong. It also checks if the certificate was revoked via OCSP. However I can't imagine that it consumes much resources.
        schoen 7y ago
        They're also different resources, and cacheable ones.
      - ceejayoz 7y ago
        That free service has rate limits, which are presumably set to a sustainable level for them.
Animats 7y ago
I stopped using Let's Encrypt when they broke Apache servers not using virtual hosting.
- oneplane 7y ago
  Good for you? I guess? Not sure what to do with that comment of yours. Is it an attempt at venting or constructing a situation where Let's Encrypt is responsible for your server?
- wolf550e 7y ago
  1. The certbot client is not a product of the Let's Encrypt CA and is not mandatory. You can use another client. 2. With the certbot client, instead of using a deep integration with your http server, you can use `certbot certonly --webroot --webroot-path /var/www/html/.well-known/ -d example.com` (or similar) and provided you told your web server to serve file from the /var/www/html/.well-known/ filesystem directory under the /.well-known/ url path prefix on your domain, cert issuance and renewal will work. This should be easy to do with any web server. certbot will not know which web server you happen to use and will not care. You can replace the web server and renewal will keep working.
- vbezhenar 7y ago
  Just don't use auto stuff. Be responsible for your configuration. certbot certonly --webroot works fine.
- privateSFacct 7y ago
  Certbot is from EFF not let's encrypt - you can actually write your own certbot if it doesn't do what you want - nothing to do with lets encrypts. Lots of instructions for many common situations here: https://certbot.eff.org/all-instructions https://certbot.eff.org/all-instructions Did you check that you were running it with the right command flags? If certbot has a bug that breaks things in a situation it's intended for do file a bug or even better a patch to fix the bug.
- sschueller 7y ago
  You can use dehydrated [1] which is a single shell script that you can trigger via cron to do the whole let's encrypt stuff for you. No dependancies other than curl. [1] https://github.com/lukas2511/dehydrated https://github.com/lukas2511/dehydrated
j_h_o 7y ago
Isn't it WAI that the Service Disruption API is unavailable?
- Thorrez 7y ago
  It's quite confusing, but I don't think it's a "Service Disruption API", but instead a Service Disruption in the API.
DarkmSparks 7y ago
I'm going out on a limb and saying that "emergency response" is a thing of beauty. timestamped. To the point. Examplar of how it should be done.
lifeisstillgood 7y ago
These look like "canned" responses - one of the terror inducing parts of managing an emergency response is, in the middle of everything, choosing the right set of words to use when telling loads of really worried people - and it is dry easy to put it off and that lack of communication makes things bad this smells like a tickbox on an incident response webform - the responder is just clicking "we have identified", "production database" (or network issue or ...) and the response handler does the rest would love to hear more
- icebraining 7y ago
  One of the features of status.io is "Create reusable templates to streamline your incident and maintenance updates": https://kb.status.io/planned-maintenance/status-update-templates/ https://kb.status.io/planned-maintenance/status-update-templ...
  - WrtCdEvrydy 7y ago
    Yeah, we have our 'Shit's on fire' one.... generates a message like 'We have some unscheduled maintenance of databases to improve your customer experience'.
    - 9nGQluzmnq3M 7y ago
      That's borderline insulting. When a service is down, it does not "improve customer experience".
      - dghughes 7y ago
        Maybe it was intermittent or very slow before taking it down to fix the issue.
      - dimino 7y ago
        Bringing the database back up with a fix does improve your customer experience, however!
      - 7y ago
z92 7y ago
It was down for 5 hours since when they posted the first update. And now it's fixed. > November 17, 2019 19:33 UTC > [Resolved] We have resolved the issue that was affecting production certificate issuance. Service is restored.
gumby 7y ago
Some of these can (and should) be written ahead of time, at least as templates. Not only does doing so save someone from having to write them on the fly when they're worrying about other things, they can encourage you to communicate specific factors ("OK, we think we've found the problem, let's publish the message to that effect now"), remind you to think of certain things ("we think we'll be up within the hour" "we don't know yet how long it will take to fix" etc) and for that matter, if you have further failures, help someone look back to see how the process went.

141 points by romseb 7y ago

Is there a detour here or is using letsencrypt reliant on these apis? https://medium.com/enigma-shards/lets-encrypt-uptime-and-operation-811cc1e018c0 https://medium.com/enigma-shards/lets-encrypt-uptime-and-ope...

ownagefool 7y ago
You need the API up; that's how it works. However, should you fail to renew a cert, your script should just keep retrying until the API is back and you should allow enough time for it to fail. I believe most clients renew certs ~10 days in advance. Obviously if you're relying on getting a new cert to bring up something like a preview environment or to hand out your own subdomains, this will result in a downtime/delay in provisioning, but most people would be fine with a single wildcard and never really experience a problem, as long as their script runs and they keep it reasonably up-to-date.
- TimWolla 7y ago
  > I believe most clients renew certs ~10 days in advance. The recommendation and the time certbot uses is 30 days.
  - tehbeard 7y ago
    I've heard 10 days before, possibly older documentation?
    - Arnavion 7y ago
      Based on the reminder emails they've sent me, they recommend renewing when one-third of the current cert's validity is remaining. That is, 1 month for their 3-month certs.
- BrandoElFollito 7y ago
  What is the problem with attempting the renewal every day? It will predictibly fail until D-30,and then there will be 30 attempts to renew (in case something went wrong firth the 1st,2nd etc attempt)

kissgyorgy 7y ago

A small outage should not matter at all in case of renewal, because clients are configured to renew days or weeks before cert expiration.

GetOutOfBed 7y ago
Yeah... my own setup for a hobby project of mine renew 30 days in advance but I don't think the way it is set up will try again tomorrow if it fails today.
- Filligree 7y ago
  I've got a systemd timer that tries to renew it once per day. It just aborts early if the existing cert has more than 45 days left in it.
  - copperx 7y ago
    Isn't that placing a lot of load on a free service?
    - shawnz 7y ago
      With certbot, the check to see if certificates are close to expiry is done offline. So the service is only hit if a renewal is needed. The developers of certbot actually recommend that you schedule the cron twice a day. EDIT: Actually I'm wrong. It also checks if the certificate was revoked via OCSP. However I can't imagine that it consumes much resources.
      - schoen 7y ago
        They're also different resources, and cacheable ones.
    - ceejayoz 7y ago
      That free service has rate limits, which are presumably set to a sustainable level for them.

Animats 7y ago

I stopped using Let's Encrypt when they broke Apache servers not using virtual hosting.

oneplane 7y ago
Good for you? I guess? Not sure what to do with that comment of yours. Is it an attempt at venting or constructing a situation where Let's Encrypt is responsible for your server?
wolf550e 7y ago
1. The certbot client is not a product of the Let's Encrypt CA and is not mandatory. You can use another client. 2. With the certbot client, instead of using a deep integration with your http server, you can use `certbot certonly --webroot --webroot-path /var/www/html/.well-known/ -d example.com` (or similar) and provided you told your web server to serve file from the /var/www/html/.well-known/ filesystem directory under the /.well-known/ url path prefix on your domain, cert issuance and renewal will work. This should be easy to do with any web server. certbot will not know which web server you happen to use and will not care. You can replace the web server and renewal will keep working.
vbezhenar 7y ago
Just don't use auto stuff. Be responsible for your configuration. certbot certonly --webroot works fine.
privateSFacct 7y ago
Certbot is from EFF not let's encrypt - you can actually write your own certbot if it doesn't do what you want - nothing to do with lets encrypts. Lots of instructions for many common situations here: https://certbot.eff.org/all-instructions https://certbot.eff.org/all-instructions Did you check that you were running it with the right command flags? If certbot has a bug that breaks things in a situation it's intended for do file a bug or even better a patch to fix the bug.
sschueller 7y ago
You can use dehydrated [1] which is a single shell script that you can trigger via cron to do the whole let's encrypt stuff for you. No dependancies other than curl. [1] https://github.com/lukas2511/dehydrated https://github.com/lukas2511/dehydrated

j_h_o 7y ago

Isn't it WAI that the Service Disruption API is unavailable?

Thorrez 7y ago
It's quite confusing, but I don't think it's a "Service Disruption API", but instead a Service Disruption in the API.

DarkmSparks 7y ago

I'm going out on a limb and saying that "emergency response" is a thing of beauty. timestamped. To the point. Examplar of how it should be done.

lifeisstillgood 7y ago

These look like "canned" responses - one of the terror inducing parts of managing an emergency response is, in the middle of everything, choosing the right set of words to use when telling loads of really worried people - and it is dry easy to put it off and that lack of communication makes things bad this smells like a tickbox on an incident response webform - the responder is just clicking "we have identified", "production database" (or network issue or ...) and the response handler does the rest would love to hear more

icebraining 7y ago
One of the features of status.io is "Create reusable templates to streamline your incident and maintenance updates": https://kb.status.io/planned-maintenance/status-update-templates/ https://kb.status.io/planned-maintenance/status-update-templ...
- WrtCdEvrydy 7y ago
  Yeah, we have our 'Shit's on fire' one.... generates a message like 'We have some unscheduled maintenance of databases to improve your customer experience'.
  - 9nGQluzmnq3M 7y ago
    That's borderline insulting. When a service is down, it does not "improve customer experience".
    - dghughes 7y ago
      Maybe it was intermittent or very slow before taking it down to fix the issue.
    - dimino 7y ago
      Bringing the database back up with a fix does improve your customer experience, however!
    - 7y ago

z92 7y ago

It was down for 5 hours since when they posted the first update. And now it's fixed. > November 17, 2019 19:33 UTC > [Resolved] We have resolved the issue that was affecting production certificate issuance. Service is restored.

gumby 7y ago

Some of these can (and should) be written ahead of time, at least as templates. Not only does doing so save someone from having to write them on the fly when they're worrying about other things, they can encourage you to communicate specific factors ("OK, we think we've found the problem, let's publish the message to that effect now"), remind you to think of certain things ("we think we'll be up within the hour" "we don't know yet how long it will take to fix" etc) and for that matter, if you have further failures, help someone look back to see how the process went.