4 ms·

Let's Encrypt stopped its certificate expiration email notification service a while ago, and I hadn't found a replacement yet. As a result, I didn't receive an

by Kholin 1y ago

Let's Encrypt stopped its certificate expiration email notification service a while ago, and I hadn't found a replacement yet. As a result, I didn't receive an expiration notice this time and failed to renew my certificate in advance. The certificate expired today, making my website inaccessible. I logged into my VPS to renew it manually, but the process failed every time. I then checked my cloud provider's platform and saw a notification at the top, which made me realize the problem was with the certificate provider. A quick look at Hacker News confirmed it: Let's Encrypt was having an outage. I want to post this news on my website, but I can't, because my site is down due to the expired certificate.

compumike 1y ago
Oof, you're right, that's rough that it's so soon after they discontinued their email service! I wrote this blog post a few weeks ago: "Minimal, cron-ready scripts in Bash, Python, Ruby, Node.js (JavaScript), Go, and Powershell to check when your website's SSL certificate expires." https://heiioncall.com/blog/barebone-scripts-to-check-ssl-certificate-expiration https://heiioncall.com/blog/barebone-scripts-to-check-ssl-ce... which may be helpful if you want to roll your own. (Disclosure: at Heii On-Call we also offer free SSL certificate expiration monitoring, among other things.)
mixdup 1y ago
They have been communicating the ending of the email notices for quite a while and have been telling users that you should have some other monitoring in place to avoid just this situation
- CamperBob2 1y ago
  Also, beware of the leopard.
  - jojobas 1y ago
    If you didn't see their sunset notification emails you wouldn't have seen your cert expiration email either.
- andrewmcwatters 1y ago
  Yes, but what’s weird is the recommended service they referred people to for new email notifications was not… sending me emails. So, what gives?
  - kevincox 1y ago
    Yeah the recommended service is awful and not nearly as useful as the one they had is. Which is disappointing because you should be able to recreate the service they had nearly exactly with certificate transparency logs.
  - deleted 1y ago
    [deleted]
yjftsjthsd-h 1y ago
> As a result, I didn't receive an expiration notice this time and failed to renew my certificate in advance. Shouldn't that happen automatically a bit beforehand?
- Kholin 1y ago
  Due to some legacy reasons, my service runs using a docker + nginx setup. However, certbot was initially used in its native nginx mode to generate the certificate, which prevented it from auto-renewing. I later switched it to standalone mode, but I'm not sure if I configured the auto-renewal correctly. In any case, the certificate happened to expire today, and it didn't renew automatically. On a side note, I was actually planning to see what an expired website certificate looked like first and then deal with the auto-renewal issue. After all, it's just a small hobby website, so it's not that big of a deal.
  - dylan604 1y ago
    that sounds like a you're holding it wrong type of a situation to me. a major point of Let's Encrypt (besides the obvious free) is that it deliberately keeps the cert times short to avoid the "someone that no longer works here set this up two years ago" type of situation with certbot checking twice a day and updating when necessary. so to break what Let's Encrypt is doing with not using certbot definitely feels like you're holding it wrong
mattbillenstein 1y ago
I run https://ismycertexpired.com/ https://ismycertexpired.com/ - you can sign up for email alerts.
mr_toad 1y ago
Isn’t the recommended practice to update every ~60 days or so, regardless?
- cpach 1y ago
  Either that, or use an ACME client that has support for ARI so that the CA can signal to the client when it’s time to renew. https://letsencrypt.org/2024/04/25/guide-to-integrating-ari-into-existing-acme-clients/ https://letsencrypt.org/2024/04/25/guide-to-integrating-ari-...
- mholt 1y ago
  Yeah but if you need a cert for a new site you have to use another CA.
tbrownaw 1y ago
Haven't they always, from day one, insisted that their primary goal was to encourage (force) automation of certificate maintenance, as a mechanism to make tls ubiquitous (mandatory everywhere)?
- ffsm8 1y ago
  Yes, we had lengthy discussions in itops (I had a admin role when LE was launched) about it. The team lead couldn't get over the slogan "devops, automating downtimes since 2010" whenever someone wanted to add a new nonessential automation that does things on prod servers. I mean he wasn't completely wrong, it was a non essential automation with high risk and very little reward (<1h saved every 2 yrs), which is why we never switched to LE for our main site, only internal tooling was allowed to use it
  - cpach 1y ago
    Perhaps you know this already but in the future, certs issued by a “real” CA will not be allowed to live for more than 47 days. https://www.digicert.com/blog/tls-certificate-lifetimes-will-officially-reduce-to-47-days https://www.digicert.com/blog/tls-certificate-lifetimes-will...
    - ffsm8 1y ago
      I didn't know that, interesting. I was merely retelling an anecdote about how LE was always positioned to be exclusively about refreshing certs automatically, though. As I've moved out of (dev-)ops roles around 2016/2017 so I'm really not up to date with operations topics
- throw0101d 1y ago
  > Haven't they always, from day one, insisted that their primary goal was to encourage (force) automation of certificate maintenance, as a mechanism to make tls ubiquitous (mandatory everywhere)? And? Automation sometimes breaks, both for internal reasons (OS patching) or external. For the latter, LE at some point in the past changed CDNs, and this caused JWST headers to be sent back differently, which broke different clients, e.g.: * https://community.letsencrypt.org/t/jws-has-no-anti-replay-nonce/103324 https://community.letsencrypt.org/t/jws-has-no-anti-replay-n... * https://github.com/dehydrated-io/dehydrated/issues/684 https://github.com/dehydrated-io/dehydrated/issues/684 Being able to get e-mails was an extra level of monitoring that was handy, even if you had automation.
rfv6723 1y ago
I use self-hosted gatus to monitor my certs and other services' status. It can send alerts to multiple alerting providers. https://github.com/TwiN/gatus https://github.com/TwiN/gatus
- hoherd 1y ago
  I use uptime-kuma[1] with notifications sent out through the included Apprise integration[2] 1. https://github.com/louislam/uptime-kuma https://github.com/louislam/uptime-kuma 2. https://github.com/caronc/apprise https://github.com/caronc/apprise
Jedd 1y ago
> Let's Encrypt stopped its certificate expiration email notification service a while ago, and I hadn't found a replacement yet. This sounds like an easy problem to identify root cause for. I think I received about 15 'we're disabling email notifications soon' emails over the past several months - one of which was interesting, but none were needed, as I'd originally set this up, per documentation, to auto-renew every 30 days. Perhaps create a calendar reminder for the short term?
superkuh 1y ago
If it's a personal website you should consider HTTP+HTTPS. It offers the best of both worlds and your website would always be accessible even if some third party CA is not (or if there's some local issue, or if the HTTP client connecting has cert issues). MITM attacks on personal websites are extremely, extremely rare.
zerof1l 1y ago
Because you're not supposed to rely on emails. You should have an automated certificate renewal in place. I'm under the impression that Let's Encrypt wants to reduce certificate validity even further from the current 90 days.
attentive 1y ago
Monitoring the health of your site is your job. You should have it on auto-renewal anyway. You can grab a cert from ZeroSSL and probably some others. You can also get 1year cert from aws for like $15 though I'd stick with auto renews.
- cpach 1y ago
  And quite soon, 1-year certs will not be available from any vendor at all: https://news.ycombinator.com/item?id=44643826 https://news.ycombinator.com/item?id=44643826

by Kholin 1y ago

compumike 1y ago

Oof, you're right, that's rough that it's so soon after they discontinued their email service! I wrote this blog post a few weeks ago: "Minimal, cron-ready scripts in Bash, Python, Ruby, Node.js (JavaScript), Go, and Powershell to check when your website's SSL certificate expires." https://heiioncall.com/blog/barebone-scripts-to-check-ssl-certificate-expiration https://heiioncall.com/blog/barebone-scripts-to-check-ssl-ce... which may be helpful if you want to roll your own. (Disclosure: at Heii On-Call we also offer free SSL certificate expiration monitoring, among other things.)

mixdup 1y ago

They have been communicating the ending of the email notices for quite a while and have been telling users that you should have some other monitoring in place to avoid just this situation

CamperBob2 1y ago
Also, beware of the leopard.
- jojobas 1y ago
  If you didn't see their sunset notification emails you wouldn't have seen your cert expiration email either.
andrewmcwatters 1y ago
Yes, but what’s weird is the recommended service they referred people to for new email notifications was not… sending me emails. So, what gives?
- kevincox 1y ago
  Yeah the recommended service is awful and not nearly as useful as the one they had is. Which is disappointing because you should be able to recreate the service they had nearly exactly with certificate transparency logs.
- deleted 1y ago
  [deleted]

yjftsjthsd-h 1y ago

> As a result, I didn't receive an expiration notice this time and failed to renew my certificate in advance. Shouldn't that happen automatically a bit beforehand?

Kholin 1y ago
Due to some legacy reasons, my service runs using a docker + nginx setup. However, certbot was initially used in its native nginx mode to generate the certificate, which prevented it from auto-renewing. I later switched it to standalone mode, but I'm not sure if I configured the auto-renewal correctly. In any case, the certificate happened to expire today, and it didn't renew automatically. On a side note, I was actually planning to see what an expired website certificate looked like first and then deal with the auto-renewal issue. After all, it's just a small hobby website, so it's not that big of a deal.
- dylan604 1y ago
  that sounds like a you're holding it wrong type of a situation to me. a major point of Let's Encrypt (besides the obvious free) is that it deliberately keeps the cert times short to avoid the "someone that no longer works here set this up two years ago" type of situation with certbot checking twice a day and updating when necessary. so to break what Let's Encrypt is doing with not using certbot definitely feels like you're holding it wrong

mattbillenstein 1y ago

I run https://ismycertexpired.com/ https://ismycertexpired.com/ - you can sign up for email alerts.

mr_toad 1y ago

Isn’t the recommended practice to update every ~60 days or so, regardless?

cpach 1y ago
Either that, or use an ACME client that has support for ARI so that the CA can signal to the client when it’s time to renew. https://letsencrypt.org/2024/04/25/guide-to-integrating-ari-into-existing-acme-clients/ https://letsencrypt.org/2024/04/25/guide-to-integrating-ari-...
mholt 1y ago
Yeah but if you need a cert for a new site you have to use another CA.

tbrownaw 1y ago

Haven't they always, from day one, insisted that their primary goal was to encourage (force) automation of certificate maintenance, as a mechanism to make tls ubiquitous (mandatory everywhere)?

ffsm8 1y ago
Yes, we had lengthy discussions in itops (I had a admin role when LE was launched) about it. The team lead couldn't get over the slogan "devops, automating downtimes since 2010" whenever someone wanted to add a new nonessential automation that does things on prod servers. I mean he wasn't completely wrong, it was a non essential automation with high risk and very little reward (<1h saved every 2 yrs), which is why we never switched to LE for our main site, only internal tooling was allowed to use it
- cpach 1y ago
  Perhaps you know this already but in the future, certs issued by a “real” CA will not be allowed to live for more than 47 days. https://www.digicert.com/blog/tls-certificate-lifetimes-will-officially-reduce-to-47-days https://www.digicert.com/blog/tls-certificate-lifetimes-will...
  - ffsm8 1y ago
    I didn't know that, interesting. I was merely retelling an anecdote about how LE was always positioned to be exclusively about refreshing certs automatically, though. As I've moved out of (dev-)ops roles around 2016/2017 so I'm really not up to date with operations topics
throw0101d 1y ago
> Haven't they always, from day one, insisted that their primary goal was to encourage (force) automation of certificate maintenance, as a mechanism to make tls ubiquitous (mandatory everywhere)? And? Automation sometimes breaks, both for internal reasons (OS patching) or external. For the latter, LE at some point in the past changed CDNs, and this caused JWST headers to be sent back differently, which broke different clients, e.g.: * https://community.letsencrypt.org/t/jws-has-no-anti-replay-nonce/103324 https://community.letsencrypt.org/t/jws-has-no-anti-replay-n... * https://github.com/dehydrated-io/dehydrated/issues/684 https://github.com/dehydrated-io/dehydrated/issues/684 Being able to get e-mails was an extra level of monitoring that was handy, even if you had automation.

rfv6723 1y ago

I use self-hosted gatus to monitor my certs and other services' status. It can send alerts to multiple alerting providers. https://github.com/TwiN/gatus https://github.com/TwiN/gatus

hoherd 1y ago
I use uptime-kuma[1] with notifications sent out through the included Apprise integration[2] 1. https://github.com/louislam/uptime-kuma https://github.com/louislam/uptime-kuma 2. https://github.com/caronc/apprise https://github.com/caronc/apprise

Jedd 1y ago

> Let's Encrypt stopped its certificate expiration email notification service a while ago, and I hadn't found a replacement yet. This sounds like an easy problem to identify root cause for. I think I received about 15 'we're disabling email notifications soon' emails over the past several months - one of which was interesting, but none were needed, as I'd originally set this up, per documentation, to auto-renew every 30 days. Perhaps create a calendar reminder for the short term?

superkuh 1y ago

If it's a personal website you should consider HTTP+HTTPS. It offers the best of both worlds and your website would always be accessible even if some third party CA is not (or if there's some local issue, or if the HTTP client connecting has cert issues). MITM attacks on personal websites are extremely, extremely rare.

zerof1l 1y ago

Because you're not supposed to rely on emails. You should have an automated certificate renewal in place. I'm under the impression that Let's Encrypt wants to reduce certificate validity even further from the current 90 days.

attentive 1y ago

Monitoring the health of your site is your job. You should have it on auto-renewal anyway. You can grab a cert from ZeroSSL and probably some others. You can also get 1year cert from aws for like $15 though I'd stick with auto renews.

cpach 1y ago
And quite soon, 1-year certs will not be available from any vendor at all: https://news.ycombinator.com/item?id=44643826 https://news.ycombinator.com/item?id=44643826