6 ms·

Last week's Let's Encrypt downtime

218 points by agwa 3y ago

bullen 3y ago
Use https://datatracker.ietf.org/doc/html/rfc2289 https://datatracker.ietf.org/doc/html/rfc2289 instead. And please don't down vote without comment.
ElongatedMusket 3y ago
Thanks for following through on this writeup! I knew LE certs were publicly logged but didn't know the logs were decentralized or how they hold the CA accountable. Appreciate the layman explanation.
mjw1007 3y ago
> these certificates were already being rejected by Chrome and Safari for having invalid SCTs What's a good way to make an equivalent check from a script, if I want (in future) to be able to check whether I have a website whose certificate has such a problem?
- agwa 3y ago
  Excellent question! The sctcheck command from https://github.com/google/certificate-transparency-go/ https://github.com/google/certificate-transparency-go/ can be used to check the signatures of the embedded SCTs in a certificate. I've also got an online tool which you can use to test a site for CT policy compliance: https://sslmate.com/labs/ct_policy_analyzer/ https://sslmate.com/labs/ct_policy_analyzer/ Example of a working site: https://sslmate.com/labs/ct_policy_analyzer/?sslmate.com https://sslmate.com/labs/ct_policy_analyzer/?sslmate.com Example of one of the sites affected by the Let's Encrypt incident: https://sslmate.com/labs/ct_policy_analyzer/?thecandyshake.com https://sslmate.com/labs/ct_policy_analyzer/?thecandyshake.c...
jimmyl02 3y ago
This is a great writeup and intro to certificate transparency overall. Glad to see that certificate authorities are being held accountable and learn more about how its done!
jrpelkonen 3y ago
> I find it alarming that a week after the incident, 40% of the affected certificates are still in use, despite being rejected by the most popular browsers and despite affected subscribers being emailed by Let's Encrypt. This is perhaps a consequence on how well-oiled of a machine LE typically is: people stop paying attention to it.
- tredre3 3y ago
  That is true but how come certbot had no awareness of revoked/withdrawn certificates before now? It seems like one of the things a CA is supposed to solve for you, and the fact that it doesn't is bit alarming in itself. Though, as the following sentence points out, they were already working on it before the outage, so clearly they knew it was needed. 1. https://datatracker.ietf.org/doc/draft-ietf-acme-ari/ https://datatracker.ietf.org/doc/draft-ietf-acme-ari/
  - 411111111111111 3y ago
    The CA can't solve it for you. The certificate authority signs certificate requests, creating certificates. The revocation process is necessary as well, but the CA doesn't have the ability to change the already issued certificate, thus it cannot take action. A software like certbot can solve it for you, but that's not affiliated with your CA
    - agwa 3y ago
      The CA is part of the solution by using ARI to inform ACME clients to replace impacted certificates.
      - mcpherrinm 3y ago
        Even before ARI, some integrated ACME/Web servers use OCSP as a way of knowing to renew if a cert was revoked. Plus if you're doing that you can pin the OCSP response while you're at it.
fruitreunion1 3y ago
Will non-browser clients like curl/requests ever support checking CT logs? It's great that some browsers have it, but browsers are not the only clients using TLS with CAs. Also doesn't help that a lot of software can't use CA root stores with much granularity: https://news.ycombinator.com/item?id=33876949 https://news.ycombinator.com/item?id=33876949
- agwa 3y ago
  Hopefully, although there are challenges to overcome. CT is a fast-moving ecosystem, with logs coming and going, and policies changing regularly. This requires CT-enforcing clients to be very on-the-ball with updates, both in the sense that the developers need to pay attention and update their code in time, and any users of the apps need to upgrade frequently. Browser makers can handle this because they are competently-staffed and well-resourced. The authors of non-browser apps need to know what they're getting into. A cautionary tale: there is a library for adding CT enforcement to Android apps. Earlier this year, every app using this library was suddenly unable to establish any TLS connections because Google stopped publishing a JSON file which the library should never have been consuming in the first place. There was plenty of warning that this would happen, but the author of the library was not on-the-ball. https://groups.google.com/g/certificate-transparency/c/38Lr9K46cCA/m/mtwe68NTHwAJ https://groups.google.com/g/certificate-transparency/c/38Lr9...
- NovemberWhiskey 3y ago
  The elephant in the room is that TLS implementations for browsers and those in the libraries of common programming languages have diverged really substantially: Web PKI is massively more restrictive and depends on a bunch of technology that's not in the baseline PKI.
AdamJacobMuller 3y ago
Did we kill crt.sh? FATAL: terminating connection due to conflict with recovery DETAIL: User query might have needed to see row versions that must be removed. CONTEXT: SQL statement "SELECT c.ID, x509_print(c.CERTIFICATE, NULL, 196608), ca.ID, cac.CA_ID, digest(c.CERTIFICATE, 'sha1'::text), digest(c.CERTIFICATE, 'sha256'::text), x509_serialNumber(c.CERTIFICATE), digest(x509_publicKey(c.CERTIFICATE), 'sha256'::text), x509_rsamodulus(c.CERTIFICATE), x509_hasROCAFingerprint(c.CERTIFICATE), x509_hasClosePrimes(c.CERTIFICATE), c.CERTIFICATE FROM certificate c LEFT OUTER JOIN ca ON (c.ISSUER_CA_ID = ca.ID) LEFT OUTER JOIN ca_certificate cac ON (c.ID = cac.CERTIFICATE_ID) WHERE digest(c.CERTIFICATE, 'sha256') = t_bytea" PL/pgSQL function web_apis(text,text[],text[]) line 1757 at SQL statement ERROR: server conn crashed? server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. and now it's just a 502 error!
- agwa 3y ago
  Unfortunately, crt.sh is chronically overloaded.
  - faangsticle 3y ago
    Are there other ways to get these logs?
    - AdamJacobMuller 3y ago
      I haven't found any, yet. I would love to have a list of domains affected by this to cross-check that none of my issued certificates were affected by this.
      - schoen 3y ago
francislavoie 3y ago
FWIW, if those websites used Caddy as their ACME client, then it would have detected the certificate being revoked as soon as possible via OCSP stapling and would have had the certificate renewed. It's a shame that other ACME clients aren't as robust to problems like this. (Disclaimer: I work on Caddy as a volunteer)
- agwa 3y ago
  Note that the certificates were not revoked until 2023-06-19 at 18:00. In contrast, ARI was updated on 2023-06-15 at 22:43 to tell ARI-supporting clients (such as lego) to renew immediately. That means Caddy served broken certificates for almost 4 days longer than necessary. Are there plans for Caddy to support ARI?
  - francislavoie 3y ago
    > Note that the certificates were not revoked until 2023-06-19 at 18:00. Ah okay, I missed that. > Are there plans for Caddy to support ARI? It's... complicated. Matt argues that ARI does not make sense for a variety of reasons. You can find the complex and deep discussions about it on the LE forums. Do a Ctrl+F for ARI in https://community.letsencrypt.org/c/client-dev/14 https://community.letsencrypt.org/c/client-dev/14 to find them, there's a lot.
  - mholt 3y ago
    > That means Caddy served broken certificates for almost 4 days longer than necessary. This would be news to me. Do you have a source for Caddy serving any of the affected certificates? I'd like as much info as possible. > Are there plans for Caddy to support ARI? If ARI can be made into an effective mechanism, then yes. ACMEz already supports the current draft. I know Francis linked to a forum category, here's some more specific links for background: - https://community.letsencrypt.org/t/can-ari-conforming-clients-be-granted-exemptions-to-relevant-rate-limits/195600 https://community.letsencrypt.org/t/can-ari-conforming-clien... - https://community.letsencrypt.org/t/thoughts-from-starting-to-play-with-ari/200276 https://community.letsencrypt.org/t/thoughts-from-starting-t...

218 points by agwa 3y ago

Use https://datatracker.ietf.org/doc/html/rfc2289 https://datatracker.ietf.org/doc/html/rfc2289 instead. And please don't down vote without comment.

ElongatedMusket 3y ago

Thanks for following through on this writeup! I knew LE certs were publicly logged but didn't know the logs were decentralized or how they hold the CA accountable. Appreciate the layman explanation.

mjw1007 3y ago

> these certificates were already being rejected by Chrome and Safari for having invalid SCTs What's a good way to make an equivalent check from a script, if I want (in future) to be able to check whether I have a website whose certificate has such a problem?

agwa 3y ago
Excellent question! The sctcheck command from https://github.com/google/certificate-transparency-go/ https://github.com/google/certificate-transparency-go/ can be used to check the signatures of the embedded SCTs in a certificate. I've also got an online tool which you can use to test a site for CT policy compliance: https://sslmate.com/labs/ct_policy_analyzer/ https://sslmate.com/labs/ct_policy_analyzer/ Example of a working site: https://sslmate.com/labs/ct_policy_analyzer/?sslmate.com https://sslmate.com/labs/ct_policy_analyzer/?sslmate.com Example of one of the sites affected by the Let's Encrypt incident: https://sslmate.com/labs/ct_policy_analyzer/?thecandyshake.com https://sslmate.com/labs/ct_policy_analyzer/?thecandyshake.c...

jimmyl02 3y ago

This is a great writeup and intro to certificate transparency overall. Glad to see that certificate authorities are being held accountable and learn more about how its done!

jrpelkonen 3y ago

> I find it alarming that a week after the incident, 40% of the affected certificates are still in use, despite being rejected by the most popular browsers and despite affected subscribers being emailed by Let's Encrypt. This is perhaps a consequence on how well-oiled of a machine LE typically is: people stop paying attention to it.

tredre3 3y ago
That is true but how come certbot had no awareness of revoked/withdrawn certificates before now? It seems like one of the things a CA is supposed to solve for you, and the fact that it doesn't is bit alarming in itself. Though, as the following sentence points out, they were already working on it before the outage, so clearly they knew it was needed. 1. https://datatracker.ietf.org/doc/draft-ietf-acme-ari/ https://datatracker.ietf.org/doc/draft-ietf-acme-ari/
- 411111111111111 3y ago
  The CA can't solve it for you. The certificate authority signs certificate requests, creating certificates. The revocation process is necessary as well, but the CA doesn't have the ability to change the already issued certificate, thus it cannot take action. A software like certbot can solve it for you, but that's not affiliated with your CA
  - agwa 3y ago
    The CA is part of the solution by using ARI to inform ACME clients to replace impacted certificates.
    - mcpherrinm 3y ago
      Even before ARI, some integrated ACME/Web servers use OCSP as a way of knowing to renew if a cert was revoked. Plus if you're doing that you can pin the OCSP response while you're at it.

fruitreunion1 3y ago

Will non-browser clients like curl/requests ever support checking CT logs? It's great that some browsers have it, but browsers are not the only clients using TLS with CAs. Also doesn't help that a lot of software can't use CA root stores with much granularity: https://news.ycombinator.com/item?id=33876949 https://news.ycombinator.com/item?id=33876949

agwa 3y ago
Hopefully, although there are challenges to overcome. CT is a fast-moving ecosystem, with logs coming and going, and policies changing regularly. This requires CT-enforcing clients to be very on-the-ball with updates, both in the sense that the developers need to pay attention and update their code in time, and any users of the apps need to upgrade frequently. Browser makers can handle this because they are competently-staffed and well-resourced. The authors of non-browser apps need to know what they're getting into. A cautionary tale: there is a library for adding CT enforcement to Android apps. Earlier this year, every app using this library was suddenly unable to establish any TLS connections because Google stopped publishing a JSON file which the library should never have been consuming in the first place. There was plenty of warning that this would happen, but the author of the library was not on-the-ball. https://groups.google.com/g/certificate-transparency/c/38Lr9K46cCA/m/mtwe68NTHwAJ https://groups.google.com/g/certificate-transparency/c/38Lr9...
NovemberWhiskey 3y ago
The elephant in the room is that TLS implementations for browsers and those in the libraries of common programming languages have diverged really substantially: Web PKI is massively more restrictive and depends on a bunch of technology that's not in the baseline PKI.

AdamJacobMuller 3y ago

Did we kill crt.sh? FATAL: terminating connection due to conflict with recovery DETAIL: User query might have needed to see row versions that must be removed. CONTEXT: SQL statement "SELECT c.ID, x509_print(c.CERTIFICATE, NULL, 196608), ca.ID, cac.CA_ID, digest(c.CERTIFICATE, 'sha1'::text), digest(c.CERTIFICATE, 'sha256'::text), x509_serialNumber(c.CERTIFICATE), digest(x509_publicKey(c.CERTIFICATE), 'sha256'::text), x509_rsamodulus(c.CERTIFICATE), x509_hasROCAFingerprint(c.CERTIFICATE), x509_hasClosePrimes(c.CERTIFICATE), c.CERTIFICATE FROM certificate c LEFT OUTER JOIN ca ON (c.ISSUER_CA_ID = ca.ID) LEFT OUTER JOIN ca_certificate cac ON (c.ID = cac.CERTIFICATE_ID) WHERE digest(c.CERTIFICATE, 'sha256') = t_bytea" PL/pgSQL function web_apis(text,text[],text[]) line 1757 at SQL statement ERROR: server conn crashed? server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. and now it's just a 502 error!

agwa 3y ago
Unfortunately, crt.sh is chronically overloaded.
- faangsticle 3y ago
  Are there other ways to get these logs?
  - AdamJacobMuller 3y ago
    I haven't found any, yet. I would love to have a list of domains affected by this to cross-check that none of my issued certificates were affected by this.
    - schoen 3y ago

francislavoie 3y ago

FWIW, if those websites used Caddy as their ACME client, then it would have detected the certificate being revoked as soon as possible via OCSP stapling and would have had the certificate renewed. It's a shame that other ACME clients aren't as robust to problems like this. (Disclaimer: I work on Caddy as a volunteer)

agwa 3y ago
Note that the certificates were not revoked until 2023-06-19 at 18:00. In contrast, ARI was updated on 2023-06-15 at 22:43 to tell ARI-supporting clients (such as lego) to renew immediately. That means Caddy served broken certificates for almost 4 days longer than necessary. Are there plans for Caddy to support ARI?
- francislavoie 3y ago
  > Note that the certificates were not revoked until 2023-06-19 at 18:00. Ah okay, I missed that. > Are there plans for Caddy to support ARI? It's... complicated. Matt argues that ARI does not make sense for a variety of reasons. You can find the complex and deep discussions about it on the LE forums. Do a Ctrl+F for ARI in https://community.letsencrypt.org/c/client-dev/14 https://community.letsencrypt.org/c/client-dev/14 to find them, there's a lot.
- mholt 3y ago
  > That means Caddy served broken certificates for almost 4 days longer than necessary. This would be news to me. Do you have a source for Caddy serving any of the affected certificates? I'd like as much info as possible. > Are there plans for Caddy to support ARI? If ARI can be made into an effective mechanism, then yes. ACMEz already supports the current draft. I know Francis linked to a forum category, here's some more specific links for background: - https://community.letsencrypt.org/t/can-ari-conforming-clients-be-granted-exemptions-to-relevant-rate-limits/195600 https://community.letsencrypt.org/t/can-ari-conforming-clien... - https://community.letsencrypt.org/t/thoughts-from-starting-to-play-with-ari/200276 https://community.letsencrypt.org/t/thoughts-from-starting-t...