15 ms·
Intent to end OCSP service
- arsome 2y agoDisappointing to hear considering the limitations of CRLs - is there any intention to go forward with OCSP stapling or is that completely abandoned at this point?
- tialaramex 2y agoMy understanding is that stapling is the victim of the usual incompetence and laziness that infects a lot of systems where if one in a billion fail closed that would be considered a disaster but one in ten fail open is considered fine. You can't achieve meaningful security this way. The browser vendors have learned that you have to do it yourself or it won't be done well enough to be useful. So you pull every CRL, do a bunch of compression or other tricks, then give your users that data and now they have working revocation. When Bob's CA and Kebab Shop breaks their revocation stack, instead of dozens of poor individual users or web site owners confused and calling Bob's outsourced call centre in Pakistan with no sign of a fix, now a Google account exec asks Bob's CTO whether they forgot to say they were getting out of the CA business... I agree this isn't a desirable outcome, but it might be all we have.
- eqvinox 2y ago> The browser vendors have learned that you have to do it yourself Cool. We already got the internet ossified on TCP + UDP, other L4 protocols just get stuck in firewalls and whatnot. Now we're progressing in ossification of HTTP. <insert expletives here> To be clear: this OCSP decision seems to be driven directly and only by web/HTTP consumers. Anything else is just not considered.
- tialaramex 2y agoIt is called the Web PKI after all. If somebody else actually wants to do all the hard work they're welcome, but my impression is that there's only enthusiasm for bitching and whining which won't get the work done.
- eqvinox 2y ago> It is called the Web PKI after all. Let's Encrypt issues X.509 certificates, not "Web PKI certificates".
- mcpherrinm 2y agox.509 is a certificate format used by many PKIs. Let's Encrypt is a CA that is part of the WebPKI. We follow standards set by the CA/B forum, undergo WebTrust Audits, and are accepted into the root programs run by the browser vendors (Primarily: Apple, Mozilla, Microsoft, and Chrome). That is the WebPKI.
- eqvinox 2y agoAre you implying Let's Encrypt certificates should not be used for non-HTTP services?
- nickf 2y agoWhat non-HTTP services need publicly-trusted certificates and care about revocation?
- formerly_proven 2y agomail
- nickf 2y agoLike SMTP/IMAP etc? That would make sense, though I'm not sure how much revocation checking even happens there.
- eqvinox 2y agoOCSP stapling: free feature of TLS library, works OCSP must-staple: free feature of TLS library, works plain OCSP: hit & miss, depends on the client software using the TLS library correctly CRL: no. … that's the crux of this entire thread.
- c0balt 2y agoInteresting, how does the support for CRLs in web servers look? Afaik, NGINX and Apache only have OCSP stapling support.
- c0balt 2y agoNote: for nginx there is https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_crl https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_... which needs to be pointed to a pem-encoded file with the CRL list (list of revoked certs). So no API-based configuration, like OCSP stapling, that just works. I can probably try to configure this with a cronjob/systemd timer but this is significantly less ergonomic.
- Arnavion 2y agoYou were previously using OCSP stapling for your server cert. The CRLs containing your server cert have nothing to do with your server. The server config you have found is for nginx to verify client certs.
- Asmod4n 2y agoWhy would CRL require web server support?
- iso8859-1 2y agoSo that admins find out that the certificate they have configured is no longer valid.
- Asmod4n 2y agoYou’d need a monitoring solution for that, your web server won’t just send you a mail when it’s cert got revoked. The client does that for you, by checking back at the CA. No need for any configuration server side.
- mcpherrinm 2y agoFor webservers, our recommendation is that your ACME client queries the ACME Renewal Information endpoint to find out if the certificate needs to be renewed: https://letsencrypt.org/2023/03/23/improving-resliiency-and-reliability-with-ari.html https://letsencrypt.org/2023/03/23/improving-resliiency-and-...
- codegeek 2y agoCan someone ELI5 what does this mean for people using LetsEncrypt today with servers like Nginx or Caddy ? Do we need to make any changes to adjust ?
- mcpherrinm 2y agoFor regular webserver users, accessed by web browsers, no changes are needed. Note this is still a long ways out. At this time, only people who are writing code against OCSP need to be aware of the future roadmap.
- candiddevmike 2y agoThis mostly only impacts verification of Lets Encrypt certificates. No changes for ACME users.
- eqvinox 2y agoI… er… what? First of all, privacy was one of the points of OCSP stapling. Second, this breaks all non-http applications in the sense that they could previously work through OCSP stapling which would be communicated in-line. CRLs need to be fetched by the client, generally over HTTP. Third, most non-browser TLS clients simply do not fetch CRLs, the implementation hurdle was too high. … I'm left seriously befuddled by this decision by Let's Encrypt [edit: or rather the CA/B forum] :(
- deleted 2y ago[deleted]
- andrewaylett 2y agoOCSP pretty much has to fail open, and stapling doesn't fix this. If you're in a position to MITM a client using an exfiltrated certificate that no longer passes OSCP, you're probably also in a position to block the request to the OSCP server. And you're not going to staple while you MITM. As a client, you can't really tell the difference between "this certificate is valid but not stapled and the OSCP server is down" and "this certificate isn't stapled because I'm being MITM'd, and the OSCP server is blocked". For those who could successfully staple, really short-lived certificates might be a suitable answer -- that's effectively what OSCP gave you, only without actually ensuring that the certificate would cleanly expire.
- mhils 2y agoThere's OCSP Must-Staple, which makes MITM without stapling impossible. That is, if the client implements it and does not fail open. :)
- thayne 2y agoOCSP has problems too. Al ot of non-browser applications just don't bother checking revocation. Which is terrible, but there aren't currently any good options. The best solution would be making cert lifetimes short enough that you don't need revocation, or making OCSP stapling, or something similar mandatory. But rolling either of those out across the whole internet would be slow and difficult.
- datadrivenangel 2y ago"As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse." Interesting to see Microsoft dragging here.
- nick238 2y agoI don't see anything about OCSP policy changing in the Mozilla CA Program Policy release notes [1] or Chrome's [2], so I wonder if Microsoft was more proactive (for better or worse) in requiring it vs. other popular root programs. [1]: https://wiki.mozilla.org/CA/Root_Store_Policy_Archive https://wiki.mozilla.org/CA/Root_Store_Policy_Archive [2]: https://www.chromium.org/Home/chromium-security/root-ca-policy/ https://www.chromium.org/Home/chromium-security/root-ca-poli...
- agwa 2y agoMozilla and Chrome were requiring it via the incorporation of the Baseline Requirements into their policies. The removal of the requirement from the BRs won't appear in the release notes of their policies.
- mcpherrinm 2y agoMicrosoft has always had stricter requirements for OCSP than the baseline requirements, which the other root programs have not had. With the CA/Browser forum baseline requirements amended to no longer require OCSP, it is Microsoft's requirements alone that require OCSP.
- agwa 2y agoAfter I created OCSP Watch[1], I regularly detected CAs returning an OCSP response of unknown or unauthorized for certificates found in CT logs, indicating that they had basically forgotten that they had issued a certificate. I find that rather troubling. Indeed, OCSP Watch is currently reporting several forgotten NETLOCK certificates. (The certificates from other CAs are recently issued and will probably have OCSP responses provisioned in the near future.) CRLs can't be used to detect this, because they only list revoked certificates rather than providing a definitive status for every issued certificate. I do wish the root programs had merely removed the requirement to include an OCSP URL in certificates, but required OCSP URLs for every issuer to be disclosed in the CCADB. That would have solved the privacy problems and made OCSP responders much cheaper to operate, while continuing to provide transparency into the status of certificates. [1] https://sslmate.com/labs/ocsp_watch/ https://sslmate.com/labs/ocsp_watch/
- nine_k 2y agoBTW what is the need for a CA to remember issued certificates? Certificates should work as long as they are not expired, not on a CRL, and while the same applies to their.CA's certificate used to sign it. The only good use for remembering issued certificates that I see is that a CA could detect a compromise if it's handed a certificate it does not remember issuing. It looks far-fetched to me, but I know nothing about CA operation.
- phasmantistes 2y agoCAs need to know every certificate they've issued so that a) if they receive a request to revoke that certificate, they can actually do so; and b) if they need to revoke all of their certs (e.g. due to discovering a validation process failure) they don't miss any.
- masklinn 2y agoIsn't it more "a lot" of certs? Surely if they need to revoke all certs they can just revoke the intermediate.
- trillic 2y agoAre there any free or very inexpensive certificate providers which support ACME, DNS-01 challenges, and OCSP other than ZeroSSL?
- psz 2y agoacme.sh has a list in their Wiki: https://github.com/acmesh-official/acme.sh/wiki/CA https://github.com/acmesh-official/acme.sh/wiki/CA
- hlieberman 2y agoWhy not simply add the Must Staple restriction unconditionally to all certificates (aka "status_request")? That will eliminate privacy concerns -- no compliant TLS implementation should fetch a OCSP ticket given a stapled response -- while still allowing for broad non-browser support.
- blahgeek 2y agoI don't get the OCSP Stampling protocol: if the server needs / is able to to contact the CA frequently anyway, why not simply use very short-lived certificates?
- eqvinox 2y agoIssuing a certificate has significantly more overhead (e.g. Certificate Transparency logs) than signing a statement that an existing certificate is still valid.
- briansmith 2y agoWhich CA's will issue short-lived certificates without negotiating a custom ($$$) contract with them?
- 2y ago
- Sohcahtoa82 2y agoI'm curious how much of a factor this is. How often do certificates get revoked before expiration? I would think the only reason to revoke a certificate is if it was compromised.
- agwa 2y agoCurrently only 20,830,034 revoked certificates per my daily CRL download, versus ~1 billion active certificates. But the number of revoked certs could balloon if there's another Heartbleed-like event, or a mass misissuance.
- mintplant 2y agoSo are embedded devices just expected to not check for certificate revocation?
- BertoldVdb 2y agoIf you can tolerate some level of false positives you can turn the CRL into a bloom filter and save quite a bit of storage.
- kibwen 2y agoI don't think this comment deserves to be downvoted. For the specific constraints imposed by embedded environments, a bloom filter seems perfectly cromulent here. The odds of a false positive scale according to the amount of memory that you can afford, and even in the event of a false positive it would fail closed, which is a better security posture than not checking at all.
- less_less 2y agoDepends how embedded and how many servers it's talking to. I'm also not sure you can really tolerate false positives, since the main domain you talk to might get revoked, but there are alternatives to Bloom filters that eliminate those. In any case it's going to be hard to go below about 0.8-1 byte per revoked cert (common case where most certs aren't revoked) even with a fancy data structure. You will surely want to shard the CRL, so if the device only accesses a few servers (and thus only has a few shards) then it'll be fine, but if it accesses many servers then there will be trouble, and you'll want to reserve extra space in case of a mass revocation.
- eduction 2y agoIt would be nice if one didn't need to be a TLS expert to understand the post -- particularly since the whole point of Let's Encrypt was to democratize TLS access. I have no idea if this means my setup will break even after consulting the docs of my ACME client. Can I still use ACME Tiny[1] with nginx? Any reason to think that will break? How can I tell if I'm using OCSP or CRL? Totally incomprehensible blog post. [1] https://github.com/diafygi/acme-tiny https://github.com/diafygi/acme-tiny
- mcpherrinm 2y agoFor regular webserver users, accessed by web browsers, no changes are needed. Note this is probably at least a year, if not more, away. I'm sorry this post wasn't accessible enough, and we'll have more communications in the coming years as this gets closer. (I work at Let's Encrypt and proofread this post before publishing)
- phasmantistes 2y agoNothing will change for you, and nothing will break. The point of this post is to give a maximum-lead-time heads-up to the folks who _do_ need to care (the folks writing revocation-checking code in clients) so that later, more specific announcements don't come as a surprise.
- jrochkind1 2y agoletsencrypt is the nonprofit community-focused infrastruture that we imagined the internet would consist of 20 years ago. I love you letsencrypt.
- klysm 2y agoCertificate management is an interesting problem along the intersection of human behavior and computer science that feels similar to BGP. In theory, it's simple, but when met with reality things get messy really really fast.
- remram 2y agoSo if I'm not using Chrome or Firefox, how do I check for revoked certificates? This just makes the web less open again.
- agwa 2y agoTo check if a certificate is revoked, you can look up its issuer in the CCADB (https://ccadb.my.salesforce-sites.com/ccadb/AllCertificateRecordsCSVFormatv2 https://ccadb.my.salesforce-sites.com/ccadb/AllCertificateRe...) to find the URL(s) of its CRL(s), and then check the CRL(s) for the serial number. It's true this isn't an IETF standard (sometimes the IETF standards just don't work in practice) but the data is still very much in the open.
- mcpherrinm 2y agoLet’s Encrypt will put CRL URLs in certificates before or when removing OCSP URLs, so no CCADB lookup should be required
- agwa 2y agoIndeed! I forgot that the BRs mandate the CRLDP extension if the certificate lacks OCSP AIA. So all (non-short-lived) certificates will continue to have a standard revocation checking mechanism encoded in them.
- remram 2y agoSo my client should download the 100 MB CRL to check? Is that the secure workflow?
- warhorse10_9 2y agoModern crls are partitioned so this really isn’t a problem you will run into.
- remram 2y ago
- notepad0x90 2y agoCRLs don't scale, that's their problem and they take too long to update. Why isn't there a standard binary format for CRLs that is a cuckoo filter or a similar data structure? That way you don't have to worry about a CRL ballooning to gigabytes and you can expect clients to fetch the latest binary blob frequently.
- creatonez 2y ago> Why isn't there a standard binary format for CRLs that is a cuckoo filter or a similar data structure? We're getting there -- browser-summarized CRLs are available in Chrome and Firefox now - https://letsencrypt.org/2022/09/07/new-life-for-crls.html https://letsencrypt.org/2022/09/07/new-life-for-crls.html - https://blog.mozilla.org/security/2020/01/09/crlite-part-1-all-web-pki-revocations-compressed/ https://blog.mozilla.org/security/2020/01/09/crlite-part-1-a... - https://www.imperialviolet.org/2012/02/05/crlsets.html https://www.imperialviolet.org/2012/02/05/crlsets.html
- notepad0x90 2y agoNeat! and thanks for the response.
- less_less 2y agoAs a side project I also developed something like CRLite, but approximately 40% smaller: https://docs.rs/compressed_map/latest/compressed_map/ https://docs.rs/compressed_map/latest/compressed_map/ It's probably not state-of-the-art anymore, but it gets fairly close to the Shannon entropy of the {revoked, not revoked} distribution, supports non-Boolean types (like revocation reason), and is basically fast enough on both compression and queries. The main missing functionality is incremental update. To implement daily updates you'd just compress "which certs were revoked today", and to check a 50-day-old cert you'd do 50 queries. (The queries are really fast tho, so that's probably fine.)
- newman314 2y agoDoes anyone have a ready to go hosts blocklist for OCSP hostnames? Else, I could make one...
- nickf 2y agoYou could start with this: https://crt.sh/ocsp-responders https://crt.sh/ocsp-responders
- throwaway290232 2y ago[dead]
- kroeckx 2y agoFor the major browsers, this probably makes little difference, but for anything else, this will most likely result in not verifying the revocation status of certificates anymore or making things slower. As far as I know, most browser vendors already download the CRLs, and then update the browsers based on what they downloaded. For instance firefox seems to be using CRLite. There is a lack of support for something like that in the non-major browsers and non-browsers. The alternative they have is to download the CRL instead of the OCSP reply, which is larger, probably making things slower. Or they could just not check the status, which is most likely what will happen. CRLite changes the failure mode of the status check, it no longer just ignores error in downloading the status information. We need better support for something like CRLite.
- dochtman 2y agoMaking something like CRLite more like a standard seems worthwhile. I looked at the Mozilla bits but AFAICT there’s not much if any documentation of the low-level bits and pieces.
- thayne 2y agoI did some research a while ago into ensuring up to date CRLs for a non-browser use case. Besides the problem of the massive size of CRLs, I couldn't find good tools for automatic updates across all trusted CRLs. My conclusion was that it isn't really practical unless you only trust one or two CAs.
- new23d 2y agoWe collected some data [1] on the viability of only CRLs as the future (phasing out OCSP) - motivated by Let's Encrypt's announcement today [2]. Data is on CRL availability, number of entries, expiry & refresh times, etc. from various x509 leaf server SSL certificates. [1] https://news.ycombinator.com/item?id=41058138 https://news.ycombinator.com/item?id=41058138 [2] https://news.ycombinator.com/item?id=41046956 https://news.ycombinator.com/item?id=41046956
- 7bit 2y agoHave we gone full circle? Wasn't the point of OCSP to provide a low-latency and low-data alternative to CRLs, which might become massively huge?