21 ms·

Shortening the Let's Encrypt chain of trust

544 points by healsdata 3y ago

rafaelturk 3y ago
TL:DR: If your webste clientes use Android 7.0 or earlier (released 2016), you may need to take action to ensure you can still access websites secured by Let’s Encrypt certificates. By Thursday, June 6th, 2024.
- rafaelturk 3y ago
  I don't know about you, but looks like this will cause some impact.
  - berbec 3y ago
    By the article's infographic, it's 4.6% of Android users. Anyone not using FireFox Mobile on Android 7.0 or earlier. Some, but not major.
    - lesuorac 3y ago
      3 billion seems like decent number to use for how many Android users there are [1]. 4.5% of 3 billion is 135 million or about 13x more than Google Domains [2] so I guess not major indeed. [1]: https://www.google.com/search?q=number+of+android+users https://www.google.com/search?q=number+of+android+users [2]: https://www.theverge.com/2023/6/16/23763340/google-domains-sunset-sell-squarespace https://www.theverge.com/2023/6/16/23763340/google-domains-s...
    - deleted 3y ago
      [deleted]
    - morepork 3y ago
      6.1%, those using 7.0 and older will be affected. But that number will have fallen a bit by the time the certs actually expire in a year.
    - quickthrower2 3y ago
      I wonder how skewed to poorer countries that is? Also I imagine some skew towards smart tvs (since they outlive older phones, you could easily have a 2013 smart tv still working, I had a 2009 dumb TV until recently and sold it on).
- morepork 3y ago
  There's also the possibility that Chrome decides to bundle this CA like Firefox mobile does, which would mean the majority of users won't have a problem. From 5.0 the builtin webview is updatable too - I think it uses Chrome's engine. That leaves other browser's such as Samsung's one which probably has the most usage, no idea if they bundle root CAs or not.
compumike 3y ago
Is there any proposed solution (other than just using unencrypted HTTP) that would make it so TLS is no longer the most brittle component of the web? The constant churn of protocol deprecations, certificate expirations, rotations, etc. is like planned obsolescence on steroids.
- woodruffw 3y ago
  I don’t think TLS is the most brittle component of the web. That award probably goes to DNS, BGP, or (depending on how you qualify it) us-east-1.
  - lbotos 3y ago
    Whats brittle about DNS? Caching? BGP and us-east-1 fair :D
    - bawolff 3y ago
      DNSSec is pretty brittle. I think this is a very different definition of brittle than the original poster meant
      - tptacek 3y ago
        Yeah, but almost nobody uses it (it's got low single digits uptake in the US, and has actually decline in some years), so that's not the part of DNS that's making it brittle for deployments.
      - talideon 3y ago
        Describing DNSSEC as brittle is the height of understatement.
        AdamJacobMuller 3y ago
        Brittle, painful and with a tiny minority who scream at you for being stupid because you're not using it and complain about these problems.
bawolff 3y ago
> Finally, it will significantly reduce our operating costs, allowing us to focus our funding on continuing to improve your privacy and security. Does that imply they are paying millions or something for the cross-sign?
- nmjohn 3y ago
  Based on their 2021 (most recent year available) Form 990 (nonprofit public tax filing) [0] they paid Identrust $434,000 for "Internet Services." Not sure if they are getting more than just the cross-sign from Identrust - but it seems likely that may just be what they are paying for the cross-sign. In that same year, their total expenses were $5.1M - so that expense would make up almost 10% of their budget. [0]: https://beta.candid.org/profile/9328188?keyword=46-3344200&action=Search https://beta.candid.org/profile/9328188?keyword=46-3344200&a...
  - bawolff 3y ago
    Wow. Traditional CAs are such a rent seeking business :(
    - jaas 3y ago
      As the person who negotiated the agreements between Let's Encrypt and Identrust I can tell you that they have provided valuable services, including but not limited to cross-signs. I would not describe it as rent seeking. We are sincerely glad to have them as partners, and grateful for their contributions to helping get Let's Encrypt going. We could not have done what we did without them. Running a publicly trusted CA is not easy, and cross-signing others involves work and liability, particularly if the entity asking for a cross-sign is an upstart with a strange plan and little to no experience running a CA.
    - profmonocle 3y ago
      Cross-signing a CA is many orders of magnitude more work than signing a single domain leaf cert. Sure, on a technical level the result is similar - a signed X.509 cert, just with the "CA" flag set to true, but it's a very different proposition. Imagine if a CA cross-signed some new, upstart CA to get them browser compatibility (like IdenTrust did for LE), and then the new upstart went rogue and started issuing phony certs for google.com, wikipedia.org, etc. on behalf of [insert totalitarian nation here] state security. Those certs would chain up to the cross-signer's root, and they're responsible for it. They could face removal from root programs if they were reckless about cross-signatures. So if a root CA wants to cross-sign a new CA, they need to make sure that the new CA follows the same policies and gets the same audits as a root CA, because their ability to break things will be basically equivalent to a root CA. Honestly, <$500k for all the admin work on this sounds reasonable to me. It probably took a huge portion of several people's time throughout the year.
throw0101a 3y ago
Meta-ish: to attain coverage from 95% of Android devices, you have to support 7.0 ("Nougat"), which dates back to August 2016 (~7 years ago): * https://en.wikipedia.org/wiki/Android_Nougat https://en.wikipedia.org/wiki/Android_Nougat To attain coverage of 95% of iOS devices, you have to support iOS 14 which dates back to September 2020 (~3 years ago): * https://iosref.com/ios-usage https://iosref.com/ios-usage * https://en.wikipedia.org/wiki/IOS_14 https://en.wikipedia.org/wiki/IOS_14 Even for 'only' 90% coverage, it's 8.1 (2017) versus iOS 15 (2021). Seems like Apple is doing a better job of convincing / allowing folks to move to a more recent version of their operating system.
- paxys 3y ago
  > Seems like Apple is doing a better job of convincing / allowing folks to move to a more recent version of their operating system. Apple isn't selling $10 phones in developing countries. If you compare phones at the same price points from major manufacturers/carriers the difference won't be nearly as drastic.
  - redox99 3y ago
    Meh. Samsung (and Huawei) are still the best selling phones in developing countries. Samsung could definitely both reduce the unnecessarily large number of SKUs, and better unify their software stack so they could update for 5 years all of their phones. It's crazy how we went from PCs having the right abstractions, where you can easily keep a 10 year old PC updated, to this awful Android situation.
    - paxys 3y ago
      They are not. Samsung is #1 but after that there is a very long tail of Android smartphone manufacturers that most people in western countries haven't even heard of. Xiaomi, Oppo and Vivo for example all have double digit market shares globally, and combined sell more than Samsung. Realme and ZTE have >5%. The "others" section in most graphs is >30%. If you narrow down the price to say <$100 there is an even larger spread. Expecting continued multi-year support from manufacturers and carriers is impossible at this range when the sole focus is on driving down the price and nothing else.
8organicbits 3y ago
The way they got that old cross sign to keep working was quite interesting: > The new cross-sign will be somewhat novel because it extends beyond the expiration of DST Root CA X3. This solution works because Android intentionally does not enforce the expiration dates of certificates used as trust anchors. [1] Trust anchors work really differently than other certificates in practice, which can be surprising [2]. [1] https://letsencrypt.org/2020/12/21/extending-android-compatibility.html https://letsencrypt.org/2020/12/21/extending-android-compati... [2] https://alexsci.com/blog/name-non-constraint/ https://alexsci.com/blog/name-non-constraint/
- profmonocle 3y ago
  This solution wasn't perfect. Although things were mostly resolved pretty quick, it led to one of the longest threads I've ever seen on the LE forums: https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190 https://community.letsencrypt.org/t/help-thread-for-dst-root... IIRC one of the bigger problems was that older versions of OpenSSL did check root anchor expiration. But that wasn't all - at my then-employer we had a brief outage on some of our systems because Ubuntu had to patch something (I don't recall what) to deal with this, and they only released the patch a few days before the expiration. We had to mass-rebuild all our Docker images to fix the issue. This workaround was so wild and unprecedented that I assume the cost difference vs. getting cross-sig from an unexpired (and widely compatible) root was massive for them to use it. There must have been a huge amount of testing involved. The fact that it went as smoothly as it did (mostly, but not completely) was impressive.
- tzs 3y ago
  I was a bit surprised that the Android way is not how it works everywhere. I had assumed that the time validation of a TLS certificate chain C0 -> C1 -> C2 ... -> Cn went something like this (in pseudocode): 1 time_check = now() 2 for cert in Cn to C0 3 if time_check < cert.valid_from || time_check > cert.valid_to 4 return EXPIRED 5 time_check = cert.issue_time 6 return NOT_EXPIRED but a bit of Googling shows tells me that it works like that pseudocode with line #5 omitted so that all the time checks are against the current time. All certificates in the chain must be valid now. With code signing certificates it does work the way I assumed TLS work. Timestamped code signed with an expired root certificate is still valid as long as the root certificate was valid at the time of the timestamp.
vlovich123 3y ago
Does anyone know the backstory behind how they found a cert company to cross-sign? Doesn’t letsencrypt completely kill their business model?
- paxys 3y ago
  > Doesn’t letsencrypt completely kill their business model? Not at all. Major cert authorities are all selling to businesses, and that will continue to be the case. Letsencrypt's users are almost all in the personal/hobbyist space.
  - galleywest200 3y ago
    Our company uses Let's Encrypt for HTTPS traffic on our own internal networks. It is free, convenient, and cert-manager handles it all "auto-magically" for us.
  - necubi 3y ago
    > Letsencrypt's users are almost all in the personal/hobbyist space That may have been true in the early days of Let's encrypt. But today, if you're using any of a number of newer hosting options (Netlify, Vercel, Fly.io, etc.) you're probably using a let's encrypt certificate. Older/more staid companies will mostly still be using certs from Digisign et al., but there's little reason for new companies to pay for that.
    - paxys 3y ago
      > AWS, Google Cloud How so? > Netlify, Vercel, Fly, etc. For the default URLs generated when you deploy an app, sure, but companies using them commercially are all bringing their own domains (with their own certs).
      - nmjohn 3y ago
        Nope - having built out the SSL termination stack for a similar company - they are serving millions of commercial sites on their own custom domains with let's encrypt issued certs.
r1ch 3y ago
I've been stripping off the cross signed certificate on my desktop-targeted sites shortly after it was introduced. I found it caused compatibility issues when there were none before as some certificate validators were tripping up on the expired root certificate. I was unfortunately never able find out the root cause as the affected users weren't very technical.
profmonocle 3y ago
I remember when Let's Encrypt announced they were going to do this in summer of 2019. They listened to community feedback and postponed. I really want to shout out the LE team for how they handled this. Back in 2019 I was one of the loud people urging to you to reconsider. You massively overshot my wildest hopes on this issue - I never expected you to delay this switch for 4 1/2 years. That's incredible. Thank you for showing this level of care for the TLS ecosystem.
- hospitalJail 3y ago
  Can you explain? I use it and forget Let's Encrypt is so critical to my website.
  - Cerium 3y ago
    Had they gone ahead with their original plan as scheduled, then 1/3 of those using Android would have encountered certificate errors while attempting to browse Let's Encrypt secured websites. Since then people have moved to newer phones and the number of effected devices has drooped down to about 6%. There is still a bit more than a year left before the transition which will of course continue to move the numbers in a favorable direction.
    - profmonocle 3y ago
      > then 1/3 of those using Android would have encountered certificate errors while attempting to browse Let's Encrypt secured websites. It was 50% when they announced the plan initially. :-|
      - chrisdotcode 3y ago
        So in 2019, you're saying that LE would have been fine with serving 50% of all Android users certificate errors, is that correct? If so, what would make them suggest such a plan in the first place?
        3y ago
PaywallBuster 3y ago
> In addition, dropping the cross-sign will reduce the number of certificate bytes sent in a TLS handshake by over 40%. Finally, it will significantly reduce our operating costs, allowing us to focus our funding on continuing to improve your privacy and security. Are they using public cloud?
deathanatos 3y ago
> the cross-signed certificate will expire. This should be a non-event for most people Let's hope so. But the last DST cross-sign expiration wasn't a non-event. IIRC, GnuTLS failed to correctly path-build after the expiration (AFAICT, it would only build a path to the expired cert, see that it was expired, and then abort, ignoring all other possible paths); worse, GnuTLS is the TLS library used by apt (when using HTTPS, which isn't the default, but my security team wanted all packages vendored and served securely — which … sensible, but yeah. An outage was the cost). (This was fixed in Bullseye, I think? Which was literally like a mere week or so before, by sheer luck.) Azure also had multiple outages caused by or surrounding that expiration.
- codewiz 3y ago
  Wouldn't the operators of all apt mirrors be renewing their LE certificates with certbot every month or so? Then, after June 6th, 2024, they will get a new certificate signed by the new LE root, which isn't cross-signed and isn't expired. Or am I missing something?
  - deathanatos 3y ago
    Hmm. I do think you're correct, at least in theory, but somehow we had DST certs in the path that got built. I forget the exact details of the two — I think it is the — R3 certs at that time. All I know is that apt (through GnuTLS) stopped being able to install, and it was a massive headache. We upgraded to Bullseye (where apt seemed to not have whatever bug ailed it) at that time, and that fixed things. One would think the Authority Key ID info would be the only thing that could define a path up the tree. There's also an option in ACME to get "alternate" chains, and I forget which chain was the alternate at the time. (I feel like it might have been the non-DST one, b/c the DST one was being preferred to get compatibility w/ Android, who would not realize the cross had expired, or something. But I also didn't learn about alternate chains in ACME until the expiration forced me to out of necessity.) (I'm not sure how rigorous path building is, but the general state of TLS tooling leads me to believe it's a giant ball of yarn. Even since then, I have seen a bizarre apparent path to the expired DST root get built, although the circumstance in which I saw it afterwards was contrived, and involved a vendor (Hashicorp) adding a leaf (i.e., non-CA) cert to the list of CA certs, and doing it in such a way that leaves OpenSSL's internal structures in that dir corrupt. OpenSSL, in that situation, somehow, found the DST cert once again. Removing the leaf cert form the store caused the validation to succeed, but … OpenSSL's docs are not really clear about how you can or cannot screw about with its on-disk data. I call such shenanigans UB, and you reap what you sow, but I couldn't convince them of that. We lived with the issues that bug caused until it was apparently fixed in a later version.)
nubinetwork 3y ago
> Come late 2021, our cross-signed intermediates and DST Root CA X3 itself were expiring. And while all up-to-date browsers at that time trusted our root, over a third of Android devices were still running old versions of the OS which would suddenly stop trusting websites using our certificates I only noticed this a few weeks ago, apparently ubiquiti users were also affected.
motbus3 3y ago
In the long run, are there implications regarding who can view the internet?
l33tman 3y ago
FWIW I recently had to switch to ZeroSSL from Letsencrypt which I normally use, while porting a backend from AWS (that used their certs) to a local server, as the IoT gear from 2016 we're supporting didn't have the root certs that validating LE certs required (this was related to the 2021 expiration of the R3 root cert I think it was that LE used). It was kind of an eye-opener that you could potentially brick a whole fleet of sold products by a cert expiring. In this case it was no real problem as other providers had certs with valid root certs.
- mananaysiempre 3y ago
  When SHA-1 was being phased out, there was a lot of whining on mozilla.dev.security.policy from CAs who had issued SHA-1 certs installed into medical and point-of-sale systems with no real update options, then continued to issue them well after the dates that the CA/Browser forum voted on. I thought everybody realized at that time that using the Web PKI and having no way to push updates are mutually exclusive, but guess not.
lofaszvanitt 3y ago
So this was one of the trap aspects of the free cert. lol. This is how you shape the landscape, by offering something for free, then when you have a big user base, just start making your own rules. No wonder LE was supported by so many waterheaded IT companies. "If you see a sudden drop in visits from Android, it is likely because you have a significant population of users on Android 7.0 or earlier. We encourage you to provide the same advice to them as we provided above." Good luck doing that.
- xmaayy 3y ago
  "I want the free service to work for my edge case" If you want to continue serving your clients, you can still pay for a certificate from an authority trusted by your outdated clients. If you want to continue serving the old clients for free, ask them to use another free browser that will allow them to do so.
- awestroke 3y ago
  It's literally free. How are you complaining about a free service? Just buy a paid cert
- creatonez 3y ago
  This isn't just making up rules on the spot. Certificate expiration are always known in advance, and the cross-signing was never a guarantee -- especially when that cross-sign extends the lifespan of the root.
NelsonMinar 3y ago
A post-mortem from the last similar change in 2021. https://scotthelme.co.uk/lets-encrypt-root-expiration-post-mortem/ https://scotthelme.co.uk/lets-encrypt-root-expiration-post-m...
jrochkind1 3y ago
14 months advance notice is treating the community right.
carlotte 3y ago
[dead]
Nellyz 3y ago
[dead]

544 points by healsdata 3y ago

TL:DR: If your webste clientes use Android 7.0 or earlier (released 2016), you may need to take action to ensure you can still access websites secured by Let’s Encrypt certificates. By Thursday, June 6th, 2024.

rafaelturk 3y ago
I don't know about you, but looks like this will cause some impact.
- berbec 3y ago
  By the article's infographic, it's 4.6% of Android users. Anyone not using FireFox Mobile on Android 7.0 or earlier. Some, but not major.
  - lesuorac 3y ago
    3 billion seems like decent number to use for how many Android users there are [1]. 4.5% of 3 billion is 135 million or about 13x more than Google Domains [2] so I guess not major indeed. [1]: https://www.google.com/search?q=number+of+android+users https://www.google.com/search?q=number+of+android+users [2]: https://www.theverge.com/2023/6/16/23763340/google-domains-sunset-sell-squarespace https://www.theverge.com/2023/6/16/23763340/google-domains-s...
  - deleted 3y ago
    [deleted]
  - morepork 3y ago
    6.1%, those using 7.0 and older will be affected. But that number will have fallen a bit by the time the certs actually expire in a year.
  - quickthrower2 3y ago
    I wonder how skewed to poorer countries that is? Also I imagine some skew towards smart tvs (since they outlive older phones, you could easily have a 2013 smart tv still working, I had a 2009 dumb TV until recently and sold it on).
morepork 3y ago
There's also the possibility that Chrome decides to bundle this CA like Firefox mobile does, which would mean the majority of users won't have a problem. From 5.0 the builtin webview is updatable too - I think it uses Chrome's engine. That leaves other browser's such as Samsung's one which probably has the most usage, no idea if they bundle root CAs or not.

compumike 3y ago

Is there any proposed solution (other than just using unencrypted HTTP) that would make it so TLS is no longer the most brittle component of the web? The constant churn of protocol deprecations, certificate expirations, rotations, etc. is like planned obsolescence on steroids.

woodruffw 3y ago
I don’t think TLS is the most brittle component of the web. That award probably goes to DNS, BGP, or (depending on how you qualify it) us-east-1.
- lbotos 3y ago
  Whats brittle about DNS? Caching? BGP and us-east-1 fair :D
  - bawolff 3y ago
    DNSSec is pretty brittle. I think this is a very different definition of brittle than the original poster meant
    - tptacek 3y ago
      Yeah, but almost nobody uses it (it's got low single digits uptake in the US, and has actually decline in some years), so that's not the part of DNS that's making it brittle for deployments.
    - talideon 3y ago
      Describing DNSSEC as brittle is the height of understatement.
      - AdamJacobMuller 3y ago
        Brittle, painful and with a tiny minority who scream at you for being stupid because you're not using it and complain about these problems.

bawolff 3y ago

> Finally, it will significantly reduce our operating costs, allowing us to focus our funding on continuing to improve your privacy and security. Does that imply they are paying millions or something for the cross-sign?

nmjohn 3y ago
Based on their 2021 (most recent year available) Form 990 (nonprofit public tax filing) [0] they paid Identrust $434,000 for "Internet Services." Not sure if they are getting more than just the cross-sign from Identrust - but it seems likely that may just be what they are paying for the cross-sign. In that same year, their total expenses were $5.1M - so that expense would make up almost 10% of their budget. [0]: https://beta.candid.org/profile/9328188?keyword=46-3344200&action=Search https://beta.candid.org/profile/9328188?keyword=46-3344200&a...
- bawolff 3y ago
  Wow. Traditional CAs are such a rent seeking business :(
  - jaas 3y ago
    As the person who negotiated the agreements between Let's Encrypt and Identrust I can tell you that they have provided valuable services, including but not limited to cross-signs. I would not describe it as rent seeking. We are sincerely glad to have them as partners, and grateful for their contributions to helping get Let's Encrypt going. We could not have done what we did without them. Running a publicly trusted CA is not easy, and cross-signing others involves work and liability, particularly if the entity asking for a cross-sign is an upstart with a strange plan and little to no experience running a CA.
  - profmonocle 3y ago
    Cross-signing a CA is many orders of magnitude more work than signing a single domain leaf cert. Sure, on a technical level the result is similar - a signed X.509 cert, just with the "CA" flag set to true, but it's a very different proposition. Imagine if a CA cross-signed some new, upstart CA to get them browser compatibility (like IdenTrust did for LE), and then the new upstart went rogue and started issuing phony certs for google.com, wikipedia.org, etc. on behalf of [insert totalitarian nation here] state security. Those certs would chain up to the cross-signer's root, and they're responsible for it. They could face removal from root programs if they were reckless about cross-signatures. So if a root CA wants to cross-sign a new CA, they need to make sure that the new CA follows the same policies and gets the same audits as a root CA, because their ability to break things will be basically equivalent to a root CA. Honestly, <$500k for all the admin work on this sounds reasonable to me. It probably took a huge portion of several people's time throughout the year.

throw0101a 3y ago

Meta-ish: to attain coverage from 95% of Android devices, you have to support 7.0 ("Nougat"), which dates back to August 2016 (~7 years ago): * https://en.wikipedia.org/wiki/Android_Nougat https://en.wikipedia.org/wiki/Android_Nougat To attain coverage of 95% of iOS devices, you have to support iOS 14 which dates back to September 2020 (~3 years ago): * https://iosref.com/ios-usage https://iosref.com/ios-usage * https://en.wikipedia.org/wiki/IOS_14 https://en.wikipedia.org/wiki/IOS_14 Even for 'only' 90% coverage, it's 8.1 (2017) versus iOS 15 (2021). Seems like Apple is doing a better job of convincing / allowing folks to move to a more recent version of their operating system.

paxys 3y ago
> Seems like Apple is doing a better job of convincing / allowing folks to move to a more recent version of their operating system. Apple isn't selling $10 phones in developing countries. If you compare phones at the same price points from major manufacturers/carriers the difference won't be nearly as drastic.
- redox99 3y ago
  Meh. Samsung (and Huawei) are still the best selling phones in developing countries. Samsung could definitely both reduce the unnecessarily large number of SKUs, and better unify their software stack so they could update for 5 years all of their phones. It's crazy how we went from PCs having the right abstractions, where you can easily keep a 10 year old PC updated, to this awful Android situation.
  - paxys 3y ago
    They are not. Samsung is #1 but after that there is a very long tail of Android smartphone manufacturers that most people in western countries haven't even heard of. Xiaomi, Oppo and Vivo for example all have double digit market shares globally, and combined sell more than Samsung. Realme and ZTE have >5%. The "others" section in most graphs is >30%. If you narrow down the price to say <$100 there is an even larger spread. Expecting continued multi-year support from manufacturers and carriers is impossible at this range when the sole focus is on driving down the price and nothing else.

8organicbits 3y ago

The way they got that old cross sign to keep working was quite interesting: > The new cross-sign will be somewhat novel because it extends beyond the expiration of DST Root CA X3. This solution works because Android intentionally does not enforce the expiration dates of certificates used as trust anchors. [1] Trust anchors work really differently than other certificates in practice, which can be surprising [2]. [1] https://letsencrypt.org/2020/12/21/extending-android-compatibility.html https://letsencrypt.org/2020/12/21/extending-android-compati... [2] https://alexsci.com/blog/name-non-constraint/ https://alexsci.com/blog/name-non-constraint/

profmonocle 3y ago
This solution wasn't perfect. Although things were mostly resolved pretty quick, it led to one of the longest threads I've ever seen on the LE forums: https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190 https://community.letsencrypt.org/t/help-thread-for-dst-root... IIRC one of the bigger problems was that older versions of OpenSSL did check root anchor expiration. But that wasn't all - at my then-employer we had a brief outage on some of our systems because Ubuntu had to patch something (I don't recall what) to deal with this, and they only released the patch a few days before the expiration. We had to mass-rebuild all our Docker images to fix the issue. This workaround was so wild and unprecedented that I assume the cost difference vs. getting cross-sig from an unexpired (and widely compatible) root was massive for them to use it. There must have been a huge amount of testing involved. The fact that it went as smoothly as it did (mostly, but not completely) was impressive.
tzs 3y ago
I was a bit surprised that the Android way is not how it works everywhere. I had assumed that the time validation of a TLS certificate chain C0 -> C1 -> C2 ... -> Cn went something like this (in pseudocode): 1 time_check = now() 2 for cert in Cn to C0 3 if time_check < cert.valid_from || time_check > cert.valid_to 4 return EXPIRED 5 time_check = cert.issue_time 6 return NOT_EXPIRED but a bit of Googling shows tells me that it works like that pseudocode with line #5 omitted so that all the time checks are against the current time. All certificates in the chain must be valid now. With code signing certificates it does work the way I assumed TLS work. Timestamped code signed with an expired root certificate is still valid as long as the root certificate was valid at the time of the timestamp.

vlovich123 3y ago

Does anyone know the backstory behind how they found a cert company to cross-sign? Doesn’t letsencrypt completely kill their business model?

paxys 3y ago
> Doesn’t letsencrypt completely kill their business model? Not at all. Major cert authorities are all selling to businesses, and that will continue to be the case. Letsencrypt's users are almost all in the personal/hobbyist space.
- galleywest200 3y ago
  Our company uses Let's Encrypt for HTTPS traffic on our own internal networks. It is free, convenient, and cert-manager handles it all "auto-magically" for us.
- necubi 3y ago
  > Letsencrypt's users are almost all in the personal/hobbyist space That may have been true in the early days of Let's encrypt. But today, if you're using any of a number of newer hosting options (Netlify, Vercel, Fly.io, etc.) you're probably using a let's encrypt certificate. Older/more staid companies will mostly still be using certs from Digisign et al., but there's little reason for new companies to pay for that.
  - paxys 3y ago
    > AWS, Google Cloud How so? > Netlify, Vercel, Fly, etc. For the default URLs generated when you deploy an app, sure, but companies using them commercially are all bringing their own domains (with their own certs).
    - nmjohn 3y ago
      Nope - having built out the SSL termination stack for a similar company - they are serving millions of commercial sites on their own custom domains with let's encrypt issued certs.

r1ch 3y ago

I've been stripping off the cross signed certificate on my desktop-targeted sites shortly after it was introduced. I found it caused compatibility issues when there were none before as some certificate validators were tripping up on the expired root certificate. I was unfortunately never able find out the root cause as the affected users weren't very technical.

profmonocle 3y ago

I remember when Let's Encrypt announced they were going to do this in summer of 2019. They listened to community feedback and postponed. I really want to shout out the LE team for how they handled this. Back in 2019 I was one of the loud people urging to you to reconsider. You massively overshot my wildest hopes on this issue - I never expected you to delay this switch for 4 1/2 years. That's incredible. Thank you for showing this level of care for the TLS ecosystem.

hospitalJail 3y ago
Can you explain? I use it and forget Let's Encrypt is so critical to my website.
- Cerium 3y ago
  Had they gone ahead with their original plan as scheduled, then 1/3 of those using Android would have encountered certificate errors while attempting to browse Let's Encrypt secured websites. Since then people have moved to newer phones and the number of effected devices has drooped down to about 6%. There is still a bit more than a year left before the transition which will of course continue to move the numbers in a favorable direction.
  - profmonocle 3y ago
    > then 1/3 of those using Android would have encountered certificate errors while attempting to browse Let's Encrypt secured websites. It was 50% when they announced the plan initially. :-|
    - chrisdotcode 3y ago
      So in 2019, you're saying that LE would have been fine with serving 50% of all Android users certificate errors, is that correct? If so, what would make them suggest such a plan in the first place?
      - 3y ago

PaywallBuster 3y ago

> In addition, dropping the cross-sign will reduce the number of certificate bytes sent in a TLS handshake by over 40%. Finally, it will significantly reduce our operating costs, allowing us to focus our funding on continuing to improve your privacy and security. Are they using public cloud?

deathanatos 3y ago

> the cross-signed certificate will expire. This should be a non-event for most people Let's hope so. But the last DST cross-sign expiration wasn't a non-event. IIRC, GnuTLS failed to correctly path-build after the expiration (AFAICT, it would only build a path to the expired cert, see that it was expired, and then abort, ignoring all other possible paths); worse, GnuTLS is the TLS library used by apt (when using HTTPS, which isn't the default, but my security team wanted all packages vendored and served securely — which … sensible, but yeah. An outage was the cost). (This was fixed in Bullseye, I think? Which was literally like a mere week or so before, by sheer luck.) Azure also had multiple outages caused by or surrounding that expiration.

codewiz 3y ago
Wouldn't the operators of all apt mirrors be renewing their LE certificates with certbot every month or so? Then, after June 6th, 2024, they will get a new certificate signed by the new LE root, which isn't cross-signed and isn't expired. Or am I missing something?
- deathanatos 3y ago
  Hmm. I do think you're correct, at least in theory, but somehow we had DST certs in the path that got built. I forget the exact details of the two — I think it is the — R3 certs at that time. All I know is that apt (through GnuTLS) stopped being able to install, and it was a massive headache. We upgraded to Bullseye (where apt seemed to not have whatever bug ailed it) at that time, and that fixed things. One would think the Authority Key ID info would be the only thing that could define a path up the tree. There's also an option in ACME to get "alternate" chains, and I forget which chain was the alternate at the time. (I feel like it might have been the non-DST one, b/c the DST one was being preferred to get compatibility w/ Android, who would not realize the cross had expired, or something. But I also didn't learn about alternate chains in ACME until the expiration forced me to out of necessity.) (I'm not sure how rigorous path building is, but the general state of TLS tooling leads me to believe it's a giant ball of yarn. Even since then, I have seen a bizarre apparent path to the expired DST root get built, although the circumstance in which I saw it afterwards was contrived, and involved a vendor (Hashicorp) adding a leaf (i.e., non-CA) cert to the list of CA certs, and doing it in such a way that leaves OpenSSL's internal structures in that dir corrupt. OpenSSL, in that situation, somehow, found the DST cert once again. Removing the leaf cert form the store caused the validation to succeed, but … OpenSSL's docs are not really clear about how you can or cannot screw about with its on-disk data. I call such shenanigans UB, and you reap what you sow, but I couldn't convince them of that. We lived with the issues that bug caused until it was apparently fixed in a later version.)

nubinetwork 3y ago

> Come late 2021, our cross-signed intermediates and DST Root CA X3 itself were expiring. And while all up-to-date browsers at that time trusted our root, over a third of Android devices were still running old versions of the OS which would suddenly stop trusting websites using our certificates I only noticed this a few weeks ago, apparently ubiquiti users were also affected.

motbus3 3y ago

In the long run, are there implications regarding who can view the internet?

l33tman 3y ago

FWIW I recently had to switch to ZeroSSL from Letsencrypt which I normally use, while porting a backend from AWS (that used their certs) to a local server, as the IoT gear from 2016 we're supporting didn't have the root certs that validating LE certs required (this was related to the 2021 expiration of the R3 root cert I think it was that LE used). It was kind of an eye-opener that you could potentially brick a whole fleet of sold products by a cert expiring. In this case it was no real problem as other providers had certs with valid root certs.

mananaysiempre 3y ago
When SHA-1 was being phased out, there was a lot of whining on mozilla.dev.security.policy from CAs who had issued SHA-1 certs installed into medical and point-of-sale systems with no real update options, then continued to issue them well after the dates that the CA/Browser forum voted on. I thought everybody realized at that time that using the Web PKI and having no way to push updates are mutually exclusive, but guess not.

lofaszvanitt 3y ago

So this was one of the trap aspects of the free cert. lol. This is how you shape the landscape, by offering something for free, then when you have a big user base, just start making your own rules. No wonder LE was supported by so many waterheaded IT companies. "If you see a sudden drop in visits from Android, it is likely because you have a significant population of users on Android 7.0 or earlier. We encourage you to provide the same advice to them as we provided above." Good luck doing that.

xmaayy 3y ago
"I want the free service to work for my edge case" If you want to continue serving your clients, you can still pay for a certificate from an authority trusted by your outdated clients. If you want to continue serving the old clients for free, ask them to use another free browser that will allow them to do so.
awestroke 3y ago
It's literally free. How are you complaining about a free service? Just buy a paid cert
creatonez 3y ago
This isn't just making up rules on the spot. Certificate expiration are always known in advance, and the cross-signing was never a guarantee -- especially when that cross-sign extends the lifespan of the root.

NelsonMinar 3y ago

A post-mortem from the last similar change in 2021. https://scotthelme.co.uk/lets-encrypt-root-expiration-post-mortem/ https://scotthelme.co.uk/lets-encrypt-root-expiration-post-m...

jrochkind1 3y ago

14 months advance notice is treating the community right.

carlotte 3y ago

[dead]

Nellyz 3y ago

[dead]