6 ms·

Workaround is Firefox Mobile (because it ships with its own root certs), but that's a significant burden to place on the user.

est31 6y ago
Also the post says that Firefox doesn't work on Androids older than 5.0 which according to the dashboard are still 5.9% of devices. For those older devices, the only option is to install the new root certificate. Anyways, there are billions of Android devices out there. 33% of those is a large number. You can't just tell all of them that they are wrong. If this happens, people will move away from Let's encrypt in masses. They don't realize yet how self-harming this really is.
- maxerickson 6y ago
  In the post, they advertise that they are going to continue to offer a service that provides certificates signed with the old one.
  - est31 6y ago
    ... which will work until September 2021.
  - capableweb 6y ago
    That's not gonna help as the root will expire, so the devices stuck with the older root will still shows errors when trying to use them.
- baggy_trough 6y ago
  Not sure about that. Move away from Let's Encrypt to what? More likely, most smaller to medium sized sites will say forget those old Android guys.
  - sroussey 6y ago
    Too cheap to update can be seen as too cheap to buy your product or service, so I can see this happening.
  - est31 6y ago
    I found at least Buypass offering a gratis ACME product "Buypass Go SSL". They have roots which are deployed at least since Android 4.1, which covers way more Android devices (according to the Android Studio statistics, >99%): https://android.googlesource.com/platform/libcore/+/android-4.1.1_r1/luni/src/main/files/cacerts/eb375c3e.0 https://android.googlesource.com/platform/libcore/+/android-... I'm not sure whether that particular root is being used for their Go SSL product. If so, Buypass might be a good alternative to migrate to. From what I read though, they do require an E-Mail address, so you've got to keep that in mind.
- WorldMaker 6y ago
  Root certificate updates are a massive security issue. Blaming Let's Encrypt is blaming one of the canaries for the coal mine disaster. 33% of Android devices don't and can't get up to date root certificates is an impressive security crisis that grows worse by the year (look at the other root expirations and the crazy workarounds that for instance Netflix has been doing to still work on older Android devices). Shouldn't the blame squarely be on Google, the Android OEMs, and the phone carriers for allowing this disaster to happen in the first place? I realize that is a tough message to get out to users and site owners are going to be in the cross-fire, but it seems better to try to work for solidarity in pointing fingers at the right direction and the right direction certainly isn't Let's Encrypt.
  - cpach 6y ago
    That makes me curious, what workarounds did Netflix employ?
    - Aissen 6y ago
      When you control the client, it's simple, you can do pretty much anything: embed your own HTTP stack, TLS stack, your QUIC stack, or simply your PKI, or subset of the webPKI.
    - WorldMaker 6y ago
      It was the BBC I was actually thinking about, and it's a part of this article (which also mentions this Let's Encrypt root change): https://scotthelme.co.uk/impending-doom-root-ca-expiring-legacy-clients/ https://scotthelme.co.uk/impending-doom-root-ca-expiring-leg... Previous HN discussion on that article: https://news.ycombinator.com/item?id=23455463 https://news.ycombinator.com/item?id=23455463
  - est31 6y ago
    Yes the issue is really severe of most deployed Android devices not getting security updates, either at all, or the devices are used well beyond the update period. But this is not up to Let's Encrypt to solve. They market themselves to build products for the mass market instead of small niches of the market, say, everyone who buys a new phone every year. But then they also have to treat their product like a mass market product, and if Android users still use older versions of the OS, then Let's Encrypt should adopt for that.
- veeti 6y ago
  You can still install the last version of Firefox to support Android 4.x.
- notatoad 6y ago
  on androids older than 5, the browser is the old android browser instead of chrome. how many sites out there still test compatability with that? even without having to click through security warnings, the web is horribly broken on old android devices. the overlap of sites using letsencrypt and sites that care about people using android <5 has got to be vanishingly small. this isn't going to cause a move away from letsencrypt.
- merlyn 6y ago
  How many other root certificates are going to be expiring in the next 3 years that older Android won't have updates for as well? Probably more than 1.
- TheKIngofBelAir 6y ago
  > Also the post says that Firefox doesn't work on Androids older than 5.0 which according to the dashboard are still 5.9% of devices. For those older devices, the only option is to install the new root certificate. Microsoft Edge still gets updates on Android 4.4 KitKat
  - w3ll_w3ll_w3ll 6y ago
    I don't think Microsft Edge embeds its own root store on Android.
    - tacon 6y ago
      Is there enough incentive for Microsoft to add a root store to Edge by next September? How hard is it to make that addition?
      - jsjohnst 6y ago
        I may be wrong, but the effort to switch to your own root store is more doing it securely, than the difficulty of switching from system frameworks to your own SSL/HTTP transport layers. So to put another way, straight forward to do mediocre job, not as trivial to do a good or great job.
        toast0 6y ago
        Root store and TLS/HTTP library are separate concerns. You can use the system root store with your own libraries, or you can use your own root store with the system libraries. On an Android 4.4 device, you should probably skip the system root store and the system libraries, and if you're already doing it for those phones, you might as well do it for all the phones.
        jsjohnst 6y ago
        > Root store and TLS/HTTP library are separate concerns. In the context of this thread (aka older Android devices), they aren’t truly separate concerns. You really need to do both. My point is that doing both is relatively straightforward, but doing the root store part is fairly easy to do it in a mediocre way and be brittle / insecure.
- fuzxi 6y ago
  33% of devices, but per the article, only 1-5% of traffic on sites using LetsEncrypt. I don't see any site owners moving to a different CA when this affects less than 5% of their visitors, who are likely the poorest fraction of their userbase , i.e. probably not many paying users.
- toyg 6y ago
  I don’t think they have a choice. Reading between the lines, the CA who cross-signed their previous root doesn’t really want to continue doing so (or asked for a lot more money) because LE usage reached levels that are just too risky. I don’t blame them: a single bad actor found doing something particularly nefarious with a LE certificate might lose them the trust their business is literally built upon. I don’t see any other CA queueing up to help what is typically their commercial nemesis. And as they say, at some point they would have to do it anyway, might as well rip the plaster off.
  - t0astbread 6y ago
    Agreed, but what could someone do with a domain validation cert (as issued by LE) that would be so nefarious to damage IdenTrust's reputation?
- kelnos 6y ago
  If you're running Android 5.0, you also haven't benefited from updates that remove CAs that have since been shown to be untrustworthy. I think that's far worse than being unable to visit sites that use LetsEncrypt certificates. It was really short-sighted of Google to make the system cert bundle something that can't be updated without a full OS update. There should be an OTA mechanism that allows it to be updated through the Play Store or through some other means that isn't reliant upon lazy device manufacturers.
  - thayne 6y ago
    To be fair they haven't gotten mant other security updates either. It's just a bad situation all around.
stefan_ 6y ago
A Workaround is not something that the user does.
nyanpasu64 6y ago
Unfortunately, on my Moto G5 Plus (which is not an high-end device), I've found that Firefox on Android is slower than Chrome (specifically the Bromite fork). I think that Firefox may not be a good solution for low-spec or older phones running older Android versions.
neop1x 6y ago
Firefox is not a workaround for non-web mobile apps. Lots of them will stop working unless they do cert pinning / have their own CA bundle or simply stop using LE..

est31 6y ago

Also the post says that Firefox doesn't work on Androids older than 5.0 which according to the dashboard are still 5.9% of devices. For those older devices, the only option is to install the new root certificate. Anyways, there are billions of Android devices out there. 33% of those is a large number. You can't just tell all of them that they are wrong. If this happens, people will move away from Let's encrypt in masses. They don't realize yet how self-harming this really is.

maxerickson 6y ago
In the post, they advertise that they are going to continue to offer a service that provides certificates signed with the old one.
- est31 6y ago
  ... which will work until September 2021.
- capableweb 6y ago
  That's not gonna help as the root will expire, so the devices stuck with the older root will still shows errors when trying to use them.
baggy_trough 6y ago
Not sure about that. Move away from Let's Encrypt to what? More likely, most smaller to medium sized sites will say forget those old Android guys.
- sroussey 6y ago
  Too cheap to update can be seen as too cheap to buy your product or service, so I can see this happening.
- est31 6y ago
  I found at least Buypass offering a gratis ACME product "Buypass Go SSL". They have roots which are deployed at least since Android 4.1, which covers way more Android devices (according to the Android Studio statistics, >99%): https://android.googlesource.com/platform/libcore/+/android-4.1.1_r1/luni/src/main/files/cacerts/eb375c3e.0 https://android.googlesource.com/platform/libcore/+/android-... I'm not sure whether that particular root is being used for their Go SSL product. If so, Buypass might be a good alternative to migrate to. From what I read though, they do require an E-Mail address, so you've got to keep that in mind.
WorldMaker 6y ago
Root certificate updates are a massive security issue. Blaming Let's Encrypt is blaming one of the canaries for the coal mine disaster. 33% of Android devices don't and can't get up to date root certificates is an impressive security crisis that grows worse by the year (look at the other root expirations and the crazy workarounds that for instance Netflix has been doing to still work on older Android devices). Shouldn't the blame squarely be on Google, the Android OEMs, and the phone carriers for allowing this disaster to happen in the first place? I realize that is a tough message to get out to users and site owners are going to be in the cross-fire, but it seems better to try to work for solidarity in pointing fingers at the right direction and the right direction certainly isn't Let's Encrypt.
- cpach 6y ago
  That makes me curious, what workarounds did Netflix employ?
  - Aissen 6y ago
    When you control the client, it's simple, you can do pretty much anything: embed your own HTTP stack, TLS stack, your QUIC stack, or simply your PKI, or subset of the webPKI.
  - WorldMaker 6y ago
    It was the BBC I was actually thinking about, and it's a part of this article (which also mentions this Let's Encrypt root change): https://scotthelme.co.uk/impending-doom-root-ca-expiring-legacy-clients/ https://scotthelme.co.uk/impending-doom-root-ca-expiring-leg... Previous HN discussion on that article: https://news.ycombinator.com/item?id=23455463 https://news.ycombinator.com/item?id=23455463
- est31 6y ago
  Yes the issue is really severe of most deployed Android devices not getting security updates, either at all, or the devices are used well beyond the update period. But this is not up to Let's Encrypt to solve. They market themselves to build products for the mass market instead of small niches of the market, say, everyone who buys a new phone every year. But then they also have to treat their product like a mass market product, and if Android users still use older versions of the OS, then Let's Encrypt should adopt for that.
veeti 6y ago
You can still install the last version of Firefox to support Android 4.x.
notatoad 6y ago
on androids older than 5, the browser is the old android browser instead of chrome. how many sites out there still test compatability with that? even without having to click through security warnings, the web is horribly broken on old android devices. the overlap of sites using letsencrypt and sites that care about people using android <5 has got to be vanishingly small. this isn't going to cause a move away from letsencrypt.
merlyn 6y ago
How many other root certificates are going to be expiring in the next 3 years that older Android won't have updates for as well? Probably more than 1.
TheKIngofBelAir 6y ago
> Also the post says that Firefox doesn't work on Androids older than 5.0 which according to the dashboard are still 5.9% of devices. For those older devices, the only option is to install the new root certificate. Microsoft Edge still gets updates on Android 4.4 KitKat
- w3ll_w3ll_w3ll 6y ago
  I don't think Microsft Edge embeds its own root store on Android.
  - tacon 6y ago
    Is there enough incentive for Microsoft to add a root store to Edge by next September? How hard is it to make that addition?
    - jsjohnst 6y ago
      I may be wrong, but the effort to switch to your own root store is more doing it securely, than the difficulty of switching from system frameworks to your own SSL/HTTP transport layers. So to put another way, straight forward to do mediocre job, not as trivial to do a good or great job.
      - toast0 6y ago
        Root store and TLS/HTTP library are separate concerns. You can use the system root store with your own libraries, or you can use your own root store with the system libraries. On an Android 4.4 device, you should probably skip the system root store and the system libraries, and if you're already doing it for those phones, you might as well do it for all the phones.
        jsjohnst 6y ago
        > Root store and TLS/HTTP library are separate concerns. In the context of this thread (aka older Android devices), they aren’t truly separate concerns. You really need to do both. My point is that doing both is relatively straightforward, but doing the root store part is fairly easy to do it in a mediocre way and be brittle / insecure.
fuzxi 6y ago
33% of devices, but per the article, only 1-5% of traffic on sites using LetsEncrypt. I don't see any site owners moving to a different CA when this affects less than 5% of their visitors, who are likely the poorest fraction of their userbase , i.e. probably not many paying users.
toyg 6y ago
I don’t think they have a choice. Reading between the lines, the CA who cross-signed their previous root doesn’t really want to continue doing so (or asked for a lot more money) because LE usage reached levels that are just too risky. I don’t blame them: a single bad actor found doing something particularly nefarious with a LE certificate might lose them the trust their business is literally built upon. I don’t see any other CA queueing up to help what is typically their commercial nemesis. And as they say, at some point they would have to do it anyway, might as well rip the plaster off.
- t0astbread 6y ago
  Agreed, but what could someone do with a domain validation cert (as issued by LE) that would be so nefarious to damage IdenTrust's reputation?
kelnos 6y ago
If you're running Android 5.0, you also haven't benefited from updates that remove CAs that have since been shown to be untrustworthy. I think that's far worse than being unable to visit sites that use LetsEncrypt certificates. It was really short-sighted of Google to make the system cert bundle something that can't be updated without a full OS update. There should be an OTA mechanism that allows it to be updated through the Play Store or through some other means that isn't reliant upon lazy device manufacturers.
- thayne 6y ago
  To be fair they haven't gotten mant other security updates either. It's just a bad situation all around.

stefan_ 6y ago

A Workaround is not something that the user does.

nyanpasu64 6y ago

Unfortunately, on my Moto G5 Plus (which is not an high-end device), I've found that Firefox on Android is slower than Chrome (specifically the Bromite fork). I think that Firefox may not be a good solution for low-spec or older phones running older Android versions.

neop1x 6y ago

Firefox is not a workaround for non-web mobile apps. Lots of them will stop working unless they do cert pinning / have their own CA bundle or simply stop using LE..