15 ms·

Standing on our own two feet

584 points by gpff 6y ago

gpff 6y ago
Let’s Encrypt cross-signature with IdenTrust "DST Root X3" is ending on September 1, 2021 but 33.8% of Android devices are running versions under 7.1 which don't trust Let’s Encrypt new root certificate "ISRG Root X1"
- brendanclune 6y ago
  Workaround is Firefox Mobile (because it ships with its own root certs), but that's a significant burden to place on the user.
  - est31 6y ago
    Also the post says that Firefox doesn't work on Androids older than 5.0 which according to the dashboard are still 5.9% of devices. For those older devices, the only option is to install the new root certificate. Anyways, there are billions of Android devices out there. 33% of those is a large number. You can't just tell all of them that they are wrong. If this happens, people will move away from Let's encrypt in masses. They don't realize yet how self-harming this really is.
    - maxerickson 6y ago
      In the post, they advertise that they are going to continue to offer a service that provides certificates signed with the old one.
      - est31 6y ago
        ... which will work until September 2021.
      - capableweb 6y ago
        That's not gonna help as the root will expire, so the devices stuck with the older root will still shows errors when trying to use them.
    - baggy_trough 6y ago
      Not sure about that. Move away from Let's Encrypt to what? More likely, most smaller to medium sized sites will say forget those old Android guys.
      - sroussey 6y ago
        Too cheap to update can be seen as too cheap to buy your product or service, so I can see this happening.
      - est31 6y ago
        I found at least Buypass offering a gratis ACME product "Buypass Go SSL". They have roots which are deployed at least since Android 4.1, which covers way more Android devices (according to the Android Studio statistics, >99%): https://android.googlesource.com/platform/libcore/+/android-4.1.1_r1/luni/src/main/files/cacerts/eb375c3e.0 https://android.googlesource.com/platform/libcore/+/android-... I'm not sure whether that particular root is being used for their Go SSL product. If so, Buypass might be a good alternative to migrate to. From what I read though, they do require an E-Mail address, so you've got to keep that in mind.
    - WorldMaker 6y ago
      Root certificate updates are a massive security issue. Blaming Let's Encrypt is blaming one of the canaries for the coal mine disaster. 33% of Android devices don't and can't get up to date root certificates is an impressive security crisis that grows worse by the year (look at the other root expirations and the crazy workarounds that for instance Netflix has been doing to still work on older Android devices). Shouldn't the blame squarely be on Google, the Android OEMs, and the phone carriers for allowing this disaster to happen in the first place? I realize that is a tough message to get out to users and site owners are going to be in the cross-fire, but it seems better to try to work for solidarity in pointing fingers at the right direction and the right direction certainly isn't Let's Encrypt.
      - cpach 6y ago
        That makes me curious, what workarounds did Netflix employ?
        Aissen 6y ago
        When you control the client, it's simple, you can do pretty much anything: embed your own HTTP stack, TLS stack, your QUIC stack, or simply your PKI, or subset of the webPKI.
        WorldMaker 6y ago
        It was the BBC I was actually thinking about, and it's a part of this article (which also mentions this Let's Encrypt root change): https://scotthelme.co.uk/impending-doom-root-ca-expiring-legacy-clients/ https://scotthelme.co.uk/impending-doom-root-ca-expiring-leg... Previous HN discussion on that article: https://news.ycombinator.com/item?id=23455463 https://news.ycombinator.com/item?id=23455463
      - est31 6y ago
        Yes the issue is really severe of most deployed Android devices not getting security updates, either at all, or the devices are used well beyond the update period. But this is not up to Let's Encrypt to solve. They market themselves to build products for the mass market instead of small niches of the market, say, everyone who buys a new phone every year. But then they also have to treat their product like a mass market product, and if Android users still use older versions of the OS, then Let's Encrypt should adopt for that.
    - veeti 6y ago
      You can still install the last version of Firefox to support Android 4.x.
    - notatoad 6y ago
      on androids older than 5, the browser is the old android browser instead of chrome. how many sites out there still test compatability with that? even without having to click through security warnings, the web is horribly broken on old android devices. the overlap of sites using letsencrypt and sites that care about people using android <5 has got to be vanishingly small. this isn't going to cause a move away from letsencrypt.
    - merlyn 6y ago
      How many other root certificates are going to be expiring in the next 3 years that older Android won't have updates for as well? Probably more than 1.
    - TheKIngofBelAir 6y ago
      > Also the post says that Firefox doesn't work on Androids older than 5.0 which according to the dashboard are still 5.9% of devices. For those older devices, the only option is to install the new root certificate. Microsoft Edge still gets updates on Android 4.4 KitKat
      - w3ll_w3ll_w3ll 6y ago
        I don't think Microsft Edge embeds its own root store on Android.
        tacon 6y ago
        Is there enough incentive for Microsoft to add a root store to Edge by next September? How hard is it to make that addition?
        jsjohnst 6y ago
        I may be wrong, but the effort to switch to your own root store is more doing it securely, than the difficulty of switching from system frameworks to your own SSL/HTTP transport layers. So to put another way, straight forward to do mediocre job, not as trivial to do a good or great job.
        toast0 6y ago
        Root store and TLS/HTTP library are separate concerns. You can use the system root store with your own libraries, or you can use your own root store with the system libraries. On an Android 4.4 device, you should probably skip the system root store and the system libraries, and if you're already doing it for those phones, you might as well do it for all the phones.
        jsjohnst 6y ago
        > Root store and TLS/HTTP library are separate concerns. In the context of this thread (aka older Android devices), they aren’t truly separate concerns. You really need to do both. My point is that doing both is relatively straightforward, but doing the root store part is fairly easy to do it in a mediocre way and be brittle / insecure.
    - fuzxi 6y ago
      33% of devices, but per the article, only 1-5% of traffic on sites using LetsEncrypt. I don't see any site owners moving to a different CA when this affects less than 5% of their visitors, who are likely the poorest fraction of their userbase , i.e. probably not many paying users.
    - toyg 6y ago
      I don’t think they have a choice. Reading between the lines, the CA who cross-signed their previous root doesn’t really want to continue doing so (or asked for a lot more money) because LE usage reached levels that are just too risky. I don’t blame them: a single bad actor found doing something particularly nefarious with a LE certificate might lose them the trust their business is literally built upon. I don’t see any other CA queueing up to help what is typically their commercial nemesis. And as they say, at some point they would have to do it anyway, might as well rip the plaster off.
      - t0astbread 6y ago
        Agreed, but what could someone do with a domain validation cert (as issued by LE) that would be so nefarious to damage IdenTrust's reputation?
    - kelnos 6y ago
      If you're running Android 5.0, you also haven't benefited from updates that remove CAs that have since been shown to be untrustworthy. I think that's far worse than being unable to visit sites that use LetsEncrypt certificates. It was really short-sighted of Google to make the system cert bundle something that can't be updated without a full OS update. There should be an OTA mechanism that allows it to be updated through the Play Store or through some other means that isn't reliant upon lazy device manufacturers.
      - thayne 6y ago
        To be fair they haven't gotten mant other security updates either. It's just a bad situation all around.
  - stefan_ 6y ago
    A Workaround is not something that the user does.
  - nyanpasu64 6y ago
    Unfortunately, on my Moto G5 Plus (which is not an high-end device), I've found that Firefox on Android is slower than Chrome (specifically the Bromite fork). I think that Firefox may not be a good solution for low-spec or older phones running older Android versions.
  - neop1x 6y ago
    Firefox is not a workaround for non-web mobile apps. Lots of them will stop working unless they do cert pinning / have their own CA bundle or simply stop using LE..
- tyingq 6y ago
  "https almost everywhere". Thanks G!
- sbierwagen 6y ago
  Good. Those devices are unsafe, and should not be used.
  - mehrdadn 6y ago
    Would you like to buy new phones for their owners?
    - oh_sigh 6y ago
      The owners can just click through the prompt if they want...
      - neop1x 6y ago
        and get used to clicking on the prompt because who cares about MITM
  - ed25519FUUU 6y ago
    I think you might be ignorant with regards to how the rest of the world uses the internet.
    - Nextgrid 6y ago
      But for certain businesses, there is a case to be made for not serving these kinds of customers. Someone using a shitty old Android phone might not even be able/willing to pay for your product, and if they do, might cost more in technical support and/or fraud (due to their device being vulnerable) than what they bring in revenue.
      - berkes 6y ago
        When you are a bank or a crypto exchange, that might make sense. When you are the municipality of a village in rural India or the company that handles all vehicle registrations, it does not. You cannot gatekeep with such sweeping statements. People have old phones for lots of reasons. Others will have to serve those people for lots of other reasons.
- mrweasel 6y ago
  The most impressive thing it that Let’s Encrypt are the ones who are trying to fix a problem that should be fixed by phone manufactures and telcos. I can see why, but I would also have like the phones to “break” so the owners would avoid those brands in the future, and pick one who care enough to push out update. Still, I can blame Let’s Encrypt, they just want to be the good guys, and the do it so beautifully and transparently.
  - nsgi 6y ago
    It probably wouldn't encourage many people to upgrade their phones, though. They would probably just see the website as broken and either click through the warning or abandon it completely
    - kalleboo 6y ago
      Nearly 60% of the web is using Let's Encrypt certs now. It's not just the one of two sites are going to break, it's going to be half the web.
onlinejk 6y ago
Nice TL;DR, thx
gioele 6y ago
> Without IdenTrust, Let’s Encrypt may have never happened and we are grateful to them for their partnership. What I have never understood is why IdenTrust accepted to cross-sign Let’s Encrypt's root certificate. With that move, IdenTrust basically broke the CA cartel and helped driving the price of basic certificates to zero. How did they, as a for-profit organization, justify "doing the right thing" when that meant disrupting their own market? Anyway, kudos to IdenTrust.
- admax88q 6y ago
  Always better to be doing the disruption rather be the one being disrupted. Perhaps they saw the writing on the walls and wanted to boost their standing in the post LetsEncrypt world.
  - est31 6y ago
    Yeah they likely got a stash of money for it while the other CAs got nothing. First mover advantage I guess. It's not that they'd have been able to prevent it anyways, as the root stores are not under their control, especially when Let's encrypt is being supported, even started, by the entities which run the root stores (Mozilla, through Firefox, and Google, through Android). Ultimately it's the entities which run and deploy the root stores who have ultimate say on which CA gets to issue certs and which doesn't.
- dubcanada 6y ago
  IdenTrust doesn't care about random websites, they care about HIPPA, enterprise, government, securing documents and emails, etc.
  - juliend2 6y ago
    Exactly, but the funny thing is that Chrome and Firefox (Desktop at least) are no longer showing the differentiating green lock for those high-end (EV & OV) certs, but the same neutral-looking lock as those Domain-Validated (DV) certs that Let's Encrypt is issuing. I'm very grateful for IdenTrust for having made that move. I just hope it won't hurt their business too much because of that.
Aissen 6y ago
This might also concern you if your app is running on older android devices, and you have a backend that uses letsencrypt certificates, while the app is relying on the system root CAs. A fix for this could be to add Let's Encrypt's root CA to your app.
nevi-me 6y ago
Could Google possibly be able (before were discuss willingness) to push an update to root certificate via Play Services? I'd like to think that anyone not using Play Services (i.e. Android with no Play) is likely using a custom browser, and would heed a call to switch to Firefox. The problem with some devices in Africa would be that many people will using older phone often don't have enough data for the big Play updates to succeed. Teens often buy 10-100MB of data so they can use WhatsApp. (If you're from Southern Africa, and disagree with this, hit me up, you probably need to spend some time in a village ;) )
- trillic 6y ago
  How much does that much data typically cost??
  - DominikPeters 6y ago
    Here are some South Africa prices: https://www.mtn.co.za/recharge/data https://www.mtn.co.za/recharge/data Valid for a day: 25MB for $0.32; Valid for a week: 50MB for $0.64; Valid for a month: 100MB for $1.28, 1GB for $6.35
- cassepipe 6y ago
  Actually I was surprised that there would be such an easy fix : Switching to Firefox. Which is apparently around 70MB, apparently affordable from what you wrote and definitely worth it if it allows you to unlock a chunk of the internet. So no need for an improbable and costly Play update.
  - qqii 6y ago
    That won't fix any other apps though will it? Anything that uses chrome webview for example.
    - cassepipe 6y ago
      Oh, I did not think about that. On Android 8, you make firefox the default ... (renderer ?) for other apps but idk if it's the case for older versions of Android. On the other hand, as others have said, these devices are generally quite painful to browse on so accessing the wweb version of many apps could be a solution, plus firefox will let you place shortcuts on your home screen . I also wonder how LineageOS works on those old devices. Could be another solution.
zero37z 6y ago
now the corporates got a valid point, why you dont want to use lets encrypt? still the 33% of the devices is a quite a large number to consider.
- juliend2 6y ago
  But like they said in the article, those 33% of Android phones represent "1-5% of the traffic" of the "large integrators" websites that LE communicated with.
  - stefan_ 6y ago
    Well that's an easy choice, lose 1-5% of traffic or pay $100 for a certificate from a vendor whose root doesn't expire next year?
    - fuzxi 6y ago
      1-5% of traffic that comes from people using devices that are at least 4 years old. Someone who can't or won't upgrade from a phone that still uses Android Marshmallow is probably not bringing in much revenue.
      - josephg 6y ago
        Exactly. The “easy choice” is to drop support for those users. I suspect most web developers won’t even notice. It’s a pity - there’s no essential reason why old phones with new batteries should get worse over time. But software kills them in so many ways.
    - neop1x 6y ago
      As long as this statement is true. They also dont's say who these big integrators are and the situation may be widely different between integrators and fields.
lixtra 6y ago
They propose to install Firefox to work around the root certificate problem on old android devices. But can’t you just manually install their root certificate on most phones?
- lights0123 6y ago
  It probably looks a lot less sketchy to your users to tell them to install a browser they've probably heard of and may have used in the past than install a root certificate. Plus, that doesn't fix the problem of other certificates expiring, only extends it.
  - regecks 6y ago
    >Plus, that doesn't fix the problem of other certificates expiring, only extends it. Although I agree fully on the sketchiness part, installing the root is a fix. ISRG Root X1 expires in 2035. Not one of these problematic Android devices will be online anymore.
    - iudqnolq 6y ago
      But to have the whole modern web work you'll need to install more than one root.
- fuzxi 6y ago
  I'm not sure if it's the same on older versions, but on recent Android versions, that requires a rooted device.
  - Latty 6y ago
    Interestingly, it appears to be back in 11. You are right it wasn't possible for some set of versions, not sure how far you have to go back for it to be possible again. They appear to have added it back with a big warning screen similar to what they do for VPNs and stuff telling users it could compromise them, which is reasonable. It was a pain you couldn't before.
  - 6y ago
renewiltord 6y ago
> The remaining 33.8% of Android devices will eventually start getting certificate errors when users visit sites that have a Let’s Encrypt certificate. In our communications with large integrators, we have found that this represents around 1-5% of traffic to their sites. This one-third of Android devices only yields 5% of traffic? Interesting.
- degenerate 6y ago
  I have a 2010 Android smartphone and it's painfully slow to navigate the modern web, almost unbearable. Browsing news websites is simply not worth my time of waiting for the phone to download and process 22MB of JS, CSS, and graphics. Being on wifi makes no difference; it's the CPU choking to render all that cruft. So yeah, 5% of traffic makes sense.
  - GoblinSlayer 6y ago
    In firefox (at least in desktop version) you can disable javascript and css. Not sure if they are still downloaded.
    - bzbarsky 6y ago
      For scripts: if script is disabled for a document, scripts are not downloaded. See https://searchfox.org/mozilla-central/rev/a5d9abfda1e26b1207db9549549ab0bdd73f735d/dom/script/ScriptLoader.cpp#1620-1627 https://searchfox.org/mozilla-central/rev/a5d9abfda1e26b1207... as of today For stylesheets, I'm not certain how you're disabling them. Depending on how you do it, they may or may not get downloaded. The most common ways of disabling them result in them not being downloaded.
      - GoblinSlayer 6y ago
        View - Page style - No style
        bzbarsky 6y ago
tothrowaway 6y ago
> As of January 11, 2021, we’re planning to make a change to our API so that ACME clients will, by default, serve a certificate chain that leads to ISRG Root X1. Ouch. So anyone who wants to support older devices 3 months from now needs to add `--preferred-chain "DST Root CA X3"` to their certbot command. When that chain is completely retired in September next year, certbot appears to fallback on the default (even with that argument present).
rsweeney21 6y ago
I don't understand the motivation for making this change now. Why not keep the universally accepted root certificate as the default chain? Why does Let's Encrypt need to switch to their own root certificate now, and cause thousands of websites to break on older devices? I don't even control the certificate provisioning process. We use Heroku and Webflow. This is frustrating.
- tingletech 6y ago
  "the DST Root X3 root certificate that we relied on to get us off the ground is going to expire - on September 1, 2021."
- toast0 6y ago
  Their cross cert expires in 10 months. Switching in January gives most people time to notice the problem while there's an easy temporary fix (switch to the soon to be expiring cross cert while you evaluate the root compatability of commercial CAs across the devices you support). I imagine the cross cert cost a bunch of money, and they may not have the money to do that again.
- laughinghan 6y ago
  Did you miss the part where the "universally accepted" root certificate is going to be universally rejected in 10 months?
paulpauper 6y ago
i hate how google puts warnings on non-ssl sites. why doe a static page that has no forms need ssl? non-ssl worked fine for 20 years for webpages and google comes along and says noooo not good enough.
- superdisk 6y ago
  The NSA can see what pages you read, and men in the middle can modify the page to insert malicious JS or ads or whatever without HTTPS.
  - kickscondor 6y ago
    Ok - next question then: why do browsers block self-signed certs? If Lets Encrypt now allows any domain to get a cert, what's the harm in a self-signed cert? Seems like a step up from plain HTTP.
    - ocdtrekkie 6y ago
      The theory here is a self-signed cert could be from anyone (including the NSA) and you wouldn't know. Unless you explicitly trusted the certificate you were using, like enterprises do.
      - neop1x 6y ago
        NSA can probably already issue certs from some of the widely-trusted roots. If someone as big as NSA wanted to MITM you, you wouldn't notice...
    - closeparen 6y ago
      What you are trusting, when you trust a CA, is that it will only issue a certificate for a domain to someone it has verified as the owner. A self-signed cert could be from a man in the middle attacker. An active MITM is a relatively more exotic threat than a passive observer. In the days when certificates cost money, there was a legitimate argument that you shouldn't need the whole elaborate active-MITM defense just to get protection against passive snooping. But now that you can get both for free... just use Let's Encrypt.
legulere 6y ago
Now that they’re standing on their own feet can they also offer S/MIME certificates for email encryption?
DanielDent 6y ago
I think a fairly manageable path forward is underway which makes this a smaller issue than it first seems: (1) Firefox already uses its own root store (2) App developers can include additional roots in addition to the system root store: https://developer.android.com/training/articles/security-config https://developer.android.com/training/articles/security-con... (3) Chrome is migrating to using it's own store: "Historically, Chrome has integrated with the Root Store provided by the platform on which it is running. Chrome is in the process of transitioning certificate verification to use a common implementation on all platforms where it's under application control, namely Android, Chrome OS, Linux, Windows, and macOS. Apple policies prevent the Chrome Root Store and verifier from being used on Chrome for iOS." https://www.chromium.org/Home/chromium-security/root-ca-policy https://www.chromium.org/Home/chromium-security/root-ca-poli...
colinclerk 6y ago
Does anyone have experiences with ZeroSSL? Caddy has been building in support so I think it could be a drop-in replacement for Caddy/CertMagic/ACMEx users.
- regecks 6y ago
  ZeroSSL is operated by Sectigo (Comodo). It works fine with Certbot and should work fine with any ACME client. I did run into some random 503s occasionally, but that was pretty soon after it was launched. Maybe just a few teething issues.
- mholt 6y ago
  ACMEz* ;) Seconding regecks' comment. We're gradually making ZeroSSL a default CA for Caddy. (I am currently implementing multi-CA support into Caddy and CertMagic, so that Caddy will be able to use both Let's Encrypt and ZeroSSL for redundancy. It's the first server to support this!) This is a good thing for the ecosystem.
  - yjftsjthsd-h 6y ago
    > We're gradually making ZeroSSL a default CA for Caddy. As in, replacing LE as the default, or supplementing it? (And if the former, why?)
    - mholt 6y ago
      Note how I mentioned that Caddy will be the first server to support redundant ACME CAs, so we'll use both ZeroSSL, and Let's Encrypt for redundancy.
      - ptman 6y ago
        Why ZeroSSL and not e.g. BuyPass?
        mholt 6y ago
        BuyPass doesn't support wildcards, and only allows up to 5 subjects per certificate (Caddy only uses 1 SAN, but still) -- and Caddy is a ZeroSSL project. We also prefer shorter cert lifetimes.
Havoc 6y ago
That seems reasonable. LE is doing good work & hardly their fault Android is a fragmented mess.
miohtama 6y ago
Somewhat related, but in other thread two weeks ago people complain Google has too much power over Android ecosystem: https://news.ycombinator.com/item?id=24917918 https://news.ycombinator.com/item?id=24917918 Now, here people are suggesting Google should somehow update the old Androids. Be damned one way or the other.
- maxmcd 6y ago
  People complaining is pretty constant, yeah. People even complain about complaining.
- laksdjfkasljdf 6y ago
  Yes and Yes. And both are reasonable and not excluding. Google sells you a pocket computer with a locked down OS, not for your safety but to control the ability to run ads. If they cared about user security, they would provide updates, no matter how "slow" (their excuse) the device gets. If they didn't want full control to show ads (ads are downloaded by the GooglePlayServices, which is pretty much the kernel of all your android experience) then it would be trivial to install other android distributions like replicant. hence, both a reasonable and google is evil. They want full control and do not care about (your) security updates.
  - paxys 6y ago
    Google does provide updates. It's the device manufacturers and/or mobile operators who choose not to push them.
    - hannob 6y ago
      But that model is flawed, and it's been more than a decade to learn that. The interesting thing is: We already have a model that works much better, and it's been around for longer. If you buy a computer with Windows it is completely normal that you still get your updates from Microsoft, even if your computer is built by a company that may no longer exist by the time you install the update. (That's not to say Windows and Microsoft don't have their own security issues - but the idea that "we provide an OS and we add a middle man for OS updates that by all experience doesn't do his job" is a good model is at this point preposterous.)
vhiremath4 6y ago
The company I work at is in a high-growth phase and we are going to be expanding our global audience this coming year through various channels (SEO, performance marketing, sales, etc.). A 1-5% hit in potential customer traffic is not going to fly. Is my only option here to get off LetsEncrypt? A bit of a vent, but I would 100% had paid for a version of LetsEncrypt that supported their costs for the x-signature with IdenTrust, although I know and respect that would be off-brand for LE.
- mafuy 6y ago
  You certainly can't go to IdenTrust now either. No matter what you do, you'll lose access to those old-phone-people eventually, and paying would only lengthen the time slightly. Just like you already lost access to people with even older android phones. Some of my family is still on androids as old as version 2.
- CameronNemo 6y ago
  Is there any way you can measure how many of your users are on an unsupported Android version?
  - Aachen 6y ago
    I think most phones include it in their user agent string.
- user5994461 6y ago
  Are you on LetsEncrypt currently? From my experience working on legacy enterprise, I'd say to stay on there. Add the flag and you will get a few more months out of it '--preferred-chain "DST Root CA X3"' Android 6 is 2015. root and intermediates CA have a 10 and 5 year lifespan. I am afraid you might not be able to find something that work on old phones and new phones. Even if you do find an older CA vendor that has an ancient CA and is willing to sign (you will be forced into an enterprise contract that will take months to negotiate), it's going to be retired anytime soon and break everywhere. Last but not least. Old phones are stuck on old versions of SSL/TLS, they're not able to connect to recent websites irrelevant of the certificates. Your site is probably no exception and cut the old protocols a long time ago.
deleted 6y ago
[deleted]
wdb 6y ago
What about iOS? No word about it in this article.
- 1over137 6y ago
  Or macOS, or Windows, or the BSDs. Would be nice to see a table of which versions will have issues or not.
  - zinekeller 6y ago
    At least for Windows: if browser support is being considered, Firefox is the last browser that somewhat works (both IE and Chtome fails because of TLS 1.2) and it includes the ISRG root (because duh), and if an application still updates on XP there is a good chance that it uses OpenSSL or derivatives (which uses a different root set, likely Mozilla's). Natively? Windows 7, assuming you have installed all updates before January. Microsoft can update the roots as they please (and historically issued updates up unto Windows 2000, so root updates are a demonstrable solved problem). BSDs and Linuxes: (Usually) Uses Mozilla's trust list. Also updates separately from system updates, so unless you stick somehow with unsupported systems you already have this (and can be manually included into the trusted roots if you insist on using that outdated version). macOS: bundled with the system updates (see caveat with Firefox independently managing roots). Here, I don't know what version is the oldest one with ISRG root certs.
- floo 6y ago
  According to apple [1] ISRG Root X1 is available all the way down to iOS 10. So that would be every iPhone since the 5. As far as marketshare goes iOS 13 and 12 make up 94% of devices [2]. So I am guessing its an insignificant amount of actual users. [1] https://support.apple.com/en-us/HT204132 https://support.apple.com/en-us/HT204132 [2] https://developer.apple.com/support/app-store/ https://developer.apple.com/support/app-store/
nojvek 6y ago
Good. Let’s Encrypt is giving a massive headstart. They’re also giving an easy escape hatch for those who can’t get things ready in time. Good judgment.
cratermoon 6y ago
The real story here is the state of Android and the devices it runs on.
javajosh 6y ago
My vote is for trusting Let's Encrypt, because SSL protects my users (and myself, and each other) from MITM attacks! However, it is interesting to ask what "trust" means in this context, since I don't know the people behind Let's Encrypt, so I don't trust them in that way. What are we trusting the organizations associated with those certs to do, or not to do?
amluto 6y ago
Someone could plausibly write a high-quality exploit for Android 7 that installs the ISRG root CA certificate. /me runs
kelnos 6y ago
Does IdenTrust have another root cert that has a later expiration date, that is also included in the trust store of OSes farther back than 2016? If so, why can't LetsEncrypt ask them to start cross-signing with a different root cert? (Obviously IdenTrust is under no obligation to do so, but since they've done this much, it's not a stretch to hope they'd do more, even if they want to charge for it.)
- tialaramex 6y ago
  IdenTrust owns two newer roots which are widely trusted today, IdenTrust Commercial Root CA 1 and IdenTrust Public Sector Root CA 1 but they were only created in 2014. Perhaps somebody has a list of what's in the trust store for various historical Android builds, I do not. But I think you can assume that the team at Let's Encrypt have considered any options that might work and decided that either they wouldn't make enough difference to be worthwhile or have asked and been told it isn't possible or would be too expensive to make sense.
rkagerer 6y ago
Are there no companies out there with very old root signatures that could be acquired? It seems to me like those sigs are valuable IP just like well-known domains (like example.com) - I wonder how much one is worth.
tzs 6y ago
Question: If your site certificate is signed by a chain of intermediate certificates leading to a root that your browser knows, when you browser checks all the signatures at time T does it require that all the certificates be valid at time T, or does it just require that each certificate was valid at the time it was used for signing?
boris 6y ago
I find the marketing spin in the title and the article unfortunate (standing on our own feet). The simple truth is they've decided getting another cross-signature not worth their trouble so now us users must expect some breakages the scale of which doesn't appear to be well understood. Also, while the article focuses on the Android, my first thought was about all those outdated CentOS/Debian/etc boxes/VMs/containers that silently curl something from a cron job or such and that will all of a sudden stop working. So I expect a lot of infrastructure breakages.
jonathanoliver 6y ago
I too have wondered why IdenTrust would want to do this. As has been mentioned in this thread, it appears they focus on enterprise-level customers, governments, medical, among others. Generally speaking, the way that new competitors enter a given market is with low-cost options that are often inferior to established players. Then, as those entrants expand upmarket by offering better and improved products, the existing/established players abandon parts of their downmarket products to the new entrants. This cycle repeats until there's nowhere left for the established players to go. At that point, these upstarts can often replace the existing players and become the dominant ones. I'm not sure if the above was a deliberate strategic move on the part of IdenTrust or not, but in cross-signing the Let's Encrypt certificate, it effectively killed off the potential for new players in the low-cost TLS/SSL certificate market because there's no margin in $0. [Citation needed] Further, because the purpose of Let's Encrypt is to serve the base level of the market [1] with no apparent desire (as per the parent organization which is effectively a non-profit/public-benefit organization) to expand upmarket. This move would appear to solidify (whether intentional or not) the position of larger players who cater to larger customers while keeping any potential newer players from disrupting the space. Aside: I love Let's Encrypt and have about a dozen or so certificates issued through them that I am in charge of. They're awesome and kudos to their team for what they've been able to accomplish. When they first offered certificates, the 90-day validity period felt very restrictive. Now it feels great because the certificates are automatically rotated every 60 days per various automation tools and painful certificate renewals are very much a thing of the past for me. [1] https://letsencrypt.org/about/ https://letsencrypt.org/about/
mamborambo 6y ago
It appears the built-in time expiry period in the trust relationships between various organisations and clients will increasing become a weak point in the world in the future. Whether these moving parts are in trust only for three months, three years or more is not the issue, it is the fact that periodic resync and retrust needs to happen, means that it may result in a cascadable failure of our infrastructure in the future. Say if a nuclear disaster / earthquake take down one city, will it lead to paralysis of servers and services across the world? Perhaps it is time to start auditing the Internet and identify the weak points.
- ViViDboarder 6y ago
  That doesn’t feel like a weak point to me. In the case you described, trust likely is broken. As a user, I may not care and instruct my browser to ignore the fact that the certificate is broken because I understand the likely cause.
usr1106 6y ago
The answer there would be strong "rights to repair" legislation. In the most affected markets it would be a viable option to go to the phone shop and have a new operating system image installed for 20 Euros/Dollars. But those markets don't have the legal power against Google or Samsung. And markets that would have the legal power, don't care about unnecessary electronic waste ruining the planet. (Sent from Android 4.1 without Playstore.)
- epse 6y ago
  The scenario you describe unfortunately requires more than just right to repair. It requires the SoC vendors to either keep updating their kernels, or to open source their driver blobs. Without these, you can't build a proper new OS image, you're stuck on the latest one provided by the SoC manufacturer.
  - berkes 6y ago
    In practice, updates might be possible, but you'll often see hardware stopping working. If it is "just" bluetooth, that may be fine. When networking (wifi/GSM) or the screen stops working, it probably is not. I've done a fair bit of upgrades of ancient android phones to CyanogenMod, LineageOs and even an accidental Ubuntu Touch. Fairly common that things like the camera or bluetooth stop working (partly), but for many owners of old phones, that is certainly a trade to make. "I never use Bluetooth, what could you use it for?", "Oh, but then I'll just use this jack-cable for my sonos"). What I'm trying to say: yes: updates require SoC vendors to help, or at least stay out of the way. But no, that does not mean one cannot ever update at all.
sorenstoutner 6y ago
For some reason, Let's Encrypt failed to mention that older versions of Android can just import the ISRG Root X1 certificate, which solves the problem for all apps. https://www.stoutner.com/lets-encrypt-isrg-root-x1-and-privacy-browser/ https://www.stoutner.com/lets-encrypt-isrg-root-x1-and-priva...

584 points by gpff 6y ago

gpff 6y ago

Let’s Encrypt cross-signature with IdenTrust "DST Root X3" is ending on September 1, 2021 but 33.8% of Android devices are running versions under 7.1 which don't trust Let’s Encrypt new root certificate "ISRG Root X1"

brendanclune 6y ago
Workaround is Firefox Mobile (because it ships with its own root certs), but that's a significant burden to place on the user.
- est31 6y ago
  Also the post says that Firefox doesn't work on Androids older than 5.0 which according to the dashboard are still 5.9% of devices. For those older devices, the only option is to install the new root certificate. Anyways, there are billions of Android devices out there. 33% of those is a large number. You can't just tell all of them that they are wrong. If this happens, people will move away from Let's encrypt in masses. They don't realize yet how self-harming this really is.
  - maxerickson 6y ago
    In the post, they advertise that they are going to continue to offer a service that provides certificates signed with the old one.
    - est31 6y ago
      ... which will work until September 2021.
    - capableweb 6y ago
      That's not gonna help as the root will expire, so the devices stuck with the older root will still shows errors when trying to use them.
  - baggy_trough 6y ago
    Not sure about that. Move away from Let's Encrypt to what? More likely, most smaller to medium sized sites will say forget those old Android guys.
    - sroussey 6y ago
      Too cheap to update can be seen as too cheap to buy your product or service, so I can see this happening.
    - est31 6y ago
      I found at least Buypass offering a gratis ACME product "Buypass Go SSL". They have roots which are deployed at least since Android 4.1, which covers way more Android devices (according to the Android Studio statistics, >99%): https://android.googlesource.com/platform/libcore/+/android-4.1.1_r1/luni/src/main/files/cacerts/eb375c3e.0 https://android.googlesource.com/platform/libcore/+/android-... I'm not sure whether that particular root is being used for their Go SSL product. If so, Buypass might be a good alternative to migrate to. From what I read though, they do require an E-Mail address, so you've got to keep that in mind.
  - WorldMaker 6y ago
    Root certificate updates are a massive security issue. Blaming Let's Encrypt is blaming one of the canaries for the coal mine disaster. 33% of Android devices don't and can't get up to date root certificates is an impressive security crisis that grows worse by the year (look at the other root expirations and the crazy workarounds that for instance Netflix has been doing to still work on older Android devices). Shouldn't the blame squarely be on Google, the Android OEMs, and the phone carriers for allowing this disaster to happen in the first place? I realize that is a tough message to get out to users and site owners are going to be in the cross-fire, but it seems better to try to work for solidarity in pointing fingers at the right direction and the right direction certainly isn't Let's Encrypt.
    - cpach 6y ago
      That makes me curious, what workarounds did Netflix employ?
      - Aissen 6y ago
        When you control the client, it's simple, you can do pretty much anything: embed your own HTTP stack, TLS stack, your QUIC stack, or simply your PKI, or subset of the webPKI.
      - WorldMaker 6y ago
        It was the BBC I was actually thinking about, and it's a part of this article (which also mentions this Let's Encrypt root change): https://scotthelme.co.uk/impending-doom-root-ca-expiring-legacy-clients/ https://scotthelme.co.uk/impending-doom-root-ca-expiring-leg... Previous HN discussion on that article: https://news.ycombinator.com/item?id=23455463 https://news.ycombinator.com/item?id=23455463
    - est31 6y ago
      Yes the issue is really severe of most deployed Android devices not getting security updates, either at all, or the devices are used well beyond the update period. But this is not up to Let's Encrypt to solve. They market themselves to build products for the mass market instead of small niches of the market, say, everyone who buys a new phone every year. But then they also have to treat their product like a mass market product, and if Android users still use older versions of the OS, then Let's Encrypt should adopt for that.
  - veeti 6y ago
    You can still install the last version of Firefox to support Android 4.x.
  - notatoad 6y ago
    on androids older than 5, the browser is the old android browser instead of chrome. how many sites out there still test compatability with that? even without having to click through security warnings, the web is horribly broken on old android devices. the overlap of sites using letsencrypt and sites that care about people using android <5 has got to be vanishingly small. this isn't going to cause a move away from letsencrypt.
  - merlyn 6y ago
    How many other root certificates are going to be expiring in the next 3 years that older Android won't have updates for as well? Probably more than 1.
  - TheKIngofBelAir 6y ago
    > Also the post says that Firefox doesn't work on Androids older than 5.0 which according to the dashboard are still 5.9% of devices. For those older devices, the only option is to install the new root certificate. Microsoft Edge still gets updates on Android 4.4 KitKat
    - w3ll_w3ll_w3ll 6y ago
      I don't think Microsft Edge embeds its own root store on Android.
      - tacon 6y ago
        Is there enough incentive for Microsoft to add a root store to Edge by next September? How hard is it to make that addition?
        jsjohnst 6y ago
        I may be wrong, but the effort to switch to your own root store is more doing it securely, than the difficulty of switching from system frameworks to your own SSL/HTTP transport layers. So to put another way, straight forward to do mediocre job, not as trivial to do a good or great job.
        toast0 6y ago
        Root store and TLS/HTTP library are separate concerns. You can use the system root store with your own libraries, or you can use your own root store with the system libraries. On an Android 4.4 device, you should probably skip the system root store and the system libraries, and if you're already doing it for those phones, you might as well do it for all the phones.
        jsjohnst 6y ago
        > Root store and TLS/HTTP library are separate concerns. In the context of this thread (aka older Android devices), they aren’t truly separate concerns. You really need to do both. My point is that doing both is relatively straightforward, but doing the root store part is fairly easy to do it in a mediocre way and be brittle / insecure.
  - fuzxi 6y ago
    33% of devices, but per the article, only 1-5% of traffic on sites using LetsEncrypt. I don't see any site owners moving to a different CA when this affects less than 5% of their visitors, who are likely the poorest fraction of their userbase , i.e. probably not many paying users.
  - toyg 6y ago
    I don’t think they have a choice. Reading between the lines, the CA who cross-signed their previous root doesn’t really want to continue doing so (or asked for a lot more money) because LE usage reached levels that are just too risky. I don’t blame them: a single bad actor found doing something particularly nefarious with a LE certificate might lose them the trust their business is literally built upon. I don’t see any other CA queueing up to help what is typically their commercial nemesis. And as they say, at some point they would have to do it anyway, might as well rip the plaster off.
    - t0astbread 6y ago
      Agreed, but what could someone do with a domain validation cert (as issued by LE) that would be so nefarious to damage IdenTrust's reputation?
  - kelnos 6y ago
    If you're running Android 5.0, you also haven't benefited from updates that remove CAs that have since been shown to be untrustworthy. I think that's far worse than being unable to visit sites that use LetsEncrypt certificates. It was really short-sighted of Google to make the system cert bundle something that can't be updated without a full OS update. There should be an OTA mechanism that allows it to be updated through the Play Store or through some other means that isn't reliant upon lazy device manufacturers.
    - thayne 6y ago
      To be fair they haven't gotten mant other security updates either. It's just a bad situation all around.
- stefan_ 6y ago
  A Workaround is not something that the user does.
- nyanpasu64 6y ago
  Unfortunately, on my Moto G5 Plus (which is not an high-end device), I've found that Firefox on Android is slower than Chrome (specifically the Bromite fork). I think that Firefox may not be a good solution for low-spec or older phones running older Android versions.
- neop1x 6y ago
  Firefox is not a workaround for non-web mobile apps. Lots of them will stop working unless they do cert pinning / have their own CA bundle or simply stop using LE..
tyingq 6y ago
"https almost everywhere". Thanks G!
sbierwagen 6y ago
Good. Those devices are unsafe, and should not be used.
- mehrdadn 6y ago
  Would you like to buy new phones for their owners?
  - oh_sigh 6y ago
    The owners can just click through the prompt if they want...
    - neop1x 6y ago
      and get used to clicking on the prompt because who cares about MITM
- ed25519FUUU 6y ago
  I think you might be ignorant with regards to how the rest of the world uses the internet.
  - Nextgrid 6y ago
    But for certain businesses, there is a case to be made for not serving these kinds of customers. Someone using a shitty old Android phone might not even be able/willing to pay for your product, and if they do, might cost more in technical support and/or fraud (due to their device being vulnerable) than what they bring in revenue.
    - berkes 6y ago
      When you are a bank or a crypto exchange, that might make sense. When you are the municipality of a village in rural India or the company that handles all vehicle registrations, it does not. You cannot gatekeep with such sweeping statements. People have old phones for lots of reasons. Others will have to serve those people for lots of other reasons.
mrweasel 6y ago
The most impressive thing it that Let’s Encrypt are the ones who are trying to fix a problem that should be fixed by phone manufactures and telcos. I can see why, but I would also have like the phones to “break” so the owners would avoid those brands in the future, and pick one who care enough to push out update. Still, I can blame Let’s Encrypt, they just want to be the good guys, and the do it so beautifully and transparently.
- nsgi 6y ago
  It probably wouldn't encourage many people to upgrade their phones, though. They would probably just see the website as broken and either click through the warning or abandon it completely
  - kalleboo 6y ago
    Nearly 60% of the web is using Let's Encrypt certs now. It's not just the one of two sites are going to break, it's going to be half the web.

onlinejk 6y ago

Nice TL;DR, thx

gioele 6y ago

> Without IdenTrust, Let’s Encrypt may have never happened and we are grateful to them for their partnership. What I have never understood is why IdenTrust accepted to cross-sign Let’s Encrypt's root certificate. With that move, IdenTrust basically broke the CA cartel and helped driving the price of basic certificates to zero. How did they, as a for-profit organization, justify "doing the right thing" when that meant disrupting their own market? Anyway, kudos to IdenTrust.

admax88q 6y ago
Always better to be doing the disruption rather be the one being disrupted. Perhaps they saw the writing on the walls and wanted to boost their standing in the post LetsEncrypt world.
- est31 6y ago
  Yeah they likely got a stash of money for it while the other CAs got nothing. First mover advantage I guess. It's not that they'd have been able to prevent it anyways, as the root stores are not under their control, especially when Let's encrypt is being supported, even started, by the entities which run the root stores (Mozilla, through Firefox, and Google, through Android). Ultimately it's the entities which run and deploy the root stores who have ultimate say on which CA gets to issue certs and which doesn't.
dubcanada 6y ago
IdenTrust doesn't care about random websites, they care about HIPPA, enterprise, government, securing documents and emails, etc.
- juliend2 6y ago
  Exactly, but the funny thing is that Chrome and Firefox (Desktop at least) are no longer showing the differentiating green lock for those high-end (EV & OV) certs, but the same neutral-looking lock as those Domain-Validated (DV) certs that Let's Encrypt is issuing. I'm very grateful for IdenTrust for having made that move. I just hope it won't hurt their business too much because of that.

Aissen 6y ago

This might also concern you if your app is running on older android devices, and you have a backend that uses letsencrypt certificates, while the app is relying on the system root CAs. A fix for this could be to add Let's Encrypt's root CA to your app.

nevi-me 6y ago

Could Google possibly be able (before were discuss willingness) to push an update to root certificate via Play Services? I'd like to think that anyone not using Play Services (i.e. Android with no Play) is likely using a custom browser, and would heed a call to switch to Firefox. The problem with some devices in Africa would be that many people will using older phone often don't have enough data for the big Play updates to succeed. Teens often buy 10-100MB of data so they can use WhatsApp. (If you're from Southern Africa, and disagree with this, hit me up, you probably need to spend some time in a village ;) )

trillic 6y ago
How much does that much data typically cost??
- DominikPeters 6y ago
  Here are some South Africa prices: https://www.mtn.co.za/recharge/data https://www.mtn.co.za/recharge/data Valid for a day: 25MB for $0.32; Valid for a week: 50MB for $0.64; Valid for a month: 100MB for $1.28, 1GB for $6.35
cassepipe 6y ago
Actually I was surprised that there would be such an easy fix : Switching to Firefox. Which is apparently around 70MB, apparently affordable from what you wrote and definitely worth it if it allows you to unlock a chunk of the internet. So no need for an improbable and costly Play update.
- qqii 6y ago
  That won't fix any other apps though will it? Anything that uses chrome webview for example.
  - cassepipe 6y ago
    Oh, I did not think about that. On Android 8, you make firefox the default ... (renderer ?) for other apps but idk if it's the case for older versions of Android. On the other hand, as others have said, these devices are generally quite painful to browse on so accessing the wweb version of many apps could be a solution, plus firefox will let you place shortcuts on your home screen . I also wonder how LineageOS works on those old devices. Could be another solution.

zero37z 6y ago

now the corporates got a valid point, why you dont want to use lets encrypt? still the 33% of the devices is a quite a large number to consider.

juliend2 6y ago
But like they said in the article, those 33% of Android phones represent "1-5% of the traffic" of the "large integrators" websites that LE communicated with.
- stefan_ 6y ago
  Well that's an easy choice, lose 1-5% of traffic or pay $100 for a certificate from a vendor whose root doesn't expire next year?
  - fuzxi 6y ago
    1-5% of traffic that comes from people using devices that are at least 4 years old. Someone who can't or won't upgrade from a phone that still uses Android Marshmallow is probably not bringing in much revenue.
    - josephg 6y ago
      Exactly. The “easy choice” is to drop support for those users. I suspect most web developers won’t even notice. It’s a pity - there’s no essential reason why old phones with new batteries should get worse over time. But software kills them in so many ways.
  - neop1x 6y ago
    As long as this statement is true. They also dont's say who these big integrators are and the situation may be widely different between integrators and fields.

lixtra 6y ago

They propose to install Firefox to work around the root certificate problem on old android devices. But can’t you just manually install their root certificate on most phones?

lights0123 6y ago
It probably looks a lot less sketchy to your users to tell them to install a browser they've probably heard of and may have used in the past than install a root certificate. Plus, that doesn't fix the problem of other certificates expiring, only extends it.
- regecks 6y ago
  >Plus, that doesn't fix the problem of other certificates expiring, only extends it. Although I agree fully on the sketchiness part, installing the root is a fix. ISRG Root X1 expires in 2035. Not one of these problematic Android devices will be online anymore.
  - iudqnolq 6y ago
    But to have the whole modern web work you'll need to install more than one root.
fuzxi 6y ago
I'm not sure if it's the same on older versions, but on recent Android versions, that requires a rooted device.
- Latty 6y ago
  Interestingly, it appears to be back in 11. You are right it wasn't possible for some set of versions, not sure how far you have to go back for it to be possible again. They appear to have added it back with a big warning screen similar to what they do for VPNs and stuff telling users it could compromise them, which is reasonable. It was a pain you couldn't before.
- 6y ago

renewiltord 6y ago

> The remaining 33.8% of Android devices will eventually start getting certificate errors when users visit sites that have a Let’s Encrypt certificate. In our communications with large integrators, we have found that this represents around 1-5% of traffic to their sites. This one-third of Android devices only yields 5% of traffic? Interesting.

degenerate 6y ago
I have a 2010 Android smartphone and it's painfully slow to navigate the modern web, almost unbearable. Browsing news websites is simply not worth my time of waiting for the phone to download and process 22MB of JS, CSS, and graphics. Being on wifi makes no difference; it's the CPU choking to render all that cruft. So yeah, 5% of traffic makes sense.
- GoblinSlayer 6y ago
  In firefox (at least in desktop version) you can disable javascript and css. Not sure if they are still downloaded.
  - bzbarsky 6y ago
    For scripts: if script is disabled for a document, scripts are not downloaded. See https://searchfox.org/mozilla-central/rev/a5d9abfda1e26b1207db9549549ab0bdd73f735d/dom/script/ScriptLoader.cpp#1620-1627 https://searchfox.org/mozilla-central/rev/a5d9abfda1e26b1207... as of today For stylesheets, I'm not certain how you're disabling them. Depending on how you do it, they may or may not get downloaded. The most common ways of disabling them result in them not being downloaded.
    - GoblinSlayer 6y ago
      View - Page style - No style
      - bzbarsky 6y ago

tothrowaway 6y ago

> As of January 11, 2021, we’re planning to make a change to our API so that ACME clients will, by default, serve a certificate chain that leads to ISRG Root X1. Ouch. So anyone who wants to support older devices 3 months from now needs to add `--preferred-chain "DST Root CA X3"` to their certbot command. When that chain is completely retired in September next year, certbot appears to fallback on the default (even with that argument present).

rsweeney21 6y ago

I don't understand the motivation for making this change now. Why not keep the universally accepted root certificate as the default chain? Why does Let's Encrypt need to switch to their own root certificate now, and cause thousands of websites to break on older devices? I don't even control the certificate provisioning process. We use Heroku and Webflow. This is frustrating.

tingletech 6y ago
"the DST Root X3 root certificate that we relied on to get us off the ground is going to expire - on September 1, 2021."
toast0 6y ago
Their cross cert expires in 10 months. Switching in January gives most people time to notice the problem while there's an easy temporary fix (switch to the soon to be expiring cross cert while you evaluate the root compatability of commercial CAs across the devices you support). I imagine the cross cert cost a bunch of money, and they may not have the money to do that again.
laughinghan 6y ago
Did you miss the part where the "universally accepted" root certificate is going to be universally rejected in 10 months?

paulpauper 6y ago

i hate how google puts warnings on non-ssl sites. why doe a static page that has no forms need ssl? non-ssl worked fine for 20 years for webpages and google comes along and says noooo not good enough.

superdisk 6y ago
The NSA can see what pages you read, and men in the middle can modify the page to insert malicious JS or ads or whatever without HTTPS.
- kickscondor 6y ago
  Ok - next question then: why do browsers block self-signed certs? If Lets Encrypt now allows any domain to get a cert, what's the harm in a self-signed cert? Seems like a step up from plain HTTP.
  - ocdtrekkie 6y ago
    The theory here is a self-signed cert could be from anyone (including the NSA) and you wouldn't know. Unless you explicitly trusted the certificate you were using, like enterprises do.
    - neop1x 6y ago
      NSA can probably already issue certs from some of the widely-trusted roots. If someone as big as NSA wanted to MITM you, you wouldn't notice...
  - closeparen 6y ago
    What you are trusting, when you trust a CA, is that it will only issue a certificate for a domain to someone it has verified as the owner. A self-signed cert could be from a man in the middle attacker. An active MITM is a relatively more exotic threat than a passive observer. In the days when certificates cost money, there was a legitimate argument that you shouldn't need the whole elaborate active-MITM defense just to get protection against passive snooping. But now that you can get both for free... just use Let's Encrypt.

legulere 6y ago

Now that they’re standing on their own feet can they also offer S/MIME certificates for email encryption?

DanielDent 6y ago

I think a fairly manageable path forward is underway which makes this a smaller issue than it first seems: (1) Firefox already uses its own root store (2) App developers can include additional roots in addition to the system root store: https://developer.android.com/training/articles/security-config https://developer.android.com/training/articles/security-con... (3) Chrome is migrating to using it's own store: "Historically, Chrome has integrated with the Root Store provided by the platform on which it is running. Chrome is in the process of transitioning certificate verification to use a common implementation on all platforms where it's under application control, namely Android, Chrome OS, Linux, Windows, and macOS. Apple policies prevent the Chrome Root Store and verifier from being used on Chrome for iOS." https://www.chromium.org/Home/chromium-security/root-ca-policy https://www.chromium.org/Home/chromium-security/root-ca-poli...

colinclerk 6y ago

Does anyone have experiences with ZeroSSL? Caddy has been building in support so I think it could be a drop-in replacement for Caddy/CertMagic/ACMEx users.

regecks 6y ago
ZeroSSL is operated by Sectigo (Comodo). It works fine with Certbot and should work fine with any ACME client. I did run into some random 503s occasionally, but that was pretty soon after it was launched. Maybe just a few teething issues.
mholt 6y ago
ACMEz* ;) Seconding regecks' comment. We're gradually making ZeroSSL a default CA for Caddy. (I am currently implementing multi-CA support into Caddy and CertMagic, so that Caddy will be able to use both Let's Encrypt and ZeroSSL for redundancy. It's the first server to support this!) This is a good thing for the ecosystem.
- yjftsjthsd-h 6y ago
  > We're gradually making ZeroSSL a default CA for Caddy. As in, replacing LE as the default, or supplementing it? (And if the former, why?)
  - mholt 6y ago
    Note how I mentioned that Caddy will be the first server to support redundant ACME CAs, so we'll use both ZeroSSL, and Let's Encrypt for redundancy.
    - ptman 6y ago
      Why ZeroSSL and not e.g. BuyPass?
      - mholt 6y ago
        BuyPass doesn't support wildcards, and only allows up to 5 subjects per certificate (Caddy only uses 1 SAN, but still) -- and Caddy is a ZeroSSL project. We also prefer shorter cert lifetimes.

Havoc 6y ago

That seems reasonable. LE is doing good work & hardly their fault Android is a fragmented mess.

miohtama 6y ago

Somewhat related, but in other thread two weeks ago people complain Google has too much power over Android ecosystem: https://news.ycombinator.com/item?id=24917918 https://news.ycombinator.com/item?id=24917918 Now, here people are suggesting Google should somehow update the old Androids. Be damned one way or the other.

maxmcd 6y ago
People complaining is pretty constant, yeah. People even complain about complaining.
laksdjfkasljdf 6y ago
Yes and Yes. And both are reasonable and not excluding. Google sells you a pocket computer with a locked down OS, not for your safety but to control the ability to run ads. If they cared about user security, they would provide updates, no matter how "slow" (their excuse) the device gets. If they didn't want full control to show ads (ads are downloaded by the GooglePlayServices, which is pretty much the kernel of all your android experience) then it would be trivial to install other android distributions like replicant. hence, both a reasonable and google is evil. They want full control and do not care about (your) security updates.
- paxys 6y ago
  Google does provide updates. It's the device manufacturers and/or mobile operators who choose not to push them.
  - hannob 6y ago
    But that model is flawed, and it's been more than a decade to learn that. The interesting thing is: We already have a model that works much better, and it's been around for longer. If you buy a computer with Windows it is completely normal that you still get your updates from Microsoft, even if your computer is built by a company that may no longer exist by the time you install the update. (That's not to say Windows and Microsoft don't have their own security issues - but the idea that "we provide an OS and we add a middle man for OS updates that by all experience doesn't do his job" is a good model is at this point preposterous.)

vhiremath4 6y ago

The company I work at is in a high-growth phase and we are going to be expanding our global audience this coming year through various channels (SEO, performance marketing, sales, etc.). A 1-5% hit in potential customer traffic is not going to fly. Is my only option here to get off LetsEncrypt? A bit of a vent, but I would 100% had paid for a version of LetsEncrypt that supported their costs for the x-signature with IdenTrust, although I know and respect that would be off-brand for LE.

mafuy 6y ago
You certainly can't go to IdenTrust now either. No matter what you do, you'll lose access to those old-phone-people eventually, and paying would only lengthen the time slightly. Just like you already lost access to people with even older android phones. Some of my family is still on androids as old as version 2.
CameronNemo 6y ago
Is there any way you can measure how many of your users are on an unsupported Android version?
- Aachen 6y ago
  I think most phones include it in their user agent string.
user5994461 6y ago
Are you on LetsEncrypt currently? From my experience working on legacy enterprise, I'd say to stay on there. Add the flag and you will get a few more months out of it '--preferred-chain "DST Root CA X3"' Android 6 is 2015. root and intermediates CA have a 10 and 5 year lifespan. I am afraid you might not be able to find something that work on old phones and new phones. Even if you do find an older CA vendor that has an ancient CA and is willing to sign (you will be forced into an enterprise contract that will take months to negotiate), it's going to be retired anytime soon and break everywhere. Last but not least. Old phones are stuck on old versions of SSL/TLS, they're not able to connect to recent websites irrelevant of the certificates. Your site is probably no exception and cut the old protocols a long time ago.

deleted 6y ago

[deleted]

wdb 6y ago

What about iOS? No word about it in this article.

1over137 6y ago
Or macOS, or Windows, or the BSDs. Would be nice to see a table of which versions will have issues or not.
- zinekeller 6y ago
  At least for Windows: if browser support is being considered, Firefox is the last browser that somewhat works (both IE and Chtome fails because of TLS 1.2) and it includes the ISRG root (because duh), and if an application still updates on XP there is a good chance that it uses OpenSSL or derivatives (which uses a different root set, likely Mozilla's). Natively? Windows 7, assuming you have installed all updates before January. Microsoft can update the roots as they please (and historically issued updates up unto Windows 2000, so root updates are a demonstrable solved problem). BSDs and Linuxes: (Usually) Uses Mozilla's trust list. Also updates separately from system updates, so unless you stick somehow with unsupported systems you already have this (and can be manually included into the trusted roots if you insist on using that outdated version). macOS: bundled with the system updates (see caveat with Firefox independently managing roots). Here, I don't know what version is the oldest one with ISRG root certs.
floo 6y ago
According to apple [1] ISRG Root X1 is available all the way down to iOS 10. So that would be every iPhone since the 5. As far as marketshare goes iOS 13 and 12 make up 94% of devices [2]. So I am guessing its an insignificant amount of actual users. [1] https://support.apple.com/en-us/HT204132 https://support.apple.com/en-us/HT204132 [2] https://developer.apple.com/support/app-store/ https://developer.apple.com/support/app-store/

nojvek 6y ago

Good. Let’s Encrypt is giving a massive headstart. They’re also giving an easy escape hatch for those who can’t get things ready in time. Good judgment.

cratermoon 6y ago

The real story here is the state of Android and the devices it runs on.

javajosh 6y ago

My vote is for trusting Let's Encrypt, because SSL protects my users (and myself, and each other) from MITM attacks! However, it is interesting to ask what "trust" means in this context, since I don't know the people behind Let's Encrypt, so I don't trust them in that way. What are we trusting the organizations associated with those certs to do, or not to do?

amluto 6y ago

Someone could plausibly write a high-quality exploit for Android 7 that installs the ISRG root CA certificate. /me runs

kelnos 6y ago

Does IdenTrust have another root cert that has a later expiration date, that is also included in the trust store of OSes farther back than 2016? If so, why can't LetsEncrypt ask them to start cross-signing with a different root cert? (Obviously IdenTrust is under no obligation to do so, but since they've done this much, it's not a stretch to hope they'd do more, even if they want to charge for it.)

tialaramex 6y ago
IdenTrust owns two newer roots which are widely trusted today, IdenTrust Commercial Root CA 1 and IdenTrust Public Sector Root CA 1 but they were only created in 2014. Perhaps somebody has a list of what's in the trust store for various historical Android builds, I do not. But I think you can assume that the team at Let's Encrypt have considered any options that might work and decided that either they wouldn't make enough difference to be worthwhile or have asked and been told it isn't possible or would be too expensive to make sense.

rkagerer 6y ago

Are there no companies out there with very old root signatures that could be acquired? It seems to me like those sigs are valuable IP just like well-known domains (like example.com) - I wonder how much one is worth.

tzs 6y ago

Question: If your site certificate is signed by a chain of intermediate certificates leading to a root that your browser knows, when you browser checks all the signatures at time T does it require that all the certificates be valid at time T, or does it just require that each certificate was valid at the time it was used for signing?

boris 6y ago

I find the marketing spin in the title and the article unfortunate (standing on our own feet). The simple truth is they've decided getting another cross-signature not worth their trouble so now us users must expect some breakages the scale of which doesn't appear to be well understood. Also, while the article focuses on the Android, my first thought was about all those outdated CentOS/Debian/etc boxes/VMs/containers that silently curl something from a cron job or such and that will all of a sudden stop working. So I expect a lot of infrastructure breakages.

jonathanoliver 6y ago

I too have wondered why IdenTrust would want to do this. As has been mentioned in this thread, it appears they focus on enterprise-level customers, governments, medical, among others. Generally speaking, the way that new competitors enter a given market is with low-cost options that are often inferior to established players. Then, as those entrants expand upmarket by offering better and improved products, the existing/established players abandon parts of their downmarket products to the new entrants. This cycle repeats until there's nowhere left for the established players to go. At that point, these upstarts can often replace the existing players and become the dominant ones. I'm not sure if the above was a deliberate strategic move on the part of IdenTrust or not, but in cross-signing the Let's Encrypt certificate, it effectively killed off the potential for new players in the low-cost TLS/SSL certificate market because there's no margin in $0. [Citation needed] Further, because the purpose of Let's Encrypt is to serve the base level of the market [1] with no apparent desire (as per the parent organization which is effectively a non-profit/public-benefit organization) to expand upmarket. This move would appear to solidify (whether intentional or not) the position of larger players who cater to larger customers while keeping any potential newer players from disrupting the space. Aside: I love Let's Encrypt and have about a dozen or so certificates issued through them that I am in charge of. They're awesome and kudos to their team for what they've been able to accomplish. When they first offered certificates, the 90-day validity period felt very restrictive. Now it feels great because the certificates are automatically rotated every 60 days per various automation tools and painful certificate renewals are very much a thing of the past for me. [1] https://letsencrypt.org/about/ https://letsencrypt.org/about/

mamborambo 6y ago

It appears the built-in time expiry period in the trust relationships between various organisations and clients will increasing become a weak point in the world in the future. Whether these moving parts are in trust only for three months, three years or more is not the issue, it is the fact that periodic resync and retrust needs to happen, means that it may result in a cascadable failure of our infrastructure in the future. Say if a nuclear disaster / earthquake take down one city, will it lead to paralysis of servers and services across the world? Perhaps it is time to start auditing the Internet and identify the weak points.

ViViDboarder 6y ago
That doesn’t feel like a weak point to me. In the case you described, trust likely is broken. As a user, I may not care and instruct my browser to ignore the fact that the certificate is broken because I understand the likely cause.

usr1106 6y ago

The answer there would be strong "rights to repair" legislation. In the most affected markets it would be a viable option to go to the phone shop and have a new operating system image installed for 20 Euros/Dollars. But those markets don't have the legal power against Google or Samsung. And markets that would have the legal power, don't care about unnecessary electronic waste ruining the planet. (Sent from Android 4.1 without Playstore.)

epse 6y ago
The scenario you describe unfortunately requires more than just right to repair. It requires the SoC vendors to either keep updating their kernels, or to open source their driver blobs. Without these, you can't build a proper new OS image, you're stuck on the latest one provided by the SoC manufacturer.
- berkes 6y ago
  In practice, updates might be possible, but you'll often see hardware stopping working. If it is "just" bluetooth, that may be fine. When networking (wifi/GSM) or the screen stops working, it probably is not. I've done a fair bit of upgrades of ancient android phones to CyanogenMod, LineageOs and even an accidental Ubuntu Touch. Fairly common that things like the camera or bluetooth stop working (partly), but for many owners of old phones, that is certainly a trade to make. "I never use Bluetooth, what could you use it for?", "Oh, but then I'll just use this jack-cable for my sonos"). What I'm trying to say: yes: updates require SoC vendors to help, or at least stay out of the way. But no, that does not mean one cannot ever update at all.

sorenstoutner 6y ago

For some reason, Let's Encrypt failed to mention that older versions of Android can just import the ISRG Root X1 certificate, which solves the problem for all apps. https://www.stoutner.com/lets-encrypt-isrg-root-x1-and-privacy-browser/ https://www.stoutner.com/lets-encrypt-isrg-root-x1-and-priva...