27 ms·

The Impending Doom of Expiring Root CAs and Legacy Clients

234 points by fabian2k 6y ago

jakub_g 6y ago
Smart TVs are the worst. Unrelated to TLS: I work for a video streaming company, and while almost all of desktop/mobile Chromium-based traffic is Chrome v80+ or at least v70+, the smart TVs have webviews based on Chromium v45~55 (most TVs) or ~v60 (cutting edge modern Samsungs), and some older ones (~2016) are even worse, based on a very outdated webkit which is more or less Safari 8.0 - basically hardly better than IE11. But coming back to TLS again: even if the old devices have an up-to-date root store, they typically have an outdated TLS stack which does not support TLS 1.2 ¯\_(ツ)_/¯ and this one is probably even harder to update. And since most of the companies want to drop TLS 1.1- support on their servers soon, this should also doom those outdated devices apart from the root stores issues.
- deleted 6y ago
  [deleted]
- kstrauser 6y ago
  If this is the newspaper-whack-to-the-nose that gets people to stop using sly TVs as more than dumb monitors, then I think we'd all be better off for it.
deleted 6y ago
[deleted]
geerlingguy 6y ago
I begrudgingly bought a 'Smart TV' just because I wanted a 4K monitor. But I use it as a dumb monitor and watch app-based content through an Apple TV. At least I'm reasonably sure the Apple TV will be updated for 5-10 years, and is a lot more disposable than a giant TV. My last TV worked great for about 12 years. No way Panasonic would've kept supporting it that long. Honestly, the thing I hate most about smart TVs, especially budget-conscious ones like the one I bought, is they have terribly slow processors so just to turn on the TV takes like 12 seconds. My old TV had a picture up within 3 seconds of turning it on. Does any manufacturer make a good TV, or are they all smart now?
- pletnes 6y ago
  What we did for our old tv, was to get a chromecast after 3-4 years. This might be a cheaper and more environmentally friendly approach than yours, as we then buy one «smart-device», but you might end up with 2, over 12 years. But then some (all?) smart tv offerings have issues with terms and conditions, privacy, etc which may make them undesirable for you.
  - briffle 6y ago
    I never hooked up the smart tv to my wireless network. I have heard too many stories about data getting sent back to the manufacturers to sell your watching habits, etc. Instead, I use a Roku, that I know gets updates (and yes, they also sell your watching habits, but so far, nextdns.io's lists block the DNS that roku uses to do this.)
    - dathinab 6y ago
      Problem some do connect to any non secured WiFi they can find.
      - katbyte 6y ago
        I purposely bought and expensive one that’s pure android tv where you can disable all the tracking bits
        jannes 6y ago
        You bought an Android device to avoid tracking? lol As long as you're using binaries that were compiled by Google I wouldn't trust them one bit. If you use LineageOS on your TV you might be a little better off.
        katbyte 6y ago
        Android by Sony verses Samsung or webos? I’m happy with my choice.
devit 6y ago
Having expiring certificates/key rotation might be a net negative: if you keep the private key secure, there is no need to rotate it, and it avoids a lot of hassle. Also, if you have a revocation mechanism, then rotation doesn't add that much for keys that are only used for signing and not for encryption (like the CA keys). Of course in some cases like domains it's necessary since the domain can be transferred, but this is solvable by DNSSEC+DANE.
- gregmac 6y ago
  It's a good point. At any given time, the key is either compromised or it's not. And you may or may not know either way. If it's compromised, and you know, then it should be revoked -- if your only mechanism to revoke it is waiting some amount of time (several days, months, or even a decade) you have a pretty big problem. If you don't have a way to detect if it's compromised, rotating a precaution almost makes sense -- but then the valid time should be very short. I'm not exactly sure how short, but definitely the decade or two used for CA certs is too long. Make it shorter than a year and automated renewal becomes necessary, but if you have that mechanism why not check for revocations? The only other reason I can think of to expire non-compromised certificates is to force them to be regenerated to use newer signature algorithms, but even that's difficult to predict. We're currently using SHA-2 for PKI, but is that still going to be reasonable in 5, 10 years? Someone could figure out a potential weakness at any time, which would start the move to something else.
  - tialaramex 6y ago
    What we're talking about here are trust roots and one of the things people usually get wrong when they try to draw a diagram of how this works is they draw certificates as nodes and then just connect those nodes together with abstract lines. Actually the correct mental model is a graph of public keys and the lines between them (joining two nodes directionally) are the certificates. So although we traditionally handle the root trust set as a bunch of self-signed certificates, those certificates are largely unimportant, what's vital is the public keys baked inside them. As a result what makes older roots obsolete is not a signature algorithm, but the type and size of key chosen when they were created, that key is in a very real sense the root. This AddTrust root for example was 2048-bit RSA. You can use that size of RSA key today for your funny cat video site, no problem, but it's clearly an inadequate choice for a CA root. Fortunately roots like this are gradually expiring. If I figure out how to build a machine that can break 2048-bit RSA keys for $10M per key it makes no sense to target your cat videos. But a CA root is an attractive target. So we'd like to have more margin for the roots not the same or less. Some years ago Mozilla finally prohibited 1024-bit RSA keys in roots. Some of the oldest roots in the business were 1024-bit RSA, which today would not be considered acceptable even on your cat video site, but when those roots were created 1024-bit RSA seemed safe enough. In 5-10 years you'd probably want as much as possible for roots to be the more compact elliptic curve public keys, maybe there will be some better (more secure) curves in use by then, maybe not.
tialaramex 6y ago
> If we take a look at similar data for iOS, it's a very different story. This fits the prevailing narrative, and for all I know it's true, but Scott does not in fact illustrate it. The "similar data" to a table of all Android API versions versus popularity turns out to be a chart of recent iOS versions in which everything that isn't current is just grouped as "Other" at 16.28% Maybe if we ungrouped "Other" it would prove Scott's point, maybe not, we shan't find out answers in his article. If those "Other" iOS users are all running 10.3 it's a very different story than if they're on iOS 6...
- idbehold 6y ago
  He links directly to the source of that data and on that site you can download a CSV of the aggregated data. "iOS 13.3",27.03 "iOS 12.4",16.19 "iOS 12.3",14.2 "iOS 13.1",10.22 "iOS 12.2",7.89 "iOS 12.1",4.16 "iOS 13.4",4.02 "iOS 13.2",2.68 "iOS 10.3",2.31 "iOS 11.4",2.14 "iOS 9.3",1.94 "iOS 11.2",1.03 "iOS 12.0",0.86 "iOS 13.0",0.82 "iOS 11.0",0.81 "iOS 11.3",0.72 "iOS 6.0",0.45 "iOS 10.2",0.4 "iOS 11.1",0.32 "iOS 7.1",0.18 "iOS 13.5",0.17 "iOS 10.1",0.17 "iOS 9.1",0.15 "iOS 7.0",0.15 "iOS 10.0",0.14 "iOS 6.1",0.13 "iOS 5.1",0.12 "iOS 9.2",0.12 "iOS 8.4",0.11 "iOS 8.1",0.1 "iOS 8.3",0.06 "iOS 9.0",0.06 "iOS 5.0",0.04 "iOS 8.0",0.03 "iOS 4.3",0.03 "iOS 8.2",0.02 "iOS 3.2",0.01 "Other",0.02
gambler 6y ago
A working device should never stop working because of an expiration of some made-up entity. It is a signs of compete failure in basic system design.
- withinboredom 6y ago
  For sure. Add to the fact that in every crypto-library I’ve used, you can’t tell it to ignore the expiration date without also ignoring actually harmful certificates, and this is what you end up with. If devs could just say: yeah we are not ever going to update the certificate, I’d be ok with that. An expired certificate is not inherently untrustworthy, it should just mean that it can’t sign anything. They’d then die of their own accord or be revoked. Tin foil hat: To me, it smells more like CA’s wanted to make some cash by forcing their customers to buy certificates every so often instead of actually solving the problem and are now being bitten by their own rules.
- deleted 6y ago
  [deleted]
tialaramex 6y ago
If you have a current Firefox that first diagram may not be correct for you. Let's tell a brief story about why. Servers, as Scott explains, are supposed to present a "chain" of certificates, in practice it's one leaf and then just anything else that might be useful for clients to assemble a trust path. If you do this everything works. But lots of servers are misconfigured and present only the leaf certificate, not least because a lot of older software makes configuring the chain needlessly confusing/ complicated and it often "seems" to work anyway in some client software. Given just a leaf, and a set of roots it trusts, a client can't make a useful trust decision, there's a gap. One option in this circumstance is to just give up and just tell the client's user this isn't secure. That's a poor UX. Another idea is called AIA Chasing, many popular web browsers do this. When it can't get from the presented certificates to a trusted root the browser inspects the Authority Information Access field of a certificate to find a URL, hopefully this URL points to an intermediate certificate that plugs the gap. A problem with AIA chasing is that it reveals information about your browsing to the CA who otherwise have no reason to know. This privacy concern led Mozilla to reject AIA Chasing. Yet another idea is to cache intermediate certificates for some period of time, then use that cache to fill any "gap" when making a trust path for other servers you connect to. This causes hard-to-predict errors for users, maybe the funny cat video site you enjoy works fine... unless you try to visit it first thing after turning on the PC. It can potentially represent a privacy risk within a session. So ultimately Mozilla's workaround in current Firefox is to give Firefox a fairly complete set of intermediates, just as if it had cached them already. Essentially that right hand "embedded in browser" part of the diagram extends left to the intermediates too. It made that happen through two means: Firstly m.d.s.policy required all root CAs to tell it about all unconstrained intermediates they had ever created, which was itself a revealing process because there sure are a lot of (mostly older) garbage intermediates that clearly are untrustworthy and shouldn't exist. More recently Certificate Transparency means any intermediates still in use must be in the CT logs because otherwise the leaf certificates won't be accepted by those logs, so Mozilla can "just" fetch that data.
- diroussel 6y ago
  Interesting note about Firefox there. I've certainly expirenced the missing "missing intermediate cert" problem before. It is a bit of a nightmare to debug. 1. Had issue, raise support ticket: TLS not working 2. Ticket closed as can't reproduce 3. Try myself again locally, also can't reproduce. Hmpf! 4. 2 months goes by.. 5. Experience same issue. Debug more carefully locally, use openssl go get proof of missing intermediate 6. Raise support ticket with platform team, then try to convince them that just because most people are not affected, it still needs to be fixed!
topkai22 6y ago
Certificates suck. They are monumentally user unfriendly and complex. How many major internet properties have had outages driven by certificate renewal in the last several years? Half? I know that enterprises I work with seem to have some sort of work impacting certificate problem every month. The ecosystem needs to become a lot more robust and user friendly in general. Unfortunately, I’m just a user of certificates not a cryptographer, so I’m not really qualified to design security critical aspects of the system, but here are some rough features that I’m looking for: - certificate warning date: a time period defined like expiry date that indicates that clients should warn of impending expiration but still click though - Make it easier for people to acquire and renew certs. Let’s encrypt is an amazing start here, but it’s not a universal solution yet - Formalize a way to solve the key distribution problem described in the article. Again, I’m not a security expert, but perhaps embedding a “replaces” concept into a certificate would work.
- deleted 6y ago
  [deleted]
- danpalmer 6y ago
  While I empathise with your viewpoint, that the ecosystem is hard to work with, I'm afraid I don't think there are any quick solutions to this. > certificate warning date If this is user-visible then for most companies this would be nearly as bad as expiry, and if it's not user visible then it's not useful. > Make it easier for people to acquire and renew certs Let's Encrypt pretty much solves this problem. The places where it doesn't work are companies that lock down the use of that sort of software. I don't think there's an easy way around that. > Formalize a way to solve the key distribution problem described in the article This problem isn't just the problem described in the article, it's _THE_ Key Distribution Problem. It has a Wikipedia entry, it's a chapter in most security textbooks, it's a lecture in most cryptography modules at universities. It's a _hard problem_. The practice of including Root CA certs on devices is the best thing we've come up with yet that has the security and scalability trade-offs necessary. If the "replaces" concept was viable in the general case I think we'd have likely implemented it, but I suspect it's too vulnerable, and likely only pushes back the problem.
stormdennis 6y ago
We have our smart TV just acting as a display for a mini PC via HDMI connection. TV knows nothing about our WiFi so no phoning home.
- tomxor 6y ago
  Have you had fun hunting for and turning off all of the awful picture "enhancing" features yet? and then figuring out how to make it persistent. Modern TVs really seem to ruin the video even though they are perfectly capable of just showing a color adjusted signal, quickly becomes obvious when you plug in a computer.
  - stormdennis 6y ago
    I don't know to be truthful. The TV remote has loads of buttons bottom the only ones I use are on/off, source to find the HDMI and volume sometimes. I'd love just to have a dumb monitor of the same size but they're not available. A bonus then would be no need to have a tv licence.
    - jfk13 6y ago
      If you're referring to a UK TV licence, it doesn't matter what kind of monitor you're using; it's about what you watch (live TV, whether over the air or online, or any use of BBC iPlayer). Watching TV via a computer and a dumb monitor still requires a licence. https://www.tvlicensing.co.uk/check-if-you-need-one https://www.tvlicensing.co.uk/check-if-you-need-one (Edited to add: Conversely, merely owning a TV does not require a license; if you're only using it as a computer or gaming monitor, and not watching [online] TV programmes on it, you should be fine.)
      - stormdennis 6y ago
        Unfortunately not. Having a tv set in the house here requires a licence even the thing is out of order
        6y ago
mschuster91 6y ago
I got hit at work by the AddTrust issue too, quite a bit of a nightmare to chase down, especially Java keystores. And that was mostly because there exist clients who barf if they have a valid working chain but also another chain where one of the certs has expired! For embedded devices - it's not just smart TVs but all IoT devices! - I think that the solution is a legal requirement: a condition of a device being able to be sold on the EU / US market must be the commit of the whole source tree and build chain including any and all key material for signing firmwares to the national library or other government-run secure escrow, to be released to the public once the manufacturer discontinues the product support. Similar to like you can't sell a product without adhering to the CE and electrical safety standards, manufacturers should not be able to sell products without having them certified for IT safety and e-waste/"planned obsolescence" issues!
afwaller 6y ago
We had to deal with the sectigo / addtrust expiration. It was annoying because we reached out to our cert vendor and they had claimed it would not affect us (only affecting “legacy” systems or old browsers), but then of course it did affect us. It was an easy resolution, but still annoying. Of course, we still have people working on our stuff. If it were some abandoned hardware product I could see this being a disaster.
- rmetzler 6y ago
  We too were affected by the Sectigo / Addtrust expiration. Google's Uptime Checks couldn't verify the validity of the certificate and reported our website was down. While this was a false alert and it was easy to verify and circumvent it just shows that even companies like Google aren't prepared for something like this.
teddyh 6y ago
To quote myself¹ from two years ago: Any Internet-connected device is, in fact, a server, and must be seen and managed as one. This means strict control of installed services and, first and foremost, regular updates of all its software components (including firmware). If you acquire and install such a server which either can’t be updated or one which you know, realistically, won’t get any updates six months after installation, that’s asking to lose. 1. https://news.ycombinator.com/item?id=18019343 https://news.ycombinator.com/item?id=18019343
superkuh 6y ago
Once again the persuit of security theater to meet the requirements of commerce (which most of the web doesn't share) cause problems accessing the large parts of the normal web. When are people going to learn that centralized cert authorities are just for commerce and only hurt the non-commercial web?
- Drdrdrq 6y ago
  Curious - what do you suggest for non-commercial web, taking into account both privacy and security?
  - superkuh 6y ago
    If those are your motivations then making the site accessible as an onion service over tor in addition to the http interface (and https self signed) seems reasonable.
Androider 6y ago
TIL about Network Error Logging[1], which is having browsers send you reports that your DNS isn't resolving, your load balancer is timing out (HTTP / TCP errors in general), or your TLS certificate is invalid. Basically all types of issues that don't end up in your own application server logs. Report URI should lead with this (it's not even mentioned on the front page)! It's super useful for every site, I can't think why anyone wouldn't want this. CSP monitoring on the other hand, which seems to be their focus, is a much harder sell, a pain to setup and maintain, and a bit questionable value in my opinion. [1] https://report-uri.com/products/network_error_logging https://report-uri.com/products/network_error_logging
- parliament32 6y ago
  This is the actual tech, rather than a commercial solution: https://w3c.github.io/network-error-logging/#introduction https://w3c.github.io/network-error-logging/#introduction https://developer.mozilla.org/en-US/docs/Web/HTTP/Network_Error_Logging https://developer.mozilla.org/en-US/docs/Web/HTTP/Network_Er...
  - Androider 6y ago
    Report URI is run by Scott and was featured in the article. By definition you have to run such a service entirely separate from your own infrastructure: you should use a different CA than your other site(s), a different domain name for sure, preferably a different infrastructure host etc. It makes a lot of sense to use something like the Report URI service for it.
  - nitrogen 6y ago
    This looks incredibly useful! It'll also probably tell you who is using Pi-hole though.
deleted 6y ago
[deleted]
dehrmann 6y ago
You almost need something in HTTP for a client to signal which root certificate is being used, then if it's old, push the new one (signed by the old one) in the response. There's plenty of potential for abuse with this, though. Maybe root certs need to expire more often so implementors have to do OTA updates.
rootedbox 6y ago
Who hasn't had a root ca issue in production. It's v painful.
syats 6y ago
This will also be an issue with some docker images built some years ago with an outdated cert store somewhere in there. Unmaintained images are, in effect, frozen in time. I'll just mention glassfish here in case someone in the near future wonders why this or that application is failing: it has it's own cert store (different from the JVM one), and it is a bit out of date. Yes, I just ran into a github project whose last commit was 3 months ago (march 2020) and that wouldn't work with LE certificates, all because deep down the docker matrioshka, there was a glassfish with outdated certs.
hamiltont 6y ago
Has anyone taken the offered training? I'm pretty good with TLS, but could use more hands-on time with openssl and PKI, especially a deeper understanding of certificate chaining issues. Would be interesting to know if the training is targeted at "total beginner" or "intermediate looking to be advanced"
- payamb 6y ago
  I’ve taken the training and I recommend it, Keep in mind though that TLS and PKI are broad subjects and covering every intermediate/advanced use case in a two days training won’t be possible. But yeah it does cover certificate chaining issues, cross signing etc.
AshamedCaptain 6y ago
"Impending" doom? Devices from 10 years ago are _already_ useless online. Not only are the root certificates completely expired, but let's see which protocols does an openssl 0.9.8 build from a decade ago have in common with, say, current Gmail's IMAPS server: NONE.
- imhoguy 6y ago
  My TP-LINK Wifi-N router bought in 2010 still works nicely on latest OpenWRT and I assure you it runs software more up to date than 99% of closed firmware devices currently on the market. Open-source firmware is a way to go to stay current and to avoid producing more electronic waste.
- GoblinSlayer 6y ago
  ISRG root is 2 years old. And even uses RSA.
- account42 6y ago
  I am really conflicted when configuring my servers. I can see the need to deprecate insecure protocols but deliberately cutting off compatibility with old clients feels wrong.
imhoguy 6y ago
That is why we need open hardware specs, open-source firmware and a right to reverse-engineer out of market devices. Like OpenWRT or LineageOS, they keep old devices usable for long and reduce electronic waste.
grishka 6y ago
This might be a stupid question, but... Why do root certificates have to expire at all? The article presents this as if it's an immutable law of physics.
- MaxBarraclough 6y ago
  We have no real means of revoking certificates whose private keys have leaked. Eventual expiration seems better than nothing. Also, ciphers tend to become obsolete, so enforcing churn will help keep things moving forward. I'm not an expert here though, there may be other reasons.
  - radnom 6y ago
    > Eventual expiration seems better than nothing. Does it? Because that is the cause of the 'impending doom' from TFA
  - grishka 6y ago
    I feel like if a certificate gets compromised, the fact that it expires in 5 years won't be of any help from the security standpoint. There are already mechanisms in place to revoke certificates before they expire. https://en.wikipedia.org/wiki/Certificate_revocation_list https://en.wikipedia.org/wiki/Certificate_revocation_list https://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol https://en.wikipedia.org/wiki/Online_Certificate_Status_Prot...
lol768 6y ago
This post honestly reads like an ad for the training course.. did it really need 8 mentions on the page?

234 points by fabian2k 6y ago

Smart TVs are the worst. Unrelated to TLS: I work for a video streaming company, and while almost all of desktop/mobile Chromium-based traffic is Chrome v80+ or at least v70+, the smart TVs have webviews based on Chromium v45~55 (most TVs) or ~v60 (cutting edge modern Samsungs), and some older ones (~2016) are even worse, based on a very outdated webkit which is more or less Safari 8.0 - basically hardly better than IE11. But coming back to TLS again: even if the old devices have an up-to-date root store, they typically have an outdated TLS stack which does not support TLS 1.2 ¯\_(ツ)_/¯ and this one is probably even harder to update. And since most of the companies want to drop TLS 1.1- support on their servers soon, this should also doom those outdated devices apart from the root stores issues.

deleted 6y ago
[deleted]
kstrauser 6y ago
If this is the newspaper-whack-to-the-nose that gets people to stop using sly TVs as more than dumb monitors, then I think we'd all be better off for it.

deleted 6y ago

[deleted]

geerlingguy 6y ago

I begrudgingly bought a 'Smart TV' just because I wanted a 4K monitor. But I use it as a dumb monitor and watch app-based content through an Apple TV. At least I'm reasonably sure the Apple TV will be updated for 5-10 years, and is a lot more disposable than a giant TV. My last TV worked great for about 12 years. No way Panasonic would've kept supporting it that long. Honestly, the thing I hate most about smart TVs, especially budget-conscious ones like the one I bought, is they have terribly slow processors so just to turn on the TV takes like 12 seconds. My old TV had a picture up within 3 seconds of turning it on. Does any manufacturer make a good TV, or are they all smart now?

pletnes 6y ago
What we did for our old tv, was to get a chromecast after 3-4 years. This might be a cheaper and more environmentally friendly approach than yours, as we then buy one «smart-device», but you might end up with 2, over 12 years. But then some (all?) smart tv offerings have issues with terms and conditions, privacy, etc which may make them undesirable for you.
- briffle 6y ago
  I never hooked up the smart tv to my wireless network. I have heard too many stories about data getting sent back to the manufacturers to sell your watching habits, etc. Instead, I use a Roku, that I know gets updates (and yes, they also sell your watching habits, but so far, nextdns.io's lists block the DNS that roku uses to do this.)
  - dathinab 6y ago
    Problem some do connect to any non secured WiFi they can find.
    - katbyte 6y ago
      I purposely bought and expensive one that’s pure android tv where you can disable all the tracking bits
      - jannes 6y ago
        You bought an Android device to avoid tracking? lol As long as you're using binaries that were compiled by Google I wouldn't trust them one bit. If you use LineageOS on your TV you might be a little better off.
        katbyte 6y ago
        Android by Sony verses Samsung or webos? I’m happy with my choice.

devit 6y ago

Having expiring certificates/key rotation might be a net negative: if you keep the private key secure, there is no need to rotate it, and it avoids a lot of hassle. Also, if you have a revocation mechanism, then rotation doesn't add that much for keys that are only used for signing and not for encryption (like the CA keys). Of course in some cases like domains it's necessary since the domain can be transferred, but this is solvable by DNSSEC+DANE.

gregmac 6y ago
It's a good point. At any given time, the key is either compromised or it's not. And you may or may not know either way. If it's compromised, and you know, then it should be revoked -- if your only mechanism to revoke it is waiting some amount of time (several days, months, or even a decade) you have a pretty big problem. If you don't have a way to detect if it's compromised, rotating a precaution almost makes sense -- but then the valid time should be very short. I'm not exactly sure how short, but definitely the decade or two used for CA certs is too long. Make it shorter than a year and automated renewal becomes necessary, but if you have that mechanism why not check for revocations? The only other reason I can think of to expire non-compromised certificates is to force them to be regenerated to use newer signature algorithms, but even that's difficult to predict. We're currently using SHA-2 for PKI, but is that still going to be reasonable in 5, 10 years? Someone could figure out a potential weakness at any time, which would start the move to something else.
- tialaramex 6y ago
  What we're talking about here are trust roots and one of the things people usually get wrong when they try to draw a diagram of how this works is they draw certificates as nodes and then just connect those nodes together with abstract lines. Actually the correct mental model is a graph of public keys and the lines between them (joining two nodes directionally) are the certificates. So although we traditionally handle the root trust set as a bunch of self-signed certificates, those certificates are largely unimportant, what's vital is the public keys baked inside them. As a result what makes older roots obsolete is not a signature algorithm, but the type and size of key chosen when they were created, that key is in a very real sense the root. This AddTrust root for example was 2048-bit RSA. You can use that size of RSA key today for your funny cat video site, no problem, but it's clearly an inadequate choice for a CA root. Fortunately roots like this are gradually expiring. If I figure out how to build a machine that can break 2048-bit RSA keys for $10M per key it makes no sense to target your cat videos. But a CA root is an attractive target. So we'd like to have more margin for the roots not the same or less. Some years ago Mozilla finally prohibited 1024-bit RSA keys in roots. Some of the oldest roots in the business were 1024-bit RSA, which today would not be considered acceptable even on your cat video site, but when those roots were created 1024-bit RSA seemed safe enough. In 5-10 years you'd probably want as much as possible for roots to be the more compact elliptic curve public keys, maybe there will be some better (more secure) curves in use by then, maybe not.

tialaramex 6y ago

> If we take a look at similar data for iOS, it's a very different story. This fits the prevailing narrative, and for all I know it's true, but Scott does not in fact illustrate it. The "similar data" to a table of all Android API versions versus popularity turns out to be a chart of recent iOS versions in which everything that isn't current is just grouped as "Other" at 16.28% Maybe if we ungrouped "Other" it would prove Scott's point, maybe not, we shan't find out answers in his article. If those "Other" iOS users are all running 10.3 it's a very different story than if they're on iOS 6...

idbehold 6y ago
He links directly to the source of that data and on that site you can download a CSV of the aggregated data. "iOS 13.3",27.03 "iOS 12.4",16.19 "iOS 12.3",14.2 "iOS 13.1",10.22 "iOS 12.2",7.89 "iOS 12.1",4.16 "iOS 13.4",4.02 "iOS 13.2",2.68 "iOS 10.3",2.31 "iOS 11.4",2.14 "iOS 9.3",1.94 "iOS 11.2",1.03 "iOS 12.0",0.86 "iOS 13.0",0.82 "iOS 11.0",0.81 "iOS 11.3",0.72 "iOS 6.0",0.45 "iOS 10.2",0.4 "iOS 11.1",0.32 "iOS 7.1",0.18 "iOS 13.5",0.17 "iOS 10.1",0.17 "iOS 9.1",0.15 "iOS 7.0",0.15 "iOS 10.0",0.14 "iOS 6.1",0.13 "iOS 5.1",0.12 "iOS 9.2",0.12 "iOS 8.4",0.11 "iOS 8.1",0.1 "iOS 8.3",0.06 "iOS 9.0",0.06 "iOS 5.0",0.04 "iOS 8.0",0.03 "iOS 4.3",0.03 "iOS 8.2",0.02 "iOS 3.2",0.01 "Other",0.02

gambler 6y ago

A working device should never stop working because of an expiration of some made-up entity. It is a signs of compete failure in basic system design.

withinboredom 6y ago
For sure. Add to the fact that in every crypto-library I’ve used, you can’t tell it to ignore the expiration date without also ignoring actually harmful certificates, and this is what you end up with. If devs could just say: yeah we are not ever going to update the certificate, I’d be ok with that. An expired certificate is not inherently untrustworthy, it should just mean that it can’t sign anything. They’d then die of their own accord or be revoked. Tin foil hat: To me, it smells more like CA’s wanted to make some cash by forcing their customers to buy certificates every so often instead of actually solving the problem and are now being bitten by their own rules.
deleted 6y ago
[deleted]

tialaramex 6y ago

If you have a current Firefox that first diagram may not be correct for you. Let's tell a brief story about why. Servers, as Scott explains, are supposed to present a "chain" of certificates, in practice it's one leaf and then just anything else that might be useful for clients to assemble a trust path. If you do this everything works. But lots of servers are misconfigured and present only the leaf certificate, not least because a lot of older software makes configuring the chain needlessly confusing/ complicated and it often "seems" to work anyway in some client software. Given just a leaf, and a set of roots it trusts, a client can't make a useful trust decision, there's a gap. One option in this circumstance is to just give up and just tell the client's user this isn't secure. That's a poor UX. Another idea is called AIA Chasing, many popular web browsers do this. When it can't get from the presented certificates to a trusted root the browser inspects the Authority Information Access field of a certificate to find a URL, hopefully this URL points to an intermediate certificate that plugs the gap. A problem with AIA chasing is that it reveals information about your browsing to the CA who otherwise have no reason to know. This privacy concern led Mozilla to reject AIA Chasing. Yet another idea is to cache intermediate certificates for some period of time, then use that cache to fill any "gap" when making a trust path for other servers you connect to. This causes hard-to-predict errors for users, maybe the funny cat video site you enjoy works fine... unless you try to visit it first thing after turning on the PC. It can potentially represent a privacy risk within a session. So ultimately Mozilla's workaround in current Firefox is to give Firefox a fairly complete set of intermediates, just as if it had cached them already. Essentially that right hand "embedded in browser" part of the diagram extends left to the intermediates too. It made that happen through two means: Firstly m.d.s.policy required all root CAs to tell it about all unconstrained intermediates they had ever created, which was itself a revealing process because there sure are a lot of (mostly older) garbage intermediates that clearly are untrustworthy and shouldn't exist. More recently Certificate Transparency means any intermediates still in use must be in the CT logs because otherwise the leaf certificates won't be accepted by those logs, so Mozilla can "just" fetch that data.

diroussel 6y ago
Interesting note about Firefox there. I've certainly expirenced the missing "missing intermediate cert" problem before. It is a bit of a nightmare to debug. 1. Had issue, raise support ticket: TLS not working 2. Ticket closed as can't reproduce 3. Try myself again locally, also can't reproduce. Hmpf! 4. 2 months goes by.. 5. Experience same issue. Debug more carefully locally, use openssl go get proof of missing intermediate 6. Raise support ticket with platform team, then try to convince them that just because most people are not affected, it still needs to be fixed!

topkai22 6y ago

Certificates suck. They are monumentally user unfriendly and complex. How many major internet properties have had outages driven by certificate renewal in the last several years? Half? I know that enterprises I work with seem to have some sort of work impacting certificate problem every month. The ecosystem needs to become a lot more robust and user friendly in general. Unfortunately, I’m just a user of certificates not a cryptographer, so I’m not really qualified to design security critical aspects of the system, but here are some rough features that I’m looking for: - certificate warning date: a time period defined like expiry date that indicates that clients should warn of impending expiration but still click though - Make it easier for people to acquire and renew certs. Let’s encrypt is an amazing start here, but it’s not a universal solution yet - Formalize a way to solve the key distribution problem described in the article. Again, I’m not a security expert, but perhaps embedding a “replaces” concept into a certificate would work.

deleted 6y ago
[deleted]
danpalmer 6y ago
While I empathise with your viewpoint, that the ecosystem is hard to work with, I'm afraid I don't think there are any quick solutions to this. > certificate warning date If this is user-visible then for most companies this would be nearly as bad as expiry, and if it's not user visible then it's not useful. > Make it easier for people to acquire and renew certs Let's Encrypt pretty much solves this problem. The places where it doesn't work are companies that lock down the use of that sort of software. I don't think there's an easy way around that. > Formalize a way to solve the key distribution problem described in the article This problem isn't just the problem described in the article, it's _THE_ Key Distribution Problem. It has a Wikipedia entry, it's a chapter in most security textbooks, it's a lecture in most cryptography modules at universities. It's a _hard problem_. The practice of including Root CA certs on devices is the best thing we've come up with yet that has the security and scalability trade-offs necessary. If the "replaces" concept was viable in the general case I think we'd have likely implemented it, but I suspect it's too vulnerable, and likely only pushes back the problem.

stormdennis 6y ago

We have our smart TV just acting as a display for a mini PC via HDMI connection. TV knows nothing about our WiFi so no phoning home.

tomxor 6y ago
Have you had fun hunting for and turning off all of the awful picture "enhancing" features yet? and then figuring out how to make it persistent. Modern TVs really seem to ruin the video even though they are perfectly capable of just showing a color adjusted signal, quickly becomes obvious when you plug in a computer.
- stormdennis 6y ago
  I don't know to be truthful. The TV remote has loads of buttons bottom the only ones I use are on/off, source to find the HDMI and volume sometimes. I'd love just to have a dumb monitor of the same size but they're not available. A bonus then would be no need to have a tv licence.
  - jfk13 6y ago
    If you're referring to a UK TV licence, it doesn't matter what kind of monitor you're using; it's about what you watch (live TV, whether over the air or online, or any use of BBC iPlayer). Watching TV via a computer and a dumb monitor still requires a licence. https://www.tvlicensing.co.uk/check-if-you-need-one https://www.tvlicensing.co.uk/check-if-you-need-one (Edited to add: Conversely, merely owning a TV does not require a license; if you're only using it as a computer or gaming monitor, and not watching [online] TV programmes on it, you should be fine.)
    - stormdennis 6y ago
      Unfortunately not. Having a tv set in the house here requires a licence even the thing is out of order
      - 6y ago

mschuster91 6y ago

I got hit at work by the AddTrust issue too, quite a bit of a nightmare to chase down, especially Java keystores. And that was mostly because there exist clients who barf if they have a valid working chain but also another chain where one of the certs has expired! For embedded devices - it's not just smart TVs but all IoT devices! - I think that the solution is a legal requirement: a condition of a device being able to be sold on the EU / US market must be the commit of the whole source tree and build chain including any and all key material for signing firmwares to the national library or other government-run secure escrow, to be released to the public once the manufacturer discontinues the product support. Similar to like you can't sell a product without adhering to the CE and electrical safety standards, manufacturers should not be able to sell products without having them certified for IT safety and e-waste/"planned obsolescence" issues!

afwaller 6y ago

We had to deal with the sectigo / addtrust expiration. It was annoying because we reached out to our cert vendor and they had claimed it would not affect us (only affecting “legacy” systems or old browsers), but then of course it did affect us. It was an easy resolution, but still annoying. Of course, we still have people working on our stuff. If it were some abandoned hardware product I could see this being a disaster.

rmetzler 6y ago
We too were affected by the Sectigo / Addtrust expiration. Google's Uptime Checks couldn't verify the validity of the certificate and reported our website was down. While this was a false alert and it was easy to verify and circumvent it just shows that even companies like Google aren't prepared for something like this.

teddyh 6y ago

To quote myself¹ from two years ago: Any Internet-connected device is, in fact, a server, and must be seen and managed as one. This means strict control of installed services and, first and foremost, regular updates of all its software components (including firmware). If you acquire and install such a server which either can’t be updated or one which you know, realistically, won’t get any updates six months after installation, that’s asking to lose. 1. https://news.ycombinator.com/item?id=18019343 https://news.ycombinator.com/item?id=18019343

superkuh 6y ago

Once again the persuit of security theater to meet the requirements of commerce (which most of the web doesn't share) cause problems accessing the large parts of the normal web. When are people going to learn that centralized cert authorities are just for commerce and only hurt the non-commercial web?

Drdrdrq 6y ago
Curious - what do you suggest for non-commercial web, taking into account both privacy and security?
- superkuh 6y ago
  If those are your motivations then making the site accessible as an onion service over tor in addition to the http interface (and https self signed) seems reasonable.

Androider 6y ago

TIL about Network Error Logging[1], which is having browsers send you reports that your DNS isn't resolving, your load balancer is timing out (HTTP / TCP errors in general), or your TLS certificate is invalid. Basically all types of issues that don't end up in your own application server logs. Report URI should lead with this (it's not even mentioned on the front page)! It's super useful for every site, I can't think why anyone wouldn't want this. CSP monitoring on the other hand, which seems to be their focus, is a much harder sell, a pain to setup and maintain, and a bit questionable value in my opinion. [1] https://report-uri.com/products/network_error_logging https://report-uri.com/products/network_error_logging

parliament32 6y ago
This is the actual tech, rather than a commercial solution: https://w3c.github.io/network-error-logging/#introduction https://w3c.github.io/network-error-logging/#introduction https://developer.mozilla.org/en-US/docs/Web/HTTP/Network_Error_Logging https://developer.mozilla.org/en-US/docs/Web/HTTP/Network_Er...
- Androider 6y ago
  Report URI is run by Scott and was featured in the article. By definition you have to run such a service entirely separate from your own infrastructure: you should use a different CA than your other site(s), a different domain name for sure, preferably a different infrastructure host etc. It makes a lot of sense to use something like the Report URI service for it.
- nitrogen 6y ago
  This looks incredibly useful! It'll also probably tell you who is using Pi-hole though.

deleted 6y ago

[deleted]

dehrmann 6y ago

You almost need something in HTTP for a client to signal which root certificate is being used, then if it's old, push the new one (signed by the old one) in the response. There's plenty of potential for abuse with this, though. Maybe root certs need to expire more often so implementors have to do OTA updates.

rootedbox 6y ago

Who hasn't had a root ca issue in production. It's v painful.

syats 6y ago

This will also be an issue with some docker images built some years ago with an outdated cert store somewhere in there. Unmaintained images are, in effect, frozen in time. I'll just mention glassfish here in case someone in the near future wonders why this or that application is failing: it has it's own cert store (different from the JVM one), and it is a bit out of date. Yes, I just ran into a github project whose last commit was 3 months ago (march 2020) and that wouldn't work with LE certificates, all because deep down the docker matrioshka, there was a glassfish with outdated certs.

hamiltont 6y ago

Has anyone taken the offered training? I'm pretty good with TLS, but could use more hands-on time with openssl and PKI, especially a deeper understanding of certificate chaining issues. Would be interesting to know if the training is targeted at "total beginner" or "intermediate looking to be advanced"

payamb 6y ago
I’ve taken the training and I recommend it, Keep in mind though that TLS and PKI are broad subjects and covering every intermediate/advanced use case in a two days training won’t be possible. But yeah it does cover certificate chaining issues, cross signing etc.

AshamedCaptain 6y ago

"Impending" doom? Devices from 10 years ago are _already_ useless online. Not only are the root certificates completely expired, but let's see which protocols does an openssl 0.9.8 build from a decade ago have in common with, say, current Gmail's IMAPS server: NONE.

imhoguy 6y ago
My TP-LINK Wifi-N router bought in 2010 still works nicely on latest OpenWRT and I assure you it runs software more up to date than 99% of closed firmware devices currently on the market. Open-source firmware is a way to go to stay current and to avoid producing more electronic waste.
GoblinSlayer 6y ago
ISRG root is 2 years old. And even uses RSA.
account42 6y ago
I am really conflicted when configuring my servers. I can see the need to deprecate insecure protocols but deliberately cutting off compatibility with old clients feels wrong.

imhoguy 6y ago

That is why we need open hardware specs, open-source firmware and a right to reverse-engineer out of market devices. Like OpenWRT or LineageOS, they keep old devices usable for long and reduce electronic waste.

grishka 6y ago

This might be a stupid question, but... Why do root certificates have to expire at all? The article presents this as if it's an immutable law of physics.

MaxBarraclough 6y ago
We have no real means of revoking certificates whose private keys have leaked. Eventual expiration seems better than nothing. Also, ciphers tend to become obsolete, so enforcing churn will help keep things moving forward. I'm not an expert here though, there may be other reasons.
- radnom 6y ago
  > Eventual expiration seems better than nothing. Does it? Because that is the cause of the 'impending doom' from TFA
- grishka 6y ago
  I feel like if a certificate gets compromised, the fact that it expires in 5 years won't be of any help from the security standpoint. There are already mechanisms in place to revoke certificates before they expire. https://en.wikipedia.org/wiki/Certificate_revocation_list https://en.wikipedia.org/wiki/Certificate_revocation_list https://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol https://en.wikipedia.org/wiki/Online_Certificate_Status_Prot...

lol768 6y ago

This post honestly reads like an ad for the training course.. did it really need 8 mentions on the page?