8 ms·

Missing TLS intermediate certificates can create mysterious browser problems

80 points by zdw 4y ago

omnicognate 4y ago
This is a common server misconfiguration that clients can often work in spite of, due to certificate caching and explicit workarounds (AIA Fetching), and I sort of wish they wouldn't. I'm no expert so I'm sure people will tell me why this isn't workable, but it seems to me that if browsers were to actively require that a server provides any required intermediate certificates people would fix their servers, as they wouldn't work at all.
- tinus_hn 4y ago
  It seems extremely easy to detect this problem and pop-up an annoying, ugly but dismissible warning.
- jeroenhd 4y ago
  I don't think servers should be breaking working websites just to be pedantic if they can already work around these issues. They should definitely put warnings about these kinds of faults in the dev console, though. I don't think developers that don't get their certs in order will even bother with opening the dev tools on their website, but it's the best place I can think of. Google and other partnering websites also have a position to prevent these failures. Many websites decide to include Google's stalking scripts and it should be trivial for Google to put scan the domains these scripts run on and warn the website administrators about their bad configuration. This poses a problem for split horizon intranet websites, but in those cases you'll probably never run into the issue due to the weird setup anyway.
  - mschuster91 4y ago
    > Many websites decide to include Google's stalking scripts and it should be trivial for Google to put scan the domains these scripts run on and warn the website administrators about their bad configuration. The Google Webmaster Console actually does warn you about all issues regarding your site's ability to be indexed, including SSL issues.
- mcpherrinm 4y ago
  One issue is that there’s no “one true list of intermediates” for a server to include in the handshake, as it’s not possible to always know what roots a client trusts. The client authors (of browsers) tend to care about making their software easy to use and avoid errors. And there’s a pretty small set of clients. TLS servers on the other hand often just serve whatever you put in a config file. A chain that works for them on an up to date Firefox may not work for an older android phone, but the server administrator may not be aware of the trade off they’ve made. My personal opinion is that the tools for debugging for server authors aren’t good enough yet. I’m one of the authors of the certigo tool mentioned in the article, which was written out of frustration from a lack of other tools. Even then, I think it’s not good enough and I hope this class of issue is one we can illustrate better. SSL Labs probably does the best here.
  - hamburglar 4y ago
    > One issue is that there’s no “one true list of intermediates” for a server to include in the handshake This doesn't make sense. Given a cert that was issued from an intermediate, there is exactly one intermediate that issued it, and that intermediate must be in the handshake. If that intermediate was issued from another intermediate, that next intermediate must also be in the handshake. And so on. There's no ambiguity here. Your server is required to send a fully-formed cert chain whose final cert is signed by one of the roots the client trusts. It's entirely possible to run a PKI where the set of intermediates is known by nobody and is unbounded and ephemeral, so long as everyone who is given a cert is given the intermediate chain that issued it. The burden is on the bearer of the cert to include it in the handshake. All problems here have arisen from a misunderstanding that causes people to think the intermediates should be well-known or obtainable publicly. That is simply not true. > it’s not possible to always know what roots a client trusts It doesn't matter. You send the chain that issued your cert, not what you expect the client to trust. If the client trusts it, you're good. If not, you're not.
    - hamburglar 4y ago
      TFA complains about browser intermediate cache behavior but doesn't seem to realize that if you're relying on browser intermediate caching at all, your server is misconfigured. The lesson should be "fix your server", not "those darn browsers."
mcpherrinm 4y ago
One thing that wasn’t clear to me from this post: the certificate in question had an AIA url, and chrome will download intermediates from that. Why didn’t that work in this case?
- lbriner 4y ago
  It sounded like the intermediate cert was very old so any browsers that had visited a site that used it previously would have cached it but anyone trying to obtain it now didn't work. Seems weird though, you would think that if the intermediate had expired, the client cert would also have expired.
  - cipherboy 4y ago
    Strictly it isn't true that a leaf cert's validity must be capped by the parent issuer's, but I'm also surprised this worked. I suspect the intermediate is still valid though. The PKIX RFC 5280 says: https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2.5 https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2.... I'll leave you to parse it, but it notably lacks the requirement that CAs must truncate validity. Let's Encrypt is a great example: https://letsencrypt.org/certificates/ https://letsencrypt.org/certificates/ -- DST Root CA X3 -> ISRG Root X1 is a cross-signed root, and DST Root CA X3 expired recently (breaking some validation on Android devices). However, by issuing that cross-signed ISRG root before the DST root expired, Let's Encrypt is still working on older Android devices: https://letsencrypt.org/2020/12/21/extending-android-compatibility.html https://letsencrypt.org/2020/12/21/extending-android-compati... In practice though, most chain validation I've looked at (OpenSSL, NSS+pkix, Go) all validate the validity period of the intermediate (relative to now, not of the leaf) and will err if it isn't still valid. What they don't validate is _root_ validity (since it is ultimately trusted and in the trust store, and is again how the Android support worked). So typically you'd want to permit longer validity from roots (pending the caveat about supporting CRL/OCSP/... throughout the longer lifetime of any leaf), but truncate validity on leaves issued from intermediates.
christkv 4y ago
This tool saved me when we realized that the ssl cert bundle we had included the wrong intermediate certificates by default. https://www.digicert.com/help/ https://www.digicert.com/help/ I really would have love to find a tool that would just spit out the right intermediate certificate for your ssl certificate.
- perryh2 4y ago
  Have you seen this site: https://whatsmychaincert.com/ https://whatsmychaincert.com/
  - christkv 4y ago
    Awesome. Going straight on my list of essential tools to sleep well at night.
davidgerard 4y ago
have been caught by this one a number of times. Nowadays when I'm sending certs to vendors (that is, vendors who can't work LE), I include the fullchain cert by preference.
francislavoie 4y ago
This seems solved if everyone just uses ACME
- cpach 4y ago
  Yep. Either ACME, or some other reliable method to automate cert issuing. Manual cert deployment is IMHO bad hygiene in this day and age. (Yes, I know there’s esoteric hardware and stuff like that where automation isn’t straight-forward, but that’s the exception, not the rule.) E.g. on AWS there’s Certificate Manager, which is similar. (But obviously only usable for people on AWS.)
  - blueflow 4y ago
    Its not dependend on manual deployment, you can as well automate it wrongly.
    - cpach 4y ago
      Sure. But then you have the chance to fix your automation logic.
  - lbriner 4y ago
    > Yes, I know there’s esoteric hardware and stuff like that where automation isn’t straight-forward, but that’s the exception, not the rule. Maybe in your world but plenty of us work in places that have 1000s of servers, many of them internal, they don't support LE or there are security concerns with opening external firewalls just to support certificate updates so it still has to be done manually. You might be lucky to work somewhere much smaller/newer that doesn't have the problem but please don't make disparaging statements about those of us who have to do this manually.
    - cpach 4y ago
      It’s still possible to automate cert minting. For example, you can have your own internal CA that’s not connected to the Internet.
bumper_crop 4y ago
A similar odd case is when the intermediate cert is expired, but the root and leaf are still valid. Different TLS implementations forgive this differently!
- supertrope 4y ago
  https://letsencrypt.org/2020/12/21/extending-android-compatibility.html https://letsencrypt.org/2020/12/21/extending-android-compati...
planede 4y ago
Umm, can this intermediate certificate caching be used for fingerprinting? Looks like a privacy concern.
- jeroenhd 4y ago
  Only by generating certificate authorities on the fly. You'd also find fingerprinting attempts in the certificate transparancy log pretty quickly. I think browser vendors would take action against that stuff. I suppose it's possible for one website to pay a whole bunch of CAs for certificates, but I don't think they'd be able to get enough to actually track anyone.
  - planede 4y ago
    I don't see why you would need to generate CAs on the fly. You could have a few end-points with a certificate signed by common intermediates, but not serving the intermediate themselves. Then you can check if resources from those end-points load or not, effectively fingerprinting the user's cache for those common intermediates. I'm not sure how feasible this is, but certainly seems more feasible than generating CAs on the fly.
    - jeroenhd 4y ago
      That's possible but for that you'd need a valid certificate from every intermediate you fingerprint. You can maybe get a few bits out of it, but the moment you fingerprint the device it's got the intermediates practically forever. If the user clears all cookies, storage, etc. and you try to match the device with the old profile it'll still keep the old certs cached and it won't appear the same.
      - planede 4y ago
        > the moment you fingerprint the device it's got the intermediates practically forever That's not true, the point is that you fingerprint using misconfigured hosts, they never get the intermediates through them. Of course it's possibly hard to use this practically, but anyway, apparently Firefox deemed it to be concerning enough to eventually fix it according to https://bugzilla.mozilla.org/show_bug.cgi?id=1334485 https://bugzilla.mozilla.org/show_bug.cgi?id=1334485.
xg15 4y ago
> browsers silently cache intermediate TLS certificates that they've seen before, in addition to being willing to download them under the right circumstances A bit OT but how would you even go about downloading a certificate? From my understanding, certificates are transmitted as part of the TLS handshake, so it's not like there is some url you could send a GET request to. Or is there some facility in TLS where a client can request additional certificates?
- cipherboy 4y ago
  If you visit the site in most (desktop) browsers, you can usually click the lock icon, navigate through various intermediate steps (Firefox is Connection Secure -> View Certificate) and find details about the cert or download it (hidden in the middle). $ openssl s_client -showcerts -connect <host>:<port> also does it from the command line.
  - xg15 4y ago
    I didn't know about that command line invocation. Thanks a lot! That's just the UI side though. I'm wondering what is actually happening on the wire when I run that command. Is there some TLS opcode which lets a client request certificate from the server?
    - cipherboy 4y ago
      Nah, no op code. You just open a regular connection and then terminate it. This forces the TLS server to auth (as the CLI OpenSSL lacks session resumption tickets).
      - xg15 4y ago
        Well yes, but the whole point of this thread is servers which don't include intermediate certs in the regular handshake. The article says about this: > The website worked in other browsers because browsers silently cache intermediate TLS certificates that they've seen before, in addition to being willing to download them under the right circumstances. I understood that as "some clients can recover from that situation by either caching the certs from a previous handshake or somehow downloading them on their own". And I was curious about the last point.
whisdol 4y ago
Safari on iOS/iPadOS displays a great "white page fast reload loop" when opening a website missing the intermediate certificate of an internal CA, even when that intermediate CA is installed in the device's keychain. That behavior is so weirdly specific, it's at least easy to identify the cause if you have seen it before.
ramesh213 4y ago
[dead]

80 points by zdw 4y ago

This is a common server misconfiguration that clients can often work in spite of, due to certificate caching and explicit workarounds (AIA Fetching), and I sort of wish they wouldn't. I'm no expert so I'm sure people will tell me why this isn't workable, but it seems to me that if browsers were to actively require that a server provides any required intermediate certificates people would fix their servers, as they wouldn't work at all.

tinus_hn 4y ago
It seems extremely easy to detect this problem and pop-up an annoying, ugly but dismissible warning.
jeroenhd 4y ago
I don't think servers should be breaking working websites just to be pedantic if they can already work around these issues. They should definitely put warnings about these kinds of faults in the dev console, though. I don't think developers that don't get their certs in order will even bother with opening the dev tools on their website, but it's the best place I can think of. Google and other partnering websites also have a position to prevent these failures. Many websites decide to include Google's stalking scripts and it should be trivial for Google to put scan the domains these scripts run on and warn the website administrators about their bad configuration. This poses a problem for split horizon intranet websites, but in those cases you'll probably never run into the issue due to the weird setup anyway.
- mschuster91 4y ago
  > Many websites decide to include Google's stalking scripts and it should be trivial for Google to put scan the domains these scripts run on and warn the website administrators about their bad configuration. The Google Webmaster Console actually does warn you about all issues regarding your site's ability to be indexed, including SSL issues.
mcpherrinm 4y ago
One issue is that there’s no “one true list of intermediates” for a server to include in the handshake, as it’s not possible to always know what roots a client trusts. The client authors (of browsers) tend to care about making their software easy to use and avoid errors. And there’s a pretty small set of clients. TLS servers on the other hand often just serve whatever you put in a config file. A chain that works for them on an up to date Firefox may not work for an older android phone, but the server administrator may not be aware of the trade off they’ve made. My personal opinion is that the tools for debugging for server authors aren’t good enough yet. I’m one of the authors of the certigo tool mentioned in the article, which was written out of frustration from a lack of other tools. Even then, I think it’s not good enough and I hope this class of issue is one we can illustrate better. SSL Labs probably does the best here.
- hamburglar 4y ago
  > One issue is that there’s no “one true list of intermediates” for a server to include in the handshake This doesn't make sense. Given a cert that was issued from an intermediate, there is exactly one intermediate that issued it, and that intermediate must be in the handshake. If that intermediate was issued from another intermediate, that next intermediate must also be in the handshake. And so on. There's no ambiguity here. Your server is required to send a fully-formed cert chain whose final cert is signed by one of the roots the client trusts. It's entirely possible to run a PKI where the set of intermediates is known by nobody and is unbounded and ephemeral, so long as everyone who is given a cert is given the intermediate chain that issued it. The burden is on the bearer of the cert to include it in the handshake. All problems here have arisen from a misunderstanding that causes people to think the intermediates should be well-known or obtainable publicly. That is simply not true. > it’s not possible to always know what roots a client trusts It doesn't matter. You send the chain that issued your cert, not what you expect the client to trust. If the client trusts it, you're good. If not, you're not.
  - hamburglar 4y ago
    TFA complains about browser intermediate cache behavior but doesn't seem to realize that if you're relying on browser intermediate caching at all, your server is misconfigured. The lesson should be "fix your server", not "those darn browsers."

mcpherrinm 4y ago

One thing that wasn’t clear to me from this post: the certificate in question had an AIA url, and chrome will download intermediates from that. Why didn’t that work in this case?

lbriner 4y ago
It sounded like the intermediate cert was very old so any browsers that had visited a site that used it previously would have cached it but anyone trying to obtain it now didn't work. Seems weird though, you would think that if the intermediate had expired, the client cert would also have expired.
- cipherboy 4y ago
  Strictly it isn't true that a leaf cert's validity must be capped by the parent issuer's, but I'm also surprised this worked. I suspect the intermediate is still valid though. The PKIX RFC 5280 says: https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2.5 https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2.... I'll leave you to parse it, but it notably lacks the requirement that CAs must truncate validity. Let's Encrypt is a great example: https://letsencrypt.org/certificates/ https://letsencrypt.org/certificates/ -- DST Root CA X3 -> ISRG Root X1 is a cross-signed root, and DST Root CA X3 expired recently (breaking some validation on Android devices). However, by issuing that cross-signed ISRG root before the DST root expired, Let's Encrypt is still working on older Android devices: https://letsencrypt.org/2020/12/21/extending-android-compatibility.html https://letsencrypt.org/2020/12/21/extending-android-compati... In practice though, most chain validation I've looked at (OpenSSL, NSS+pkix, Go) all validate the validity period of the intermediate (relative to now, not of the leaf) and will err if it isn't still valid. What they don't validate is _root_ validity (since it is ultimately trusted and in the trust store, and is again how the Android support worked). So typically you'd want to permit longer validity from roots (pending the caveat about supporting CRL/OCSP/... throughout the longer lifetime of any leaf), but truncate validity on leaves issued from intermediates.

christkv 4y ago

This tool saved me when we realized that the ssl cert bundle we had included the wrong intermediate certificates by default. https://www.digicert.com/help/ https://www.digicert.com/help/ I really would have love to find a tool that would just spit out the right intermediate certificate for your ssl certificate.

perryh2 4y ago
Have you seen this site: https://whatsmychaincert.com/ https://whatsmychaincert.com/
- christkv 4y ago
  Awesome. Going straight on my list of essential tools to sleep well at night.

davidgerard 4y ago

have been caught by this one a number of times. Nowadays when I'm sending certs to vendors (that is, vendors who can't work LE), I include the fullchain cert by preference.

francislavoie 4y ago

This seems solved if everyone just uses ACME

cpach 4y ago
Yep. Either ACME, or some other reliable method to automate cert issuing. Manual cert deployment is IMHO bad hygiene in this day and age. (Yes, I know there’s esoteric hardware and stuff like that where automation isn’t straight-forward, but that’s the exception, not the rule.) E.g. on AWS there’s Certificate Manager, which is similar. (But obviously only usable for people on AWS.)
- blueflow 4y ago
  Its not dependend on manual deployment, you can as well automate it wrongly.
  - cpach 4y ago
    Sure. But then you have the chance to fix your automation logic.
- lbriner 4y ago
  > Yes, I know there’s esoteric hardware and stuff like that where automation isn’t straight-forward, but that’s the exception, not the rule. Maybe in your world but plenty of us work in places that have 1000s of servers, many of them internal, they don't support LE or there are security concerns with opening external firewalls just to support certificate updates so it still has to be done manually. You might be lucky to work somewhere much smaller/newer that doesn't have the problem but please don't make disparaging statements about those of us who have to do this manually.
  - cpach 4y ago
    It’s still possible to automate cert minting. For example, you can have your own internal CA that’s not connected to the Internet.

bumper_crop 4y ago

A similar odd case is when the intermediate cert is expired, but the root and leaf are still valid. Different TLS implementations forgive this differently!

supertrope 4y ago
https://letsencrypt.org/2020/12/21/extending-android-compatibility.html https://letsencrypt.org/2020/12/21/extending-android-compati...

planede 4y ago

Umm, can this intermediate certificate caching be used for fingerprinting? Looks like a privacy concern.

jeroenhd 4y ago
Only by generating certificate authorities on the fly. You'd also find fingerprinting attempts in the certificate transparancy log pretty quickly. I think browser vendors would take action against that stuff. I suppose it's possible for one website to pay a whole bunch of CAs for certificates, but I don't think they'd be able to get enough to actually track anyone.
- planede 4y ago
  I don't see why you would need to generate CAs on the fly. You could have a few end-points with a certificate signed by common intermediates, but not serving the intermediate themselves. Then you can check if resources from those end-points load or not, effectively fingerprinting the user's cache for those common intermediates. I'm not sure how feasible this is, but certainly seems more feasible than generating CAs on the fly.
  - jeroenhd 4y ago
    That's possible but for that you'd need a valid certificate from every intermediate you fingerprint. You can maybe get a few bits out of it, but the moment you fingerprint the device it's got the intermediates practically forever. If the user clears all cookies, storage, etc. and you try to match the device with the old profile it'll still keep the old certs cached and it won't appear the same.
    - planede 4y ago
      > the moment you fingerprint the device it's got the intermediates practically forever That's not true, the point is that you fingerprint using misconfigured hosts, they never get the intermediates through them. Of course it's possibly hard to use this practically, but anyway, apparently Firefox deemed it to be concerning enough to eventually fix it according to https://bugzilla.mozilla.org/show_bug.cgi?id=1334485 https://bugzilla.mozilla.org/show_bug.cgi?id=1334485.

xg15 4y ago

> browsers silently cache intermediate TLS certificates that they've seen before, in addition to being willing to download them under the right circumstances A bit OT but how would you even go about downloading a certificate? From my understanding, certificates are transmitted as part of the TLS handshake, so it's not like there is some url you could send a GET request to. Or is there some facility in TLS where a client can request additional certificates?

cipherboy 4y ago
If you visit the site in most (desktop) browsers, you can usually click the lock icon, navigate through various intermediate steps (Firefox is Connection Secure -> View Certificate) and find details about the cert or download it (hidden in the middle). $ openssl s_client -showcerts -connect <host>:<port> also does it from the command line.
- xg15 4y ago
  I didn't know about that command line invocation. Thanks a lot! That's just the UI side though. I'm wondering what is actually happening on the wire when I run that command. Is there some TLS opcode which lets a client request certificate from the server?
  - cipherboy 4y ago
    Nah, no op code. You just open a regular connection and then terminate it. This forces the TLS server to auth (as the CLI OpenSSL lacks session resumption tickets).
    - xg15 4y ago
      Well yes, but the whole point of this thread is servers which don't include intermediate certs in the regular handshake. The article says about this: > The website worked in other browsers because browsers silently cache intermediate TLS certificates that they've seen before, in addition to being willing to download them under the right circumstances. I understood that as "some clients can recover from that situation by either caching the certs from a previous handshake or somehow downloading them on their own". And I was curious about the last point.

whisdol 4y ago

Safari on iOS/iPadOS displays a great "white page fast reload loop" when opening a website missing the intermediate certificate of an internal CA, even when that intermediate CA is installed in the device's keychain. That behavior is so weirdly specific, it's at least easy to identify the cause if you have seen it before.

ramesh213 4y ago

[dead]