4 ms·

As part of this effort, Mozilla requires disclosure of all intermediate CA certificates in the WebPKI. They bundle that list in Firefox, so that even if a serve

As part of this effort, Mozilla requires disclosure of all intermediate CA certificates in the WebPKI. They bundle that list in Firefox, so that even if a server is misconfigured and sends the wrong certificate chain (but a valid leaf certificate), they can successfully establish a TLS connection. It's pretty cool, and less confusing than the caching approach of other browsers, which leads to non-deterministic behavior. Using the list they publish [1] I built a Go package that provides the same feature, as both a x509.CertPool or a tls.VerifyConnection callback, to allow clients to connect to misconfigured servers: https://pkg.go.dev/filippo.io/intermediates https://pkg.go.dev/filippo.io/intermediates The pool is regenerated [2] by a GitHub Action [3] every night, and embedded into the package so it requires no network connection. If tests fail on the new pool, it doesn't get committed. It's actually kinda interesting watching the intermediates come and go [4], and it's very satisfying to have a self-maintaining package. [1] https://ccadb-public.secure.force.com/mozilla/MozillaIntermediateCertsCSVReport https://ccadb-public.secure.force.com/mozilla/MozillaInterme... [2] https://github.com/FiloSottile/intermediates/blob/7dfa9179/generate.go https://github.com/FiloSottile/intermediates/blob/7dfa9179/g... [3] https://github.com/FiloSottile/intermediates/blob/7dfa91796/.github/workflows/generate.yml https://github.com/FiloSottile/intermediates/blob/7dfa91796/... [4] https://github.com/FiloSottile/intermediates/commits/main https://github.com/FiloSottile/intermediates/commits/main

tialaramex 5y ago
I would have liked the world where AIA chasing happened in servers. Today several browsers which don't have Mozilla's list of intermediates instead (if they don't have a satisfactory chain offered to them) do AIA chasing, meaning that the browser consults the Authority Information Access fields in the X.509 certificate, finds a URL in there and fetches that URL hoping to get a certificate it trusts, often it iterates several times before giving up or reaching a dead end. AIA chasing in browsers is a (small) privacy risk and a (non-trivial) performance cost, plus it can just fail if you don't have as much Internet access, perhaps because you need to talk to a server, whose certificate you are trying to validate, before getting access; or of course it can just fail because temporarily some server is down somewhere. If server software did AIA chasing (e.g. nginx, or your SMTP server) it would see during startup that you configured a bare certificate, and do the same AIA chasing that browsers do today but only once for the lifetime of the service and with the privilege of being a server (likely better Internet access, and more likely somebody technical sees the error output if there are problems) and it could have retry/fallback behaviour and if it doesn't work well the server can tell its administrator to fix the problem, whereas the browser doesn't have a way to contact that admin. But Mozilla's choice here was pragmatic. We get the world we deserve.
- jcims 5y ago
  Your suggestion would definitely be more efficient. I’d split the difference and just say the server could just abort startup if it can’t validate its own cert. Letting servers indiscriminately connect to the internet isn’t great security practice, something that was substantially reinforced with the recent log4j issue.
- ayende 5y ago
  I hate this idea. What happens if the server starts at a point in time when it cannot do the AIA chasing? That means that you have a (silent) dependency that you are absolutely unaware of. When this fails, your service is down. Far better to have it setup so the service fails if the certificate is not configured with the whole chain.
  - tialaramex 5y ago
    What I propose (AIA chasing in servers) can be dropped into existing servers and it won't make anything worse, in most cases it will make things better. What you propose either can't be dropped in (so it's an interesting experiment for a new web server, are you writing one? No? Then it's of no value) or if it's dropped in it breaks working systems, making you a nuisance. If you have a time machine so you can go back to the mid-1990s and ensure all web servers work your way your idea is better or at least good. But you don't have a time machine, we must work with what exists today.
  - patmorgan23 5y ago
    If the AIA fails it could just continue as normal.
hamburglar 5y ago
> even if a server is misconfigured and sends the wrong certificate chain (but a valid leaf certificate), they can successfully establish a TLS connection. It's pretty cool I don’t really get why we want this. What is wrong with insisting that servers have their cert chains configured properly? This feels like one of those early web problems caused by implementations being too permissive with invalid input. Set your server up properly (it isn’t hard) and your TLS will work.
- tialaramex 5y ago
  Mozilla's decision avoids tumbling all the way down the slope in a must-win race to the bottom to reduce user pushback for mis-configured sites. Previously the situation was many sites seem to work fine in (for example) Edge, but don't work all the time in Firefox. What do you do if you're the average user? Do you track down every site owner whose site malfunctions, drive over and insist they fix it? No. You delete Firefox and switch to Edge. Nothing is "wrong" with your perfect world outcome, Mozilla's change doesn't prevent it - it just won't happen anyway.
  - hamburglar 5y ago
    I hear you, but I also am encouraged that, for example, in 2017-ish, Chrome was able to turn on enforcement of deprecating CommonName in favor of SAN for single-name serverauth certs. They flipped a switch and a lot of sites broke, then fixed themselves. There was no mass exodus. You could argue that this was because of Chrome's dominance and I'd agree to an extent, but I like the fact that the solution was not just "have browsers pretend this is ok." And that was for a change that really did not matter from a security standpoint, it was just catching up to an RFC change.
    - tialaramex 5y ago
      > They flipped a switch and a lot of sites broke, then fixed themselves. No. This isn't a thing that happened. Certificates which lack SANs matching a DNS name claimed as Common Name had been non-compliant since PKIX last century. So in the subsequent decade plus the big problem was identifying CAs which were flouting these rules and then getting them to understand what they'd done wrong and stop doing it. That's something which CA/B (the CA/B BRs forbade this from the outset) and m.d.s.policy were getting good traction on by 2013 or so, but Certificate Transparency (initially seeded by Google's "Googlebot" robots before the Chrome mandate) represents a sea change beyond that because now finding non-compliant certificates was a trivial SQL query. And that took such non-compliant certificates from "rare but you will find them if you look hard" to "extinct" very quickly indeed. Even when there were still a few technically non-compliant certificates out there, they wouldn't break Chrome, the usual non-compliance went like this: Say you own a site with an IDN https://президент.рф/ https://xn--d1abbgf6aiiy.xn--p1ai/ and so the DNS entry for that is xn--d1abbgf6aiiy.xn--p1ai - your SAN DnsName will say xn--d1abbgf6aiiy.xn--p1ai because that's how SANs work, but you persuade the issuing CA to write президент.рф in the Common Name instead of xn--d1abbgf6aiiy.xn--p1ai because it looks nicer. [ NB Hacker News won't show the IDN to most (any?) of you but imagine you're a Russian and your web browser shows this URL in Cyrillic not as Punycode ] This is non-compliant with PKIX, because this Common Name doesn't match any of the SANs. It can't because it's Unicode and none of the SANs can express Unicode [IpAddress SANs are numbers, and DnsName SANs are restricted to a simple alphabet suitable for writing names of hosts in DNS]. But it doesn't cause any problems for Chrome because the SANs match the real DNS names of the server, the Common Name is just decorative garbage that doesn't match anything.
cortesoft 5y ago
> They bundle that list in Firefox, so that even if a server is misconfigured and sends the wrong certificate chain (but a valid leaf certificate), they can successfully establish a TLS connection. It's pretty cool, and less confusing than the caching approach of other browsers, which leads to non-deterministic behavior. This seems like it could lead to even more confusion... now the site works on Mozilla but not other browsers. If the person setting up the server happens to use Mozilla to test, they won't realize they misconfigured it. Meanwhile, everyone using different browsers if failing to load their site. I would prefer a loud failure than a semi-failure.
- floatingatoll 5y ago
  The last time I reported a Letsencrypt CA misconfiguration to a major open source software ops team that was resulting in SSL errors in some of their clients, they decided they knew better than I did how to operate TLS on the web and refused my report about sending the wrong intermediate, because they were expert users. As far as I know, they’re still inaccessible to this day in any browser that doesn’t do what Firefox does, that’s running on any slightly older unpatched OS, and they still think it’s everyone else’s fault that they’re receiving errors. SSL is too complex to successfully validate all possible errors locally over time without reports of semi-failures, and human superiority at dismissing semi-failures as non-issues remains unchallenged. So while I appreciate the spirit of the argument, I don’t think it works out that way in practice, and I can’t really find fault with the Firefox team for deciding (TIL) to ship a better intermediate navigator for doing so. The alternative is directing user complaints to often-uncaring operators.
- tialaramex 5y ago
  > This seems like it could lead to even more confusion... They say making predictions about the future is difficult, but you're predicting something about the past. Where was the "even more confusion" in the long period where Mozilla shipped this feature? In this world, Mozilla's Firefox browser has to cope with the stupid but true fact that millions of servers are mis-configured, and being "loud" about that just ensures users will switch to a different browser.
throw0101a 5y ago
> The pool is regenerated [2] by a GitHub Action [3] every night, and embedded into the package so it requires no network connection. Once someone pulls the code and compiles it into a binary, how does the list of certs get updated? Do you have have a tzcode and tzdata split à la the IANA/Olson timezone database?

tialaramex 5y ago

I would have liked the world where AIA chasing happened in servers. Today several browsers which don't have Mozilla's list of intermediates instead (if they don't have a satisfactory chain offered to them) do AIA chasing, meaning that the browser consults the Authority Information Access fields in the X.509 certificate, finds a URL in there and fetches that URL hoping to get a certificate it trusts, often it iterates several times before giving up or reaching a dead end. AIA chasing in browsers is a (small) privacy risk and a (non-trivial) performance cost, plus it can just fail if you don't have as much Internet access, perhaps because you need to talk to a server, whose certificate you are trying to validate, before getting access; or of course it can just fail because temporarily some server is down somewhere. If server software did AIA chasing (e.g. nginx, or your SMTP server) it would see during startup that you configured a bare certificate, and do the same AIA chasing that browsers do today but only once for the lifetime of the service and with the privilege of being a server (likely better Internet access, and more likely somebody technical sees the error output if there are problems) and it could have retry/fallback behaviour and if it doesn't work well the server can tell its administrator to fix the problem, whereas the browser doesn't have a way to contact that admin. But Mozilla's choice here was pragmatic. We get the world we deserve.

jcims 5y ago
Your suggestion would definitely be more efficient. I’d split the difference and just say the server could just abort startup if it can’t validate its own cert. Letting servers indiscriminately connect to the internet isn’t great security practice, something that was substantially reinforced with the recent log4j issue.
ayende 5y ago
I hate this idea. What happens if the server starts at a point in time when it cannot do the AIA chasing? That means that you have a (silent) dependency that you are absolutely unaware of. When this fails, your service is down. Far better to have it setup so the service fails if the certificate is not configured with the whole chain.
- tialaramex 5y ago
  What I propose (AIA chasing in servers) can be dropped into existing servers and it won't make anything worse, in most cases it will make things better. What you propose either can't be dropped in (so it's an interesting experiment for a new web server, are you writing one? No? Then it's of no value) or if it's dropped in it breaks working systems, making you a nuisance. If you have a time machine so you can go back to the mid-1990s and ensure all web servers work your way your idea is better or at least good. But you don't have a time machine, we must work with what exists today.
- patmorgan23 5y ago
  If the AIA fails it could just continue as normal.

hamburglar 5y ago

> even if a server is misconfigured and sends the wrong certificate chain (but a valid leaf certificate), they can successfully establish a TLS connection. It's pretty cool I don’t really get why we want this. What is wrong with insisting that servers have their cert chains configured properly? This feels like one of those early web problems caused by implementations being too permissive with invalid input. Set your server up properly (it isn’t hard) and your TLS will work.

tialaramex 5y ago
Mozilla's decision avoids tumbling all the way down the slope in a must-win race to the bottom to reduce user pushback for mis-configured sites. Previously the situation was many sites seem to work fine in (for example) Edge, but don't work all the time in Firefox. What do you do if you're the average user? Do you track down every site owner whose site malfunctions, drive over and insist they fix it? No. You delete Firefox and switch to Edge. Nothing is "wrong" with your perfect world outcome, Mozilla's change doesn't prevent it - it just won't happen anyway.
- hamburglar 5y ago
  I hear you, but I also am encouraged that, for example, in 2017-ish, Chrome was able to turn on enforcement of deprecating CommonName in favor of SAN for single-name serverauth certs. They flipped a switch and a lot of sites broke, then fixed themselves. There was no mass exodus. You could argue that this was because of Chrome's dominance and I'd agree to an extent, but I like the fact that the solution was not just "have browsers pretend this is ok." And that was for a change that really did not matter from a security standpoint, it was just catching up to an RFC change.
  - tialaramex 5y ago
    > They flipped a switch and a lot of sites broke, then fixed themselves. No. This isn't a thing that happened. Certificates which lack SANs matching a DNS name claimed as Common Name had been non-compliant since PKIX last century. So in the subsequent decade plus the big problem was identifying CAs which were flouting these rules and then getting them to understand what they'd done wrong and stop doing it. That's something which CA/B (the CA/B BRs forbade this from the outset) and m.d.s.policy were getting good traction on by 2013 or so, but Certificate Transparency (initially seeded by Google's "Googlebot" robots before the Chrome mandate) represents a sea change beyond that because now finding non-compliant certificates was a trivial SQL query. And that took such non-compliant certificates from "rare but you will find them if you look hard" to "extinct" very quickly indeed. Even when there were still a few technically non-compliant certificates out there, they wouldn't break Chrome, the usual non-compliance went like this: Say you own a site with an IDN https://президент.рф/ https://xn--d1abbgf6aiiy.xn--p1ai/ and so the DNS entry for that is xn--d1abbgf6aiiy.xn--p1ai - your SAN DnsName will say xn--d1abbgf6aiiy.xn--p1ai because that's how SANs work, but you persuade the issuing CA to write президент.рф in the Common Name instead of xn--d1abbgf6aiiy.xn--p1ai because it looks nicer. [ NB Hacker News won't show the IDN to most (any?) of you but imagine you're a Russian and your web browser shows this URL in Cyrillic not as Punycode ] This is non-compliant with PKIX, because this Common Name doesn't match any of the SANs. It can't because it's Unicode and none of the SANs can express Unicode [IpAddress SANs are numbers, and DnsName SANs are restricted to a simple alphabet suitable for writing names of hosts in DNS]. But it doesn't cause any problems for Chrome because the SANs match the real DNS names of the server, the Common Name is just decorative garbage that doesn't match anything.

cortesoft 5y ago

> They bundle that list in Firefox, so that even if a server is misconfigured and sends the wrong certificate chain (but a valid leaf certificate), they can successfully establish a TLS connection. It's pretty cool, and less confusing than the caching approach of other browsers, which leads to non-deterministic behavior. This seems like it could lead to even more confusion... now the site works on Mozilla but not other browsers. If the person setting up the server happens to use Mozilla to test, they won't realize they misconfigured it. Meanwhile, everyone using different browsers if failing to load their site. I would prefer a loud failure than a semi-failure.

floatingatoll 5y ago
The last time I reported a Letsencrypt CA misconfiguration to a major open source software ops team that was resulting in SSL errors in some of their clients, they decided they knew better than I did how to operate TLS on the web and refused my report about sending the wrong intermediate, because they were expert users. As far as I know, they’re still inaccessible to this day in any browser that doesn’t do what Firefox does, that’s running on any slightly older unpatched OS, and they still think it’s everyone else’s fault that they’re receiving errors. SSL is too complex to successfully validate all possible errors locally over time without reports of semi-failures, and human superiority at dismissing semi-failures as non-issues remains unchallenged. So while I appreciate the spirit of the argument, I don’t think it works out that way in practice, and I can’t really find fault with the Firefox team for deciding (TIL) to ship a better intermediate navigator for doing so. The alternative is directing user complaints to often-uncaring operators.
tialaramex 5y ago
> This seems like it could lead to even more confusion... They say making predictions about the future is difficult, but you're predicting something about the past. Where was the "even more confusion" in the long period where Mozilla shipped this feature? In this world, Mozilla's Firefox browser has to cope with the stupid but true fact that millions of servers are mis-configured, and being "loud" about that just ensures users will switch to a different browser.

throw0101a 5y ago

> The pool is regenerated [2] by a GitHub Action [3] every night, and embedded into the package so it requires no network connection. Once someone pulls the code and compiles it into a binary, how does the list of certs get updated? Do you have have a tzcode and tzdata split à la the IANA/Olson timezone database?