5 ms·

Avoiding downtime: modern alternatives to outdated certificate pinning practices

45 points by cpach 2y ago

ivewonyoung 2y ago
> However, what we’ve learned is that almost all customers that were impacted by the change were unaware that they had a certificate pin in place. All that lengthy blog post and no explanation of what configuration is causing this and in what kinds of software. And how to find certificate pins in your infrastructure. Anyone know?
- aflukasz 2y ago
  Well, it's on a software component by component basis, so not much they could have done here, I guess. Look for any settings or in-code references related to ssl/tls. You then may find references to some local paths, pointing to certificate files. Or to cert strings compiled into the binaries. Depending on the details, this may indicate pinned certs.
njtransit 2y ago
The article mentions certificate pinning is used to prevent MITM attacks, but none of the alternatives listed actually prevent this class of attacks. I’ve heard multiple times that “certificate pinning is obsolete.” Who is pushing this narrative?
- ekr____ 2y ago
  Well, requiring a WebPKI valid certificate itself prevents MITM attacks. What pinning does is prevent MITM attacks in the case where the attacker has a valid (but misissued) certificate. It's true that CT doesn't prevent this class of attack; instead what it does is require that valid certificates be public, thus -- at least in principle -- allowing for detection of misissuance.
  - patrakov 2y ago
    Not really true. Please also consider the case where the user and the attacker are one and the same person trying to reverse-engineer your mobile API. Then the certificate is misissued from the viewpoint of the app developer but not from the user's viewpoint.
    - lokar 2y ago
      If you don’t want people to reverse engineer your software provide it on tamper proof hardware.
      - azinman2 2y ago
        How would Facebook, say, in this case work? No longer available on mobile devices and you have to buy Facebook specific hardware?
- tprynn 2y ago
  I am. Pinning is a footgun with negligible real world security impact: https://tprynn.github.io/2022/12/06/cert-pinning-bad.html https://tprynn.github.io/2022/12/06/cert-pinning-bad.html
  - njtransit 2y ago
    A few thoughts: 1. Almost all mobile devices used by adults to access their work email have provisioning profiles that allow trusted certificates to be installed by one’s employer. 2. Plenty of authoritarian counties require trusting CAs operated by the government. If you have users in those countries, they are vulnerable to snooping. Your blog post makes it seem like users vulnerable to MITM attacks are in the minority, when in fact they are likely in the vast majority.
aflukasz 2y ago
If you pin it, you may as well use self signed cert, with whatever expiration date you want. It has nothing to do with what browsers or popular CAs do - you may be the CA in this case. Not saying this is a good idea, just that it's the same idea as before (vide their "why now" argument and shortening rotation times). Is pinning to certs signed by CAs you don't control, but that are public and trusted, beneficial over using self signed ones? I guess you won't (easily) land in CT log. Anything else?
- ekr____ 2y ago
  This won't really work technically, because pins can only be installed after the client has formed a TLS connection to the server, so if you have a self-signed certificate, the client will refuse to connect. In principle you can override this refusal, but in practice browsers make it very hard. Moreover, this isn't what you want people to do, because unless you have some out of band way of distributing the certificate or its fingerprint (and users check it) then they are open to a MITM attack by anyone on the network on first use, not just by attackers who have a valid (but misissued) WebPKI certificate.
  - csande17 2y ago
    The article mentions that pinning is often used by mobile apps and IoT devices. In those contexts, you have an easy out-of-band mechanism to deliver the cert: baking it into the app/firmware binary. (The user generally receives this from the App Store, or in a cardboard box from a retailer, so no initial TLS connection to your server is required.) If you're pinning the leaf certificate this way, really the only benefit I see of using a WebPKI cert is if you want to reuse the same API endpoint for a web app. Otherwise you're mostly getting a bunch of restrictions and downsides (information leaks from CT, revocation drama, etc) that don't make sense if the cert is hard coded in the client.
  - aflukasz 2y ago
    I was thinking about certificates distributed with the software itself, so extra bootstrapping connection is not needed. Although in some cases it could still be done, just using other, trusted and not pinned context, I guess. But yes, for the specific narrow use case of web pages and web browsers, this means out of band installation into the browser trust store. And if we focus on this particular use case, what difference does it make if the cert is self signed or not?
aboringusername 2y ago
I have a question. How does the client learn about certificate update? I understand the "leaf" or "domain" update is done when the request is made, and that I as the end user don't need to do anything. However if an intermediate certificate is updated, do I now need to update the app or client (OS/browser etc)? Because I can sure see it in Google's interest to ensure clients only use the latest browsers, OS'. I suppose it becomes another "control" point. If your client cannot resolve a domain (outdated hardware/software) you're out of luck. If Google pushes changes to, for example, data collection within Chrome, they're going to want all users to update ASAP (e.g manifest V3, break older versions of chrome by rotating certificates).
- fjasdddwsfasd 2y ago
  > I have a question. How does the client learn about certificate update? This is what I want to know as well especially with respect to many applications running in Kubernetes.
  - JackSlateur 2y ago
    The client must be upgraded when the certificate changes
captn3m0 2y ago
I've dealt with cert pinning in the past, and the most frustrating part was invisible pins - any client can pin your certificate in their code, without ever having told you about it. If you run a large enough API over TLS, always assume that some random client will have pinned your certificate at their end. The only way to deal with it is to setup fast rotating certs, so these enterprise clients just give up, but that might not always be possible We ended up doing slow brown-outs over a long time period to deal with it.
peanut-walrus 2y ago
Surprisingly low quality article from Cloudflare. None of the solutions offered address the risk that pinning tries to mitigate, namely that you have a client who trusts a misbehaving/rogue/malicious CA (perhaps you installed a VPN or AV or something that added a system-wide CA to your machine?) and their connection to your service gets mitm-d. Shorter certificate lifetimes - does nothing to mitigate that risk. CAA records - not checked by client, only ensures that if a malicious actor somehow gets a hold of your domain, they need to use a particular CA to issue their publicly-trusted malicious cert. CT monitoring - yes, for WebPKI certs, but does not protect against the risk above, as the bad CA-s are likely not sending certs to CT log. Distributed DCV checks - sure, this is something CA-s should be doing, not something you have control over. ACME URI in CAA - operational issues quite likely, good if you can pull it off, but again, does not address the risk above. What is an actual alternative to cert pinning? Your API should have an additional layer of encryption with keys that you have full control over.
- JackSlateur 2y ago
  Yes Also, the "Understanding the trade-offs of different certificate pinning implementations" misses one important thing : you can pin the CA, and still requires the cert to be valid for your FQDN I've seen this a couple of time, and this is wonderful : fully secure, and denies the managed mitm setup by the security team.
  - patrakov 2y ago
    And the only one time where I have seen this (in a private discussion of the mobile app with a bank employee), they said that they see no valid use case for a managed MitM against their production website. Preventing proxies like Charles and mitmproxy from working was one of their explicit goals for the app.
- hn_throwaway_99 2y ago
  While I agree this is a low quality article in desperate need of an editor (there were large sections that were essentially just repeated), I'm not sure I agree with your analysis, especially as it relates to CAA records. I think the fundamental issue is that all of the players in the PKI world want to essentially enforce that services must be able to deal with short-lived certificates and certificate revocations. As an example, see the recent HN article where Digicert tried to do mass certificate revocations due to a bug with their certificate issuance, and were then sued by a client (Alegeus) who did not have things set up to deal with revocations quickly, so Digicert postponed the revocation: https://news.ycombinator.com/item?id=41114794 https://news.ycombinator.com/item?id=41114794 . So basically, the powers that be are trying to say "certificate cycling is part of the process - 'set it and forget it' is a thing of the past." Now, in your objections, you state "None of the solutions offered address the risk that pinning tries to mitigate, namely that you have a client who trusts a misbehaving/rogue/malicious CA", but CAA with CT monitoring should largely prevent that. With CAA monitoring you state "only ensures that if a malicious actor somehow gets a hold of your domain..." - well, if a malicious actor somehow gets ahold of your domain, you're generally fucked in any case - if they've got control of your domain it's probably not a huge step up to get control of your signing cert. I think your comment brings up good points to someone like me who's not an expert, but it still feels like the overall point that needs to be addressed is that "certificates should be expected to expire or be revoked at any instant" and certificate pinning breaks that.
g40694 2y ago
cloudflare continues to destroy the free web, this time around they are gaslighting you into thinking that an extremely important security feature, that happens to break cloudflare mitm is actually useless and outdated and there's all kinds of solutions, that incidentally all rely on trusting cloudflare, that you should be using instead.
grayhatter 2y ago
This is a super disappointing read coming from cloudflare. Even if I didn't disagree with the author about who exactly 'caused' these outages (I'm not blaming the users for the breaking change). I have to disagree with the suggestions. If the goal is availability; why is automation and constant cert and key rotation preferred? It's better to have to get it right every single time, over getting it right once? That's a wild take. I guess it's the behavior and recommendation I should expect from cloudflare, they are after all looking to sell, what's effectively a maintenance contract in concert with their expertise. But if I was gonna buy cloudflare, it would be for their expertise. They're industry leading for a reason. They used to be right. So I guess it's disappointing not only that they're wrong here in this article, but it's also disappointing they'd sell off some of their credibility earned by their expertise to trick people into an implementation with a much larger maintenance cost. Anyone have any alternatives to cloudflare, is there anyone who still values correctness over making money?
- agwa 2y ago
  Every single CA - including non-profit CAs like Let's Encrypt - have made mistakes that require cert rotation. And sometimes it isn't mistakes that require cert rotation, but cryptographic breakthroughs that render the algorithms in the cert insecure. It's much better engineering to accept that certs will need to be rotated and make sure it happens regularly so it's not a huge emergency when it's required.
  - grayhatter 2y ago
    > And sometimes it isn't mistakes that require cert rotation, but cryptographic breakthroughs that render the algorithms in the cert insecure. Do you have an example of this happening? I know I remember a few cases that sound similar, but weakening of a signature, or key is very different from it breaking. I'll also point out, if you find the signature made by a CA is weakened such that it can be forged; certificate pinning would actually protect existing users from such an attack. > It's much better engineering to accept that certs will need to be rotated that's a very debatable assertion, but I don't disagree outright > and make sure it happens regularly so it's not a huge emergency when it's required. No, that's not good engineering. Do something needlessly, so that it's easy to do it when it becomes required is operations not engineering. Engineering would be designing a system that requires the minimal amount of operational support. Certs do need to be updated, and maintanted, they don't need to be rotated.
mikeocool 2y ago
I got Certificate Transparency notifications for a domain of a relatively small company for a while. After you setup custom domains for a few SaaS products, and have a few hosting providers issuing short lived certs, you’re getting notifications fairly regularly. The notifications don’t really come with enough information for you have to any idea if the certs are being issued by trusted parties. At a large company I have to imagine that monitoring them would just end up being noise.
CodesInChaos 2y ago
How I'd like to replace certificate pinning: * The client requests additional certificate chains * The server returns multiple certificate chains (typically two), one chaining to a public CA, once chaining to a private CA * The connection only succeeds if all expected chains are present and valid * The public-key of the leaf certificates must be identical This means: * The public chain can follow standard practices around certificate transparency, renewal, revocation, expiration, etc. * Clients which don't care about pinning but expect a standard public chain will work * The chain to the private CA protects against a compromised public CA for clients which request/validate it * Requiring identical public-keys means that this only impacts chain validation, not the key-exchange
Hizonner 2y ago
Can we please just take X.509 out and shoot it?

45 points by cpach 2y ago

> However, what we’ve learned is that almost all customers that were impacted by the change were unaware that they had a certificate pin in place. All that lengthy blog post and no explanation of what configuration is causing this and in what kinds of software. And how to find certificate pins in your infrastructure. Anyone know?

aflukasz 2y ago
Well, it's on a software component by component basis, so not much they could have done here, I guess. Look for any settings or in-code references related to ssl/tls. You then may find references to some local paths, pointing to certificate files. Or to cert strings compiled into the binaries. Depending on the details, this may indicate pinned certs.

njtransit 2y ago

The article mentions certificate pinning is used to prevent MITM attacks, but none of the alternatives listed actually prevent this class of attacks. I’ve heard multiple times that “certificate pinning is obsolete.” Who is pushing this narrative?

ekr____ 2y ago
Well, requiring a WebPKI valid certificate itself prevents MITM attacks. What pinning does is prevent MITM attacks in the case where the attacker has a valid (but misissued) certificate. It's true that CT doesn't prevent this class of attack; instead what it does is require that valid certificates be public, thus -- at least in principle -- allowing for detection of misissuance.
- patrakov 2y ago
  Not really true. Please also consider the case where the user and the attacker are one and the same person trying to reverse-engineer your mobile API. Then the certificate is misissued from the viewpoint of the app developer but not from the user's viewpoint.
  - lokar 2y ago
    If you don’t want people to reverse engineer your software provide it on tamper proof hardware.
    - azinman2 2y ago
      How would Facebook, say, in this case work? No longer available on mobile devices and you have to buy Facebook specific hardware?
tprynn 2y ago
I am. Pinning is a footgun with negligible real world security impact: https://tprynn.github.io/2022/12/06/cert-pinning-bad.html https://tprynn.github.io/2022/12/06/cert-pinning-bad.html
- njtransit 2y ago
  A few thoughts: 1. Almost all mobile devices used by adults to access their work email have provisioning profiles that allow trusted certificates to be installed by one’s employer. 2. Plenty of authoritarian counties require trusting CAs operated by the government. If you have users in those countries, they are vulnerable to snooping. Your blog post makes it seem like users vulnerable to MITM attacks are in the minority, when in fact they are likely in the vast majority.

aflukasz 2y ago

If you pin it, you may as well use self signed cert, with whatever expiration date you want. It has nothing to do with what browsers or popular CAs do - you may be the CA in this case. Not saying this is a good idea, just that it's the same idea as before (vide their "why now" argument and shortening rotation times). Is pinning to certs signed by CAs you don't control, but that are public and trusted, beneficial over using self signed ones? I guess you won't (easily) land in CT log. Anything else?

ekr____ 2y ago
This won't really work technically, because pins can only be installed after the client has formed a TLS connection to the server, so if you have a self-signed certificate, the client will refuse to connect. In principle you can override this refusal, but in practice browsers make it very hard. Moreover, this isn't what you want people to do, because unless you have some out of band way of distributing the certificate or its fingerprint (and users check it) then they are open to a MITM attack by anyone on the network on first use, not just by attackers who have a valid (but misissued) WebPKI certificate.
- csande17 2y ago
  The article mentions that pinning is often used by mobile apps and IoT devices. In those contexts, you have an easy out-of-band mechanism to deliver the cert: baking it into the app/firmware binary. (The user generally receives this from the App Store, or in a cardboard box from a retailer, so no initial TLS connection to your server is required.) If you're pinning the leaf certificate this way, really the only benefit I see of using a WebPKI cert is if you want to reuse the same API endpoint for a web app. Otherwise you're mostly getting a bunch of restrictions and downsides (information leaks from CT, revocation drama, etc) that don't make sense if the cert is hard coded in the client.
- aflukasz 2y ago
  I was thinking about certificates distributed with the software itself, so extra bootstrapping connection is not needed. Although in some cases it could still be done, just using other, trusted and not pinned context, I guess. But yes, for the specific narrow use case of web pages and web browsers, this means out of band installation into the browser trust store. And if we focus on this particular use case, what difference does it make if the cert is self signed or not?

aboringusername 2y ago

I have a question. How does the client learn about certificate update? I understand the "leaf" or "domain" update is done when the request is made, and that I as the end user don't need to do anything. However if an intermediate certificate is updated, do I now need to update the app or client (OS/browser etc)? Because I can sure see it in Google's interest to ensure clients only use the latest browsers, OS'. I suppose it becomes another "control" point. If your client cannot resolve a domain (outdated hardware/software) you're out of luck. If Google pushes changes to, for example, data collection within Chrome, they're going to want all users to update ASAP (e.g manifest V3, break older versions of chrome by rotating certificates).

fjasdddwsfasd 2y ago
> I have a question. How does the client learn about certificate update? This is what I want to know as well especially with respect to many applications running in Kubernetes.
- JackSlateur 2y ago
  The client must be upgraded when the certificate changes

captn3m0 2y ago

I've dealt with cert pinning in the past, and the most frustrating part was invisible pins - any client can pin your certificate in their code, without ever having told you about it. If you run a large enough API over TLS, always assume that some random client will have pinned your certificate at their end. The only way to deal with it is to setup fast rotating certs, so these enterprise clients just give up, but that might not always be possible We ended up doing slow brown-outs over a long time period to deal with it.

peanut-walrus 2y ago

Surprisingly low quality article from Cloudflare. None of the solutions offered address the risk that pinning tries to mitigate, namely that you have a client who trusts a misbehaving/rogue/malicious CA (perhaps you installed a VPN or AV or something that added a system-wide CA to your machine?) and their connection to your service gets mitm-d. Shorter certificate lifetimes - does nothing to mitigate that risk. CAA records - not checked by client, only ensures that if a malicious actor somehow gets a hold of your domain, they need to use a particular CA to issue their publicly-trusted malicious cert. CT monitoring - yes, for WebPKI certs, but does not protect against the risk above, as the bad CA-s are likely not sending certs to CT log. Distributed DCV checks - sure, this is something CA-s should be doing, not something you have control over. ACME URI in CAA - operational issues quite likely, good if you can pull it off, but again, does not address the risk above. What is an actual alternative to cert pinning? Your API should have an additional layer of encryption with keys that you have full control over.

JackSlateur 2y ago
Yes Also, the "Understanding the trade-offs of different certificate pinning implementations" misses one important thing : you can pin the CA, and still requires the cert to be valid for your FQDN I've seen this a couple of time, and this is wonderful : fully secure, and denies the managed mitm setup by the security team.
- patrakov 2y ago
  And the only one time where I have seen this (in a private discussion of the mobile app with a bank employee), they said that they see no valid use case for a managed MitM against their production website. Preventing proxies like Charles and mitmproxy from working was one of their explicit goals for the app.
hn_throwaway_99 2y ago
While I agree this is a low quality article in desperate need of an editor (there were large sections that were essentially just repeated), I'm not sure I agree with your analysis, especially as it relates to CAA records. I think the fundamental issue is that all of the players in the PKI world want to essentially enforce that services must be able to deal with short-lived certificates and certificate revocations. As an example, see the recent HN article where Digicert tried to do mass certificate revocations due to a bug with their certificate issuance, and were then sued by a client (Alegeus) who did not have things set up to deal with revocations quickly, so Digicert postponed the revocation: https://news.ycombinator.com/item?id=41114794 https://news.ycombinator.com/item?id=41114794 . So basically, the powers that be are trying to say "certificate cycling is part of the process - 'set it and forget it' is a thing of the past." Now, in your objections, you state "None of the solutions offered address the risk that pinning tries to mitigate, namely that you have a client who trusts a misbehaving/rogue/malicious CA", but CAA with CT monitoring should largely prevent that. With CAA monitoring you state "only ensures that if a malicious actor somehow gets a hold of your domain..." - well, if a malicious actor somehow gets ahold of your domain, you're generally fucked in any case - if they've got control of your domain it's probably not a huge step up to get control of your signing cert. I think your comment brings up good points to someone like me who's not an expert, but it still feels like the overall point that needs to be addressed is that "certificates should be expected to expire or be revoked at any instant" and certificate pinning breaks that.

g40694 2y ago

cloudflare continues to destroy the free web, this time around they are gaslighting you into thinking that an extremely important security feature, that happens to break cloudflare mitm is actually useless and outdated and there's all kinds of solutions, that incidentally all rely on trusting cloudflare, that you should be using instead.

grayhatter 2y ago

This is a super disappointing read coming from cloudflare. Even if I didn't disagree with the author about who exactly 'caused' these outages (I'm not blaming the users for the breaking change). I have to disagree with the suggestions. If the goal is availability; why is automation and constant cert and key rotation preferred? It's better to have to get it right every single time, over getting it right once? That's a wild take. I guess it's the behavior and recommendation I should expect from cloudflare, they are after all looking to sell, what's effectively a maintenance contract in concert with their expertise. But if I was gonna buy cloudflare, it would be for their expertise. They're industry leading for a reason. They used to be right. So I guess it's disappointing not only that they're wrong here in this article, but it's also disappointing they'd sell off some of their credibility earned by their expertise to trick people into an implementation with a much larger maintenance cost. Anyone have any alternatives to cloudflare, is there anyone who still values correctness over making money?

agwa 2y ago
Every single CA - including non-profit CAs like Let's Encrypt - have made mistakes that require cert rotation. And sometimes it isn't mistakes that require cert rotation, but cryptographic breakthroughs that render the algorithms in the cert insecure. It's much better engineering to accept that certs will need to be rotated and make sure it happens regularly so it's not a huge emergency when it's required.
- grayhatter 2y ago
  > And sometimes it isn't mistakes that require cert rotation, but cryptographic breakthroughs that render the algorithms in the cert insecure. Do you have an example of this happening? I know I remember a few cases that sound similar, but weakening of a signature, or key is very different from it breaking. I'll also point out, if you find the signature made by a CA is weakened such that it can be forged; certificate pinning would actually protect existing users from such an attack. > It's much better engineering to accept that certs will need to be rotated that's a very debatable assertion, but I don't disagree outright > and make sure it happens regularly so it's not a huge emergency when it's required. No, that's not good engineering. Do something needlessly, so that it's easy to do it when it becomes required is operations not engineering. Engineering would be designing a system that requires the minimal amount of operational support. Certs do need to be updated, and maintanted, they don't need to be rotated.

mikeocool 2y ago

I got Certificate Transparency notifications for a domain of a relatively small company for a while. After you setup custom domains for a few SaaS products, and have a few hosting providers issuing short lived certs, you’re getting notifications fairly regularly. The notifications don’t really come with enough information for you have to any idea if the certs are being issued by trusted parties. At a large company I have to imagine that monitoring them would just end up being noise.

CodesInChaos 2y ago

How I'd like to replace certificate pinning: * The client requests additional certificate chains * The server returns multiple certificate chains (typically two), one chaining to a public CA, once chaining to a private CA * The connection only succeeds if all expected chains are present and valid * The public-key of the leaf certificates must be identical This means: * The public chain can follow standard practices around certificate transparency, renewal, revocation, expiration, etc. * Clients which don't care about pinning but expect a standard public chain will work * The chain to the private CA protects against a compromised public CA for clients which request/validate it * Requiring identical public-keys means that this only impacts chain validation, not the key-exchange

Hizonner 2y ago

Can we please just take X.509 out and shoot it?