5 ms·

> I find it alarming that a week after the incident, 40% of the affected certificates are still in use, despite being rejected by the most popular browsers and

> I find it alarming that a week after the incident, 40% of the affected certificates are still in use, despite being rejected by the most popular browsers and despite affected subscribers being emailed by Let's Encrypt. This is perhaps a consequence on how well-oiled of a machine LE typically is: people stop paying attention to it.

tredre3 3y ago
That is true but how come certbot had no awareness of revoked/withdrawn certificates before now? It seems like one of the things a CA is supposed to solve for you, and the fact that it doesn't is bit alarming in itself. Though, as the following sentence points out, they were already working on it before the outage, so clearly they knew it was needed. 1. https://datatracker.ietf.org/doc/draft-ietf-acme-ari/ https://datatracker.ietf.org/doc/draft-ietf-acme-ari/
- 411111111111111 3y ago
  The CA can't solve it for you. The certificate authority signs certificate requests, creating certificates. The revocation process is necessary as well, but the CA doesn't have the ability to change the already issued certificate, thus it cannot take action. A software like certbot can solve it for you, but that's not affiliated with your CA
  - agwa 3y ago
    The CA is part of the solution by using ARI to inform ACME clients to replace impacted certificates.
    - mcpherrinm 3y ago
      Even before ARI, some integrated ACME/Web servers use OCSP as a way of knowing to renew if a cert was revoked. Plus if you're doing that you can pin the OCSP response while you're at it.
      - 411111111111111 3y ago
        My point was that the CA can't solve it for you, they can only give you APIs and processes with which you can solve it yourself. If your webserver supports checking the certificate validity then it's not solved by the CA, it's been solved by the developers of that software and by you installing it.
hinkley 3y ago
Vernor Vinge has dominated the Singularity space in science fiction pretty much from the beginning of the concept. Rainbow’s End plays around in time frame right around where we are now, just a bit before the sorts of doglegs we predict would presage a Singularity in your lifetime. At one point the protagonists need to attack a bad actor, and to make it work they need chaos on the internet. I don’t recall exactly how this plays out, but the way they decide to achieve it is that one of the collaborators believes that they can reject a CA cert that affects 10% of all certificates in the wild, and the resulting pandemonium will give them approximately the sort of chaos they need. Sounds to me like maybe that is either no longer true, or never was.
- tialaramex 3y ago
  [Spoilers] They don't need Chaos. They want to disable Rabbit, and they know Rabbit's certificates mostly tie back to a single CA, Credit Suisse. So they "revoke" Credit Suisse and accept the consequences, which (they acknowledge) are career ending for the Europeans. This is mostly a plot convenience because Rabbit is much too powerful to allow what Vinge wants to happen next. No, you can't actually "revoke" a root CA, the decision to trust (or not) a root is local. So this part of the novel is a fantasy. But even if you assume it means that the European authorities can somehow reach into Credit Suisse and cause it to revoke all the intermediates (which maybe is a plausible reading) and so on down to end entity certificates, that doesn't really work either. Not on the time scale Vinge needs for the novel. Hours are conceivable but unlikely. Days maybe. A week. But the novel needs it to be seconds. There are two big obstacles to even the revocation which does really exist. Firstly humans are much more enthusiastic about seeing Dancing Pigs than they are about safety, because safety is a very abstract idea, whereas seeing dancing pigs is an immediate reward. This is the Dancing Pigs problem, and we've put some effort in, it's less likely a random Chrome user would get their face ripped to pieces because they wanted Dancing Pigs and so they bypassed the security checks that would protected them - than say - fifteen years ago, but only somewhat. Secondly though, there's not a great enthusiasm technically for this sort of counter-measure. It's so rarely beneficial in practice. Most of the time those humans were right, we were just denying them Dancing Pigs. Their face might get ripped to pieces, but to be honest it's as likely to be because they deliberate went to "Rip My Face To Pieces.example" as through anything we could have prevented. This is only barely a technical problem. So, when there are things we could do to get closer to what's in the novel, why would we? Building the PKI which exists in Vinge's novel is probably a bad expenditure of resources.
NovemberWhiskey 3y ago
Based on my experience, the capability model for certificate management usually went like: 1) Chaos: certificates requested and installed manually, either in response to incidents caused by expiration or calendar reminders 2) Monitoring: certificates requested and installed manually, in response to noisy alerting by probers looking for indications of pending expiration or other ill-health 3) Automation: continuous certificate provisioning, distribution and enablement either through platform or integration The Let's Encrypt revolution has taken a lot of people from stage 1 to stage 3 without stage 2 in between.
tialaramex 3y ago
I haven't looked at a list of revoked certificates, because I was busy, (and I no longer operate my own CT auditing software, so I'd have to poke around in crt.sh which is not much fun) but lets suppose these are a random sample of Let's Encrypt's ~2 million issuances per day. What %age of the world's HTTPS web sites are "parked" and so there is nobody who expects them to actually work? BrandFromATVShow.example ? TeenDanceISawOnTikTok.example ? SomeShortEnglishWord.example ? Nobody cares, if they do visit, and there's a certificate failure, they realise that's not where they meant to go and leave. Then what %age are somebody's fever dream / retirement plan / abandoned start-up idea and so although the owner may notice eventually that it's broken, that might not happen before automatic renewal "fixes" the problem anyway if ever. MyTownOlympicSwimmingPool.example JimAndBethsCakeShop.example and LikeAWSForDogsSomehow.example And then how about all the outfits which folded weeks, months, even in some cases years ago, but the ISP bill was paid, so, the web site continues to exist until somebody removes it, but of course nobody cares ? BoughtByGoogle.example and YetAnotherBayAreaCryptoStartup.example together with DefinitelyViableProduct.example and OopsWalmartAlreadySellsThatForLessMoney.example If it was 95% I'd be more worried, at 40% I'd need to actually check at least a decent sample and see for myself. In the time I was writing this post I checked one, it wasn't replaced... exactly, because the actual web site uses a certificate issued five days earlier. Chances are they've got a bunch of duplicate certificates, so the fact that some they don't use are broken has never come up - that's just rude (wastes other people's resources) but it works fine technically.
- tedunangst 3y ago
  Renewing a cert without immediately deploying it seems like a reasonable practice in the face of CAs that will misissue through no fault of your own.
  - schoen 3y ago
    When we wrote Certbot, we thought (by analogy with prior practice) that many sysadmins would want to manually inspect certificates before deploying them! That's one reason that we kept old certificates around and used a symlink-updating system. As it turned out, misissued and invalid certs account for an incredibly small fraction of Let's Encrypt's issuance volume (I'm going to say < 1/10⁸ offhand?) and manual inspection kind of gets in the way of automation, so the idea of separating these steps has come to seem kind of quaint, for me at least. I've also helped thousands of people on the Let's Encrypt forum and I think at most 2 have said they were interested in looking at their new certs' contents before starting to use them.

tredre3 3y ago

That is true but how come certbot had no awareness of revoked/withdrawn certificates before now? It seems like one of the things a CA is supposed to solve for you, and the fact that it doesn't is bit alarming in itself. Though, as the following sentence points out, they were already working on it before the outage, so clearly they knew it was needed. 1. https://datatracker.ietf.org/doc/draft-ietf-acme-ari/ https://datatracker.ietf.org/doc/draft-ietf-acme-ari/

411111111111111 3y ago
The CA can't solve it for you. The certificate authority signs certificate requests, creating certificates. The revocation process is necessary as well, but the CA doesn't have the ability to change the already issued certificate, thus it cannot take action. A software like certbot can solve it for you, but that's not affiliated with your CA
- agwa 3y ago
  The CA is part of the solution by using ARI to inform ACME clients to replace impacted certificates.
  - mcpherrinm 3y ago
    Even before ARI, some integrated ACME/Web servers use OCSP as a way of knowing to renew if a cert was revoked. Plus if you're doing that you can pin the OCSP response while you're at it.
    - 411111111111111 3y ago
      My point was that the CA can't solve it for you, they can only give you APIs and processes with which you can solve it yourself. If your webserver supports checking the certificate validity then it's not solved by the CA, it's been solved by the developers of that software and by you installing it.

hinkley 3y ago

Vernor Vinge has dominated the Singularity space in science fiction pretty much from the beginning of the concept. Rainbow’s End plays around in time frame right around where we are now, just a bit before the sorts of doglegs we predict would presage a Singularity in your lifetime. At one point the protagonists need to attack a bad actor, and to make it work they need chaos on the internet. I don’t recall exactly how this plays out, but the way they decide to achieve it is that one of the collaborators believes that they can reject a CA cert that affects 10% of all certificates in the wild, and the resulting pandemonium will give them approximately the sort of chaos they need. Sounds to me like maybe that is either no longer true, or never was.

tialaramex 3y ago
[Spoilers] They don't need Chaos. They want to disable Rabbit, and they know Rabbit's certificates mostly tie back to a single CA, Credit Suisse. So they "revoke" Credit Suisse and accept the consequences, which (they acknowledge) are career ending for the Europeans. This is mostly a plot convenience because Rabbit is much too powerful to allow what Vinge wants to happen next. No, you can't actually "revoke" a root CA, the decision to trust (or not) a root is local. So this part of the novel is a fantasy. But even if you assume it means that the European authorities can somehow reach into Credit Suisse and cause it to revoke all the intermediates (which maybe is a plausible reading) and so on down to end entity certificates, that doesn't really work either. Not on the time scale Vinge needs for the novel. Hours are conceivable but unlikely. Days maybe. A week. But the novel needs it to be seconds. There are two big obstacles to even the revocation which does really exist. Firstly humans are much more enthusiastic about seeing Dancing Pigs than they are about safety, because safety is a very abstract idea, whereas seeing dancing pigs is an immediate reward. This is the Dancing Pigs problem, and we've put some effort in, it's less likely a random Chrome user would get their face ripped to pieces because they wanted Dancing Pigs and so they bypassed the security checks that would protected them - than say - fifteen years ago, but only somewhat. Secondly though, there's not a great enthusiasm technically for this sort of counter-measure. It's so rarely beneficial in practice. Most of the time those humans were right, we were just denying them Dancing Pigs. Their face might get ripped to pieces, but to be honest it's as likely to be because they deliberate went to "Rip My Face To Pieces.example" as through anything we could have prevented. This is only barely a technical problem. So, when there are things we could do to get closer to what's in the novel, why would we? Building the PKI which exists in Vinge's novel is probably a bad expenditure of resources.

NovemberWhiskey 3y ago

Based on my experience, the capability model for certificate management usually went like: 1) Chaos: certificates requested and installed manually, either in response to incidents caused by expiration or calendar reminders 2) Monitoring: certificates requested and installed manually, in response to noisy alerting by probers looking for indications of pending expiration or other ill-health 3) Automation: continuous certificate provisioning, distribution and enablement either through platform or integration The Let's Encrypt revolution has taken a lot of people from stage 1 to stage 3 without stage 2 in between.

tialaramex 3y ago

I haven't looked at a list of revoked certificates, because I was busy, (and I no longer operate my own CT auditing software, so I'd have to poke around in crt.sh which is not much fun) but lets suppose these are a random sample of Let's Encrypt's ~2 million issuances per day. What %age of the world's HTTPS web sites are "parked" and so there is nobody who expects them to actually work? BrandFromATVShow.example ? TeenDanceISawOnTikTok.example ? SomeShortEnglishWord.example ? Nobody cares, if they do visit, and there's a certificate failure, they realise that's not where they meant to go and leave. Then what %age are somebody's fever dream / retirement plan / abandoned start-up idea and so although the owner may notice eventually that it's broken, that might not happen before automatic renewal "fixes" the problem anyway if ever. MyTownOlympicSwimmingPool.example JimAndBethsCakeShop.example and LikeAWSForDogsSomehow.example And then how about all the outfits which folded weeks, months, even in some cases years ago, but the ISP bill was paid, so, the web site continues to exist until somebody removes it, but of course nobody cares ? BoughtByGoogle.example and YetAnotherBayAreaCryptoStartup.example together with DefinitelyViableProduct.example and OopsWalmartAlreadySellsThatForLessMoney.example If it was 95% I'd be more worried, at 40% I'd need to actually check at least a decent sample and see for myself. In the time I was writing this post I checked one, it wasn't replaced... exactly, because the actual web site uses a certificate issued five days earlier. Chances are they've got a bunch of duplicate certificates, so the fact that some they don't use are broken has never come up - that's just rude (wastes other people's resources) but it works fine technically.

tedunangst 3y ago
Renewing a cert without immediately deploying it seems like a reasonable practice in the face of CAs that will misissue through no fault of your own.
- schoen 3y ago
  When we wrote Certbot, we thought (by analogy with prior practice) that many sysadmins would want to manually inspect certificates before deploying them! That's one reason that we kept old certificates around and used a symlink-updating system. As it turned out, misissued and invalid certs account for an incredibly small fraction of Let's Encrypt's issuance volume (I'm going to say < 1/10⁸ offhand?) and manual inspection kind of gets in the way of automation, so the idea of separating these steps has come to seem kind of quaint, for me at least. I've also helped thousands of people on the Let's Encrypt forum and I think at most 2 have said they were interested in looking at their new certs' contents before starting to use them.