5 ms·
Health industry company sues to prevent certificate revocation
- loceng 2y agoIs this only happening to this one company? It seems like it'd be a widespread problem?
- kadoban 2y agoNaw it's a bunch of people/companies. It was discussed on HN at least once recently, but best link I have is: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322
- deathanatos 2y agoThe complaint: https://www.courtlistener.com/docket/68995396/1/alegeus-technologies-llc-v-digicert/ https://www.courtlistener.com/docket/68995396/1/alegeus-tech... IANAL. > DigiCert knew or should have known about its failure to properly complete the Security Certificates for Alegeus weeks ago. 'You should have known there were bugs in your code!' > DigiCert is a voluntary member of the Certification Authority Browser Forum (CABF), which has bylaws stating that certificates with an issue in their domain validation must be revoked within 24 hours "voluntary" is technically correct, I suppose. But let's assume they'd like to be a CA? But then we go > despite that DigiCert has arbitrarily chosen this less than 24-hour window It wasn't arbitrary, it was a requirement, as the complaint itself pointed out… > Alegeus has hundreds of such Clients. Alegeus is generally required by contract to give its clients much longer than 24 hours’ notice before executing such a change regarding certification. Yet another example of that nobody ever thinks about cert rotation, ever. > Consequently, due to the errors and delays of DigiCert alone, Alegeus cannot practically make all the required changes Yeah, no, your error was not having a process to account for this in place. The TRO also states, > The threatened injury to Alegeus outweighs any injury DigiCert might suffer under the injunction. Well, if DigiCert is removed as a CA because they've violated the CA/B BRs (at least, AIUI, they have) — that could be a pretty big injury. They're probably too big to fail, and "we tried but we had a TRO" is a pretty good excuse. The TRO is scoped to only their certs, and only for 7 days (barring further orders), so it's at least pretty minimal? So I suppose DigiCert could revoke the rest.
- stackskipton 2y agoMy guess is all they want is 7 days since complaints indicates they asked for 3 days. So they filed this suit, got the TRO, will do the work while TRO is underway and at hearing, declare it all moot, pay the lawyers and move on.
- JoshTriplett 2y agoWhat is the legal process for "cannot comply with this TRO because doing so would likely cause us to cease to exist"? It certainly is the case that "any injury DigiCert might suffer under the injunction" has the possibility of including no longer being able to be a certificate provider.
- dataflow 2y ago> What is the legal process for "cannot comply Appealing, I imagine? > would likely cause > has the possibility of Doesn't sound like it's "likely" in this case, so said process probably wouldn't work?
- wbl 2y agoThis is a TRO. Digicert hasn't been heard yet. So it's not an appeal just a regular hearing.
- dataflow 2y agoI thought you can appeal a TRO, at least sometimes? Don't know if the states differ on this.
- JoshTriplett 2y agoAppealing doesn't work if you're being made to comply within 24 hours before having any opportunity to prepare and make arguments. I'm wondering what the process is for showing that a TRO could potentially cause irreparable harm.
- 2y ago
- Hizonner 2y ago1. It does look like there's very strong reason to believe that these particular certs were not misissued, in the sense that the people who have the private keys really are the people who control the domains. Stronger than the sort of verification that the CA/BF requires. It's a bug if the system can't deal with that. 2. It's also true that there are rules, they exist for a reason, these Alegeus guys aren't special, and dragging this stuff into the courts is antisocial in the extreme. So it wouldn't be unreasonable for browsers to, say, explicitly blacklist their domains for as long as there conceivably might be noncompliant certs out there.
- jandrese 2y agoThe issue isn't that someone is taking over this particular healthcare industry domain, just that in that time period there could be bad certs issued for other subdomains and there is no way to verify it. The whole setup where the verification depends on a leading underscore because subdomains aren't allowed to start with an underscore seems rather flimsy to me, but it's how the system is set up.
- michaelt 2y ago> It's also true that there are rules, they exist for a reason, [...] dragging this stuff into the courts is antisocial in the extreme [...] it wouldn't be unreasonable for browsers to, say, explicitly blacklist their domains You know who else thinks there are rules, that the rules exist for a reason, and that draconian punishments should be applied to people who try to derail attempts to enforce the rules? Judges. Like the judge that granted this temporary restraining order.
- Hizonner 2y agoThe browsers don't have any contract with Alegeus, nor any relationship at all. They owe Alegeus nothing. They're not covered in the TRO. I doubt the TRO could legally be extended to cover them. Browsers do have a relationship with their users, who may very well be relying on them to enforce the CA/BF validity rules, or if they can't do that to provide equivalent security guarantees. And in particular many of the users rely on their browsers to protect them by blocking suspicious domains in various ways. So blocking the domains wouldn't be arbitrary, capricious, discriminatory, or anything like that. Judges don't have infinite sua sponte powers to order just anybody to do just anything. Although I admit they sometimes act as though they did, especially when they feel like their authoriteh has been disrespecked. Anyway the browsers aren't gonna do it. There's zero chance. So legalities don't really arise. I'm just saying it wouldn't be unreasonable.
- newprint 2y agoIf there is a contract between DigiCert & Alegeus Technologies, stating that DigiCert could potentially revoke contracts within 24h and Alegeus client contracts state that clients have more than 24h to deal with expired certs, then I feel like it's Alegeus problem, who over-promised to their clients and should have carefully read their contracts.
- Filligree 2y agoIt would seem that the contact between DigiCert and Alegeus said nothing about revocation.
- agwa 2y agoThe Master Services Agreement (https://storage.courtlistener.com/recap/gov.uscourts.utd.149707/gov.uscourts.utd.149707.1.1.pdf https://storage.courtlistener.com/recap/gov.uscourts.utd.149...) states: > The Service Specific Terms are incorporated by reference into this Agreement > “Service Specific Terms” mean additional terms specific to certain Services as set forth at www.digicert.com/service-specific-terms, which are incorporated herein to the extent applicable to any specific Services procured by Customer hereunder (which includes the Certificate Terms of Use with respect to Customer’s use of any Certificates). https://www.digicert.com/service-specific-terms https://www.digicert.com/service-specific-terms states: > The Certificate Terms of Use available at www.digicert.com/certificate-terms (the “Certificate Terms of Use”) apply to Certificates other than Qualified Certificates and PKIoverheid Certificates requested or issued by Customer. The applicable Certificate Terms of Use are incorporated herein by reference. https://www.digicert.com/certificate-terms https://www.digicert.com/certificate-terms states: > DigiCert may revoke a Certificate without notice for the reasons stated in the CPS, including if DigiCert reasonably believes that: > Industry Standards or DigiCert’s CPS require Certificate revocation, or revocation is necessary to protect the rights, confidential information, operations, or reputation of DigiCert or a third party.
- jandrese 2y agoMore to the point, Alegeus should have a system in place that can re-issue certificates within 24 hours since that is the timetable that their registrar works under. The lawsuit smells like them trying to buy time to fix their IT problems. Even if it loses or gets tossed out all they needed was that preliminary injunction. It's really sucky for DigiCert who have to choose which party to anger, the courts or the browser manufacturers, just because this healthcare company doesn't have its act together.
- hn_throwaway_99 2y agoI see way too many comments here that are basically saying "Why aren't these thousands of contracted orgs able to drop everything at the drop of a hat to update their DNS." Here is my understanding of the issue: 1. Alegeus creates white-labeled sites for their customers, but those sites are hosted on customer domains (usually for ease of sharing login cookies across subdomains on the customer), e.g. alegeus.customerdomain.com. 2. So the issue isn't necessarily that Alegeus can't update their systems, it's that their many customers can't update their DNS records fast enough to point to the new certs. IMO, if you built a system (and I'm talking about DigiCert here) that expects thousands of customers can update everything at the drop of a hat to update their code, you built a bad system. I don't think it's too different from CrowdStrike, except it's as if CrowdStrike gave 24 hours notice instead, saying "Hey, all of our customers, just a heads up we're going to break all your machines in 24 hours, you really should have planned better if you can't handle that." Alegeus is the one suing here because they've got tons of customers, but make no mistake this would cause outtages for tons of other companies that use DigiCert.
- Hizonner 2y agoUm, what are you talking about? Unless you use DANE (which nobody does, and the browsers don't check it even if they did), you do not have to change any DNS records, or touch the DNS at all, to update a certificate. There is no DNS involvement. Period. DANE should be the norm, but it's not.
- hn_throwaway_99 2y agoPerhaps I am misunderstanding what needs to happen, but since Alegeus itself doesn't control the domains on which its sites are hosted, all of its clients would basically need to rerun this procedure, https://docs.digicert.com/en/certcentral/manage-certificates/supported-dcv-methods-for-validating-the-domains-on-ov-ev-tls-ssl-certificate-orders/use-the-dns-txt-validation-method-to-verify-domain-control.html https://docs.digicert.com/en/certcentral/manage-certificates...
- Hizonner 2y ago
- michaelt 2y ago> In any event, given that Alegeus was aware that DigiCert is required to revoke certificates within 24 hours, one wonders why Alegeus went ahead and signed agreements with their customers that required a lengthy notice period before changing certificates. It seems obvious enough how this would come about: 1. The healthcare company knows better than to do a Crowdstrike, where "just a configuration change" is rushed out with inadequate testing, for "security". As even a configuration change could trigger a latent bug, they agree configuration changes should be rolled out gradually, and not applied during clients' peak operating hours. 2. The healthcare company could have used "Let's Encrypt" to get certificates for free, if they wanted short-term certificates. But instead they paid $$$$ to DigiCert for certificates with a 12 month duration. 3. Even if Alegeus had read the small print saying mis-issued certificates can be revoked with only 24 hours of notice - they quite reasonably assumed DigiCert would have controls in place to avoid mis-issued certificates. Hasn't DigiCert told the browser vendors precisely that?
- mynameisvlad 2y ago> 3. Even if Alegeus had read the small print saying mis-issued certificates can be revoked with only 24 hours of notice - they quite reasonably assumed DigiCert would have controls in place to avoid mis-issued certificates. Hasn't DigiCert told the browser vendors precisely that? I mean, sure, in theory there should be controls in place, but no company or process is perfect. That's why there's a process in place to respond to problems with issuance in the first place. DigiCert is responding in the way the browser vendors mandate; 24-hour revocation, per the article, is a baseline requirement to be a CA.
- bastawhiz 2y ago> they quite reasonably assumed DigiCert would have controls in place to avoid mis-issued certificates This makes no sense. The revoke certs when there's been an incident. Telling customers that they might need to revoke certs in the event of an incident should mean that they are going to be free from incidents?? If Stripe tells me webhooks might be delivered more than once, that does NOT mean I would assume they have controls in place to prevent webhooks from being delivered more than once. That's just wild to assume.
- etimberg 2y agoOne thing that i think is clear from the responses here is that it’s going to be a long time before software engineering is like other engineering disciplines.
- wbl 2y agoYou can't sue the health inspector for saying "those rats got to go" or the FAA saying "you didn't keep records on this plane, it doesn't fly"
- michaelt 2y agoActually you can sue the FAA https://www.denverpost.com/2024/07/29/boulder-sues-faa-airport-closure/ https://www.denverpost.com/2024/07/29/boulder-sues-faa-airpo... You might not win - but if you just need a few extra days? The court paperwork will take time.
- psnyder 2y agoThe original email response addressed to Alegeus from the DigiCert CEO can be found here https://storage.courtlistener.com/recap/gov.uscourts.utd.149707/gov.uscourts.utd.149707.1.3.pdf https://storage.courtlistener.com/recap/gov.uscourts.utd.149...
- 8organicbits 2y agoThe bug sounds interesting. The underscore is needed to ensure the CNAME isn't a valid domain name. For services that offer subdomains for customers, an attacker could get a certificate for the parent domain. Like if example.com offered subdomains based on a name the user chooses, the attacker would start a certificate verification process and get a random value from Digicert. They'd then create an account with that random value, causing the example.com service to create a CNAME for <random>.example.com. Digicert then sees the random value and issues the certificate for example.com to the attacker. The underscore is there as _RANDOM.example.com isn't a valid domain name, so you'd expect services to reject these sorts of domains. Edit: > Therefore, this is truly a security-critical incident, as there is a real risk (not a negligible 2^-150 risk as implied by DigiCert) that this flaw could have been exploited to get unauthorized certificates. Revocation of the improperly validated certificates is security-critical. Yeah... have they proven that this wasn't exploited? With 83,000 certificates getting revoked this isn't a small scale mistake.
- michaelt 2y ago> Yeah... have they proven that this wasn't exploited? Presumably it would be fairly easy to test, for the vast majority of cases. HTTP file validation is allowed for DV certificates, so you can test a domain by connecting to it. Testing from a variety of locations around the world avoids the risk of MITM. If you issued a certificate for example.com and it's now being used by example.com, doesn't that prove the same thing? But whether it's been exploited is, in a sense, irrelevant. The baseline requirements don't say "when you detect a mis-issued cert carry out more checks and decide for yourself whether to revoke" - they say the CA must revoke.
- 8organicbits 2y agoOh sure, they need to revoke yesterday, but if a certificate was mis-issued there's downstream security impact that needs to be handled immediately. > HTTP file validation is allowed for DV certificates, so you can test a domain by connecting to it. This was validation over DNS using a CNAME record. You don't need to keep the verification records around after the certificate is issued, so post-hoc analysis won't tell you much. But worse, how does the CA know if a RANDOM.example.com CNAME record was created by a customer vs the domain owner? The underscore is supposed to solve that, and they didn't use it.
- CaliforniaKarl 2y agoFor discussion on the DigiCert issue that triggered this, have a look at the following HN discussions (which are still active!) DigiCert's blog post… DigiCert Revocation Incident (CNAME Domain Validation) (digicert.com) — https://news.ycombinator.com/item?id=41104504 https://news.ycombinator.com/item?id=41104504 — 136 points by vitaliyf 1 day ago … and the Bugzilla entry where DigiCert reported this to Mozilla: Digicert pauses mass certificate revocation due to legal action from customer (bugzilla.mozilla.org) — https://news.ycombinator.com/item?id=41114794 https://news.ycombinator.com/item?id=41114794 — 9 points by frakkingcylons 23 hours ago
- tzs 2y ago> The CA/Browser Forum Baseline Requirements (BRs) (which all CAs are required to adhere to, by virtue of their being included in various browser and OS trust stores), say that revocation is required within 24 hours when “[t]he CA obtains evidence that the validation of domain authorization or control for any Fully‐Qualified Domain Name or IP address in the Certificate should not be relied upon” (section 4.9.1.1, point 5). Does that revocation requirement actually apply to these particular certificates? If I've understood the problem it is that DigiCert was issuing challenge keys for DNS based validation that did not start with the required underscore. The underscore is required to prevent an attack that only works against sites that allow users to create subdomains. The particular sites of the plaintiff's clients are not such sites and so has the CA actually obtained "evidence that the validation [...] should not be relied upon"?
- 8organicbits 2y agoHow does the CA validate that the plaintiff's client's sites don't match that criteria? Should they take the plaintiff's lawyer's word for it? You can't really go off script with stuff like this, there's too many gotchas that you need to think through. I wonder if we could come up with a process where a customer could push another DNS record to redo the verification. They'd need to act fast to beat the 24 hour revocation window, but maybe that's better than a full re-issuance? Probably better to invest in re-issuance agility though.