15 ms·

DigiCert: Threat of legal action to stifle Bugzilla discourse

610 points by DanAtC 2y ago

nightfly 2y ago
Can someone link to (or post copies of) the offending questions/statements?
- infogulch 2y ago
  In the second paragraph: > Contrary to this statement, I received a letter from DigiCert’s lawyers, Wilson Sonsini, regarding posts made by Sectigo’s Chief Compliance Officer in bug 1910322. https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322
  - mdaniel 2y ago
    > The code worked in our original monolithic system but was not implemented properly when we moved to our micro-services systems :chefs_kiss:
    - crooked-v 2y ago
      As we all know, software isn't software unless it can immediately scale to having 12 billion users.
    - evil-olive 2y ago
      it gets even better: > We also found that the bug in the code was inadvertently remediated when engineering completed a user-experience enhancement project that collapsed multiple random value generation microservices into a single service.
      - tpxl 2y ago
        > multiple random value generation microservices Wot. They had _multiple_ services generating random numbers?
        aragilar 2y ago
        It might mean multiple instances of the same code (which would explain why there might be commonalities)?
      - intelVISA 2y ago
        Some might see horror but I see a few genius level engineers 'working' 1hour a month rebuilding the same getrandom syscall. They hiring? :]
infogulch 2y ago
This is shocking to read. Even attempting to choke the legal speech of web PKI contributors with legalistic bullying is a gross inversion of the purpose and goals of the organization, and IMO warrants revoking everything to do with DigiCert on the spot.
- terminalbraid 2y ago
  > IMO warrants revoking everything to do with DigiCert on the spot This is pretty heavy handed and I don't think you've thought through what the consequences of that may look like. The historic way to deal with a problematic CA is to prevent them from issuing new certificates or renewing certificates (after taking care of immediate damage, of course). There are lots of legitimate companies that use Digicert and they should have an expectation of being able to continue business in the short term while they find a different certificate provider.
  - DeepPPP 2y ago
    Are you saying it was okay to distrust EnTrust but DigiCert is too big to fail?
    - terminalbraid 2y ago
      No, and I'm deeply confused how you drew this conclusion from what was written. I specifically say the historical way to deprecate a certificate authority is to prevent them from issuing new certificates. Since certificates have a finite lifetime, the certificate authority would as well and would have immediately no further revenue from certificates. I am saying "pulling the plug" without notice is going to take down tens of thousands of bystanders and anyone using those certificates for business would be unable to conduct business securely online, which would be disastrous. For example amazon.com has a DigiCert-issued certificate.
      - DeepPPP 2y ago
        Agreed. That was my bad.
jchw 2y ago
Web PKI drama is always astonishing to me because it is one of the only areas of the entire world where a corporation's "fucking around" is seldom ever not followed by a sobering "finding out" period. The various entities that decide what CAs to trust can effectively dismantle any CA business in the world, basically at the drop of a hat. If DigiCert decides to play this game and lose, it would make them the biggest such loser so far. DigiCert is, as far as I know, the largest CA on the Internet. It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores. Yet, there's no particular reason it couldn't happen. How exciting. Of course, I think that is unlikely, but on the other hand, it's just as cathartic to imagine whatever idiot at DigiCert thought it was a good idea to engage legal here to have the dressing down of a lifetime. I read the thread in question. It doesn't make DigiCert look good, but this action definitely is more damning to them than anything Collan said in my opinion.
- userbinator 2y ago
  It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores. How many will agree to that removal? How many will see one more reason to forever turn off automatic updates and decide what to trust themselves, having seen yet another way some faceless entity they never knew about can break things? It will certainly send a strong message, but likely not the intended one. All it will do is increase the lack of trust in centralised PKI in general.
  - jchw 2y ago
    I'm not sure how to put this nicely, but I'll try my best: The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision. (And corporate users can't do this anyways, since it's not in their hands, and keeping things out of date is not a reasonable option for an organization that has security standards.) Most of them won't know what is going on here, and if they hear about it, probably won't care or know what to do about it. That's the Web PKI infrastructure working as intended, because it would be infeasible for billions of people to properly understand the gravity of all of these decisions. In order to ensure that TLS connections are at least minimally safe, it's pretty much necessary for things to work this way. I'm not arguing that it's good that a relatively small number of entities (mainly Google, Microsoft and Mozilla) decide which CAs are trustworthy, but that's all the more reason that it's important for all of this Web PKI work to happen completely in the open, so that the few who can spare the time and effort to scrutinize what is going on can help the rest of us have a safer Internet. We don't have a better solution. That's also why DigiCert issuing legal threats because they don't like how one of these issue reports makes them look is a serious problem that can't be tolerated.
Arnavion 2y ago
Even taking the (one-sided) depiction of the conversations in DigiCert's letter at face value, maybe Sectigo's guy was being a git at best and intentionally trolling at worst. (I don't think he was, but let's play devil's advocate.) But even then, how did DigiCert think getting legal involved would possibly go well? Sectigo stands to gain publicity and lose nothing by going public with it to the CAB as they did here, and it's not like the CAB is going to play marriage counselor and get the two companies to make up because one of them got their feefees hurt. Besides, this kind of hyper-polite passive-aggressive "erm akchually" conversation happens in every CAB incident discussion. I don't know why DigiCert got particularly upset about this one.
- akerl_ 2y ago
  > Besides, this kind of hyper-polite passive-aggressive "erm akchually" conversation happens in every CAB incident discussion. As somebody who doesn't spend much time scrolling CAB reports, this was jarring to me. Digicert's legal action seems nuts, and there seems like a real, risky issue in the idea that a company's customers can use the legal system to block the company from complying with its obligations to other entities, but it's hard to see any way that could be productively addressed given the back and forth in the thread. It's like I'm watching a theatrical production staring the most stereotypical corporate drones trading comments with the most stereotypical IRC nerds, both sides doing circles around an interesting topic but too busy trading blows to ever really get to it.
  - SpicyLemonZest 2y ago
    > there seems like a real, risky issue in the idea that a company's customers can use the legal system to block the company from complying with its obligations to other entities As Digicert has repeatedly explained, this is simply how the United States legal system works. Courts have broad and indisputable power to issue temporary restraining orders, and the parties to a case must comply even if doing so violates some promise they made to a third party. (The point of the TRO is to maintain the status quo while the court figures out details like what promises have been made to who.) People in the PKI community who believe that some carefully written policy would enable CAs to reject an invalidation TRO, or convince a court that they cannot issue it, are wrong. The reason it's never come up before is that no CA had previously attempted to enforce a widespread 24 hour revocation caused by its own error.
webprofusion 2y ago
Always two sides to a story but the guy who caused the validation bug at DigiCert already resigned because of it (which is extreme), the Sectigo guy wanted to prevent the bug being closed so he could keep pushing them (in a subjectively prickly fashion) for more answers about their general responsiveness. A bit of back and forth discourse is fine and expected, but if you keep pushing someone who has their own legal dept they're eventually going to wander over to the coffee machine and have a chat about it with them, then they're going to take a look and it becomes their problem. So the number one rule would be don't even breath the word "legal" unless you want to invoke them. This particular response is just a letter telling them to back off and it's why you have a legal dept, so they can argue with each other. This one has just found it's way into the open. There is a understandable perspective that says CAs shouldn't be burdened with legal risk in their discussions, but that's contrary to the fact these guys are commercial entities protecting their interests, so you don't get it both ways unless all your CAs are non-commercial, and even then that would only extend so far.
- willvarfar 2y ago
  You've given a detailed rundown like you've been following, so what is your sense about the resignation? Did the guy resign willingly, or is Digicert the kind of management that likely scapegoated him?
  - braiamp 2y ago
    He assumed personal responsibility rather than institutional. At the time, everyone in the community expressed alarm that that happened, because 1) and this is obvious, there's no way a single guy would be responsible for the entire fallout and 2) because it sets a bad precedent about what companies can do with their PoC wrt web PKI. It also meant a loss in institutional knowledge about how things worked.
    - smithcoin 2y ago
      Do you have any resources regarding the resignation I could read?
nneonneo 2y ago
In a nutshell: DigiCert has delayed revocation beyond what's allowed in the Baseline Requirements a few times; most recently, https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 and https://bugzilla.mozilla.org/show_bug.cgi?id=1910805 https://bugzilla.mozilla.org/show_bug.cgi?id=1910805. In the former case, it seems DigiCert chose to delay revocation to appease certain clients; in the latter case they were prohibited by a Temporary Restraining Order (TRO) from performing timely revocation. Tim Callan from Sectigo has publicly lambasted DigiCert for these delays, since in both cases it seems DigiCert hasn't pushed back hard enough on its clients. In the latter case, there's concern that measures like TROs might be employed more often to stall revocation. Sectigo (and others in the WebPKI ecosystem) seem to want DigiCert to make the revocation policies very clear to clients and to ensure that clients can actually replace their certificates in a timely manner. Sectigo is clearly the most vocal but they don't seem to be the only ones telling DigiCert to get their delayed revocation under control. So the escalation to legal threats is really uncalled for, and DigiCert could face some very significant pushback for trying this tactic.
- RHSeeger 2y ago
  What is the correct action when a TRO says a company cannot revoke the cert? Is it that the company will delay revocation, but will push the judicial system to resolve the issue as fast as possible?
  - nneonneo 2y ago
    DigiCert probably should have revoked every cert they could within 24 hours. Instead they just pushed the revocation of all 80,000+ certs out to five days. It's quite likely that many of their other clients pushed back on the 24-hour timeline (similar to what happened in their previous incident); I believe the delayed revocation issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322) hints at this. The TRO gave them a convenient excuse to delay all revocations without having to explain all over again why they made exemptions for their special clients. Heck, their status page (https://status.digicert.com/incidents/3sccz3v31lc9 https://status.digicert.com/incidents/3sccz3v31lc9) even gives instructions for how to request a delayed revocation - even though the initial incident page (https://www.digicert.com/support/certificate-revocation-incident https://www.digicert.com/support/certificate-revocation-inci...) says clearly: "Any issue with domain validation is considered a serious issue by CABF and requires immediate action. Failure to comply can result in a distrust of the Certificate Authority. As such, we must revoke all impacted certificates within 24 hours of discovery. No extensions or delays are permitted. We apologize if this causes a business disruption to you and are standing by to assist you with validating your domain and issuing replacement certificates immediately."
naitgacem 2y ago
Can someone point to where i can read some context?
- nneonneo 2y ago
  DigiCert's legal threat, while obviously biased towards DigiCert, gives some context: https://bug1950144.bmoattachments.org/attachment.cgi?id=9468038 https://bug1950144.bmoattachments.org/attachment.cgi?id=9468... For further reading, consider these two incidents which resulted in delayed revocation from DigiCert and a bunch of comments about how DigiCert should not be allowing delayed revocation: - Incident report https://bugzilla.mozilla.org/show_bug.cgi?id=1894560 https://bugzilla.mozilla.org/show_bug.cgi?id=1894560, delayed revocation report https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 - incident due to the issuance of some certificates with incorrectly-capitalized phrases in the certificate's Business Category field; baseline requirements require revocation within five days but DigiCert dragged that out much further - Incident report https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322, delayed revocation report https://bugzilla.mozilla.org/show_bug.cgi?id=1910805 https://bugzilla.mozilla.org/show_bug.cgi?id=1910805, DigiCert information page https://www.digicert.com/support/certificate-revocation-incident https://www.digicert.com/support/certificate-revocation-inci... - incident due to incorrect CNAME-based domain validation (failure to check that the CNAME started with an underscore); baseline requirements require revocation within 24 hours but DigiCert was stopped by the TRO and revoked after five days. Essentially, DigiCert has been delaying the revocation process (twice now) and people are unhappy about that. DigiCert has apparently attempted to silence those unhappy people (Sectigo and their representative Tim Callan) with legal action.
  - skylerwiernik 2y ago
    I don't believe that that's a legal filing. DigiCert never filed anything with the courts. That's just a letter to Sectigo threatening to sue.
    - nneonneo 2y ago
deleted 2y ago
[deleted]
tbrownaw 2y ago
So what's happened in the ... two months and a bit since the dates mentioned for the letters that this is apparently in reference to?
- nickburns 2y ago
  Continued back-and-forth on Bugzilla: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c63 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c63
- nneonneo 2y ago
  DigiCert posted a message fifteen days ago saying that "We have not used a legal team as a shield against accountability." (https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c74 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c74). This is clearly contradicted by their legal threat against Sectigo. Hence, Sectigo decided to post the "Threat of legal action" bug to make the community aware of what DigiCert actually did. Maybe if DigiCert had not decided to make that comment, Sectigo would have been willing to stay quiet...
xmodem 2y ago
Looking at the original report ( https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 ) I can see a couple of questions that DigiCert appears to be avoiding: > The public record of Alegeus Technologies LLC v. DigiCert shows no attempt by DigiCert to contest the court’s order prior to the end of its preferred period of nearly 120 hours, even though such a motion could have freed DigiCert to revoke the certificates days earlier. and > The other question in comment 28 was for the language establishing DigiCert’s right to revoke Alegeus Technologies certificates. DigiCert has waffled on this point, first implying that this language was to be found on its website but later refusing to confirm that the language on the site applied to Alegeus Technologies at the time. SPECULATION: Digicert may have offered special terms to Alegeus, and possibly other customers. They may have chosen not to dispute the TRO in court because they did not have grounds to do so under those agreements. They may also have included confidentiality terms in those contracts that prevented them from speaking about it. OPINION: I am surprised that the forum allowed the issue to be closed without the above quoted questions being satisfied, though it is possible they are addressed elsewhere, I have not done a complete reading of all the linked issues. EDIT to add: DigiCert has a response in a different thread here: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43 https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43 that would appear to contradict my speculation. Specifically > Even though DigiCert’s TOU and MSA prohibited Alegeus from taking the action it did, once it filed for a TRO and the court almost immediately granted it, DigiCert’s hands were tied
- thayne 2y ago
  > Even though DigiCert’s TOU and MSA prohibited Alegeus from taking the action it did, once it filed for a TRO and the court almost immediately granted it, DigiCert’s hands were tied So did the judge just not read the TOU before signing the TRO? I wonder if the CAB forum would have standing to sue Alegeus and/or that judge for interfering with the PKI process with an invalid TRO.
DarkmSparks 2y ago
From the bugzilla "The actual reason for the underscore is so that services which allow users to create DNS records at subdomains (e.g. dynamic DNS services) can block users from registering subdomains starting with an underscore and be safe from unwanted certificate issuance. It serves the same purpose that /.well-known does for Agreed-Upon Change To Website, and that admin/administrator/webmaster/hostmaster/postmaster do for Constructed Email to Domain Contact. By using DNS records without underscores, DigiCert has violated a security-critical assumption that these services have made. Therefore, this is truly a security-critical incident, " That is a pretty brutal f' up of epic proportions... not sure any digicert cert should be trusted after that.
- mtlynch 2y ago
  Direct link to the comment: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c10 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c10 The author of that comment is Andrew Ayer, who also does great writeups about CA incidents and processes on his blog: https://www.agwa.name/blog/index https://www.agwa.name/blog/index
- kevincox 2y ago
  My bigger concern is that what percentage of people will realize when delegating subdomains that having a leading underscore is a security vulnerability?
terom 2y ago
It's fascinating that we've built a system that has expended perhaps several million dollars of engineering, legal and admin etc time over the issue of a single letter not being capitalized [1], without any demonstrable impact beyond a failure to meet ambiguous specifications. I do hope that dealing with all of the underlying issues around revocation etc makes the time and effort spent useful, and the Web PKI doesn't just mire itself in squabbling that blocks progress on actually meaningful issues. [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1894560 https://bugzilla.mozilla.org/show_bug.cgi?id=1894560
- xmodem 2y ago
  I think that the Van Halen Brown M&M anecdote is relevant here. > and the Web PKI doesn't just mire itself in squabbling that blocks progress on actually meaningful issues. In your view, are there any meaningful issues going un-addressed currently?
- ocdtrekkie 2y ago
  Entrust got torpedoed basically for deploying an improvement to the requirements for its certificates slightly before the improvement got officially approved, and the browser people collectively lost their crud over the concept that Entrust didn't immediately revoke all of their... perfectly valid, secure certificates immediately. Fundamentally, there is no accountability in the web PKI stewards. You want to talk about utter waste and incredible damage to the Internet, you can see it right here, in the people determining who is allowed to issue you sets of magic numbers that browsers have agreed are trustworthy, despite everyone involved behaving like complete children. And of course, the browser operators all have their own root CAs, so basically have extremely motivated reasons to want to eliminate every commercial provider that isn't one of the monopoly companies. Meanwhile: - Compromised certificates are basically a non-issue from a threat model standpoint, every certificate error people hit are just... expired certificates people didn't rotate yet. - Expired certificates cause issues for the majority of businesses at some point or another, making the internet increasingly fragile and unreliable.
  - drudgemetal 2y ago
DeepSeaTortoise 2y ago
Why do you all think DigiCert handled this badly? 1. Bugs happen. Critical ones, too. They didn't try to brush this under the carpet, but admitted to it, acted to resolve it and were transparent about it. 2. They worked quickly to make it happen. Would 24h been nice? Sure, but 24h is not much shorter than 120h. In general, 24h is plenty of time for some exploits and 120h doesn't open the window to many more. It would have been very different if it took them months or years to resolve it. 3. They genuinely engaged with the critics on bugzilla, even after Sectigo's CCO went completely off the rails with trying to strip customers off legal recourse and demanding to blacklist those who try to make use of it. 4. They could have taken legal actions against Sectigo's CCO directly but took the extra step to ask them to stop this nonsense. They didn't demand anything more and even outlined steps Sectigo needed to take to prevent any legal problems down the line, like affirming that their CCO did not make these statements on behalf of Sectigo, an affirmation that they would notify their employees to not make any actions that would violate the laws mentioned in their letter, affirm that their CCO would be instructed not to violate any of the laws outlined in their letter and lastly confirm that, upon consulting with their CCO, they were able to conclude that his statements were not meant to harm DigiCert. The only ick is the short timeframe they expect a reply within, but that's sadly usual corporate US law practice... Basically that letter is the result of asking an US law firm for help and telling them to be nice about it and helping their opponent through the process.
- deleted 2y ago
  [deleted]
- SAI_Peregrinus 2y ago
  DigiCert miss-issued thousands of certs. Certificate authorities are required by contract (the CA/Browser Forum Baseline Requirements is a contract they agree to when being granted trust as a CA) to revoke miss-issued certificates within 24 hours of the time they learn of the miss-issuance. One of their customers filed a court case & got a temporary restraining order (TRO) preventing DigiCert from revoking ≈70 of those certs within 24 hours. DigiCert then proceeded to delay revocation beyond the 24 hour limit *for all the certs, not just the ≈70 that the TRO covered. That is a violation of the CA/Browser Forum Baseline Requirements. Failure to revoke ≈70 certs because of a court order could be accepted, but failure to revoke thousands of other certs (putting customers at risk) should not be.
mightysashiman 2y ago
Streisand effect 101?
account42 2y ago
[flagged]
kuon 2y ago
What is the actual impact on infrastructure? Is it just some old certificates being valid past expiration?
- drpossum 2y ago
  I don't think DigiCert has the power to change literally every internal validating mechanism to check if a certificate is expired or not. But you seem to know more, so I'd ask you expound on that.
aaronmdjones 2y ago
Certificate authorities have enormous trust placed upon them by every Internet user (whether they know it or not). Commensurate with this trust, they have enormous responsibilities. As the name implies, the Baseline Requirements are the minimum standard they should achieve. If they can't even do this (being unable or unwilling to revoke issued certificates within the required time-frame), then they do not deserve this trust, and it should be removed. I understand that the TRO prevented them from revoking approximately 70 certificates, and there really is nothing they could have done differently in that case. Their other revocation failings are inexcusable.
csense 2y ago
This is a really stupid situation all around. To issue a cert for a client (say example.com), the CA sends a random number challenge to the client (say 12345678). The client then responds by making a DNS record for _12345678.example.com but due to a bug, DigiCert accepted 12345678.example.com instead. Apparently there's a rule that says any such certs must be considered mis-issued and must be revoked by the issuing CA within 24 hours. The CA talked to some clients who said it would be too much trouble to replace their cert on such short notice. One of those clients filed a lawsuit with the court system, which responded by issuing a restraining order telling the CA not to revoke the cert. First of all, this is a dumb reason to revoke certs. It's difficult to imagine a situation where any of the mis-issued certificates correspond to an actual compromise of somebody's website by a bad actor. But we can perhaps give the benefit of the doubt: I think the intent here is a fail-safe procedure is implemented so revocation is triggered by a broad class of circumstances, which is constructed so that it's easy to check. Second, this is a dumb thing for a website to file a lawsuit about. If you have to replace the cert on your website because your CA made a booboo, the engineer-hours needed to change out your new cert are far less than the lawyer-hours needed to try to work through the court system. Third, it's dumb for people to blame DigiCert for the TRO. The TRO was filed by a service provider called Alegeus. If DigiCert doesn't follow the TRO, they'll be in contempt of court. Theoretically, a DigiCert employee who presses the "Revoke" button at this point could be sent to jail for it! Fourth, a certificate revocation is basically a signed message that says "News Flash: I think this certificate might not correspond to this website." A court preventing the publication of such a message is equivalent to a court preventing a newspaper from publishing an article. This is called "prior restraint" and it has a high bar in the US. Prior restraint analysis was not performed by the court before the TRO was issued. I think the court violated DigiCert's due process rights (by not analyzing whether prior restraint applies) and DigiCert's free speech rights (by not denying the TRO as it would be prior restraint). (That browsers have a hair-trigger response to certain kinds of news flashes and will take drastic action isn't the fault of the publisher of the news.) Fifth, it's obvious there ought to be some technical means to prevent this kind of situation in the future: What is the protocol to handle a case of a CA refusing (or court-ordered not to) revoke certs known to be compromised? I think you need to have a system that allows a quorum of CA's not including the issuing CA to speedily revoke certificates, or an entire CA. (A public real-time quorum of a dynamic set of validators passing messages they want to reach consensus on: Perhaps this is a good use case for a blockchain?) Of course you also have to deal with the problem of, what if the next TRO targets the entire validator set? You'd need to be sure those participants are jurisdictionally diverse (or anonymous) so it would be difficult for a single entity to compromise the entire system at once.
registeredcorn 2y ago
Fascinating. I think there was a fair amount of snark on both sides, but I do think some good points were raised by both, as well. 1) To DigiCert's point: If certs need an emergency revocation but it will impact a service which say: provides life saving services, or keeps the electricity on for the majority of a country, would it not be wise to file them as a one-off "exceptional circumstance". I think that common sense should prevail and everyone can agree that, "Yes, computer security is absolutely essential. Essential services are also essential." I wish that that was the direction the debate had gone in. For instance, What is considered an 'exceptional circumstance'? What kind of services are covered, and what are not? Personally, I would think that things like: health, heat, water, electricity, and physical security (prison and law enforcement) are all potentially essential areas. They are industries that ought be able to request an emergency, 48-hour exception if they know they can't meet it within 24 hours and their services will go down as a result. I feel like two days should be enough time for just about any organization to work through a certificate issue, unless it's a long holiday, or something very, very niche. I think that, to a degree, Tim Callan (Setigo CEO) was being unreasonable in expecting DigiCert to not offer any kind of possibility for exceptions. Some services should not go down, just because it goes against the principals of computer security. It hate saying that, but it's true. Keeping the ICU running matters more than whether the hospital is following best security practices during an emergency. Could it cause more problems by ignoring best practices? Possibly! Will enforcing best practices possibly kill someone? If the answer is anything other than a firm "No", then it is secondary to protecting that service. 2) To Sectigo's point: We should not allow any CA to hide behind Policies or poorly written MSAs. If things went the way they did because they were allowed to go that way, then that means you should learn from those things in the post mortem! Take steps to shore it up! Try and prevent other companies from following suit, otherwise more will take action whenever it meets their own best interest. It is disappointing that this part seems to fell into snarky retorts too, because there were some legitimate means to discuss this. For instance: Instead of barring from someone from being allowed to file a TRO, simply have an agreement in place that before any legal action like a TRO is filed, the customer will meet with the CA and a emergency mediator. Just take 30 minutes to one hour to see if you can work things out before the customer submits a TRO! It seems logical, right? If a customer has the cycles to file for a TRO, they should have the time to spare talking to the company they are filing a TRO against. Explain a clear reasons to a mediator why the TRO is needed, and why they can't get it done in time. Assuming that the customer can explain all of that in clear terms, it would then be obvious for DigiCert to acknowledge that level of criticality and "exceptional need", and offer their customer an emergency, temporary exemption. Neither side wants a TRO! It makes DigiCert look weak during an emergency, and it makes Alegeus (the company that filed the TRO) look incompetent, desperate, and underhanded. The crux of what Tim Callan (Sectigo) was getting at, is that there needs to be a correction to DigiCert's policies. It's blaringly obvious. DigiCert were, in a way, "legally attacked" in a manner that should be prevented in the future, as best they can prevent it. DigiCert lackadaisically shrugging their shoulders and saying "B-But...that goes against Mozilla policy!" is just deflection and meaningless. DigiCert can go to the trouble of sending legal council after Sectigo for comments on Bugzilla, but they can't use legal council to protect DigiCert from surprise TRO's? Really? Bugzilla feedback...that is the legal issue? Not DigiCert being sucker punched by their own customers? The whole thing is just so aggravating. Both sides need to get over themselves and try to work together. They don't need to like each other, but they should do what is best for the industry. Each side sending out daddy lawyer to fight for them completely misses the point, and kills the chance for constructive feedback.
ziddoap 2y ago
Bug has been updated with a DigiCert response. Obviously draw your own conclusions, but I actually laughed out loud at the following statement from DigiCert: "In reality, our letter to you was consistent with our desire to promote open and honest dialogue."

610 points by DanAtC 2y ago

nightfly 2y ago

Can someone link to (or post copies of) the offending questions/statements?

infogulch 2y ago
In the second paragraph: > Contrary to this statement, I received a letter from DigiCert’s lawyers, Wilson Sonsini, regarding posts made by Sectigo’s Chief Compliance Officer in bug 1910322. https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322
- mdaniel 2y ago
  > The code worked in our original monolithic system but was not implemented properly when we moved to our micro-services systems :chefs_kiss:
  - crooked-v 2y ago
    As we all know, software isn't software unless it can immediately scale to having 12 billion users.
  - evil-olive 2y ago
    it gets even better: > We also found that the bug in the code was inadvertently remediated when engineering completed a user-experience enhancement project that collapsed multiple random value generation microservices into a single service.
    - tpxl 2y ago
      > multiple random value generation microservices Wot. They had _multiple_ services generating random numbers?
      - aragilar 2y ago
        It might mean multiple instances of the same code (which would explain why there might be commonalities)?
    - intelVISA 2y ago
      Some might see horror but I see a few genius level engineers 'working' 1hour a month rebuilding the same getrandom syscall. They hiring? :]

infogulch 2y ago

This is shocking to read. Even attempting to choke the legal speech of web PKI contributors with legalistic bullying is a gross inversion of the purpose and goals of the organization, and IMO warrants revoking everything to do with DigiCert on the spot.

terminalbraid 2y ago
> IMO warrants revoking everything to do with DigiCert on the spot This is pretty heavy handed and I don't think you've thought through what the consequences of that may look like. The historic way to deal with a problematic CA is to prevent them from issuing new certificates or renewing certificates (after taking care of immediate damage, of course). There are lots of legitimate companies that use Digicert and they should have an expectation of being able to continue business in the short term while they find a different certificate provider.
- DeepPPP 2y ago
  Are you saying it was okay to distrust EnTrust but DigiCert is too big to fail?
  - terminalbraid 2y ago
    No, and I'm deeply confused how you drew this conclusion from what was written. I specifically say the historical way to deprecate a certificate authority is to prevent them from issuing new certificates. Since certificates have a finite lifetime, the certificate authority would as well and would have immediately no further revenue from certificates. I am saying "pulling the plug" without notice is going to take down tens of thousands of bystanders and anyone using those certificates for business would be unable to conduct business securely online, which would be disastrous. For example amazon.com has a DigiCert-issued certificate.
    - DeepPPP 2y ago
      Agreed. That was my bad.

jchw 2y ago

Web PKI drama is always astonishing to me because it is one of the only areas of the entire world where a corporation's "fucking around" is seldom ever not followed by a sobering "finding out" period. The various entities that decide what CAs to trust can effectively dismantle any CA business in the world, basically at the drop of a hat. If DigiCert decides to play this game and lose, it would make them the biggest such loser so far. DigiCert is, as far as I know, the largest CA on the Internet. It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores. Yet, there's no particular reason it couldn't happen. How exciting. Of course, I think that is unlikely, but on the other hand, it's just as cathartic to imagine whatever idiot at DigiCert thought it was a good idea to engage legal here to have the dressing down of a lifetime. I read the thread in question. It doesn't make DigiCert look good, but this action definitely is more damning to them than anything Collan said in my opinion.

userbinator 2y ago
It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores. How many will agree to that removal? How many will see one more reason to forever turn off automatic updates and decide what to trust themselves, having seen yet another way some faceless entity they never knew about can break things? It will certainly send a strong message, but likely not the intended one. All it will do is increase the lack of trust in centralised PKI in general.
- jchw 2y ago
  I'm not sure how to put this nicely, but I'll try my best: The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision. (And corporate users can't do this anyways, since it's not in their hands, and keeping things out of date is not a reasonable option for an organization that has security standards.) Most of them won't know what is going on here, and if they hear about it, probably won't care or know what to do about it. That's the Web PKI infrastructure working as intended, because it would be infeasible for billions of people to properly understand the gravity of all of these decisions. In order to ensure that TLS connections are at least minimally safe, it's pretty much necessary for things to work this way. I'm not arguing that it's good that a relatively small number of entities (mainly Google, Microsoft and Mozilla) decide which CAs are trustworthy, but that's all the more reason that it's important for all of this Web PKI work to happen completely in the open, so that the few who can spare the time and effort to scrutinize what is going on can help the rest of us have a safer Internet. We don't have a better solution. That's also why DigiCert issuing legal threats because they don't like how one of these issue reports makes them look is a serious problem that can't be tolerated.

Arnavion 2y ago

Even taking the (one-sided) depiction of the conversations in DigiCert's letter at face value, maybe Sectigo's guy was being a git at best and intentionally trolling at worst. (I don't think he was, but let's play devil's advocate.) But even then, how did DigiCert think getting legal involved would possibly go well? Sectigo stands to gain publicity and lose nothing by going public with it to the CAB as they did here, and it's not like the CAB is going to play marriage counselor and get the two companies to make up because one of them got their feefees hurt. Besides, this kind of hyper-polite passive-aggressive "erm akchually" conversation happens in every CAB incident discussion. I don't know why DigiCert got particularly upset about this one.

akerl_ 2y ago
> Besides, this kind of hyper-polite passive-aggressive "erm akchually" conversation happens in every CAB incident discussion. As somebody who doesn't spend much time scrolling CAB reports, this was jarring to me. Digicert's legal action seems nuts, and there seems like a real, risky issue in the idea that a company's customers can use the legal system to block the company from complying with its obligations to other entities, but it's hard to see any way that could be productively addressed given the back and forth in the thread. It's like I'm watching a theatrical production staring the most stereotypical corporate drones trading comments with the most stereotypical IRC nerds, both sides doing circles around an interesting topic but too busy trading blows to ever really get to it.
- SpicyLemonZest 2y ago
  > there seems like a real, risky issue in the idea that a company's customers can use the legal system to block the company from complying with its obligations to other entities As Digicert has repeatedly explained, this is simply how the United States legal system works. Courts have broad and indisputable power to issue temporary restraining orders, and the parties to a case must comply even if doing so violates some promise they made to a third party. (The point of the TRO is to maintain the status quo while the court figures out details like what promises have been made to who.) People in the PKI community who believe that some carefully written policy would enable CAs to reject an invalidation TRO, or convince a court that they cannot issue it, are wrong. The reason it's never come up before is that no CA had previously attempted to enforce a widespread 24 hour revocation caused by its own error.

webprofusion 2y ago

Always two sides to a story but the guy who caused the validation bug at DigiCert already resigned because of it (which is extreme), the Sectigo guy wanted to prevent the bug being closed so he could keep pushing them (in a subjectively prickly fashion) for more answers about their general responsiveness. A bit of back and forth discourse is fine and expected, but if you keep pushing someone who has their own legal dept they're eventually going to wander over to the coffee machine and have a chat about it with them, then they're going to take a look and it becomes their problem. So the number one rule would be don't even breath the word "legal" unless you want to invoke them. This particular response is just a letter telling them to back off and it's why you have a legal dept, so they can argue with each other. This one has just found it's way into the open. There is a understandable perspective that says CAs shouldn't be burdened with legal risk in their discussions, but that's contrary to the fact these guys are commercial entities protecting their interests, so you don't get it both ways unless all your CAs are non-commercial, and even then that would only extend so far.

willvarfar 2y ago
You've given a detailed rundown like you've been following, so what is your sense about the resignation? Did the guy resign willingly, or is Digicert the kind of management that likely scapegoated him?
- braiamp 2y ago
  He assumed personal responsibility rather than institutional. At the time, everyone in the community expressed alarm that that happened, because 1) and this is obvious, there's no way a single guy would be responsible for the entire fallout and 2) because it sets a bad precedent about what companies can do with their PoC wrt web PKI. It also meant a loss in institutional knowledge about how things worked.
  - smithcoin 2y ago
    Do you have any resources regarding the resignation I could read?

nneonneo 2y ago

In a nutshell: DigiCert has delayed revocation beyond what's allowed in the Baseline Requirements a few times; most recently, https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 and https://bugzilla.mozilla.org/show_bug.cgi?id=1910805 https://bugzilla.mozilla.org/show_bug.cgi?id=1910805. In the former case, it seems DigiCert chose to delay revocation to appease certain clients; in the latter case they were prohibited by a Temporary Restraining Order (TRO) from performing timely revocation. Tim Callan from Sectigo has publicly lambasted DigiCert for these delays, since in both cases it seems DigiCert hasn't pushed back hard enough on its clients. In the latter case, there's concern that measures like TROs might be employed more often to stall revocation. Sectigo (and others in the WebPKI ecosystem) seem to want DigiCert to make the revocation policies very clear to clients and to ensure that clients can actually replace their certificates in a timely manner. Sectigo is clearly the most vocal but they don't seem to be the only ones telling DigiCert to get their delayed revocation under control. So the escalation to legal threats is really uncalled for, and DigiCert could face some very significant pushback for trying this tactic.

RHSeeger 2y ago
What is the correct action when a TRO says a company cannot revoke the cert? Is it that the company will delay revocation, but will push the judicial system to resolve the issue as fast as possible?
- nneonneo 2y ago
  DigiCert probably should have revoked every cert they could within 24 hours. Instead they just pushed the revocation of all 80,000+ certs out to five days. It's quite likely that many of their other clients pushed back on the 24-hour timeline (similar to what happened in their previous incident); I believe the delayed revocation issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322) hints at this. The TRO gave them a convenient excuse to delay all revocations without having to explain all over again why they made exemptions for their special clients. Heck, their status page (https://status.digicert.com/incidents/3sccz3v31lc9 https://status.digicert.com/incidents/3sccz3v31lc9) even gives instructions for how to request a delayed revocation - even though the initial incident page (https://www.digicert.com/support/certificate-revocation-incident https://www.digicert.com/support/certificate-revocation-inci...) says clearly: "Any issue with domain validation is considered a serious issue by CABF and requires immediate action. Failure to comply can result in a distrust of the Certificate Authority. As such, we must revoke all impacted certificates within 24 hours of discovery. No extensions or delays are permitted. We apologize if this causes a business disruption to you and are standing by to assist you with validating your domain and issuing replacement certificates immediately."

naitgacem 2y ago

Can someone point to where i can read some context?

nneonneo 2y ago
DigiCert's legal threat, while obviously biased towards DigiCert, gives some context: https://bug1950144.bmoattachments.org/attachment.cgi?id=9468038 https://bug1950144.bmoattachments.org/attachment.cgi?id=9468... For further reading, consider these two incidents which resulted in delayed revocation from DigiCert and a bunch of comments about how DigiCert should not be allowing delayed revocation: - Incident report https://bugzilla.mozilla.org/show_bug.cgi?id=1894560 https://bugzilla.mozilla.org/show_bug.cgi?id=1894560, delayed revocation report https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 - incident due to the issuance of some certificates with incorrectly-capitalized phrases in the certificate's Business Category field; baseline requirements require revocation within five days but DigiCert dragged that out much further - Incident report https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322, delayed revocation report https://bugzilla.mozilla.org/show_bug.cgi?id=1910805 https://bugzilla.mozilla.org/show_bug.cgi?id=1910805, DigiCert information page https://www.digicert.com/support/certificate-revocation-incident https://www.digicert.com/support/certificate-revocation-inci... - incident due to incorrect CNAME-based domain validation (failure to check that the CNAME started with an underscore); baseline requirements require revocation within 24 hours but DigiCert was stopped by the TRO and revoked after five days. Essentially, DigiCert has been delaying the revocation process (twice now) and people are unhappy about that. DigiCert has apparently attempted to silence those unhappy people (Sectigo and their representative Tim Callan) with legal action.
- skylerwiernik 2y ago
  I don't believe that that's a legal filing. DigiCert never filed anything with the courts. That's just a letter to Sectigo threatening to sue.
  - nneonneo 2y ago

deleted 2y ago

[deleted]

tbrownaw 2y ago

So what's happened in the ... two months and a bit since the dates mentioned for the letters that this is apparently in reference to?

nickburns 2y ago
Continued back-and-forth on Bugzilla: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c63 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c63
nneonneo 2y ago
DigiCert posted a message fifteen days ago saying that "We have not used a legal team as a shield against accountability." (https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c74 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c74). This is clearly contradicted by their legal threat against Sectigo. Hence, Sectigo decided to post the "Threat of legal action" bug to make the community aware of what DigiCert actually did. Maybe if DigiCert had not decided to make that comment, Sectigo would have been willing to stay quiet...

xmodem 2y ago

Looking at the original report ( https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 ) I can see a couple of questions that DigiCert appears to be avoiding: > The public record of Alegeus Technologies LLC v. DigiCert shows no attempt by DigiCert to contest the court’s order prior to the end of its preferred period of nearly 120 hours, even though such a motion could have freed DigiCert to revoke the certificates days earlier. and > The other question in comment 28 was for the language establishing DigiCert’s right to revoke Alegeus Technologies certificates. DigiCert has waffled on this point, first implying that this language was to be found on its website but later refusing to confirm that the language on the site applied to Alegeus Technologies at the time. SPECULATION: Digicert may have offered special terms to Alegeus, and possibly other customers. They may have chosen not to dispute the TRO in court because they did not have grounds to do so under those agreements. They may also have included confidentiality terms in those contracts that prevented them from speaking about it. OPINION: I am surprised that the forum allowed the issue to be closed without the above quoted questions being satisfied, though it is possible they are addressed elsewhere, I have not done a complete reading of all the linked issues. EDIT to add: DigiCert has a response in a different thread here: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43 https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43 that would appear to contradict my speculation. Specifically > Even though DigiCert’s TOU and MSA prohibited Alegeus from taking the action it did, once it filed for a TRO and the court almost immediately granted it, DigiCert’s hands were tied

thayne 2y ago
> Even though DigiCert’s TOU and MSA prohibited Alegeus from taking the action it did, once it filed for a TRO and the court almost immediately granted it, DigiCert’s hands were tied So did the judge just not read the TOU before signing the TRO? I wonder if the CAB forum would have standing to sue Alegeus and/or that judge for interfering with the PKI process with an invalid TRO.

DarkmSparks 2y ago

From the bugzilla "The actual reason for the underscore is so that services which allow users to create DNS records at subdomains (e.g. dynamic DNS services) can block users from registering subdomains starting with an underscore and be safe from unwanted certificate issuance. It serves the same purpose that /.well-known does for Agreed-Upon Change To Website, and that admin/administrator/webmaster/hostmaster/postmaster do for Constructed Email to Domain Contact. By using DNS records without underscores, DigiCert has violated a security-critical assumption that these services have made. Therefore, this is truly a security-critical incident, " That is a pretty brutal f' up of epic proportions... not sure any digicert cert should be trusted after that.

mtlynch 2y ago
Direct link to the comment: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c10 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c10 The author of that comment is Andrew Ayer, who also does great writeups about CA incidents and processes on his blog: https://www.agwa.name/blog/index https://www.agwa.name/blog/index
kevincox 2y ago
My bigger concern is that what percentage of people will realize when delegating subdomains that having a leading underscore is a security vulnerability?

terom 2y ago

It's fascinating that we've built a system that has expended perhaps several million dollars of engineering, legal and admin etc time over the issue of a single letter not being capitalized [1], without any demonstrable impact beyond a failure to meet ambiguous specifications. I do hope that dealing with all of the underlying issues around revocation etc makes the time and effort spent useful, and the Web PKI doesn't just mire itself in squabbling that blocks progress on actually meaningful issues. [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1894560 https://bugzilla.mozilla.org/show_bug.cgi?id=1894560

xmodem 2y ago
I think that the Van Halen Brown M&M anecdote is relevant here. > and the Web PKI doesn't just mire itself in squabbling that blocks progress on actually meaningful issues. In your view, are there any meaningful issues going un-addressed currently?
ocdtrekkie 2y ago
Entrust got torpedoed basically for deploying an improvement to the requirements for its certificates slightly before the improvement got officially approved, and the browser people collectively lost their crud over the concept that Entrust didn't immediately revoke all of their... perfectly valid, secure certificates immediately. Fundamentally, there is no accountability in the web PKI stewards. You want to talk about utter waste and incredible damage to the Internet, you can see it right here, in the people determining who is allowed to issue you sets of magic numbers that browsers have agreed are trustworthy, despite everyone involved behaving like complete children. And of course, the browser operators all have their own root CAs, so basically have extremely motivated reasons to want to eliminate every commercial provider that isn't one of the monopoly companies. Meanwhile: - Compromised certificates are basically a non-issue from a threat model standpoint, every certificate error people hit are just... expired certificates people didn't rotate yet. - Expired certificates cause issues for the majority of businesses at some point or another, making the internet increasingly fragile and unreliable.
- drudgemetal 2y ago

DeepSeaTortoise 2y ago

Why do you all think DigiCert handled this badly? 1. Bugs happen. Critical ones, too. They didn't try to brush this under the carpet, but admitted to it, acted to resolve it and were transparent about it. 2. They worked quickly to make it happen. Would 24h been nice? Sure, but 24h is not much shorter than 120h. In general, 24h is plenty of time for some exploits and 120h doesn't open the window to many more. It would have been very different if it took them months or years to resolve it. 3. They genuinely engaged with the critics on bugzilla, even after Sectigo's CCO went completely off the rails with trying to strip customers off legal recourse and demanding to blacklist those who try to make use of it. 4. They could have taken legal actions against Sectigo's CCO directly but took the extra step to ask them to stop this nonsense. They didn't demand anything more and even outlined steps Sectigo needed to take to prevent any legal problems down the line, like affirming that their CCO did not make these statements on behalf of Sectigo, an affirmation that they would notify their employees to not make any actions that would violate the laws mentioned in their letter, affirm that their CCO would be instructed not to violate any of the laws outlined in their letter and lastly confirm that, upon consulting with their CCO, they were able to conclude that his statements were not meant to harm DigiCert. The only ick is the short timeframe they expect a reply within, but that's sadly usual corporate US law practice... Basically that letter is the result of asking an US law firm for help and telling them to be nice about it and helping their opponent through the process.

deleted 2y ago
[deleted]
SAI_Peregrinus 2y ago
DigiCert miss-issued thousands of certs. Certificate authorities are required by contract (the CA/Browser Forum Baseline Requirements is a contract they agree to when being granted trust as a CA) to revoke miss-issued certificates within 24 hours of the time they learn of the miss-issuance. One of their customers filed a court case & got a temporary restraining order (TRO) preventing DigiCert from revoking ≈70 of those certs within 24 hours. DigiCert then proceeded to delay revocation beyond the 24 hour limit *for all the certs, not just the ≈70 that the TRO covered. That is a violation of the CA/Browser Forum Baseline Requirements. Failure to revoke ≈70 certs because of a court order could be accepted, but failure to revoke thousands of other certs (putting customers at risk) should not be.

mightysashiman 2y ago

Streisand effect 101?

account42 2y ago

[flagged]

kuon 2y ago

What is the actual impact on infrastructure? Is it just some old certificates being valid past expiration?

drpossum 2y ago
I don't think DigiCert has the power to change literally every internal validating mechanism to check if a certificate is expired or not. But you seem to know more, so I'd ask you expound on that.

aaronmdjones 2y ago

Certificate authorities have enormous trust placed upon them by every Internet user (whether they know it or not). Commensurate with this trust, they have enormous responsibilities. As the name implies, the Baseline Requirements are the minimum standard they should achieve. If they can't even do this (being unable or unwilling to revoke issued certificates within the required time-frame), then they do not deserve this trust, and it should be removed. I understand that the TRO prevented them from revoking approximately 70 certificates, and there really is nothing they could have done differently in that case. Their other revocation failings are inexcusable.

csense 2y ago

This is a really stupid situation all around. To issue a cert for a client (say example.com), the CA sends a random number challenge to the client (say 12345678). The client then responds by making a DNS record for _12345678.example.com but due to a bug, DigiCert accepted 12345678.example.com instead. Apparently there's a rule that says any such certs must be considered mis-issued and must be revoked by the issuing CA within 24 hours. The CA talked to some clients who said it would be too much trouble to replace their cert on such short notice. One of those clients filed a lawsuit with the court system, which responded by issuing a restraining order telling the CA not to revoke the cert. First of all, this is a dumb reason to revoke certs. It's difficult to imagine a situation where any of the mis-issued certificates correspond to an actual compromise of somebody's website by a bad actor. But we can perhaps give the benefit of the doubt: I think the intent here is a fail-safe procedure is implemented so revocation is triggered by a broad class of circumstances, which is constructed so that it's easy to check. Second, this is a dumb thing for a website to file a lawsuit about. If you have to replace the cert on your website because your CA made a booboo, the engineer-hours needed to change out your new cert are far less than the lawyer-hours needed to try to work through the court system. Third, it's dumb for people to blame DigiCert for the TRO. The TRO was filed by a service provider called Alegeus. If DigiCert doesn't follow the TRO, they'll be in contempt of court. Theoretically, a DigiCert employee who presses the "Revoke" button at this point could be sent to jail for it! Fourth, a certificate revocation is basically a signed message that says "News Flash: I think this certificate might not correspond to this website." A court preventing the publication of such a message is equivalent to a court preventing a newspaper from publishing an article. This is called "prior restraint" and it has a high bar in the US. Prior restraint analysis was not performed by the court before the TRO was issued. I think the court violated DigiCert's due process rights (by not analyzing whether prior restraint applies) and DigiCert's free speech rights (by not denying the TRO as it would be prior restraint). (That browsers have a hair-trigger response to certain kinds of news flashes and will take drastic action isn't the fault of the publisher of the news.) Fifth, it's obvious there ought to be some technical means to prevent this kind of situation in the future: What is the protocol to handle a case of a CA refusing (or court-ordered not to) revoke certs known to be compromised? I think you need to have a system that allows a quorum of CA's not including the issuing CA to speedily revoke certificates, or an entire CA. (A public real-time quorum of a dynamic set of validators passing messages they want to reach consensus on: Perhaps this is a good use case for a blockchain?) Of course you also have to deal with the problem of, what if the next TRO targets the entire validator set? You'd need to be sure those participants are jurisdictionally diverse (or anonymous) so it would be difficult for a single entity to compromise the entire system at once.

registeredcorn 2y ago

Fascinating. I think there was a fair amount of snark on both sides, but I do think some good points were raised by both, as well. 1) To DigiCert's point: If certs need an emergency revocation but it will impact a service which say: provides life saving services, or keeps the electricity on for the majority of a country, would it not be wise to file them as a one-off "exceptional circumstance". I think that common sense should prevail and everyone can agree that, "Yes, computer security is absolutely essential. Essential services are also essential." I wish that that was the direction the debate had gone in. For instance, What is considered an 'exceptional circumstance'? What kind of services are covered, and what are not? Personally, I would think that things like: health, heat, water, electricity, and physical security (prison and law enforcement) are all potentially essential areas. They are industries that ought be able to request an emergency, 48-hour exception if they know they can't meet it within 24 hours and their services will go down as a result. I feel like two days should be enough time for just about any organization to work through a certificate issue, unless it's a long holiday, or something very, very niche. I think that, to a degree, Tim Callan (Setigo CEO) was being unreasonable in expecting DigiCert to not offer any kind of possibility for exceptions. Some services should not go down, just because it goes against the principals of computer security. It hate saying that, but it's true. Keeping the ICU running matters more than whether the hospital is following best security practices during an emergency. Could it cause more problems by ignoring best practices? Possibly! Will enforcing best practices possibly kill someone? If the answer is anything other than a firm "No", then it is secondary to protecting that service. 2) To Sectigo's point: We should not allow any CA to hide behind Policies or poorly written MSAs. If things went the way they did because they were allowed to go that way, then that means you should learn from those things in the post mortem! Take steps to shore it up! Try and prevent other companies from following suit, otherwise more will take action whenever it meets their own best interest. It is disappointing that this part seems to fell into snarky retorts too, because there were some legitimate means to discuss this. For instance: Instead of barring from someone from being allowed to file a TRO, simply have an agreement in place that before any legal action like a TRO is filed, the customer will meet with the CA and a emergency mediator. Just take 30 minutes to one hour to see if you can work things out before the customer submits a TRO! It seems logical, right? If a customer has the cycles to file for a TRO, they should have the time to spare talking to the company they are filing a TRO against. Explain a clear reasons to a mediator why the TRO is needed, and why they can't get it done in time. Assuming that the customer can explain all of that in clear terms, it would then be obvious for DigiCert to acknowledge that level of criticality and "exceptional need", and offer their customer an emergency, temporary exemption. Neither side wants a TRO! It makes DigiCert look weak during an emergency, and it makes Alegeus (the company that filed the TRO) look incompetent, desperate, and underhanded. The crux of what Tim Callan (Sectigo) was getting at, is that there needs to be a correction to DigiCert's policies. It's blaringly obvious. DigiCert were, in a way, "legally attacked" in a manner that should be prevented in the future, as best they can prevent it. DigiCert lackadaisically shrugging their shoulders and saying "B-But...that goes against Mozilla policy!" is just deflection and meaningless. DigiCert can go to the trouble of sending legal council after Sectigo for comments on Bugzilla, but they can't use legal council to protect DigiCert from surprise TRO's? Really? Bugzilla feedback...that is the legal issue? Not DigiCert being sucker punched by their own customers? The whole thing is just so aggravating. Both sides need to get over themselves and try to work together. They don't need to like each other, but they should do what is best for the industry. Each side sending out daddy lawyer to fight for them completely misses the point, and kills the chance for constructive feedback.

ziddoap 2y ago

Bug has been updated with a DigiCert response. Obviously draw your own conclusions, but I actually laughed out loud at the following statement from DigiCert: "In reality, our letter to you was consistent with our desire to promote open and honest dialogue."