6 ms·

In a nutshell: DigiCert has delayed revocation beyond what's allowed in the Baseline Requirements a few times; most recently, https://bugzilla.mozilla.org/show

In a nutshell: DigiCert has delayed revocation beyond what's allowed in the Baseline Requirements a few times; most recently, https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 and https://bugzilla.mozilla.org/show_bug.cgi?id=1910805 https://bugzilla.mozilla.org/show_bug.cgi?id=1910805. In the former case, it seems DigiCert chose to delay revocation to appease certain clients; in the latter case they were prohibited by a Temporary Restraining Order (TRO) from performing timely revocation. Tim Callan from Sectigo has publicly lambasted DigiCert for these delays, since in both cases it seems DigiCert hasn't pushed back hard enough on its clients. In the latter case, there's concern that measures like TROs might be employed more often to stall revocation. Sectigo (and others in the WebPKI ecosystem) seem to want DigiCert to make the revocation policies very clear to clients and to ensure that clients can actually replace their certificates in a timely manner. Sectigo is clearly the most vocal but they don't seem to be the only ones telling DigiCert to get their delayed revocation under control. So the escalation to legal threats is really uncalled for, and DigiCert could face some very significant pushback for trying this tactic.

RHSeeger 2y ago
What is the correct action when a TRO says a company cannot revoke the cert? Is it that the company will delay revocation, but will push the judicial system to resolve the issue as fast as possible?
- nneonneo 2y ago
  DigiCert probably should have revoked every cert they could within 24 hours. Instead they just pushed the revocation of all 80,000+ certs out to five days. It's quite likely that many of their other clients pushed back on the 24-hour timeline (similar to what happened in their previous incident); I believe the delayed revocation issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322) hints at this. The TRO gave them a convenient excuse to delay all revocations without having to explain all over again why they made exemptions for their special clients. Heck, their status page (https://status.digicert.com/incidents/3sccz3v31lc9 https://status.digicert.com/incidents/3sccz3v31lc9) even gives instructions for how to request a delayed revocation - even though the initial incident page (https://www.digicert.com/support/certificate-revocation-incident https://www.digicert.com/support/certificate-revocation-inci...) says clearly: "Any issue with domain validation is considered a serious issue by CABF and requires immediate action. Failure to comply can result in a distrust of the Certificate Authority. As such, we must revoke all impacted certificates within 24 hours of discovery. No extensions or delays are permitted. We apologize if this causes a business disruption to you and are standing by to assist you with validating your domain and issuing replacement certificates immediately."
- Dylan16807 2y ago
  I think the eventual correct option is pretty clear. A TRO like that is based on the company loudly declaring that revoking will cause real damage. That means their use of certificates is incompatible with the web PKI rules and ecosystem. That means they need to be migrated out ASAP, with every certificate authority refusing to take their business. Make that series of consequences clear, and companies will stop trying that trick.
  - _cenw 2y ago
    I think that's pretty unrealistic, considering that X.509 is a de-facto and de-jure standard in a lot of places that also ignore that requirement. It's not always up to that company to make it possible to replace certificates easily, it's an entire chain of vendors (I know at least Salesforce's process would fail here). Unless you want those people to run self-signed/private CA certs. The other option would be building in a way to revoke other CA's individual certificates if there's some consensus on them being compelled to not revoke them. Not sure if the status quo or this would be more dangerous, but if a TRO can compel a CA to not sign a revocation, can it also compel to sign a certificate?
    - nickf 2y ago
      The parent was correct - it's not about the company not using x509 certificates, but not using publicly trusted certificates. There are myriad private/internal PKI solutions available from OpenSSL & bash to millions of dollars of solutions from Big Vendor. If you can't replace the publicly-trusted certificate quickly, you probably don't need it to be publicly-trusted in the first place.
    - jeroenhd 2y ago
      A company chooses its vendors. If its vendors turn out to be incompetent, they should choose again once the damage has been done. Just because Salesforce can't get their stuff together doesn't mean the CA industry needs to bend the knee. Let Salesforce figure out how to automate certificates, they've been in the business for long enough. If running a private CA or self-signed certificates are even viable, then there are plenty of other workarounds that can be put in place (i.e. not updating the CRL so the software doesn't know about revocations). If a CA's terms and legal documentation aren't tight enough to prevent a court from compel them to sign a certificate, that CA clearly cannot be trusted. Hopefully that issue can be fixed by writing clearer terms and better agreements, like people have been telling Digicert to do, but perhaps that's not possible at all. In that case, either the company should move to a jurisdiction where that kind of nonsense isn't possible, or it should be removed from global trust stores all together.
- kevin_thibedeau 2y ago
  How can a court inhibit revocation when every CA declares their right to do so when you purchase a certificate?
  - saurik 2y ago
    I can declare my right to do something as much as I want... that doesn't mean I get to do it, certainly not if a judge decides I can't.
  - AnthonyMouse 2y ago
    The better question is, how do you prevent this tactic from working? For example, suppose there were required to be multiple parties who could issue a revocation, each in a different jurisdiction, and if any of them was ordered not to do it then the others would be required to do it, and would have the technical capacity to do it but not be subject to the jurisdiction of that court.
    - tadfisher 2y ago
      Well, you can contest the motion for a TRO, for a start. Digicert failed to do so. You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so. You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so.
      - tkfu 2y ago
        Correction, the petition for the TRO was filed ex parte. Digicert did not have any opportunity to respond before it was granted. They certainly could have filed a response contesting the TRO. Then their customer could have filed another motion, and eventually (7 days later in this case) the judge would have ruled on the substance of it. Their judgement was that it would be preferable to work with the customer to resolve the technical issues with revocation, and submit a joint request to dismiss the TRO. The stated reasoning behind this was that it would be significantly faster than contesting the TRO. This is true: the certs were revoked and the TRO dropped within 3 days. I think the communication on that point was severely lacking, as they only clarified it three months later and after significant hectoring in two different bug threads: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43 https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43 I also think it's reasonable not to take Digicert's statements at face value, given their history. But I think both of the points you made here are wrong: > You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so. Let's be clear about the timeline: Digicert notified their customers that the certs would be revoked. In between the time they notified the customer and the time of revocation (less than 24 hours), the customer got the ex parte restraining order. Are you suggesting that issuers should revoke certificates without notifying their users, so that the users don't have time to get an emergency TRO? I believe that would be in violation of the BRs. > You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so. By "revoke the cert in face of the legal consequences" do you mean "openly defy a valid and legal court order"? Because that would also violate the BRs.
- deleted 2y ago
  [deleted]
- rlpb 2y ago
  The order would normally be a matter of public record, and the ecosystem should follow these events carefully and ensure never to issue a certificate to such a company again as it undermines the entire system.
_cenw 2y ago
Isn't Sectigo Comodo? Extra rich coming from them.
- nickf 2y ago
  No, Sectigo is the PKI business that was carved out of Comodo, back in 2017. Comodo still exists, doing whatever they do. They have been totally separate entities since then.
jeroenhd 2y ago
Reads to me like Digicert is hiding behind the TRO to protect themselves from upset customers over following the procedures they should have followed. I don't think they want to update their legal work to prevent customers from legal action over revocation, legal action has been working in their favor so far. For a company whose entire business is verifying company names and handling the CA process, Digicert seems rather unwilling to actually stick to the process. They can't help having a TRO filed against them by customers with incompetent tech such as Alegeus Technologies LLC, but this isn't the first time they've failed to follow the appropriate processes. Going through the courts to stop negative discourse seems rather crass for a CA. I don't understand why a company that has been subject of suspicion and doubt lime Digicert would do such a thing unless it's a last-ditch effort to get the heat off their backs. I'm sure their clients love Digicert for not forcing them to replace their certificates at the appropriate times, but they're in for a surprise when this whole thing blows up and they suddenly need to find another cert vendor when Digicert gets delisted.
- michaelt 2y ago
  I dunno, seems to me in the TRO case there wasn't anything wrong with the certificates that had been issued. There wasn't any suggestion the the certificates had been issued to the wrong party or anything like that. When performing DNS verification there's supposed to be an underscore in a record, which protects sites that let their users control arbitrary subdomains - you know, dyndns and things like that. Digicert mistakenly didn't ask the customer to include an underscore. Could have been a problem, if Alegeus was a dyndns provider - but they aren't. Then Digicert spent a load of the 24 hour disclosure window checking things on their end, then e-mailed the customers after office hours so when they get in the next day they barely had any time to respond. The only "incompetent tech" here is the assholes who decided on the policy that certificates be urgently revoked, as if they'd been issued to the wrong person, when they hadn't been issued to the wrong customer.
  - Thorrez 2y ago
    >Could have been a problem, if Alegeus was a dyndns provider - but they aren't. AFAIUI, Alegeus was just 1 customer of Digicert. But Digicert has tons of customers. Just because there wasn't a problem for Alegeus (due to Alegeus not being a dyndns provider), doesn't mean there weren't problems for other Digicert customers.
e40 2y ago
Here is the last reply from Callan: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c73 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c73 Seems extremely reasonable to me.

RHSeeger 2y ago

What is the correct action when a TRO says a company cannot revoke the cert? Is it that the company will delay revocation, but will push the judicial system to resolve the issue as fast as possible?

nneonneo 2y ago
DigiCert probably should have revoked every cert they could within 24 hours. Instead they just pushed the revocation of all 80,000+ certs out to five days. It's quite likely that many of their other clients pushed back on the 24-hour timeline (similar to what happened in their previous incident); I believe the delayed revocation issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322) hints at this. The TRO gave them a convenient excuse to delay all revocations without having to explain all over again why they made exemptions for their special clients. Heck, their status page (https://status.digicert.com/incidents/3sccz3v31lc9 https://status.digicert.com/incidents/3sccz3v31lc9) even gives instructions for how to request a delayed revocation - even though the initial incident page (https://www.digicert.com/support/certificate-revocation-incident https://www.digicert.com/support/certificate-revocation-inci...) says clearly: "Any issue with domain validation is considered a serious issue by CABF and requires immediate action. Failure to comply can result in a distrust of the Certificate Authority. As such, we must revoke all impacted certificates within 24 hours of discovery. No extensions or delays are permitted. We apologize if this causes a business disruption to you and are standing by to assist you with validating your domain and issuing replacement certificates immediately."
Dylan16807 2y ago
I think the eventual correct option is pretty clear. A TRO like that is based on the company loudly declaring that revoking will cause real damage. That means their use of certificates is incompatible with the web PKI rules and ecosystem. That means they need to be migrated out ASAP, with every certificate authority refusing to take their business. Make that series of consequences clear, and companies will stop trying that trick.
- _cenw 2y ago
  I think that's pretty unrealistic, considering that X.509 is a de-facto and de-jure standard in a lot of places that also ignore that requirement. It's not always up to that company to make it possible to replace certificates easily, it's an entire chain of vendors (I know at least Salesforce's process would fail here). Unless you want those people to run self-signed/private CA certs. The other option would be building in a way to revoke other CA's individual certificates if there's some consensus on them being compelled to not revoke them. Not sure if the status quo or this would be more dangerous, but if a TRO can compel a CA to not sign a revocation, can it also compel to sign a certificate?
  - nickf 2y ago
    The parent was correct - it's not about the company not using x509 certificates, but not using publicly trusted certificates. There are myriad private/internal PKI solutions available from OpenSSL & bash to millions of dollars of solutions from Big Vendor. If you can't replace the publicly-trusted certificate quickly, you probably don't need it to be publicly-trusted in the first place.
  - jeroenhd 2y ago
    A company chooses its vendors. If its vendors turn out to be incompetent, they should choose again once the damage has been done. Just because Salesforce can't get their stuff together doesn't mean the CA industry needs to bend the knee. Let Salesforce figure out how to automate certificates, they've been in the business for long enough. If running a private CA or self-signed certificates are even viable, then there are plenty of other workarounds that can be put in place (i.e. not updating the CRL so the software doesn't know about revocations). If a CA's terms and legal documentation aren't tight enough to prevent a court from compel them to sign a certificate, that CA clearly cannot be trusted. Hopefully that issue can be fixed by writing clearer terms and better agreements, like people have been telling Digicert to do, but perhaps that's not possible at all. In that case, either the company should move to a jurisdiction where that kind of nonsense isn't possible, or it should be removed from global trust stores all together.
kevin_thibedeau 2y ago
How can a court inhibit revocation when every CA declares their right to do so when you purchase a certificate?
- saurik 2y ago
  I can declare my right to do something as much as I want... that doesn't mean I get to do it, certainly not if a judge decides I can't.
- AnthonyMouse 2y ago
  The better question is, how do you prevent this tactic from working? For example, suppose there were required to be multiple parties who could issue a revocation, each in a different jurisdiction, and if any of them was ordered not to do it then the others would be required to do it, and would have the technical capacity to do it but not be subject to the jurisdiction of that court.
  - tadfisher 2y ago
    Well, you can contest the motion for a TRO, for a start. Digicert failed to do so. You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so. You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so.
    - tkfu 2y ago
      Correction, the petition for the TRO was filed ex parte. Digicert did not have any opportunity to respond before it was granted. They certainly could have filed a response contesting the TRO. Then their customer could have filed another motion, and eventually (7 days later in this case) the judge would have ruled on the substance of it. Their judgement was that it would be preferable to work with the customer to resolve the technical issues with revocation, and submit a joint request to dismiss the TRO. The stated reasoning behind this was that it would be significantly faster than contesting the TRO. This is true: the certs were revoked and the TRO dropped within 3 days. I think the communication on that point was severely lacking, as they only clarified it three months later and after significant hectoring in two different bug threads: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43 https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43 I also think it's reasonable not to take Digicert's statements at face value, given their history. But I think both of the points you made here are wrong: > You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so. Let's be clear about the timeline: Digicert notified their customers that the certs would be revoked. In between the time they notified the customer and the time of revocation (less than 24 hours), the customer got the ex parte restraining order. Are you suggesting that issuers should revoke certificates without notifying their users, so that the users don't have time to get an emergency TRO? I believe that would be in violation of the BRs. > You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so. By "revoke the cert in face of the legal consequences" do you mean "openly defy a valid and legal court order"? Because that would also violate the BRs.
deleted 2y ago
[deleted]
rlpb 2y ago
The order would normally be a matter of public record, and the ecosystem should follow these events carefully and ensure never to issue a certificate to such a company again as it undermines the entire system.

_cenw 2y ago

Isn't Sectigo Comodo? Extra rich coming from them.

nickf 2y ago
No, Sectigo is the PKI business that was carved out of Comodo, back in 2017. Comodo still exists, doing whatever they do. They have been totally separate entities since then.

jeroenhd 2y ago

Reads to me like Digicert is hiding behind the TRO to protect themselves from upset customers over following the procedures they should have followed. I don't think they want to update their legal work to prevent customers from legal action over revocation, legal action has been working in their favor so far. For a company whose entire business is verifying company names and handling the CA process, Digicert seems rather unwilling to actually stick to the process. They can't help having a TRO filed against them by customers with incompetent tech such as Alegeus Technologies LLC, but this isn't the first time they've failed to follow the appropriate processes. Going through the courts to stop negative discourse seems rather crass for a CA. I don't understand why a company that has been subject of suspicion and doubt lime Digicert would do such a thing unless it's a last-ditch effort to get the heat off their backs. I'm sure their clients love Digicert for not forcing them to replace their certificates at the appropriate times, but they're in for a surprise when this whole thing blows up and they suddenly need to find another cert vendor when Digicert gets delisted.

michaelt 2y ago
I dunno, seems to me in the TRO case there wasn't anything wrong with the certificates that had been issued. There wasn't any suggestion the the certificates had been issued to the wrong party or anything like that. When performing DNS verification there's supposed to be an underscore in a record, which protects sites that let their users control arbitrary subdomains - you know, dyndns and things like that. Digicert mistakenly didn't ask the customer to include an underscore. Could have been a problem, if Alegeus was a dyndns provider - but they aren't. Then Digicert spent a load of the 24 hour disclosure window checking things on their end, then e-mailed the customers after office hours so when they get in the next day they barely had any time to respond. The only "incompetent tech" here is the assholes who decided on the policy that certificates be urgently revoked, as if they'd been issued to the wrong person, when they hadn't been issued to the wrong customer.
- Thorrez 2y ago
  >Could have been a problem, if Alegeus was a dyndns provider - but they aren't. AFAIUI, Alegeus was just 1 customer of Digicert. But Digicert has tons of customers. Just because there wasn't a problem for Alegeus (due to Alegeus not being a dyndns provider), doesn't mean there weren't problems for other Digicert customers.

e40 2y ago

Here is the last reply from Callan: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c73 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c73 Seems extremely reasonable to me.