5 ms·

Jeremy Rowley resigns from DigiCert due to mass-revocation incident

130 points by CaliforniaKarl 2y ago

deleted 2y ago
[deleted]
upon_drumhead 2y ago
For those of you who don't know who he is, he was the Chief Information Security Officer https://www.digicert.com/blog/author/jeremy-rowley https://www.digicert.com/blog/author/jeremy-rowley
_3u10 2y ago
Sounds like inside baseball
McGlockenshire 2y ago
This should probably be a direct link to the comment announcing the resignation: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c17 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c17
- hackernewds 2y ago
  > This incident made me realize that I am no longer the person for this role. As such, after consulting with Amit (CEO), we have agreed that the path forward is for me to tender my resignation at the company. I will definitely miss the community, browsers, and public interactions as the PKI ecosystem has been my home and life for such a long time. wow that is an extreme level of ownership. the debate of the resignation and the chilling effect it might lend to the incident report itself is also interesting.
hugneutron 2y ago
I imagine someone as articulate and humble as that guy is going to land on his feet. That was a really good write up.
braiamp 2y ago
I'm with amir in comment 23 and with Aaron in previous comments. Stuff happens. And when there are multiple moving pieces, the process and policies are the issue, not the individuals. Since individuals rarely have a complete overview of the entire system. Also, as noted in the comments, it sets a bad precedent for people coming forward reporting issues.
- foolfoolz 2y ago
  jeremy rowley has been C level at digi cert for over 7 years and is/was CISO. if anyone should have a completely overview of the entire system. it’s him accountability at the top is a good thing
  - Spivak 2y ago
    And this guy is the gold standard for accountability based on these comments. Whoever pressured him to resign I think is making a mistake. Bugs happen, and he's doing exactly what's expected of a leader in this situation. Anyone who thinks "this incident is personally my fault because I didn't read every line of code the devs who work for me wrote, and for this dishonor I am now unfit to lead" is a sane reaction to these events is not living the blameless postmortem life.
    - fizx 2y ago
      Seems like he fell on his own sword.
      - wizzwizz4 2y ago
        It's always the way. The people with honour tend to hold themselves to a high standard, and step back when they find they do not meet it. Their replacements are either the same in this regard, or they're not. Captains used to go down with their ships. In most organisations, this is no longer the case, because we lost all the captains willing to do so, without replacement. Resigning when you fail to prevent an incident rarely helps, directly. But it's not something the power-hungry do, unless forced (and if they expect to be forced, they will try to cover things up). It rarely fails to win my respect. As a move in a social game, I suspect that the general strategy "resign when an unacceptable failure has occurred" makes things better, overall, even if it doesn't directly benefit the organisation you're leaving. (I'm not sure whether this applies when you don't expect that your replacement feels the same way about duty.)
cebert 2y ago
> “The code worked in our original monolithic system but was not implemented properly when we moved to our micro-services systems.” This could happen to anyone, but imagine being the developer or development team that made this mistake.
23B1 2y ago
He resigned with honor, grace and responsibility – and should be applauded. This is what real accountability looks like, and doing so not only preserves the reputation and trustworthiness of his employer, but demonstrates that he is a valuable contributor and trustworthy individual. He will land on his feet as a result.
jtc331 2y ago
I don’t quite follow why a missing underscore results in a security problem. It seems like it must be somehow related to what’s valid for CNAME records?
- aaomidi 2y ago
  For DNS managers, it’s a known thing to restrict the use of domains that start with _. It’s effectively a reserved keyword. In this case, a sophisticated attacker can get certificates for domains they don’t control by abusing this.
  - jtc331 2y ago
    Thanks, that’s helpful information.
fallingsquirrel 2y ago
> We note that other customers have also initiated legal action against us to block revocation. This seems crazy to me. In what world does suing your business partner make more sense than clicking some buttons in a UI or running some shell commands to renew your cert?
- xyst 2y ago
  [flagged]
- pitched 2y ago
  Maybe some mobile apps or IoT hardware with pinned certs that would all need to be rolled first? So they could get a new cert immediately but still need the old one for a while until the roll over completes.
- semiquaver 2y ago
  Here's one of the complaints against digicert: https://storage.courtlistener.com/recap/gov.uscourts.utd.149707/gov.uscourts.utd.149707.1.0.pdf https://storage.courtlistener.com/recap/gov.uscourts.utd.149...
  - kingsloi 2y ago
    > IN THE UNITES STATES DISTRICT COURT IN AND FOR THE DISTRICT OF UTAH, CENTRAL DIVISION Is UNITES > UNITED a typo? Would that cause any weird legal consequences, or are typos (if it is a typo) like this just accepted?
    - phonon 2y ago
      Yes, no, yes.
- stackskipton 2y ago
  Ops here, there is probably plenty of companies who use Digicert for various certs and nothing is automated. Also, in any regulatory strict company, filling out all Change Controls and pushing them through approval process would be take me longer then 24 hours.
Banditoz 2y ago
> We also found that the bug in the code was inadvertently remediated when engineering completed a user-experience enhancement project that collapsed multiple random value generation microservices into a single service. Interesting. What is the value of a microservice that generates random numbers over just using a language's SecureRandom equivalent?
- pitched 2y ago
  An underscore prefix was added in some microservices but required in all. Abstracting it out fixed the code paths requiring an underscore but not adding one.
- magicalhippo 2y ago
  edit: well I got that wrong. The reason seems to be they had multiple different validation methods and wanted to be able to use the same random number across the different methods. Lots of details in the incident report[0]. I'll leave this in case some find the links interesting: For someone like DigiCert I imagine they could ensure a much higher quality source of randomness, ie multiple different hardware sources like the avalanche noise of semiconductor junctions[1], cosmic rays[2] or lava lamps[3]. It's also easier to ensure the random numbers used are of good quality when you have a single source, ie you can collect statistics as they're served. [0]: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c17 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c17 [1]: https://ieeexplore.ieee.org/document/10295491 https://ieeexplore.ieee.org/document/10295491 [2]: https://nyuscholars.nyu.edu/en/publications/muon-ra-quantum-random-number-generation-from-cosmic-rays https://nyuscholars.nyu.edu/en/publications/muon-ra-quantum-... [3]: https://blog.cloudflare.com/lavarand-in-production-the-nitty-gritty-technical-details/ https://blog.cloudflare.com/lavarand-in-production-the-nitty...
xyst 2y ago
I have low expectations from C-level executives. But this incident and his response to it has changed my perspective of them just slightly. It's a rare incident where a C-level executive actually takes accountability for their fuck up. Shit rolls down hill. He is very likely to end up taking the helm at another place or startup on his own. He is the exact opposite of the CrowdStrike CEO (George Kurtz) that caused an absolute shitstorm compared to DigiCert incident.
bryan0 2y ago
I think this comment from the thread sums it up: “When DigiCert has another incident (and while I have tremendous faith in Tim, it will happen), I would rather that they have Jeremy Rowley with his wisdom and scar tissue around to guide their response and subsequent improvement.”
mr_toad 2y ago
> We note that other customers have also initiated legal action against us to block revocation. How can it make economic sense to initiate a lawsuit rather than just get new certificates?
- vincnetas 2y ago
  they want to get payed for the extra work that they will need to do. also there might be some systems that cant be modifief fast enough and they need more time before revocation. of cours they could just ask directly.
- nucleardog 2y ago
  In previous incidents I’ve seen cases of customers who had contracted with their customers to giving them 90 days notice of any changes to certificates. Probably cheaper to file for one TRO than defend yourself against cases from a bunch of your own customers. I mean, cheaper still to not contractually obligate yourself to something you can’t guarantee or perform, but once you’re already in that position…
sneak 2y ago
Given that a revocation is simply a publication of additional data by a CA, and does not directly affect the customer’s systems, how is the TRO in this case not unconstitutional? I’m not a lawyer but it feels like prior restraint, no?
- SpicyLemonZest 2y ago
  It does directly affect the customer’s systems. Revoking the certificates would have broken the customer’s website as an immediate, foreseen, and arguably intentional consequence. I’m skeptical any judge would buy an argument that this is an expressive act. Even if one would, I very much doubt that a company which argued it has a free speech right to break your website with 24 hours notice would survive the ensuing controversy.
- immibis 2y ago
  If only all cyber attacks would be protected as free speech - publication of additional packets.
langsoul-com 2y ago
I question why we accept someone resigning after making a big mistake. Unless it's malice, or the fault truly is entirely on that person, what good would resigning do? Rowley admitted he fucked up, badly, he admitted on several layers what must be changed. How he must change. How the org must change. How the way things are presently is not good enough. Made an extremely deep dive into what happened. And now he's leaving??? Someone who royalty messes up, would not want to mess up on the same issue twice. So all that experience is now worthless and doesn't benefit Digicert in the slightest.
deleted 2y ago
[deleted]
gklitz 2y ago
> The ultimate root cause ended up being me. I have led the compliance team for the past several years. The fact this went unnoticed in our many reviews during that time shows that we need a different approach to both our internal investigations and compliance controls. I also dropped the ball on the certificate problem report by failing to escalate the issue to engineering and give it the proper attention it deserved. Although I did some investigation, I failed to treat the allegations with sufficient seriousness based on what could have been wrong. I assumed I knew the systems and what was happening in them rather than deeply investigating the report. Finally, I didn’t do enough to eliminate the silos between compliance and engineering. Really does sound like he personally dropped the ball in the handling of the report. It would be interesting to hear the story from the researcher who will undoubtably have been frustrated beyond reason that they kept acting like there was no issue despite the repeated persistent attempts at getting them to take it serious.
RevEng 2y ago
While I applaud his openness and willingness to take accountability, I agree with others that resigning shouldn't be necessary. Resigning is what you do when you are clearly not fit for your post. Jeremy has demonstrated that he is anything but unfit. People that can see where things went wrong, who can communicate such, can come up with changes to fix those issues, and can implement them are exactly what is needed at such a high level of management. Most people would bury the story or claim ignorance, but Jeremy doesn't hide anything and takes full responsibility. I wish Jeremy could have stayed and used this honesty and insight to make the necessary changes. Firing a C-level executive when things go wrong doesn't fix anything any more than finding a low level engineer to blame and fire. Experienced people learn lessons by making mistakes. It sucks that it happens, but unexpected circumstances can't be foreseen. Hindsight is 20/20. Now that they know, they know to look out for it and to change the system to prevent it next time. Perhaps he did overlook it. Perhaps he didn't respond when he should have. It's easy to get complacent. This is a wake up call. I have no doubt that he would be much more attentive and responsive as a result of this, and as such, be exactly what's needed for his post. Mistakes don't call for sacrifices; they call for systematic changes to prevent making the same mistakes again. Thank you Jeremy for being as forthcoming as you have been. I only wish more C-level execs would do the same. I hope you find a good place to land where you can take this experience and do an even better job. And I hope that whoever replaces you can bring the same rigor and professionalism that your brought.
amluto 2y ago
One thing I find odd about this: the rules for CAs are long and detailed, but they don’t seem especially complicated. If I were implementing a CA, I would have the main code (their “service oriented architecture” or a monolith or whatever) produce not just an instruction to issue a certificate but a transcript of the entire exchange. Then a completely separate code path (plain old synchronous Rust or Python or Go or Haskell or ML — no microservices) would check the transcript for compliance with each clause of the requirements and block issuance if anything fails. And raise an alert that gets noticed. One could even get fancy and use verifiable randomness for everything in the protocol that is supposed to be random. And then one could refactor some other code with much less worry about messing up. This might also reduce the blast radius from a bug in some other component. If the magic random string generator can be coerced into returning ‘www’, then a separate check would prevent this from compromising everything. (I work in a different industry, and in my industry there is plenty of complex, evolving code, that needs to do the right thing. The more competent players have separate verification code as a double-check.)

deleted 2y ago

[deleted]

For those of you who don't know who he is, he was the Chief Information Security Officer https://www.digicert.com/blog/author/jeremy-rowley https://www.digicert.com/blog/author/jeremy-rowley

_3u10 2y ago

Sounds like inside baseball

McGlockenshire 2y ago

This should probably be a direct link to the comment announcing the resignation: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c17 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c17

hackernewds 2y ago
> This incident made me realize that I am no longer the person for this role. As such, after consulting with Amit (CEO), we have agreed that the path forward is for me to tender my resignation at the company. I will definitely miss the community, browsers, and public interactions as the PKI ecosystem has been my home and life for such a long time. wow that is an extreme level of ownership. the debate of the resignation and the chilling effect it might lend to the incident report itself is also interesting.

hugneutron 2y ago

I imagine someone as articulate and humble as that guy is going to land on his feet. That was a really good write up.

braiamp 2y ago

I'm with amir in comment 23 and with Aaron in previous comments. Stuff happens. And when there are multiple moving pieces, the process and policies are the issue, not the individuals. Since individuals rarely have a complete overview of the entire system. Also, as noted in the comments, it sets a bad precedent for people coming forward reporting issues.

foolfoolz 2y ago
jeremy rowley has been C level at digi cert for over 7 years and is/was CISO. if anyone should have a completely overview of the entire system. it’s him accountability at the top is a good thing
- Spivak 2y ago
  And this guy is the gold standard for accountability based on these comments. Whoever pressured him to resign I think is making a mistake. Bugs happen, and he's doing exactly what's expected of a leader in this situation. Anyone who thinks "this incident is personally my fault because I didn't read every line of code the devs who work for me wrote, and for this dishonor I am now unfit to lead" is a sane reaction to these events is not living the blameless postmortem life.
  - fizx 2y ago
    Seems like he fell on his own sword.
    - wizzwizz4 2y ago
      It's always the way. The people with honour tend to hold themselves to a high standard, and step back when they find they do not meet it. Their replacements are either the same in this regard, or they're not. Captains used to go down with their ships. In most organisations, this is no longer the case, because we lost all the captains willing to do so, without replacement. Resigning when you fail to prevent an incident rarely helps, directly. But it's not something the power-hungry do, unless forced (and if they expect to be forced, they will try to cover things up). It rarely fails to win my respect. As a move in a social game, I suspect that the general strategy "resign when an unacceptable failure has occurred" makes things better, overall, even if it doesn't directly benefit the organisation you're leaving. (I'm not sure whether this applies when you don't expect that your replacement feels the same way about duty.)

cebert 2y ago

> “The code worked in our original monolithic system but was not implemented properly when we moved to our micro-services systems.” This could happen to anyone, but imagine being the developer or development team that made this mistake.

23B1 2y ago

He resigned with honor, grace and responsibility – and should be applauded. This is what real accountability looks like, and doing so not only preserves the reputation and trustworthiness of his employer, but demonstrates that he is a valuable contributor and trustworthy individual. He will land on his feet as a result.

jtc331 2y ago

I don’t quite follow why a missing underscore results in a security problem. It seems like it must be somehow related to what’s valid for CNAME records?

aaomidi 2y ago
For DNS managers, it’s a known thing to restrict the use of domains that start with _. It’s effectively a reserved keyword. In this case, a sophisticated attacker can get certificates for domains they don’t control by abusing this.
- jtc331 2y ago
  Thanks, that’s helpful information.

fallingsquirrel 2y ago

> We note that other customers have also initiated legal action against us to block revocation. This seems crazy to me. In what world does suing your business partner make more sense than clicking some buttons in a UI or running some shell commands to renew your cert?

xyst 2y ago
[flagged]
pitched 2y ago
Maybe some mobile apps or IoT hardware with pinned certs that would all need to be rolled first? So they could get a new cert immediately but still need the old one for a while until the roll over completes.
semiquaver 2y ago
Here's one of the complaints against digicert: https://storage.courtlistener.com/recap/gov.uscourts.utd.149707/gov.uscourts.utd.149707.1.0.pdf https://storage.courtlistener.com/recap/gov.uscourts.utd.149...
- kingsloi 2y ago
  > IN THE UNITES STATES DISTRICT COURT IN AND FOR THE DISTRICT OF UTAH, CENTRAL DIVISION Is UNITES > UNITED a typo? Would that cause any weird legal consequences, or are typos (if it is a typo) like this just accepted?
  - phonon 2y ago
    Yes, no, yes.
stackskipton 2y ago
Ops here, there is probably plenty of companies who use Digicert for various certs and nothing is automated. Also, in any regulatory strict company, filling out all Change Controls and pushing them through approval process would be take me longer then 24 hours.

Banditoz 2y ago

> We also found that the bug in the code was inadvertently remediated when engineering completed a user-experience enhancement project that collapsed multiple random value generation microservices into a single service. Interesting. What is the value of a microservice that generates random numbers over just using a language's SecureRandom equivalent?

pitched 2y ago
An underscore prefix was added in some microservices but required in all. Abstracting it out fixed the code paths requiring an underscore but not adding one.
magicalhippo 2y ago
edit: well I got that wrong. The reason seems to be they had multiple different validation methods and wanted to be able to use the same random number across the different methods. Lots of details in the incident report[0]. I'll leave this in case some find the links interesting: For someone like DigiCert I imagine they could ensure a much higher quality source of randomness, ie multiple different hardware sources like the avalanche noise of semiconductor junctions[1], cosmic rays[2] or lava lamps[3]. It's also easier to ensure the random numbers used are of good quality when you have a single source, ie you can collect statistics as they're served. [0]: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c17 https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c17 [1]: https://ieeexplore.ieee.org/document/10295491 https://ieeexplore.ieee.org/document/10295491 [2]: https://nyuscholars.nyu.edu/en/publications/muon-ra-quantum-random-number-generation-from-cosmic-rays https://nyuscholars.nyu.edu/en/publications/muon-ra-quantum-... [3]: https://blog.cloudflare.com/lavarand-in-production-the-nitty-gritty-technical-details/ https://blog.cloudflare.com/lavarand-in-production-the-nitty...

xyst 2y ago

I have low expectations from C-level executives. But this incident and his response to it has changed my perspective of them just slightly. It's a rare incident where a C-level executive actually takes accountability for their fuck up. Shit rolls down hill. He is very likely to end up taking the helm at another place or startup on his own. He is the exact opposite of the CrowdStrike CEO (George Kurtz) that caused an absolute shitstorm compared to DigiCert incident.

bryan0 2y ago

I think this comment from the thread sums it up: “When DigiCert has another incident (and while I have tremendous faith in Tim, it will happen), I would rather that they have Jeremy Rowley with his wisdom and scar tissue around to guide their response and subsequent improvement.”

mr_toad 2y ago

> We note that other customers have also initiated legal action against us to block revocation. How can it make economic sense to initiate a lawsuit rather than just get new certificates?

vincnetas 2y ago
they want to get payed for the extra work that they will need to do. also there might be some systems that cant be modifief fast enough and they need more time before revocation. of cours they could just ask directly.
nucleardog 2y ago
In previous incidents I’ve seen cases of customers who had contracted with their customers to giving them 90 days notice of any changes to certificates. Probably cheaper to file for one TRO than defend yourself against cases from a bunch of your own customers. I mean, cheaper still to not contractually obligate yourself to something you can’t guarantee or perform, but once you’re already in that position…

sneak 2y ago

Given that a revocation is simply a publication of additional data by a CA, and does not directly affect the customer’s systems, how is the TRO in this case not unconstitutional? I’m not a lawyer but it feels like prior restraint, no?

SpicyLemonZest 2y ago
It does directly affect the customer’s systems. Revoking the certificates would have broken the customer’s website as an immediate, foreseen, and arguably intentional consequence. I’m skeptical any judge would buy an argument that this is an expressive act. Even if one would, I very much doubt that a company which argued it has a free speech right to break your website with 24 hours notice would survive the ensuing controversy.
immibis 2y ago
If only all cyber attacks would be protected as free speech - publication of additional packets.

langsoul-com 2y ago

I question why we accept someone resigning after making a big mistake. Unless it's malice, or the fault truly is entirely on that person, what good would resigning do? Rowley admitted he fucked up, badly, he admitted on several layers what must be changed. How he must change. How the org must change. How the way things are presently is not good enough. Made an extremely deep dive into what happened. And now he's leaving??? Someone who royalty messes up, would not want to mess up on the same issue twice. So all that experience is now worthless and doesn't benefit Digicert in the slightest.

deleted 2y ago

[deleted]

gklitz 2y ago

> The ultimate root cause ended up being me. I have led the compliance team for the past several years. The fact this went unnoticed in our many reviews during that time shows that we need a different approach to both our internal investigations and compliance controls. I also dropped the ball on the certificate problem report by failing to escalate the issue to engineering and give it the proper attention it deserved. Although I did some investigation, I failed to treat the allegations with sufficient seriousness based on what could have been wrong. I assumed I knew the systems and what was happening in them rather than deeply investigating the report. Finally, I didn’t do enough to eliminate the silos between compliance and engineering. Really does sound like he personally dropped the ball in the handling of the report. It would be interesting to hear the story from the researcher who will undoubtably have been frustrated beyond reason that they kept acting like there was no issue despite the repeated persistent attempts at getting them to take it serious.

RevEng 2y ago

While I applaud his openness and willingness to take accountability, I agree with others that resigning shouldn't be necessary. Resigning is what you do when you are clearly not fit for your post. Jeremy has demonstrated that he is anything but unfit. People that can see where things went wrong, who can communicate such, can come up with changes to fix those issues, and can implement them are exactly what is needed at such a high level of management. Most people would bury the story or claim ignorance, but Jeremy doesn't hide anything and takes full responsibility. I wish Jeremy could have stayed and used this honesty and insight to make the necessary changes. Firing a C-level executive when things go wrong doesn't fix anything any more than finding a low level engineer to blame and fire. Experienced people learn lessons by making mistakes. It sucks that it happens, but unexpected circumstances can't be foreseen. Hindsight is 20/20. Now that they know, they know to look out for it and to change the system to prevent it next time. Perhaps he did overlook it. Perhaps he didn't respond when he should have. It's easy to get complacent. This is a wake up call. I have no doubt that he would be much more attentive and responsive as a result of this, and as such, be exactly what's needed for his post. Mistakes don't call for sacrifices; they call for systematic changes to prevent making the same mistakes again. Thank you Jeremy for being as forthcoming as you have been. I only wish more C-level execs would do the same. I hope you find a good place to land where you can take this experience and do an even better job. And I hope that whoever replaces you can bring the same rigor and professionalism that your brought.

amluto 2y ago

One thing I find odd about this: the rules for CAs are long and detailed, but they don’t seem especially complicated. If I were implementing a CA, I would have the main code (their “service oriented architecture” or a monolith or whatever) produce not just an instruction to issue a certificate but a transcript of the entire exchange. Then a completely separate code path (plain old synchronous Rust or Python or Go or Haskell or ML — no microservices) would check the transcript for compliance with each clause of the requirements and block issuance if anything fails. And raise an alert that gets noticed. One could even get fancy and use verifiable randomness for everything in the protocol that is supposed to be random. And then one could refactor some other code with much less worry about messing up. This might also reduce the blast radius from a bug in some other component. If the magic random string generator can be coerced into returning ‘www’, then a separate check would prevent this from compromising everything. (I work in a different industry, and in my industry there is plenty of complex, evolving code, that needs to do the right thing. The more competent players have separate verification code as a double-check.)