7 ms·

Comodo is communicating actively in the bug, in the email discussion, and proactively CC'd the original reporter on the bug that was filed about this without be

Comodo is communicating actively in the bug, in the email discussion, and proactively CC'd the original reporter on the bug that was filed about this without being asked to do so. The general rule in Operations work is tolerance. You don't fire someone for making a mistake, you fire them for lying about the mistake, for refusing to avoid mistake-prone behaviors, and other problem of that sort. Applying that same principle to CA operations, why on earth should Comodo be "fired" as a CA for this? They're actively working to solve it, they're responsive, and the agreements they operate under will compel a post-mortem into existence if only to meet the terms of their next CA audit. If, after investigation, they are found to be willfully violating policy, knowingly lying about CAA support, fail to produce an acceptable post-mortem, and/or fail their next CA audit, then of course they should be at risk of direct sanctions - not just a verbal reprimand. So if your initial response to this issue is "Comodo should be fired", remember that in Operations culture that specific response "X should be fired" is a harmful response to a zero-day report of an incident, and many Operations teams would fire anyone who called for a firing in response to a fresh incident, in order to protect others from that harmful posture.

tptacek 9y ago
I would bet money (maybe not a lot of money, but money) that the only outcome of this is that Comodo ends up within a month being the CA that most reliably checks CAA records. It would be shocking if Google penalized them for this. As you say, that's based in part on how they handle it.
- floatingatoll 9y ago
  The thing I'm sad to see is that there's no third-party information stated on when Comodo's CAA verification started and stopped. When they first announced CAA support, who tested it? When did it stop working? So I'm hoping this will come out of the post-mortem and be shared with us all.
  - agwa 9y ago
    Comodo is required by Mozilla to publish a public incident report: https://wiki.mozilla.org/CA/Responding_To_A_Misissuance https://wiki.mozilla.org/CA/Responding_To_A_Misissuance
  - jlgaddis 9y ago
    CAA support was only required starting a few days ago, IIRC.
- dopamean 9y ago
  Let's Encrypt is pretty good about checking CAA records. At work I built a system that allows our customers to order certs from LE and we see lots of failures for CAA query timeout (customer using a DNS provider that doesn't support CAA queries).
  - tptacek 9y ago
    I'm not saying you should use Comodo. I'd use LE too.
gregmac 9y ago
This reminds me of a parable: An employee at a big company makes a mistake that costs the company $500,000. The client is livid, and demands he be fired. The owner's response? "Why would I fire him? I just spent half a million dollars on training!"
- brandonwamboldt 9y ago
  This tends to be over-used though. Unfortunately some employees would continue making $500,000 mistakes and never learning. It really depends on the employee's attitude and response to the incident, as well as the circumstances of the incident.
  - floatingatoll 9y ago
    It depends also on the employer's attitude and response to the incident. If the employee behaves perfectly and writes an impeccable post-mortem and is then fired "at the request of a client", that's on the employer, not the employee.
mdip 9y ago
Kudos on this. Coming off of the Symantec thing, I think we're all viewing businesses in the CA space a lot more skeptically. I worked in a hybrid Infrastructure / Development job for almost 17 years. One of the reasons I could never shake the infrastructure part of things[0] was that I knew the environment from more angles than most of the staff and I tended to be key in high severity issues. We were a highly geographically distributed team and coordinated almost all activity via conference calls. We had two major rules when a critical failure was discovered. The first was that anyone asking "who caused this" was admonished and often ejected from the call. Often, the person responsible for the failure was on the calls and everyone had a pretty good understanding of who was at fault (if it was, indeed, a self-inflicted wound). The person at fault was also usually critical to the fix and putting them on the defensive caused evasiveness out of self-preservation. To help the first rule, the second rule was barring of anyone at the Director level or higher on these calls[1]. Generally, they were concerned with two things: (1) When's it going to work again and (2) why did it fail in the first place -- both result in delays. They were given updates, separately, by someone from the call at regular intervals. Keeping the politics low and focusing on the problem meant much shorter outages and calmer, more focused, brains of our technical staff. And to your point, despite working at a company that laid off 10% or so of it's staff about every 9 months, I can't think of a case where a member of technical staff was specifically let go because of an (infrequent) failure -- even situations that would fall under gross negligence of the technician were forgiven provided they were one-off. People get distracted and fail to follow testing procedures/forget to read architecture documentation and make mistakes[2]. After the situation is resolved, a post-mortem/"root cause analysis" was always performed. It was often done via documents rather than meetings, to avoid the confrontation associated with a bunch of managers pointing fingers, and we always learned from our mistakes and got better as a result. [0] I was a developer of infrastructure services, doing similar work that a DevOps might do, today but more focused on providing internal services and automating support. I was a full-time developer for 3/4 of that time with regular interruptions of sev0 incident response. [1] This didn't always happen, but I can count on one hand the number of times when a director/VP/C-Level joined and actually made his/her presence known. [2] I'll never forget the time early in my career when I was on-call and had to restore our most critical/largest file server that was destroyed as a result of a technician getting frustrated with a rack and dislodging a 60(?)-pin SCSI cable...partly...where it attempted to run for several hours in that condition. The best part was the maintenance he was doing was on the backup server that had required his attention for a few weeks and hadn't functioned since then. When I saw the condition of the cables, I was so pissed off that while restoring the weeks' old backups at 4:00 AM, I had the operators in that room show me how to review the (VHS) security tapes. Watching him kick the door several times to get it to latch, then peering in -- directly at the bent-up cable -- just before walking out the door was the best part. He should have been fired for that, and eventually a similar behavior had that result.

tptacek 9y ago

I would bet money (maybe not a lot of money, but money) that the only outcome of this is that Comodo ends up within a month being the CA that most reliably checks CAA records. It would be shocking if Google penalized them for this. As you say, that's based in part on how they handle it.

floatingatoll 9y ago
The thing I'm sad to see is that there's no third-party information stated on when Comodo's CAA verification started and stopped. When they first announced CAA support, who tested it? When did it stop working? So I'm hoping this will come out of the post-mortem and be shared with us all.
- agwa 9y ago
  Comodo is required by Mozilla to publish a public incident report: https://wiki.mozilla.org/CA/Responding_To_A_Misissuance https://wiki.mozilla.org/CA/Responding_To_A_Misissuance
- jlgaddis 9y ago
  CAA support was only required starting a few days ago, IIRC.
dopamean 9y ago
Let's Encrypt is pretty good about checking CAA records. At work I built a system that allows our customers to order certs from LE and we see lots of failures for CAA query timeout (customer using a DNS provider that doesn't support CAA queries).
- tptacek 9y ago
  I'm not saying you should use Comodo. I'd use LE too.

gregmac 9y ago

This reminds me of a parable: An employee at a big company makes a mistake that costs the company $500,000. The client is livid, and demands he be fired. The owner's response? "Why would I fire him? I just spent half a million dollars on training!"

brandonwamboldt 9y ago
This tends to be over-used though. Unfortunately some employees would continue making $500,000 mistakes and never learning. It really depends on the employee's attitude and response to the incident, as well as the circumstances of the incident.
- floatingatoll 9y ago
  It depends also on the employer's attitude and response to the incident. If the employee behaves perfectly and writes an impeccable post-mortem and is then fired "at the request of a client", that's on the employer, not the employee.

mdip 9y ago

Kudos on this. Coming off of the Symantec thing, I think we're all viewing businesses in the CA space a lot more skeptically. I worked in a hybrid Infrastructure / Development job for almost 17 years. One of the reasons I could never shake the infrastructure part of things[0] was that I knew the environment from more angles than most of the staff and I tended to be key in high severity issues. We were a highly geographically distributed team and coordinated almost all activity via conference calls. We had two major rules when a critical failure was discovered. The first was that anyone asking "who caused this" was admonished and often ejected from the call. Often, the person responsible for the failure was on the calls and everyone had a pretty good understanding of who was at fault (if it was, indeed, a self-inflicted wound). The person at fault was also usually critical to the fix and putting them on the defensive caused evasiveness out of self-preservation. To help the first rule, the second rule was barring of anyone at the Director level or higher on these calls[1]. Generally, they were concerned with two things: (1) When's it going to work again and (2) why did it fail in the first place -- both result in delays. They were given updates, separately, by someone from the call at regular intervals. Keeping the politics low and focusing on the problem meant much shorter outages and calmer, more focused, brains of our technical staff. And to your point, despite working at a company that laid off 10% or so of it's staff about every 9 months, I can't think of a case where a member of technical staff was specifically let go because of an (infrequent) failure -- even situations that would fall under gross negligence of the technician were forgiven provided they were one-off. People get distracted and fail to follow testing procedures/forget to read architecture documentation and make mistakes[2]. After the situation is resolved, a post-mortem/"root cause analysis" was always performed. It was often done via documents rather than meetings, to avoid the confrontation associated with a bunch of managers pointing fingers, and we always learned from our mistakes and got better as a result. [0] I was a developer of infrastructure services, doing similar work that a DevOps might do, today but more focused on providing internal services and automating support. I was a full-time developer for 3/4 of that time with regular interruptions of sev0 incident response. [1] This didn't always happen, but I can count on one hand the number of times when a director/VP/C-Level joined and actually made his/her presence known. [2] I'll never forget the time early in my career when I was on-call and had to restore our most critical/largest file server that was destroyed as a result of a technician getting frustrated with a rack and dislodging a 60(?)-pin SCSI cable...partly...where it attempted to run for several hours in that condition. The best part was the maintenance he was doing was on the backup server that had required his attention for a few weeks and hadn't functioned since then. When I saw the condition of the cables, I was so pissed off that while restoring the weeks' old backups at 4:00 AM, I had the operators in that room show me how to review the (VHS) security tapes. Watching him kick the door several times to get it to latch, then peering in -- directly at the bent-up cable -- just before walking out the door was the best part. He should have been fired for that, and eventually a similar behavior had that result.