3 ms·

> Also why it took 6 hrs to assign P1 to the bug Because people were staying up until the wee hours of the morning working on fixing it instead of toggling pri

> Also why it took 6 hrs to assign P1 to the bug Because people were staying up until the wee hours of the morning working on fixing it instead of toggling priorities in Bugzilla. This was treated as a five-alarm fire.

pbhjpbhj 8y ago
>This was treated as a five-alarm fire. I don't think it bothers me personally but it's funny you said that. Presumably you mean a "'no-alarm fire' because who has time to set off an alarm when there's a fire to fight"?!
- rhizome 8y ago
  Read the room, Sheldon.
- lugg 8y ago
  > One-alarm, two-alarm, three-alarm fires, etc., are categories of fires indicating the level of response by local authorities. The term multiple-alarm is a quick way of indicating that a fire is severe and is difficult to contain. https://en.wikipedia.org/wiki/Multiple-alarm_fire https://en.wikipedia.org/wiki/Multiple-alarm_fire The 5th level or a five-alarm fire, is the top level where you're basically talking all hands on deck.
  - pbhjpbhj 8y ago
    In the UK they actually sound/flash alarms at locations and in the fire-station, do they not do that in USA? They do in the movies. Still seems like an ironic choice, applies equally to the people saying it was DEFCON1. I'm pretty sure the military actually have displays indicating the status, but again that's based mainly on movies.
SlowRobotAhead 8y ago
I agree with you, it was more important to do the work than to signal. However, I bet it’s likely they have procedures and policies for work that first involve signaling like for example the priority level. I’d be willing to bet lots of things surrounding this issue weren’t handled in a by the book manner. So if you are always going to wing it, why have a book (or a public priority level system) at all?
- lugg 8y ago
  You have the book because at one time you didn't have the book. "The book" is something that needs to change and improve like anything else. Sometimes, that means you'll still need to wing it and add that thing to the book later.
- pcwalton 8y ago
  First, because priority is for things like major feature work, so that engineers can find the bugs that are useful to work on. In this case, everyone in the team responsible was already spending 100% of their time addressing the issue. Second, because we care about solving problems, not being bureaucrats.
  - yawaramin 8y ago
    Really? Because being the bottleneck (i.e. single point of failure) responsible for approving all addons is exactly what bureaucrats would want to do ;-) The non-bureaucratic thing to do, as has been pointed out many times of course, would be to give users the power to override the cert signing check as an advanced option.
    - pcwalton 8y ago
      It is, in fact, an "advanced option", in about:config.
      - yawaramin 8y ago
        But that option only works in nightly builds, not Firefox release builds. If an option doesn't honour what it claims to, imho it might as well not be there.
DannyBee 8y ago
The normal practice would be to have someone operating as an incident communication manager who would be taking care of status/things like this. Saying "we were too busy fixing to communicate" is actually a really bad sign, because it's not just about what you are communicating to the outside world, but also, for example, about making sure people that need to be brought in are getting consistent information.
- pcwalton 8y ago
  This is an incredibly uncharitable misreading of what I said. We do have incident communication managers who were working around the clock, of course, communicating through the official channels. The question is about why someone didn't set the priority setting in the internal Bugzilla sooner. There are a couple of reasons for this. First, that's not really what priority is for: priority is so that engineers on major projects like WebRender know what is most important to work on. It's not an effective tool for emergency responses. P1 doesn't summon on-call people. Second, everyone who was able to get it fixed was already on it. Bugzilla priorities don't make people who wouldn't otherwise be aware aware. Unlike that of Mozilla, your issue tracker at Google for outages is private and internal-only. One of the many reasons it is private is so that people who don't know the organization's operational processes don't come in and start making incorrect assumptions based on what they find there. I think it's laudable that Mozilla works in the open so much, but comments like this one are some of the downsides of doing so.
  - DannyBee 8y ago
    "This is an incredibly uncharitable misreading of what I said" I'm sorry if i offended you. Really. I don't really think it's that uncharitable, but hey, offending you wasn't my intent, and if you were, that is what matters. For what it is worth: your response to the parent took them to task for even attempting to state something: "Because people were staying up until the wee hours of the morning working on fixing it instead of toggling priorities in Bugzilla. This was treated as a five-alarm fire." This is a pretty rude response (which should never be happening regardless of what they wrote), and if you intended to convey that you had incident managers handling it and they didn't get around to it yet, you did not. "Unlike that of Mozilla, your issue tracker at Google for outages is private and internal-only. " I'm really unsure why you decided to add a completely irrelevant and unnecessary attack like this. Honestly, it just makes me think less of you and makes me sad. You are usually a very sane and even keeled person. I'm sure you were not having a good time, but i still don't think this was okay. " One of the many reasons it is private is so that people who don't know the organization's operational processes don't come in and start making incorrect assumptions based on what they find there." Which is incredibly ironic, since it is not in fact as private as you seem to believe . Your own assumption here is completely and totally incorrect. Maybe before you decide to add completely irrelevant and unnecessary attacks, at least verify they are correct? " I think it's laudable that Mozilla works in the open so much, but comments like this one are some of the downsides of doing so." You seem to have taken my comment incredibly personally, and your response seems very far out of proportion. If you want to have a discussion, you're gonna have to tone this down a few notches. FWIW: Right now most of your comments in this story read incredibly defensive (and i'm not just talking about this one) I would stop and give it a rest. It is not projecting a good image. Of course, that's just my perspective. Really hope tomorrow is a better day for you.

pbhjpbhj 8y ago

>This was treated as a five-alarm fire. I don't think it bothers me personally but it's funny you said that. Presumably you mean a "'no-alarm fire' because who has time to set off an alarm when there's a fire to fight"?!

rhizome 8y ago
Read the room, Sheldon.
lugg 8y ago
> One-alarm, two-alarm, three-alarm fires, etc., are categories of fires indicating the level of response by local authorities. The term multiple-alarm is a quick way of indicating that a fire is severe and is difficult to contain. https://en.wikipedia.org/wiki/Multiple-alarm_fire https://en.wikipedia.org/wiki/Multiple-alarm_fire The 5th level or a five-alarm fire, is the top level where you're basically talking all hands on deck.
- pbhjpbhj 8y ago
  In the UK they actually sound/flash alarms at locations and in the fire-station, do they not do that in USA? They do in the movies. Still seems like an ironic choice, applies equally to the people saying it was DEFCON1. I'm pretty sure the military actually have displays indicating the status, but again that's based mainly on movies.

SlowRobotAhead 8y ago

I agree with you, it was more important to do the work than to signal. However, I bet it’s likely they have procedures and policies for work that first involve signaling like for example the priority level. I’d be willing to bet lots of things surrounding this issue weren’t handled in a by the book manner. So if you are always going to wing it, why have a book (or a public priority level system) at all?

lugg 8y ago
You have the book because at one time you didn't have the book. "The book" is something that needs to change and improve like anything else. Sometimes, that means you'll still need to wing it and add that thing to the book later.
pcwalton 8y ago
First, because priority is for things like major feature work, so that engineers can find the bugs that are useful to work on. In this case, everyone in the team responsible was already spending 100% of their time addressing the issue. Second, because we care about solving problems, not being bureaucrats.
- yawaramin 8y ago
  Really? Because being the bottleneck (i.e. single point of failure) responsible for approving all addons is exactly what bureaucrats would want to do ;-) The non-bureaucratic thing to do, as has been pointed out many times of course, would be to give users the power to override the cert signing check as an advanced option.
  - pcwalton 8y ago
    It is, in fact, an "advanced option", in about:config.
    - yawaramin 8y ago
      But that option only works in nightly builds, not Firefox release builds. If an option doesn't honour what it claims to, imho it might as well not be there.

DannyBee 8y ago

The normal practice would be to have someone operating as an incident communication manager who would be taking care of status/things like this. Saying "we were too busy fixing to communicate" is actually a really bad sign, because it's not just about what you are communicating to the outside world, but also, for example, about making sure people that need to be brought in are getting consistent information.

pcwalton 8y ago
This is an incredibly uncharitable misreading of what I said. We do have incident communication managers who were working around the clock, of course, communicating through the official channels. The question is about why someone didn't set the priority setting in the internal Bugzilla sooner. There are a couple of reasons for this. First, that's not really what priority is for: priority is so that engineers on major projects like WebRender know what is most important to work on. It's not an effective tool for emergency responses. P1 doesn't summon on-call people. Second, everyone who was able to get it fixed was already on it. Bugzilla priorities don't make people who wouldn't otherwise be aware aware. Unlike that of Mozilla, your issue tracker at Google for outages is private and internal-only. One of the many reasons it is private is so that people who don't know the organization's operational processes don't come in and start making incorrect assumptions based on what they find there. I think it's laudable that Mozilla works in the open so much, but comments like this one are some of the downsides of doing so.
- DannyBee 8y ago
  "This is an incredibly uncharitable misreading of what I said" I'm sorry if i offended you. Really. I don't really think it's that uncharitable, but hey, offending you wasn't my intent, and if you were, that is what matters. For what it is worth: your response to the parent took them to task for even attempting to state something: "Because people were staying up until the wee hours of the morning working on fixing it instead of toggling priorities in Bugzilla. This was treated as a five-alarm fire." This is a pretty rude response (which should never be happening regardless of what they wrote), and if you intended to convey that you had incident managers handling it and they didn't get around to it yet, you did not. "Unlike that of Mozilla, your issue tracker at Google for outages is private and internal-only. " I'm really unsure why you decided to add a completely irrelevant and unnecessary attack like this. Honestly, it just makes me think less of you and makes me sad. You are usually a very sane and even keeled person. I'm sure you were not having a good time, but i still don't think this was okay. " One of the many reasons it is private is so that people who don't know the organization's operational processes don't come in and start making incorrect assumptions based on what they find there." Which is incredibly ironic, since it is not in fact as private as you seem to believe . Your own assumption here is completely and totally incorrect. Maybe before you decide to add completely irrelevant and unnecessary attacks, at least verify they are correct? " I think it's laudable that Mozilla works in the open so much, but comments like this one are some of the downsides of doing so." You seem to have taken my comment incredibly personally, and your response seems very far out of proportion. If you want to have a discussion, you're gonna have to tone this down a few notches. FWIW: Right now most of your comments in this story read incredibly defensive (and i'm not just talking about this one) I would stop and give it a rest. It is not projecting a good image. Of course, that's just my perspective. Really hope tomorrow is a better day for you.