6 ms·

I'm interested in the general writeup what went wrong that they missed this certificate expiring. That's a structural problem. Also why it took 6 hrs to assign

I'm interested in the general writeup what went wrong that they missed this certificate expiring. That's a structural problem. Also why it took 6 hrs to assign P1 to the bug

gpm 7y ago
They closed the trees (stopped merging other code changes to prioritize this) for the bug <22 minutes after it was opened. I would assume the delay in assigning P1 is really just a result of assigning P1 not being as high priority as fixing the damn problem.
- JamesMcGill 7y ago
  This is one of the things about this whole episode that I find baffling. Stuff like adjusting bug priorities and arranging for someone to tweet an announcement is the work of a good engineering manager. This is the right person to run interference and handle comms and deal with things outside of the critical path, like bugzilla updates.
  - dao- 7y ago
    The priority field just plays no practical role here since the bug was immediately escalated.
    - pbhjpbhj 7y ago
      If people can see it externally it definitely has an effect, "oh moz aren't taking this browser breaking bug seriously". Which is probably why the parent said it's a management issue rather than a directly technical one, per se.
  - rhizome 7y ago
    Stuff like adjusting bug priorities and arranging for someone to tweet an announcement is the work of a good engineering manager So we can come to the obvious conclusion about Mozilla, then? No good "engineering" managers? Miss one reprioritization and you're out! This is what sane people think?
    - Twirrim 7y ago
      At no stage did the parent post state there are no good engineering managers at Mozilla. They just said that adjusting bug priorities is the work of a good engineering manager. There's a world of difference. If you want to complain about knee-jerk overreactions, I think you might want to look in the mirror first.
      - rhizome 7y ago
        I have empathy for people in the middle of hair-on-fire incidents.
- makomk 7y ago
  If I understand the bug report comments correctly, they didn't close the trees to other code changes to prioritize fixing this, they did it because the cert expiry broke some important tests at the same time as it broke every end user's browser.
deleted 7y ago
[deleted]
bhauer 7y ago
I'm also interested in why existing adds-ons are failing to run due to this problem. (There was a similar question in another thread about the issue here at HN.) I understand why an add-on update or new installation would be prevented from succeeding by a certificate expiration. But why would a certificate expiration prevent an already-installed from running? Any already-installed add-ons were previously validated at installation time and should (IMO) run as-is. It seems unnecessary to continuously check the status of an add-on's certificate if it has not been changed. Am I missing something?
- 0xffff2 7y ago
  I believe it's set up this way specifically to allow revocation for addons that are initially approved but later found to be malicious.
  - pbhjpbhj 7y ago
    If it is, without requesting user authorisation, then that's an illegal act under the UK Computer Misuse Act (and the USA's CFAA I think too) - modification of a computer without authorisation.
    - usecontainers 7y ago
      except you agreed and authorized when you installed the software. Take your position to the logical extreme - software can't make any changes without explicit, interactive approval; and you thought UAC was bad. I look forward to joining your class-action lawsuit.
      - pbhjpbhj 7y ago
        When the changes are unexpected, yes, further explicit authorisation is required. Just because you installed a photo-album app doesn't let the distributor delete all your photos, say. Besides that, this sort of "but we hid something in the t&c-s so now we can shit on you" is the sort of thing I expect from over commercialised companies, not from what was once a paragon of the FOSS community. FWIW class-actions don't exist in UK.
pcwalton 7y ago
> Also why it took 6 hrs to assign P1 to the bug Because people were staying up until the wee hours of the morning working on fixing it instead of toggling priorities in Bugzilla. This was treated as a five-alarm fire.
- pbhjpbhj 7y ago
  >This was treated as a five-alarm fire. I don't think it bothers me personally but it's funny you said that. Presumably you mean a "'no-alarm fire' because who has time to set off an alarm when there's a fire to fight"?!
  - rhizome 7y ago
    Read the room, Sheldon.
  - lugg 7y ago
    > One-alarm, two-alarm, three-alarm fires, etc., are categories of fires indicating the level of response by local authorities. The term multiple-alarm is a quick way of indicating that a fire is severe and is difficult to contain. https://en.wikipedia.org/wiki/Multiple-alarm_fire https://en.wikipedia.org/wiki/Multiple-alarm_fire The 5th level or a five-alarm fire, is the top level where you're basically talking all hands on deck.
    - pbhjpbhj 7y ago
      In the UK they actually sound/flash alarms at locations and in the fire-station, do they not do that in USA? They do in the movies. Still seems like an ironic choice, applies equally to the people saying it was DEFCON1. I'm pretty sure the military actually have displays indicating the status, but again that's based mainly on movies.
- SlowRobotAhead 7y ago
  I agree with you, it was more important to do the work than to signal. However, I bet it’s likely they have procedures and policies for work that first involve signaling like for example the priority level. I’d be willing to bet lots of things surrounding this issue weren’t handled in a by the book manner. So if you are always going to wing it, why have a book (or a public priority level system) at all?
kibwen 7y ago
I'm also interested in the postmortem to explain the processes that failed to allow the certificate to expire, but let's not overdramatize the situation by nitpicking about filling in form fields on bugzilla. The fact that the tree was closed is equivalent to DEFCON-1, which is all the priority anyone needs to understand the severity of this bug.
- lugg 7y ago
  > which is all the priority anyone needs to understand the severity of this bug. Random user: What the fuck is a tree and why is the priority of this not higher yet?
  - kibwen 7y ago
    > Random user: What the fuck is a tree and why is the priority of this not higher yet? Not to be too glib, but any random user who is technically literate enough to know where to seek out Firefox's issue tracker and how to find the issue in question, and who has such a thorough understanding of issue trackers that they understand that such a thing as a priority field exists, is also going to be savvy enough to read the very first comment, and will be well aware of what it means, and will, one hopes, be rational enough to understand that the flurry of activity indicated by the issue in question is more important than a passing field in the bugzilla database. If anyone expects Mozilla to take power users seriously, then we need to focus our criticism on the things that aren't just imagined trivialites. It makes me frustrated that the people who irrationally fly off the handle at the slightest perceived provocation are also the ones who implicitly encourage Mozilla to write off power users as more trouble than we're worth (and after ten years of watching these incessant whining non-comments on HN, I don't blame them anymore).
mook 7y ago
Looking at the changeset [1], I'm curious why the explicit check for expiry (line 644/646) didn't work. Unfortunately the mentioned bug is rather light on details; presumably they were collaborating on IRC or something instead. [1] GitHub mirror to not stress their infra: https://github.com/mozilla/gecko-dev/commit/1d1260c7615f1d9a018493dc6eca189da96ebe14 https://github.com/mozilla/gecko-dev/commit/1d1260c7615f1d9a...
- deleted 7y ago
  [deleted]

I'm interested in the general writeup what went wrong that they missed this certificate expiring. That's a structural problem. Also why it took 6 hrs to assign P1 to the bug

gpm 7y ago

They closed the trees (stopped merging other code changes to prioritize this) for the bug <22 minutes after it was opened. I would assume the delay in assigning P1 is really just a result of assigning P1 not being as high priority as fixing the damn problem.

JamesMcGill 7y ago
This is one of the things about this whole episode that I find baffling. Stuff like adjusting bug priorities and arranging for someone to tweet an announcement is the work of a good engineering manager. This is the right person to run interference and handle comms and deal with things outside of the critical path, like bugzilla updates.
- dao- 7y ago
  The priority field just plays no practical role here since the bug was immediately escalated.
  - pbhjpbhj 7y ago
    If people can see it externally it definitely has an effect, "oh moz aren't taking this browser breaking bug seriously". Which is probably why the parent said it's a management issue rather than a directly technical one, per se.
- rhizome 7y ago
  Stuff like adjusting bug priorities and arranging for someone to tweet an announcement is the work of a good engineering manager So we can come to the obvious conclusion about Mozilla, then? No good "engineering" managers? Miss one reprioritization and you're out! This is what sane people think?
  - Twirrim 7y ago
    At no stage did the parent post state there are no good engineering managers at Mozilla. They just said that adjusting bug priorities is the work of a good engineering manager. There's a world of difference. If you want to complain about knee-jerk overreactions, I think you might want to look in the mirror first.
    - rhizome 7y ago
      I have empathy for people in the middle of hair-on-fire incidents.
makomk 7y ago
If I understand the bug report comments correctly, they didn't close the trees to other code changes to prioritize fixing this, they did it because the cert expiry broke some important tests at the same time as it broke every end user's browser.

deleted 7y ago

[deleted]

bhauer 7y ago

I'm also interested in why existing adds-ons are failing to run due to this problem. (There was a similar question in another thread about the issue here at HN.) I understand why an add-on update or new installation would be prevented from succeeding by a certificate expiration. But why would a certificate expiration prevent an already-installed from running? Any already-installed add-ons were previously validated at installation time and should (IMO) run as-is. It seems unnecessary to continuously check the status of an add-on's certificate if it has not been changed. Am I missing something?

0xffff2 7y ago
I believe it's set up this way specifically to allow revocation for addons that are initially approved but later found to be malicious.
- pbhjpbhj 7y ago
  If it is, without requesting user authorisation, then that's an illegal act under the UK Computer Misuse Act (and the USA's CFAA I think too) - modification of a computer without authorisation.
  - usecontainers 7y ago
    except you agreed and authorized when you installed the software. Take your position to the logical extreme - software can't make any changes without explicit, interactive approval; and you thought UAC was bad. I look forward to joining your class-action lawsuit.
    - pbhjpbhj 7y ago
      When the changes are unexpected, yes, further explicit authorisation is required. Just because you installed a photo-album app doesn't let the distributor delete all your photos, say. Besides that, this sort of "but we hid something in the t&c-s so now we can shit on you" is the sort of thing I expect from over commercialised companies, not from what was once a paragon of the FOSS community. FWIW class-actions don't exist in UK.

pcwalton 7y ago

> Also why it took 6 hrs to assign P1 to the bug Because people were staying up until the wee hours of the morning working on fixing it instead of toggling priorities in Bugzilla. This was treated as a five-alarm fire.

pbhjpbhj 7y ago
>This was treated as a five-alarm fire. I don't think it bothers me personally but it's funny you said that. Presumably you mean a "'no-alarm fire' because who has time to set off an alarm when there's a fire to fight"?!
- rhizome 7y ago
  Read the room, Sheldon.
- lugg 7y ago
  > One-alarm, two-alarm, three-alarm fires, etc., are categories of fires indicating the level of response by local authorities. The term multiple-alarm is a quick way of indicating that a fire is severe and is difficult to contain. https://en.wikipedia.org/wiki/Multiple-alarm_fire https://en.wikipedia.org/wiki/Multiple-alarm_fire The 5th level or a five-alarm fire, is the top level where you're basically talking all hands on deck.
  - pbhjpbhj 7y ago
    In the UK they actually sound/flash alarms at locations and in the fire-station, do they not do that in USA? They do in the movies. Still seems like an ironic choice, applies equally to the people saying it was DEFCON1. I'm pretty sure the military actually have displays indicating the status, but again that's based mainly on movies.
SlowRobotAhead 7y ago
I agree with you, it was more important to do the work than to signal. However, I bet it’s likely they have procedures and policies for work that first involve signaling like for example the priority level. I’d be willing to bet lots of things surrounding this issue weren’t handled in a by the book manner. So if you are always going to wing it, why have a book (or a public priority level system) at all?

kibwen 7y ago

I'm also interested in the postmortem to explain the processes that failed to allow the certificate to expire, but let's not overdramatize the situation by nitpicking about filling in form fields on bugzilla. The fact that the tree was closed is equivalent to DEFCON-1, which is all the priority anyone needs to understand the severity of this bug.

lugg 7y ago
> which is all the priority anyone needs to understand the severity of this bug. Random user: What the fuck is a tree and why is the priority of this not higher yet?
- kibwen 7y ago
  > Random user: What the fuck is a tree and why is the priority of this not higher yet? Not to be too glib, but any random user who is technically literate enough to know where to seek out Firefox's issue tracker and how to find the issue in question, and who has such a thorough understanding of issue trackers that they understand that such a thing as a priority field exists, is also going to be savvy enough to read the very first comment, and will be well aware of what it means, and will, one hopes, be rational enough to understand that the flurry of activity indicated by the issue in question is more important than a passing field in the bugzilla database. If anyone expects Mozilla to take power users seriously, then we need to focus our criticism on the things that aren't just imagined trivialites. It makes me frustrated that the people who irrationally fly off the handle at the slightest perceived provocation are also the ones who implicitly encourage Mozilla to write off power users as more trouble than we're worth (and after ten years of watching these incessant whining non-comments on HN, I don't blame them anymore).

mook 7y ago

Looking at the changeset [1], I'm curious why the explicit check for expiry (line 644/646) didn't work. Unfortunately the mentioned bug is rather light on details; presumably they were collaborating on IRC or something instead. [1] GitHub mirror to not stress their infra: https://github.com/mozilla/gecko-dev/commit/1d1260c7615f1d9a018493dc6eca189da96ebe14 https://github.com/mozilla/gecko-dev/commit/1d1260c7615f1d9a...

deleted 7y ago
[deleted]