9 ms·

Add-Ons Outage Post-Mortem Result

126 points by bzbarsky 7y ago

ars 7y ago
The Post-Mortem result for me is that I have xpinstall.signatures.required permanently disabled. I'm stuck running an old version of Firefox because I have too much data in Tab Groups, and I haven't be able to migrate it yet. I'll have to find the time someday, since Google Sheets no longer works.
- JoshTriplett 7y ago
  > I'm stuck running an old version of Firefox because I have too much data in Tab Groups Yikes! You're running a years-old version of Firefox without security updates? And you're browsing public websites? I think there are some supported add-ons for tab groups that work with current Firefox. Please, please don't run an insecure browser on the public internet.
  - CamJN 7y ago
    There are no working tab group addons post XUL, I've tried them all and they are all completely broken. While I agree that you shouldn't run without updates, please don't tell people that there is a working substitute for this functionality.
    - JoshTriplett 7y ago
      Before posting that comment, I searched not only for such addons but for the last updates to those addons and for reviews of those addons, which seemed to suggest that they work as Web Extensions and function as at least some people want. That said, I can't vouch for them personally, and I appreciate you providing some firsthand experience.
    - Vinnl 7y ago
      I used to use the XUL version and now use a replacement, and it fully fulfils the function the XUL version did for me.
  - silversconfused 7y ago
    Accepting anti-features in exchange for security hotfixes isn't going to win everyone over.
    - JoshTriplett 7y ago
      Posts that assume everyone considers certain changes to Firefox "anti-features" aren't going to win everyone over, either. There are good reasons for the move to Web Extensions, the deprecation of XUL, more careful checking/validation of extensions, and other such changes.
      - silversconfused 7y ago
        Right, so we fork.
      - JadeNB 7y ago
        > Posts that assume everyone considers certain changes to Firefox "anti-features" aren't going to win everyone over, either. This seems like a bad faith characterisation. The discussion was about ars's activities, and I think it is reasonable to assume silversconfused was guessing at ars's opinions on these changes, not at "everyone's" opinions.
- anticensor 7y ago
  That setting is an inert toggle in branded Release builds.
- SilasX 7y ago
  >The Post-Mortem result for me is that I have xpinstall.signatures.required permanently disabled. Firefox has been ignoring that preference since 2016, when it broke my workflow in the middle of the day and I wrote this Hitler parody: https://www.youtube.com/watch?v=taGARf8K5J8 https://www.youtube.com/watch?v=taGARf8K5J8
quanticle 7y ago
The thing I found somewhat baffling is that they didn't renew the certificate because they were certain that Firefox wasn't checking the certificate expiration. I'm wondering why they didn't renew the certificate and then implement a certificate expiration check. I don't mean to be too harsh, but that part of the post-mortem reads like, "Oh, we don't need to renew our intermediate signing certificate because there's a known security vulnerability in Firefox that allows us to get around it."
- Yoric 7y ago
  Well, it's the kind of errors that happens when two teams forget to communicate.
  - luckylion 7y ago
    But the communication is just the "why they thought it wouldn't be a big deal". The "why would you let a certificate expire that is actively used in code signing" is a different issue.
- TwoNineFive 7y ago
  No, it's not baffling. It's white-washing and blame-shifting. "it turns out to be more complicated" = It's even worse than you probably imagined. We knew the certificate was going to expire and willingly let it happen because we don't know how our own product works. Also look at this new update mechanism and the other problems we caused! Forget about the fact that none of this makes any sense and please don't fire me for gross incompetence. This was always a no-win situation for the people responsible. Allowing a certificate to expire has no reasonable justifications possible. It was always just going to be a question of if anyone would actually be held accountable for negligence or if they would excuse and blame-shift their way out of it. I don't mean to be so negative about it. The silver lining could have been setting an example of doing a real investigation, finding that people were negligent, and holding those negligent people accountable, but I don't see that happening here because the people doing the investigating are the ones responsible. The "it turns out to be more complicated" quote reminds me of Russ Roberts from Econ Talk. He has a number of interesting mannerisms that clearly signals his personal opinion on things. He doesn't come out and say "That's a terrible idea and I disagree!". Instead he whines "Well I'm not so sure." And when some study or evidence goes against his preconceived ideas or biases he says "Well, it's complicated".
dessant 7y ago
I believe it is essential for Firefox to allow installing extensions in a private way that does not force us to upload our source code to Mozilla, especially considering their renewed push for user privacy. It could be a feature that can only be enabled from the browser UI, with appropriate warnings, but the choice must be offered for those who need it. Until that happens, those with private extensions must resort to compiling their own browser, use unbranded or developer builds, or patch Firefox at runtime to disable signature checks. This is how you allow unsigned extensions in Firefox on Arch Linux, the same files can be edited on Windows and macOS, restart the browser after changes: sudo tee /usr/lib/firefox/defaults/pref/config-prefs.js &>/dev/null <<EOF pref("general.config.obscure_value", 0); pref("general.config.filename", "config.js"); pref("general.config.sandbox_enabled", false); EOF sudo tee /usr/lib/firefox/config.js &>/dev/null <<EOF // keep this comment try { Components.utils .import('resource://gre/modules/addons/XPIDatabase.jsm', {}) .XPIDatabase['SIGNED_TYPES'].clear(); } catch (ex) { Components.utils.reportError(ex.message); } EOF
- Wowfunhappy 7y ago
  At minimum, Unbranded should be able to automatically update itself, so that it can be a legitimate choice for users.
- Vinnl 7y ago
  > use unbranded or developer builds Is that such a problematic thing?
  - dessant 7y ago
    Unbranded builds do not auto-update, and Firefox Developer Edition is based on Firefox Beta, so it can be less stable than the release build.
    - Vinnl 7y ago
      Auto-update would be convenient indeed. That said, Developer Edition is really very stable, so while I see why you would not want that, it's not that bad a stop-gap measure until unbranded builds auto-update.
gilrain 7y ago
I'm very disappointed that although using Studies (Normandy) is briefly mentioned, it is treated as just an unusual option rather than a breach of policy. I was really hoping for an explanation of why they felt they could break their promise to users that Studies would only ever be used for, well, studies. * All Shield Studies must be designed to answer a specific question - Shield is a tool for evaluating ideas and features in the product. If you are not trying to evaluate a proposed feature or idea there are other means of shipping your code. Shield studies will always respect user privacy in accordance with our data collection policies * All Shield studies will adhere to the scientific method for answering complex questions - Generate a hypothesis, test, collect and analyze data, validate or refute hypothesis, refine, repeat.. * All Shield studies require a PHD (Product Hypothesis Doc) - A PHD or Product Hypothesis Doc outlines the overarching research question the study is trying to answer. It requires the study author to think critically about the problem and the outcomes long before the study ships. These guidelines (https://wiki.mozilla.org/Firefox/Shield/Shield_Studies#Guiding_Principles https://wiki.mozilla.org/Firefox/Shield/Shield_Studies#Guidi...) were created after the beach of trust represented by the Mr Robot cross-promotion as a means to restore confidence. Yet now they have been ignored, and again "Studies" has been used as a convenient backdoor.
- Deimorz 7y ago
  Imagine you run a very popular website, and you have a policy that says "we will never modify any site data by accessing the database directly", to try to ensure that everything goes through site functionality and can be properly logged/etc. One day, some weird data gets into the database somehow, the site's completely unable to handle it, and it crashes and goes totally down. You're faced with two options: 1. Rewrite a major section of the site code to be able to handle the bad data. This is expected to take at least two days to complete. 2. Break the policy, fix the data in the database, and have the site back up in 5 minutes. Do you truly believe that Option #2 needs to be treated as completely out of the question?
  - gilrain 7y ago
    Not nearly analogous. First, the policy was not about accountability, it was about privacy and specifically restoring a trust already broken. Second, it would only have taken a worst case of 24 hours vs the 6 hours of using Studies to follow that policy. I think the integrity of a company that trades on that very thing is worth 18 hours.
groovybits 7y ago
Looking at the Technical Report [1], I am confused... First bullet under Root Causes: "The server-side teams whose systems (e.g., Autograph) generate signatures knew the certificate was expiring, but did not see a problem because they had reason to believe that signing certificate dates were not checked by the client." So, if I understand correctly, the assumed process was: Root signs Intermediate --> Intermediate signs Add-on (whether Intermediate is valid or not) --> FF only validates the End-entity is within date range. Is this the correct procedure? You're telling me that the intended behavior is that FF allows add-ons signed by an expired cert? If that is the case, then what is the point of the signing system in the first place? Additionally, third paragraph under Recommendations: "Second, if we are committed to the current PKI and add-on signing approach, [...]" '_If_ we are committed' is non-absolute. Is the FF team actually set on this approach? 1: https://wiki.mozilla.org/Add-ons/Expired-Certificate-Technical-Report https://wiki.mozilla.org/Add-ons/Expired-Certificate-Technic... Edit: clarifications
- sciurus 7y ago
  "FF only validates the End-entity is within date range" No, this was disabled back in 2016. The technical report briefly had this worded wrong, sorry if you read it before it was corrected. "what is the point of the signing system in the first place?" Like the technical report describes, things like "monitoring add-ons not hosted on AMO, blocklisting malicious add-ons, providing cryptographic assurance by chaining add-ons to the Mozilla root." (Disclosure: I work for Mozilla)
  - groovybits 7y ago
    So, the intended behavior is that FF does not check any Add-on related cert for a valid date range? In that case, what prevents an insider threat of a Mozilla employee signing malicious add-ons with an out-of-date cert?
    - yzmtf2008 7y ago
      What prevents an insider threat of a Mozilla employee signing malicious add-ons with a not out-of-date cert?
sebazzz 7y ago
I wonder why there is no timestamping used on the binaries. If I sign a binary with Authenticode on Windows and I use a timestamp server, Windows still considers the binary signature valid, even if the certificate expired because it can verify that at the time of signing the certificate was valid. If you don't use a timestamp server when signing the binary, after certificate expiration the binary is considered invalid.
silversconfused 7y ago
There's no reason for local software to stop working when a remote server is unavailable, reconfigured, neglected, or your clock is wrong. They are doomed if they think this is sustainable.
- dralley 7y ago
  That is not what happened. A certificate built into the Firefox installation expired. It had nothing to do with "remote servers".
  - silversconfused 7y ago
    It was a time bomb to disable functionality after a certain date and should never have been there in the first place. The decision to silently disable the override in about:config is just more evidence of how sick this it. This is not software empowering the user. This is software assuming the user is a moron who may break it.
michaelt 7y ago
the Firefox QA plan didn’t incorporate testing for certificate expiration (or generalized testing of how the browser will behave at future dates) and therefore the problem wasn’t detected A few years ago I tried to find a VM or something similar that would let me test software with clock offsets applied. At the time, most of the VMs I looked at had prioritized the much more common requirement of not having the guest OS clock become inaccurate. Does anyone know the state of the art in integration testing software for future dates?
- donio 7y ago
  One way is to override the libc time functions using LD_PRELOAD using something like https://github.com/wolfcw/libfaketime https://github.com/wolfcw/libfaketime This doesn't work with everything (for example doesn't work with Go because Go doesn't use libc for such things) but if it does then it's a very simple to use: % LD_PRELOAD=src/libfaketime.so.1 FAKETIME='1999-12-31 23:59:59' date Fri Dec 31 23:59:59 PST 1999 % LD_PRELOAD=src/libfaketime.so.1 FAKETIME='1999-12-31 23:59:59' python -c 'import time;print(time.ctime())' Fri Dec 31 23:59:59 1999 or to have the fake clock ticking starting from the specified time: % LD_PRELOAD=src/libfaketime.so.1 FAKETIME='@1999-12-31 23:59:59' python -c 'import time;print(time.ctime());time.sleep(1);print(time.ctime())' Fri Dec 31 23:59:59 1999 Sat Jan 1 00:00:00 2000
mrosett 7y ago
The HN frontpage is just a long list of postmortems right now.
mook 7y ago
From the linked incident report: > Now that we understand the importance of add-on support for mobile (ratings plummeted during the incident), we might want to reconsider that strategy/priority for Fenix. Well that's promising at least. The listed Next Steps is still full of uncertainty and there isn't a firm decision (presumably the people involved in this post-mortem are not the relevant decision makers), but at least somebody is bringing it up. Given the Mozilla track history, though, I'm not optimistic.

126 points by bzbarsky 7y ago

ars 7y ago

The Post-Mortem result for me is that I have xpinstall.signatures.required permanently disabled. I'm stuck running an old version of Firefox because I have too much data in Tab Groups, and I haven't be able to migrate it yet. I'll have to find the time someday, since Google Sheets no longer works.

JoshTriplett 7y ago
> I'm stuck running an old version of Firefox because I have too much data in Tab Groups Yikes! You're running a years-old version of Firefox without security updates? And you're browsing public websites? I think there are some supported add-ons for tab groups that work with current Firefox. Please, please don't run an insecure browser on the public internet.
- CamJN 7y ago
  There are no working tab group addons post XUL, I've tried them all and they are all completely broken. While I agree that you shouldn't run without updates, please don't tell people that there is a working substitute for this functionality.
  - JoshTriplett 7y ago
    Before posting that comment, I searched not only for such addons but for the last updates to those addons and for reviews of those addons, which seemed to suggest that they work as Web Extensions and function as at least some people want. That said, I can't vouch for them personally, and I appreciate you providing some firsthand experience.
  - Vinnl 7y ago
    I used to use the XUL version and now use a replacement, and it fully fulfils the function the XUL version did for me.
- silversconfused 7y ago
  Accepting anti-features in exchange for security hotfixes isn't going to win everyone over.
  - JoshTriplett 7y ago
    Posts that assume everyone considers certain changes to Firefox "anti-features" aren't going to win everyone over, either. There are good reasons for the move to Web Extensions, the deprecation of XUL, more careful checking/validation of extensions, and other such changes.
    - silversconfused 7y ago
      Right, so we fork.
    - JadeNB 7y ago
      > Posts that assume everyone considers certain changes to Firefox "anti-features" aren't going to win everyone over, either. This seems like a bad faith characterisation. The discussion was about ars's activities, and I think it is reasonable to assume silversconfused was guessing at ars's opinions on these changes, not at "everyone's" opinions.
anticensor 7y ago
That setting is an inert toggle in branded Release builds.
SilasX 7y ago
>The Post-Mortem result for me is that I have xpinstall.signatures.required permanently disabled. Firefox has been ignoring that preference since 2016, when it broke my workflow in the middle of the day and I wrote this Hitler parody: https://www.youtube.com/watch?v=taGARf8K5J8 https://www.youtube.com/watch?v=taGARf8K5J8

quanticle 7y ago

The thing I found somewhat baffling is that they didn't renew the certificate because they were certain that Firefox wasn't checking the certificate expiration. I'm wondering why they didn't renew the certificate and then implement a certificate expiration check. I don't mean to be too harsh, but that part of the post-mortem reads like, "Oh, we don't need to renew our intermediate signing certificate because there's a known security vulnerability in Firefox that allows us to get around it."

Yoric 7y ago
Well, it's the kind of errors that happens when two teams forget to communicate.
- luckylion 7y ago
  But the communication is just the "why they thought it wouldn't be a big deal". The "why would you let a certificate expire that is actively used in code signing" is a different issue.
TwoNineFive 7y ago
No, it's not baffling. It's white-washing and blame-shifting. "it turns out to be more complicated" = It's even worse than you probably imagined. We knew the certificate was going to expire and willingly let it happen because we don't know how our own product works. Also look at this new update mechanism and the other problems we caused! Forget about the fact that none of this makes any sense and please don't fire me for gross incompetence. This was always a no-win situation for the people responsible. Allowing a certificate to expire has no reasonable justifications possible. It was always just going to be a question of if anyone would actually be held accountable for negligence or if they would excuse and blame-shift their way out of it. I don't mean to be so negative about it. The silver lining could have been setting an example of doing a real investigation, finding that people were negligent, and holding those negligent people accountable, but I don't see that happening here because the people doing the investigating are the ones responsible. The "it turns out to be more complicated" quote reminds me of Russ Roberts from Econ Talk. He has a number of interesting mannerisms that clearly signals his personal opinion on things. He doesn't come out and say "That's a terrible idea and I disagree!". Instead he whines "Well I'm not so sure." And when some study or evidence goes against his preconceived ideas or biases he says "Well, it's complicated".

dessant 7y ago

I believe it is essential for Firefox to allow installing extensions in a private way that does not force us to upload our source code to Mozilla, especially considering their renewed push for user privacy. It could be a feature that can only be enabled from the browser UI, with appropriate warnings, but the choice must be offered for those who need it. Until that happens, those with private extensions must resort to compiling their own browser, use unbranded or developer builds, or patch Firefox at runtime to disable signature checks. This is how you allow unsigned extensions in Firefox on Arch Linux, the same files can be edited on Windows and macOS, restart the browser after changes: sudo tee /usr/lib/firefox/defaults/pref/config-prefs.js &>/dev/null <<EOF pref("general.config.obscure_value", 0); pref("general.config.filename", "config.js"); pref("general.config.sandbox_enabled", false); EOF sudo tee /usr/lib/firefox/config.js &>/dev/null <<EOF // keep this comment try { Components.utils .import('resource://gre/modules/addons/XPIDatabase.jsm', {}) .XPIDatabase['SIGNED_TYPES'].clear(); } catch (ex) { Components.utils.reportError(ex.message); } EOF

Wowfunhappy 7y ago
At minimum, Unbranded should be able to automatically update itself, so that it can be a legitimate choice for users.
Vinnl 7y ago
> use unbranded or developer builds Is that such a problematic thing?
- dessant 7y ago
  Unbranded builds do not auto-update, and Firefox Developer Edition is based on Firefox Beta, so it can be less stable than the release build.
  - Vinnl 7y ago
    Auto-update would be convenient indeed. That said, Developer Edition is really very stable, so while I see why you would not want that, it's not that bad a stop-gap measure until unbranded builds auto-update.

gilrain 7y ago

I'm very disappointed that although using Studies (Normandy) is briefly mentioned, it is treated as just an unusual option rather than a breach of policy. I was really hoping for an explanation of why they felt they could break their promise to users that Studies would only ever be used for, well, studies. * All Shield Studies must be designed to answer a specific question - Shield is a tool for evaluating ideas and features in the product. If you are not trying to evaluate a proposed feature or idea there are other means of shipping your code. Shield studies will always respect user privacy in accordance with our data collection policies * All Shield studies will adhere to the scientific method for answering complex questions - Generate a hypothesis, test, collect and analyze data, validate or refute hypothesis, refine, repeat.. * All Shield studies require a PHD (Product Hypothesis Doc) - A PHD or Product Hypothesis Doc outlines the overarching research question the study is trying to answer. It requires the study author to think critically about the problem and the outcomes long before the study ships. These guidelines (https://wiki.mozilla.org/Firefox/Shield/Shield_Studies#Guiding_Principles https://wiki.mozilla.org/Firefox/Shield/Shield_Studies#Guidi...) were created after the beach of trust represented by the Mr Robot cross-promotion as a means to restore confidence. Yet now they have been ignored, and again "Studies" has been used as a convenient backdoor.

Deimorz 7y ago
Imagine you run a very popular website, and you have a policy that says "we will never modify any site data by accessing the database directly", to try to ensure that everything goes through site functionality and can be properly logged/etc. One day, some weird data gets into the database somehow, the site's completely unable to handle it, and it crashes and goes totally down. You're faced with two options: 1. Rewrite a major section of the site code to be able to handle the bad data. This is expected to take at least two days to complete. 2. Break the policy, fix the data in the database, and have the site back up in 5 minutes. Do you truly believe that Option #2 needs to be treated as completely out of the question?
- gilrain 7y ago
  Not nearly analogous. First, the policy was not about accountability, it was about privacy and specifically restoring a trust already broken. Second, it would only have taken a worst case of 24 hours vs the 6 hours of using Studies to follow that policy. I think the integrity of a company that trades on that very thing is worth 18 hours.

groovybits 7y ago

Looking at the Technical Report [1], I am confused... First bullet under Root Causes: "The server-side teams whose systems (e.g., Autograph) generate signatures knew the certificate was expiring, but did not see a problem because they had reason to believe that signing certificate dates were not checked by the client." So, if I understand correctly, the assumed process was: Root signs Intermediate --> Intermediate signs Add-on (whether Intermediate is valid or not) --> FF only validates the End-entity is within date range. Is this the correct procedure? You're telling me that the intended behavior is that FF allows add-ons signed by an expired cert? If that is the case, then what is the point of the signing system in the first place? Additionally, third paragraph under Recommendations: "Second, if we are committed to the current PKI and add-on signing approach, [...]" '_If_ we are committed' is non-absolute. Is the FF team actually set on this approach? 1: https://wiki.mozilla.org/Add-ons/Expired-Certificate-Technical-Report https://wiki.mozilla.org/Add-ons/Expired-Certificate-Technic... Edit: clarifications

sciurus 7y ago
"FF only validates the End-entity is within date range" No, this was disabled back in 2016. The technical report briefly had this worded wrong, sorry if you read it before it was corrected. "what is the point of the signing system in the first place?" Like the technical report describes, things like "monitoring add-ons not hosted on AMO, blocklisting malicious add-ons, providing cryptographic assurance by chaining add-ons to the Mozilla root." (Disclosure: I work for Mozilla)
- groovybits 7y ago
  So, the intended behavior is that FF does not check any Add-on related cert for a valid date range? In that case, what prevents an insider threat of a Mozilla employee signing malicious add-ons with an out-of-date cert?
  - yzmtf2008 7y ago
    What prevents an insider threat of a Mozilla employee signing malicious add-ons with a not out-of-date cert?

sebazzz 7y ago

I wonder why there is no timestamping used on the binaries. If I sign a binary with Authenticode on Windows and I use a timestamp server, Windows still considers the binary signature valid, even if the certificate expired because it can verify that at the time of signing the certificate was valid. If you don't use a timestamp server when signing the binary, after certificate expiration the binary is considered invalid.

silversconfused 7y ago

There's no reason for local software to stop working when a remote server is unavailable, reconfigured, neglected, or your clock is wrong. They are doomed if they think this is sustainable.

dralley 7y ago
That is not what happened. A certificate built into the Firefox installation expired. It had nothing to do with "remote servers".
- silversconfused 7y ago
  It was a time bomb to disable functionality after a certain date and should never have been there in the first place. The decision to silently disable the override in about:config is just more evidence of how sick this it. This is not software empowering the user. This is software assuming the user is a moron who may break it.

michaelt 7y ago

the Firefox QA plan didn’t incorporate testing for certificate expiration (or generalized testing of how the browser will behave at future dates) and therefore the problem wasn’t detected A few years ago I tried to find a VM or something similar that would let me test software with clock offsets applied. At the time, most of the VMs I looked at had prioritized the much more common requirement of not having the guest OS clock become inaccurate. Does anyone know the state of the art in integration testing software for future dates?

donio 7y ago
One way is to override the libc time functions using LD_PRELOAD using something like https://github.com/wolfcw/libfaketime https://github.com/wolfcw/libfaketime This doesn't work with everything (for example doesn't work with Go because Go doesn't use libc for such things) but if it does then it's a very simple to use: % LD_PRELOAD=src/libfaketime.so.1 FAKETIME='1999-12-31 23:59:59' date Fri Dec 31 23:59:59 PST 1999 % LD_PRELOAD=src/libfaketime.so.1 FAKETIME='1999-12-31 23:59:59' python -c 'import time;print(time.ctime())' Fri Dec 31 23:59:59 1999 or to have the fake clock ticking starting from the specified time: % LD_PRELOAD=src/libfaketime.so.1 FAKETIME='@1999-12-31 23:59:59' python -c 'import time;print(time.ctime());time.sleep(1);print(time.ctime())' Fri Dec 31 23:59:59 1999 Sat Jan 1 00:00:00 2000

mrosett 7y ago

The HN frontpage is just a long list of postmortems right now.

mook 7y ago

From the linked incident report: > Now that we understand the importance of add-on support for mobile (ratings plummeted during the incident), we might want to reconsider that strategy/priority for Fenix. Well that's promising at least. The listed Next Steps is still full of uncertainty and there isn't a firm decision (presumably the people involved in this post-mortem are not the relevant decision makers), but at least somebody is bringing it up. Given the Mozilla track history, though, I'm not optimistic.