18 ms·

Technical Details on the Recent Firefox Add-On Outage

467 points by headalgorithm 7y ago

ziddoap 7y ago
>First, I want to say that the team here did amazing work: they built and shipped a fix in less than 12 hours from the initial report. As someone who sat in the meeting where it happened, I can say that people were working incredibly hard in a tough situation and that very little time was wasted. It's a bit disheartening to see the "Lessons" section opened with this. I understand that everyone worked very hard to get things up and running - but that is not a lesson learned. That is self-back-patting.
- dblohm7 7y ago
  The post-mortem hasn't even started yet; now that 66.0.5 is out and the initial incident response is finally winding down, now they can move on to that.
  - ziddoap 7y ago
    I don't quite follow how this reply relates to the first paragraph under the "Lessons" section of the linked article.
- fwip 7y ago
  It's context for the next paragraph. > With that said, obviously this isn’t an ideal situation and it shouldn’t have happened in the first place. We clearly need to adjust our processes both to make this and similar incidents it less likely to happen and to make them easier to fix.
  - ziddoap 7y ago
    I did gather that much. In my _personal opinion_, it would have been much more palatable to start the lessons learned section with >We’ll be running a formal post-mortem next week and will publish the list of changes we intend to make, but in the meantime here are my initial thoughts about what we need to do. First, we should have a much better way of tracking And shoe-horn the back-patting in earlier or later. Perhaps I'm being pedantic. Fair enough. But for me, I would rather see this issue addressed in a way that the severity of the situation requires. Back-patting is not on the plate for me.
- shawnz 7y ago
  It's clearly just a disclaimer to show his respectful intent, since the paragraphs following that are addressing failures of his colleagues.
  - ziddoap 7y ago
    Understood, and I explained a bit more of my thoughts in another comment. I understand I'm being a bit pedantic. But, as a user who is pretty concerned with a CA having a certificate issue of this scale, which affected this many people, I don't care about the authors respectful intent to their colleagues. Tell them in person, in an internal memo, whatever. It's simply not a lesson learned.
    - shawnz 7y ago
      Likewise, I'm sure he cares more about keeping good working relationships with his colleagues than about what the reader thinks of his choice of paragraph order. A manager who makes his team feel insulted is not conducive to producing better software (which is presumably what you care most about here).
      - ziddoap 7y ago
        How is laying out, in a technical fashion, what went wrong and the lessons learned insulting his colleagues? I'm very curious. Just because someone made a mistake, and that mistake is acknowledged, doesn't mean they are insulting them. Also, how come it can't be done in an internal memo, email, party, pay bonus, raises, etc.?
    - tialaramex 7y ago
      Mozilla, unlike all of the other major Trust Stores does not actually operate a publicly trusted CA. Microsoft, Apple, and Google all do, the biggest Trust Store I can think of that does NOT operate a CA other than Mozilla is Oracle (who manage the default Java trust store) This puts Mozilla in a good situation of independence IMNSHO. At Google we know it takes actual effort to make sure that corporate firewalls protect the Trust Store function from being sabotaged by the convenience of, say, Google's Cloud function or people in their SRE wanting to keep the lights on for Chrome users. One day Google's CA people accidentally minted certs that Chrome wouldn't trust for Google web sites, and there had to be mechanisms in place to ensure that "Just tell the Chrome engineers to ignore it" wasn't a viable fix, because that would be _easier_ but it destroys any value in that trust store.
parliament32 7y ago
>An important feature here is that the new certificate has the same subject name and public key as the old certificate, so that its signature on the End-Entity certificate is valid. Shouldn't it be impossible to generate a new cert (with a different expiry date) that ends up having the same public key as an existing cert?
- bityard 7y ago
  No, one public/private key pair can be used to generate or sign as many certificates as you like, as often as you like.
  - parliament32 7y ago
    No I'm not talking about the root, I mean they generated a new certificate (the intermediate) (with a new private key) that had a public key identical to an existing certificate -- you shouldn't be able to do this, public keys can't be "specified" afaik, they're derived from your private key and the signer's public key.
    - rossdavidh 7y ago
      Isn't that the point of having the root cert and the intermediate? Otherwise, why not just have everything keyed to the root?
      - rndgermandude 7y ago
        Because a root typically has a longer expire date and thus has to be stored extra secure and handled extra safely. You'd usually store it in some hardware security module that is operated by an airgapped system or something like that that requires physical access at the very least. Replacing the root often requires shipping new versions of your software that disables the old root and bakes in a new root. If an intermediate certificate becomes compromised, you can revoke it and issue a new intermediate certificate with your still secure root without the need to push out new binaries.
    - 7y ago
chrisseaton 7y ago
After the last PR disaster - the Mr Robot tie-in - one of the ways Mozilla tried to make it right was that they promised that the survey system would never again be used for something that was not an experiment. https://blog.mozilla.org/firefox/retrospective-looking-glass/ https://blog.mozilla.org/firefox/retrospective-looking-glass... > A SHIELD study must be designed to answer a specific question. Why have they abused it again here to deploy a hot fix, breaking their promise and policy that they put in place last time they messed up? Or am I ignorant or some part of the story or technical details.
- mlthoughts2018 7y ago
  It is indeed a huge violation of trust. I’ve seen some folks trying to explain it away and compare the Normandy preference system to standard auto-updates, acting like this is no problem if you already trust them for auto-updates. Flat wrong. This is a dark pattern by Mozilla pure & simple. It’s confusing, hard to disable fully, and clearly can be abused for non-experiment modifications to the user’s settings.
  - silversconfused 7y ago
    Mozilla has gone full "helicopter parent" mode, and will do whatever they can to keep you safe from yourself despite your wishes. Use GNU icecat instead. It has all that junk turned off.
- fedups 7y ago
  Do they have a better channel for deploying hot-fixes? Maybe I'm a bit of a squish but I'm pretty agnostic of how they deploy fixes to me so long as my addons start working again.
  - deleted 7y ago
    [deleted]
  - bwat49 7y ago
    They don't currently, but adding a better channel to deploy hotfixes is something that's specifically mentioned in this article...
jzl 7y ago
Looks like a good read. I haven't finished reading it yet, but there's something I still don't get ... Windows and macOS both have a signing infrastructure for apps. The rules of that infrastructure dictate only that apps must have been signed by a valid certificate at the time they were signed. That way old app downloads don't need to be periodically re-signed just to account for expiring certificates. I can download a 5-year-old version of 7zip or whatever and it runs just fine because it was signed with something valid to the timestamp in the signature. The process of distributing desktop apps would be utterly insane if this were not the case. Not following this model for browser plugins seems unnecessarily cumbersome. Is it really worth requiring all browser plugins to be signed by a currently valid certificate? Is there a document or blog post where this is argued to be more appropriate? I get that it arguably leads to more stringent security, but I'm not convinced by the delta improvement of that model over the desktop model, given the additional downsides. And the "let everything expire after a few years and resign it" process should not be used as a substitution for revocation. After all, if it were determined retroactively that a malware extension was signed does it really help everyone that it can't be loaded in a year or two, given the damage it could cause right now?
- johncolanduoni 7y ago
  For the time stamp method to work, you need a trusted mechanism to attest that the timestamp is correct, otherwise the mechanism is useless (an attacker with an outdated private key can just backdate the timestamp in the executable and then sign it). Windows code signing uses a server Microsoft runs to provide this, and Mozilla would need to do the same. I’m not saying they shouldn’t, but it is a significant piece of complexity.
  - reificator 7y ago
    > Windows code signing uses a server Microsoft runs to provide this, and Mozilla would need to do the same. Mozilla already requires uploading even your private extensions to a Mozilla server to be signed for internal or external deployment. https://wiki.mozilla.org/Add-ons/Extension_Signing https://wiki.mozilla.org/Add-ons/Extension_Signing
  - 7y ago
silversconfused 7y ago
For anyone who misses the old days when a browser only did what you told it to, here is a solution: GNU icecat is a nice firefox esr fork with the mozilla call-home bits all turned off by default. It's very pleasant to use.
- djsumdog 7y ago
  It doesn't have the new Quantum rendering engine though, right?
  - silversconfused 7y ago
    I'm pretty sure it does. https://en.wikipedia.org/wiki/Quantum_(Mozilla) https://en.wikipedia.org/wiki/Quantum_(Mozilla) says quantum shipped in 57.
- bscphil 7y ago
  There's also GNU "abrowser", which (though terribly named) has the advantage of being based on the latest Firefox release rather than esr. It has the same privacy defaults as Icecat (though I believe it doesn't ship with the addons that Icecat does).
sciurus 7y ago
There is a related blog post about what Mozilla is doing with the data collected from users who enabled Studies in order to get the hot fix. https://blog.mozilla.org/blog/2019/05/09/what-we-do-when-things-go-wrong/ https://blog.mozilla.org/blog/2019/05/09/what-we-do-when-thi... TL;DR is "In order to respect our users’ potential intentions as much as possible, based on our current set up, we will be deleting all of our source Telemetry and Studies data for our entire user population collected between 2019-05-04T11:00:00Z and 2019-05-11T11:00:00Z." (Disclaimer: I work for Mozilla)
- jzl 7y ago
  As someone who reluctantly re-enabled Studies to get the fix, I appreciate this.
hartator 7y ago
> Second, we immediately pushed a hotfix which suppressed re-validating the signatures on add-ons. What’s the point of re-validating of installed add-on in the first place then?
acqq 7y ago
> Second, we need a mechanism to be able to quickly push updates to our users even when — especially when — everything else is down. They should simply streamline the "normal" updates for that purpose, not invent the new "channels." Specifically, not all binaries in the directory should be changed just to push an update where only few lines of the code are different.
hartator 7y ago
> because they want to run old-style add-ons, but many of these now work with newer versions of Firefox. Except it wasn’t working for this few days. Anyway older versions of Firefox wasn’t signing add-on, so they are not concerned by this fix.
- silversconfused 7y ago
  Wrong. I lost my addons in firefox 60 esr, which I got from debian. I'm done with mozilla disrespecting my preferences, and have switched to waterfox and icecat. Remote control of the browser is simply the last straw.
tedunangst 7y ago
> For the other groups we are developing a patch to Firefox that will install the new certificate once people update. This was released as a “dot release” so people will get it — and probably have already — through the ordinary update channel. If you have a downstream build, you’ll need to wait for your build maintainer to update. Why not link to the xpi that can be installed now?
- sciurus 7y ago
  FYI that should read "we have developed" instead of "we are developing". Firefox 66.0.4 and later have the patch. (Disclaimer: I work for Mozilla)
- yellowapple 7y ago
  > Why not link to the xpi that can be installed now? This is the crux of my remaining frustration with how Mozilla handled this issue. That XPI should've been front-and-center in all the articles that detailed the fix. And yet, instead of something like... "If you have Studies enabled, a fix should apply automatically. If it hasn't yet, or if you have Studies turned off (or are using a version which does not support Studies), you can install the hotfix add-on [here](URL to XPI)." ...pretty much all the official messaging ended up like so: "If you have Studies enabled, a fix should apply automatically. It may take up to 6 hours; please be patient and wait for it. If you don't want to (or can't) turn on Studies, you're SOL until we push out a point release (and further SOL if you're at the mercy of a Linux package maintainer or you want to use a version of Firefox that still supports XUL-based addons)." The notion that this was a deliberate ploy to get more people to turn on Studies is surely conspiracy-theorist mumbo-jumbo, but nonsense like this makes me wonder.
  - callahad 7y ago
    > The notion that this was a deliberate ploy to get more people to turn on Studies is surely conspiracy-theorist mumbo-jumbo, but nonsense like this makes me wonder. This is not the case. Please see my response downthread: https://news.ycombinator.com/item?id=19872490 https://news.ycombinator.com/item?id=19872490
jeffchien 7y ago
I would've liked to see some reflection or even just acknowledgment about the fact that they intentionally disabled the "xpinstall.signatures.required" setting on Windows and OSX. I hope it's at least in the formal postmortem.
- callahad 7y ago
  That's covered at https://blog.mozilla.org/addons/2015/04/15/the-case-for-extension-signing/ https://blog.mozilla.org/addons/2015/04/15/the-case-for-exte..., which is linked from ekr's post. I encourage you to read the rationale as a whole, but the specific question you're asking is addressed here: > Many developers have asked why we can’t make this a runtime option or preference. There is nowhere we could store that choice on the user’s machine that these greyware apps couldn’t change and plausibly claim they were acting on behalf of the user’s “choice” not to opt-out of the light grey checkbox on page 43 of their EULA. This is not a concern about hypotheticals, we have many documented cases of add-ons disabling the mechanisms through which we inform users and give them control over their add-ons. By baking the signing requirement into the executable these programs will either have to submit to our review process or take the blatant malware step of replacing or altering Firefox. We are sure some will take that step, but it won’t be an attractive option for a Fortune 500 plugin vendor, popular download sites, or the laptop vendor involved in distributing Superfish. For the ones who do, we hope that modifying another program’s executable code is blatant enough that security software vendors will take action and stop letting these programs hide behind terms buried in their user-hostile EULAs.
  - Wowfunhappy 7y ago
    Does Windows et al really not provide a mechinism for saving privileged settings in a tamper-resistant way? I frankly find that hard to believe. How does other software solve this problem? There are of course always workarounds on an open platform like Windows/Mac/Linux, but the threshold isn’t “impossible”, it’s just “as difficult as injecting into the browser’s code.” Edit: For example, what if the config file contained a checksum of the file's contents + the user's hardware? If the setting is changed by the user within Firefox, the checksum is updated and everything works—if the checksum is invalid, settings are reset to the default. Video games will occasionally do this type of thing with their config files. Modders often figure out the formula—but again, the idea here isn't to make editing the file impossible, it's to make it as difficult as injecting into the Firefox executable.
DangerousPie 7y ago
My point of view as a long-time Firefox user that cares about privacy but also knows we live in an imperfect world: It obviously sucks that this happened but I think they handled it very well. The bug was fixed so quickly that I wouldn't even have realized it had happened if it hadn't been for the thread here on HN. My extensions hadn't even been disabled yet by the time the patch came out. And pushing out the hotfix through studies followed by a new version probably ensured that a large fraction of the "average joe" userbase didn't even realize there was a problem. So obviously there are some improvements to make for the future but I think some of the criticism over the last few days has been a bit harsh. Firefox is still my preferred browser by far.
- SECProto 7y ago
  > The bug was fixed so quickly that I wouldn't even have realized it had happened if it hadn't been for the thread here on HN Maybe this is a timezone thing, but I was in East Asia, and I had to deal with the internet for close to 36 hrs (android) with no ublock. It was almost enough to look for a new browser (but browsers with adblock on Android are few and far between - so instead I just didn't use the internet as much for a day or two.). Part of that delay was play store being slow to push it, as I recall seeing the binaries somewhere a while sooner.
  - yellowapple 7y ago
    One option would've been to use Nightly and set xpinstall.signatures.required = false in about:config. That's exactly what I did.
    - SECProto 7y ago
      I view using nightly to be as much of a barrier to entry that I might as well switch to a totally different browser. If the issue had lasted longer, I would have found (someone on the internet who had found) a solution like that.
      - yellowapple 7y ago
Faark 7y ago
> In theory, fixing a problem like this looks simple: make a new, valid certificate and republish every add-on with that certificate. Unfortunately, [..] I'd expect addon usage to follow a pareto distribution. Thus resigning the most important ones would have helped a lot of users. Why didn't they start going this route anyway? Not enough manpower for this to not diverted resources from the other more important fixes?
- callahad 7y ago
  I expect the full postmortem will dive into this in more detail, but one reason mentioned in ekr's post is that many add-ons are distributed independently, outside of AMO. So while we could re-sign those, we're not in a place to re-distribute them. Instead, "users would have to manually update any add-ons that they had installed from other sources, which would be very inconvenient."
- sciurus 7y ago
  I think this is covered in the blog post, but I'll take another stab at explaining it from my perspective: Republishing was one of the options we were investigating early on. However, the problem is that it only fixes things once you check for and install the updated version signed with the new certificate. Firefox would still have disabled your installed version that had an expired certificate. Firefox checks for addon updates every 24 hours, but it checks in with Normandy every 6 hours. Thus once we had stopgap fixes shipping to users via Normandy, and a proper fix of a new Firefox release in progress, republishing addons wasn't necessary. (Disclaimer: I work for Mozilla)
  - Faark 7y ago
    Thanks for your response. An update being deactivated doesn't trigger an update check for said addon? Well, complexity of a graceful update attempt for a not loadable addon probably outweighs the benefit of such rare cases. But the main reason I had a hard time with you guys discarding this path is probably addon stats like [0]. The spike in downloads made me think it could have helped at least some users. But on second thought, it might also have been caused by extensions getting re-enabled post fix. 2m downloads at 4m DAU makes me guess this was caused by something automatic. [0] https://addons.mozilla.org/en-US/firefox/addon/ublock-origin/statistics/?last=30 https://addons.mozilla.org/en-US/firefox/addon/ublock-origin...
gpm 7y ago
THIS IS WRONG: Preserved to make below reply not seem out of context. > Second, we immediately pushed a hotfix which suppressed re-validating the signatures on add-ons. Wait, that's not right is it!? The only hotfix I'm aware of (and I was following this pretty closely, as you might guess by my dozens of comments on the topic) was the addon installed via the studies system. That addon didn't suppress the re-validation of signatures, it installed a new certificate and then triggered the re-validation of signatures immediately. It left the validation that happens on a 24 hour cycle alone.
- KwanEsq 7y ago
  There was a hotfix before the one that installed a new cert: > hotfix-reset-xpi-verification-timestamp-1548973•Complete > This study sets app.update.lastUpdateTime.xpi-signature-verification to 1556945257.
  - gpm 7y ago
    Oh, you're completely right. Oops.
ohGoodie 7y ago
Obviously this is a bike shedding thread, and it will hang around for about 12 hours, and get hundreds of replies. And nothing will change, despite all the exasperation.
lllr_finger 7y ago
It might not have helped here, but updating certs every several years always leads to problems from my experience - people move on, processes get lost or outdated, etc. I'll accept the yearly annoyance to avoid those issues every time.
- Someone1234 7y ago
  Agreed. My initial reaction when Let's Encrypt had you re-issue every 90 days was negative, but I was wrong. Very wrong. A 90 day re-issue forces you to have working re-issue infrastructure and procedures, and therefore you're less likely to get stung by an accidental expiration. Long expirations are a trap, a very easy trap to fall into.
  - drtillberg 7y ago
    Why not update long-duration keys every 90 days? That way you're never close to expiration, ever, best of both worlds.
    - cyphar 7y ago
      Because if you make it optional then nobody does it (except you) and then you're back to square one. By mandating a 90-day expiry, LetsEncrypt forced people to automate the process -- and everyone is on the same page. It should be noted that most LetsEncrypt tools will renew a certificate when it is 30 days from expiry, so if you run the renew script every week (or day) you're also never close to expiry.
      - bscphil 7y ago
        >By mandating a 90-day expiry, LetsEncrypt forced people to automate the process -- and everyone is on the same page. Ironically, I had the opposite problem. I used to be on top of things like cert expirations, but now I just let certbot do everything. The problem (at least in my case) was that even though certbot updated the cert on time, it doesn't restart / reload nginx so that it picks up the new cert. My site was up for the full ~30 days between the renewal and the expiration of the old cert. So my site went down because of letsencrypt's cert renewal policy. (I now have a script set up that reload's nginx's configuration whenever the certificate is updated.)
ghostly_s 7y ago
If anyone's wondering if this answers the Actual Question of why the cert was allowed to expired, don't waste your time, it doesn't. I guess implicitly that's a "social detail"?
- Someone1234 7y ago
  What are you after? This clearly says they screwed up and that internal procedures need to be changed so it cannot recur. If this was in fact an oversight/accident, what other kind of explanation can they give? > This was due to an error on our end: we let one of the certificates used to sign add-ons expire And: > We clearly need to adjust our processes both to make this and similar incidents it less likely to happen and to make them easier to fix.
  - tedunangst 7y ago
    Well, they could tell us what the previous process was. Was the cert renewer on vacation, or was the wrong date entered on the calendar, or? How was it supposed to work? I think a lot of people are curious about that.
    - bpfrh 7y ago
      They said they will make a formal post-mortem investigation as well as make a post about what they will change. I think the information will be contained in one of the two posts?
      - tedunangst 7y ago
        Sure? But someone basically said there's nothing more to be learned or discussed.
    - c0vfefe 7y ago
      It's understandable to be curious about that, I am myself. But I don't think they owe us that. It might be sensitive information, anyway.
- 7y ago
Tsubasachan 7y ago
My heartfelt condolences to everyone who has had to browse the internet without an ad blocker. Nobody deserves that.
- sinsterizme 7y ago
  If nothing, this outage reminded me how valuable my adblocker really is. I got interrupted three times while watching a podcast, incredibly obnoxious.
- driverdan 7y ago
  This is why you should have multi-tier defenses. An ad blocker in the browser, blackholed domains in your hosts file, and use DNS servers that also blackholes ad domains.
  - BuckRogers 7y ago
    I used to do all 3 in the past, 4 with domain blocking done in router configuration, and am no longer a fan. It made troubleshooting any issue that may pop up too much of a hassle. Now I just use single-source blocking at the application layer. When Firefox disabled extensions, I simply used Edgium until they had it fixed.
  - cyborgx7 7y ago
    I forgot I had the hosts file, and was surprised by how little ads I was seeing when the add-on was disabled, until I realised the reason.
charliebrownau 7y ago
If Firefox are willing to break all addons to force everyone to stop using GAB dissenter addon What else are they willing to do as a "WOKE" company
shanxS 7y ago
Certificate expiration, if only we can figure way to see it coming. Oh wait, metrics!
keyle 7y ago
So my first thought was that a lot of ads would have been printed during that 24 hrs? Anyone notice a blip in revenues of sorts? I mean that's the most crucial add-on to ever work: ad blockers. When I visit a site with ad-blocker off I feel compelled to clear all caches and cookies and go have a shower.
- djsumdog 7y ago
  None of the ad companies are going to publicly talk about that data. They might have seen a blip, but I doubt it was anything major. Firefox doesn't have the market share it once did either.
toyg 7y ago
> users should be able to opt-in to updates (including hot-fixes) but opt out of everything else Finally some good news. This is what I suggested in one of the previous threads: there should be a delivery channel for important updates, and a channel for experiments/telemetry/whatnot. Some other HNer said it was an unrealistic expectation "because manpower". Guess what, it isn't. This is how things should always be.
- mook 7y ago
  They did have that capability (see https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-ons/Process https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-ons/Pr... ), at least the linked repository had commits in 2016. I can't tell from a 30 second search why that no longer works, just that it is a replacement of a similar previous capability.
  - sciurus 7y ago
    This started before my time at Mozilla, but I think several different systems grew out of the "go faster" initiative. Of them I believe the fastest and most flexible means of shipping changes is Normandy, which is primarily used for our shield studies. Like the blog post says, we need to revisit this. (Disclosure: I work for Mozilla)
drtillberg 7y ago
The Firefox update required an administrative login in my windows system at work, which I don't have have. Normal updates haven't required that. So far I've just left it broken and it keeps prompting me for an administrative login on launch. The article doesn't explain why elevated privileges are required to apply the update.
- kbrosnan 7y ago
  Does the current user have rights to write to the Firefox install location? If not then elevation is required to overwrite the files.
  - silversconfused 7y ago
    If normal updates do not require admin perms then it must not have been installed in a system location, and they must be doing something system-level with this particular update.
  - hcs 7y ago
    The current user may not need to have those rights if the maintenance service is installed, which allows the updater to run with higher privileges. I don't know of any reason why the recent updates would be any different, though. There may be an unrelated issue that broke "silent" updates for drtillberg. (I work on the Firefox updater and am a Mozilla employee)
jsjohnst 7y ago
What I don’t see mentioned anywhere is why the heck was this a last minute scramble? Don’t they know when their certificates are going to expire!?
- cheeze 7y ago
  Probably not. They probably know when most of their certs schedule but likely weren't monitoring this intermediate code signing cert (which is different than a TLS/webserver leaf/intermediate cert). It seems super simple to do, but in practice IMO is harder than it seems. Most major cloud providers have been hit by at least one cert expiry causing an outage in the past year... Hell, likely in the past month. This doesn't surprise me at all, certs are hard.
throw7 7y ago
i lost all my custom multi-account containers after getting a version that has the cert fixes (firefox is still broken on fedora; i had to download the testing version today to get it back working). either today's engineers are sub-standard or the foxes rule the henhouse.
- floatingatoll 7y ago
  If you disable the addon, restart Firefox, and enable the addon, do your custom containers get erased again?
  - lysp 7y ago
    It's my understand that they'd be deleted. Current bug for issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1549204 https://bugzilla.mozilla.org/show_bug.cgi?id=1549204 This was discovered after the expiring caused a similar process to kill data.
sohkamyung 7y ago
Related to this: Mozilla has deleted Telemetry data for those users who enabled Telemetry to get the hot-fix [1] [1] https://twitter.com/firefox/status/1126593558490693632 https://twitter.com/firefox/status/1126593558490693632
- deleted 7y ago
  [deleted]
- spectramax 7y ago
  I know Mozilla is not your typical for-profit organization, but it is so nice to see "We failed and we are sorry." It felt good. This post-mortem read like a genuine story of what happened and how they solved the problem. No lawyer language. No bullshit. I love Mozilla.
- driverdan 7y ago
  Thanks for sharing this. I didn't know they were doing that. This helps restore some of my faith.
- sohkamyung 7y ago
  Edit: a error on my part. Mozilla is deleting ALL Telemetry data collected during that time period, not just from certain users. From the post linked from the tweet [1] >In order to respect our users’ potential intentions as much as possible, based on our current set up, we will be deleting all of our source Telemetry and Studies data for our entire user population collected between 2019-05-04T11:00:00Z and 2019-05-11T11:00:00Z. [1] https://blog.mozilla.org/blog/2019/05/09/what-we-do-when-things-go-wrong/ https://blog.mozilla.org/blog/2019/05/09/what-we-do-when-thi...
ekianjo 7y ago
I love how they paint the picture that certificates "unfortunately expired" as if it were an act of god or something. Surely one cannot see it coming! No mention at all why nobody was checking certificates expiry.
- lifthrasiir 7y ago
  The post seems to imply that it was a simple overlook (which is frequent when every such thing is not formally tracked). I agree that it is hard. > We’ll be running a formal post-mortem next week and will publish the list of changes we intend to make, but in the meantime here are my initial thoughts about what we need to do. First, we should have a much better way of tracking the status of everything in Firefox that is a potential time bomb and making sure that we don’t find ourselves in a situation where one goes off unexpectedly. We’re still working out the details here, but at minimum we need to inventory everything of this nature.
  - ekianjo 7y ago
    > a simple overlook I'm sorry but when your business is to enable 30 000 extensions to work your business is also to check that certificates that enable such extensions don't fail tomorrow. That's the core of one's job, not even a side project or something.
    - lifthrasiir 7y ago
      I don't claim to know what actually happened, but one possible cause is that the HSM (required for issuing and renewing intermediate certificates, mentioned in the post) might require a right person and right schedule to operate. The use of an HSM means that you don't normally use it to issue certificates and you should only touch it a few times a year. As a result there may have been only a few people with an effective knowledge of the HSM; when they can't be scheduled for some various reasons, well, it may be forgotten. While it of course sucks, something like this happens anywhere anytime.
carmate383 7y ago
Hate on Google / Chrome / Proprietary / Lock-In / Whatever buzzword (tm) you want, but this would _never_ happen in Chrome. And it's the reason why Firefox will die to Chrome. BRING ON THE IN-DENIAL DOWNVOTES.
- dingo_bat 7y ago
  That's because they actually care about users. Mozilla only cares about all its millions of inconsequential hobby projects.
some13 7y ago
I switched to falkon and hope to never go back to firefox.
James_Owens_69 7y ago
Yeah, no Dissenter App is still banned because muh Nazis.
kwccoin 7y ago
Do they follow basic PKI best practice. Do they actually know (not after the fact) the certification path validation algorithm. It shall be auto. Is Firefox use the normal PKI authentication mechanism. Their reaction is like this is a surprise and even signing intermediate cert as the first step and instead of talking about bypass or hack the whole PKI trust chain. Based on some of the comments here, I think one has to understand that it is not just timestamp and validity. The checking of PKI is per transaction and on a continuous basis. It is NOT just based on signing but also based on CRL (cert. revocation list) which is also key. I read the blog a few time. I feel frightened not enlightened. It seems they are not on the ball. A minor mistake (forget to renewal cert. like O2 (not sure but heard same issue)) gave a lot of lights on issues. Do they have CPS even ... :-) or :-(((
zelpo 7y ago
No problem for me. The only add-on I use and recommend is Pocket. Pocket helps me keep my stuff organized, when im on the Go.
sara7262 7y ago
Who is Satan and why there is evil Best video about this with proofs https://youtu.be/AWNM1-xYI5E https://youtu.be/AWNM1-xYI5E Funny monkey using phone in video Don't think just you can use phone See how he is searching and watching own monkey videos on phone http://bit.ly/2DS4p2R http://bit.ly/2DS4p2R See what happened when a lion was trying to kill man http://bit.ly/2H2zsKc http://bit.ly/2H2zsKc Best video with proofs Which showing there is creator of this universe https://youtu.be/j5HHl6-r3VU https://youtu.be/j5HHl6-r3VU Dogs are meeting their owners after a long time see in this video the reaction of dogs and try to control your cry http://bit.ly/2JmjuNw http://bit.ly/2JmjuNw Funny cats are teasing babies in video just try to control your laugh I am sure you can't http://bit.ly/2LJu58n http://bit.ly/2LJu58n
jrochkind1 7y ago
Interestingly, it seems like it's not just that they didn't realize the certificate expire date was zooming past; they didn't even realize that particular certificate expiring was a thing. If they had noticed 6 months ago it was about to expire -- they still would have had to figure out what to do about it, there wasn't an actual documented procedure in place to swap in a new non-expired cert without disruption to FF users with existing signed add-ons installed. If they had known the expire date was upcoming, they just would have had more time to figure out what to do about it in a leisurely fashion. I wonder if the engineers who implemented the original system had a clear idea of what would be done when the cert expire approached and it just never got documented (or FF accidentally changed to make it no longer applicable?), or if they just figured they'd figure that out years later when the date approached.
PudgePacket 7y ago
> Note that this is the same logic we use for validating TLS certificates, so it’s relatively well understood code that we were able to leverage. Don't know whether to be happy or scared that the TLS validation code is "relatively well understood" :D ! I assume it's just a sub-optimal choice of phrasing.
- josteink 7y ago
  I’m scared that they (still?) confuse regular HTTPS TLS-validation with code-signing. These are two entirely different domains which follows entirely different rules. Current Firefox behaviour is still broken, even after the “fix”.
josteink 7y ago
This is basic code-signing and “time-stamping” is not mentioned once in the post-mortem, despite lack of that was exactly what caused this issue in the first place. That’s not inspiring a lot of confidence, to be honest.
known 7y ago
Catalog of classic Firefox add-ons created before WebExtensions apocalypse https://github.com/JustOff/ca-archive https://github.com/JustOff/ca-archive
jokoon 7y ago
I hope this was not a government attempt to snoop and that mozilla did not miss any detail...
cyborgx7 7y ago
> We clearly need to adjust our processes both to make this and similar incidents it less likely to happen and to make them easier to fix. I knew it. They are going to use THEIR fuck-up to justify why they need even more remote control and access to firefox installed on users machines.
koliber 7y ago
I don't understand one thing. How were they able to generate a new certificate with the same subject, a different expiration date, and the same signature. I assume I missed something in reading this.
- cyborgx7 7y ago
  It's a new Date but still the same key. So the signature of the root certificate for the intermediate certificate is different, but the signature the intermediate key generates are the same.
pcvarmint 7y ago
Why is no-one talking about the Dissenter ban? Isn't that what prompted this whole fiasco? [0][1][2] 0. https://elgan.com/blog/why-mozilla-and-google-are-wrong-to-ban-gabs-dissenter-extension https://elgan.com/blog/why-mozilla-and-google-are-wrong-to-b... 1. https://discourse.mozilla.org/t/the-removal-of-the-dissenter-extention/38140/13 https://discourse.mozilla.org/t/the-removal-of-the-dissenter... 2. https://www.youtube.com/watch?v=f0Cc8RpqH1g https://www.youtube.com/watch?v=f0Cc8RpqH1g
johnchristopher 7y ago
> Even on Monday, we still had some users who hadn’t picked up either the hotfix or the dot release, which clearly isn’t ideal. My kubuntu boxes had the patch wednesday, not before.

467 points by headalgorithm 7y ago

>First, I want to say that the team here did amazing work: they built and shipped a fix in less than 12 hours from the initial report. As someone who sat in the meeting where it happened, I can say that people were working incredibly hard in a tough situation and that very little time was wasted. It's a bit disheartening to see the "Lessons" section opened with this. I understand that everyone worked very hard to get things up and running - but that is not a lesson learned. That is self-back-patting.

dblohm7 7y ago
The post-mortem hasn't even started yet; now that 66.0.5 is out and the initial incident response is finally winding down, now they can move on to that.
- ziddoap 7y ago
  I don't quite follow how this reply relates to the first paragraph under the "Lessons" section of the linked article.
fwip 7y ago
It's context for the next paragraph. > With that said, obviously this isn’t an ideal situation and it shouldn’t have happened in the first place. We clearly need to adjust our processes both to make this and similar incidents it less likely to happen and to make them easier to fix.
- ziddoap 7y ago
  I did gather that much. In my _personal opinion_, it would have been much more palatable to start the lessons learned section with >We’ll be running a formal post-mortem next week and will publish the list of changes we intend to make, but in the meantime here are my initial thoughts about what we need to do. First, we should have a much better way of tracking And shoe-horn the back-patting in earlier or later. Perhaps I'm being pedantic. Fair enough. But for me, I would rather see this issue addressed in a way that the severity of the situation requires. Back-patting is not on the plate for me.
shawnz 7y ago
It's clearly just a disclaimer to show his respectful intent, since the paragraphs following that are addressing failures of his colleagues.
- ziddoap 7y ago
  Understood, and I explained a bit more of my thoughts in another comment. I understand I'm being a bit pedantic. But, as a user who is pretty concerned with a CA having a certificate issue of this scale, which affected this many people, I don't care about the authors respectful intent to their colleagues. Tell them in person, in an internal memo, whatever. It's simply not a lesson learned.
  - shawnz 7y ago
    Likewise, I'm sure he cares more about keeping good working relationships with his colleagues than about what the reader thinks of his choice of paragraph order. A manager who makes his team feel insulted is not conducive to producing better software (which is presumably what you care most about here).
    - ziddoap 7y ago
      How is laying out, in a technical fashion, what went wrong and the lessons learned insulting his colleagues? I'm very curious. Just because someone made a mistake, and that mistake is acknowledged, doesn't mean they are insulting them. Also, how come it can't be done in an internal memo, email, party, pay bonus, raises, etc.?
  - tialaramex 7y ago
    Mozilla, unlike all of the other major Trust Stores does not actually operate a publicly trusted CA. Microsoft, Apple, and Google all do, the biggest Trust Store I can think of that does NOT operate a CA other than Mozilla is Oracle (who manage the default Java trust store) This puts Mozilla in a good situation of independence IMNSHO. At Google we know it takes actual effort to make sure that corporate firewalls protect the Trust Store function from being sabotaged by the convenience of, say, Google's Cloud function or people in their SRE wanting to keep the lights on for Chrome users. One day Google's CA people accidentally minted certs that Chrome wouldn't trust for Google web sites, and there had to be mechanisms in place to ensure that "Just tell the Chrome engineers to ignore it" wasn't a viable fix, because that would be _easier_ but it destroys any value in that trust store.

parliament32 7y ago

>An important feature here is that the new certificate has the same subject name and public key as the old certificate, so that its signature on the End-Entity certificate is valid. Shouldn't it be impossible to generate a new cert (with a different expiry date) that ends up having the same public key as an existing cert?

bityard 7y ago
No, one public/private key pair can be used to generate or sign as many certificates as you like, as often as you like.
- parliament32 7y ago
  No I'm not talking about the root, I mean they generated a new certificate (the intermediate) (with a new private key) that had a public key identical to an existing certificate -- you shouldn't be able to do this, public keys can't be "specified" afaik, they're derived from your private key and the signer's public key.
  - rossdavidh 7y ago
    Isn't that the point of having the root cert and the intermediate? Otherwise, why not just have everything keyed to the root?
    - rndgermandude 7y ago
      Because a root typically has a longer expire date and thus has to be stored extra secure and handled extra safely. You'd usually store it in some hardware security module that is operated by an airgapped system or something like that that requires physical access at the very least. Replacing the root often requires shipping new versions of your software that disables the old root and bakes in a new root. If an intermediate certificate becomes compromised, you can revoke it and issue a new intermediate certificate with your still secure root without the need to push out new binaries.
  - 7y ago

chrisseaton 7y ago

After the last PR disaster - the Mr Robot tie-in - one of the ways Mozilla tried to make it right was that they promised that the survey system would never again be used for something that was not an experiment. https://blog.mozilla.org/firefox/retrospective-looking-glass/ https://blog.mozilla.org/firefox/retrospective-looking-glass... > A SHIELD study must be designed to answer a specific question. Why have they abused it again here to deploy a hot fix, breaking their promise and policy that they put in place last time they messed up? Or am I ignorant or some part of the story or technical details.

mlthoughts2018 7y ago
It is indeed a huge violation of trust. I’ve seen some folks trying to explain it away and compare the Normandy preference system to standard auto-updates, acting like this is no problem if you already trust them for auto-updates. Flat wrong. This is a dark pattern by Mozilla pure & simple. It’s confusing, hard to disable fully, and clearly can be abused for non-experiment modifications to the user’s settings.
- silversconfused 7y ago
  Mozilla has gone full "helicopter parent" mode, and will do whatever they can to keep you safe from yourself despite your wishes. Use GNU icecat instead. It has all that junk turned off.
fedups 7y ago
Do they have a better channel for deploying hot-fixes? Maybe I'm a bit of a squish but I'm pretty agnostic of how they deploy fixes to me so long as my addons start working again.
- deleted 7y ago
  [deleted]
- bwat49 7y ago
  They don't currently, but adding a better channel to deploy hotfixes is something that's specifically mentioned in this article...

jzl 7y ago

Looks like a good read. I haven't finished reading it yet, but there's something I still don't get ... Windows and macOS both have a signing infrastructure for apps. The rules of that infrastructure dictate only that apps must have been signed by a valid certificate at the time they were signed. That way old app downloads don't need to be periodically re-signed just to account for expiring certificates. I can download a 5-year-old version of 7zip or whatever and it runs just fine because it was signed with something valid to the timestamp in the signature. The process of distributing desktop apps would be utterly insane if this were not the case. Not following this model for browser plugins seems unnecessarily cumbersome. Is it really worth requiring all browser plugins to be signed by a currently valid certificate? Is there a document or blog post where this is argued to be more appropriate? I get that it arguably leads to more stringent security, but I'm not convinced by the delta improvement of that model over the desktop model, given the additional downsides. And the "let everything expire after a few years and resign it" process should not be used as a substitution for revocation. After all, if it were determined retroactively that a malware extension was signed does it really help everyone that it can't be loaded in a year or two, given the damage it could cause right now?

johncolanduoni 7y ago
For the time stamp method to work, you need a trusted mechanism to attest that the timestamp is correct, otherwise the mechanism is useless (an attacker with an outdated private key can just backdate the timestamp in the executable and then sign it). Windows code signing uses a server Microsoft runs to provide this, and Mozilla would need to do the same. I’m not saying they shouldn’t, but it is a significant piece of complexity.
- reificator 7y ago
  > Windows code signing uses a server Microsoft runs to provide this, and Mozilla would need to do the same. Mozilla already requires uploading even your private extensions to a Mozilla server to be signed for internal or external deployment. https://wiki.mozilla.org/Add-ons/Extension_Signing https://wiki.mozilla.org/Add-ons/Extension_Signing
- 7y ago

silversconfused 7y ago

For anyone who misses the old days when a browser only did what you told it to, here is a solution: GNU icecat is a nice firefox esr fork with the mozilla call-home bits all turned off by default. It's very pleasant to use.

djsumdog 7y ago
It doesn't have the new Quantum rendering engine though, right?
- silversconfused 7y ago
  I'm pretty sure it does. https://en.wikipedia.org/wiki/Quantum_(Mozilla) https://en.wikipedia.org/wiki/Quantum_(Mozilla) says quantum shipped in 57.
bscphil 7y ago
There's also GNU "abrowser", which (though terribly named) has the advantage of being based on the latest Firefox release rather than esr. It has the same privacy defaults as Icecat (though I believe it doesn't ship with the addons that Icecat does).

sciurus 7y ago

There is a related blog post about what Mozilla is doing with the data collected from users who enabled Studies in order to get the hot fix. https://blog.mozilla.org/blog/2019/05/09/what-we-do-when-things-go-wrong/ https://blog.mozilla.org/blog/2019/05/09/what-we-do-when-thi... TL;DR is "In order to respect our users’ potential intentions as much as possible, based on our current set up, we will be deleting all of our source Telemetry and Studies data for our entire user population collected between 2019-05-04T11:00:00Z and 2019-05-11T11:00:00Z." (Disclaimer: I work for Mozilla)

jzl 7y ago
As someone who reluctantly re-enabled Studies to get the fix, I appreciate this.

hartator 7y ago

> Second, we immediately pushed a hotfix which suppressed re-validating the signatures on add-ons. What’s the point of re-validating of installed add-on in the first place then?

acqq 7y ago

> Second, we need a mechanism to be able to quickly push updates to our users even when — especially when — everything else is down. They should simply streamline the "normal" updates for that purpose, not invent the new "channels." Specifically, not all binaries in the directory should be changed just to push an update where only few lines of the code are different.

hartator 7y ago

> because they want to run old-style add-ons, but many of these now work with newer versions of Firefox. Except it wasn’t working for this few days. Anyway older versions of Firefox wasn’t signing add-on, so they are not concerned by this fix.

silversconfused 7y ago
Wrong. I lost my addons in firefox 60 esr, which I got from debian. I'm done with mozilla disrespecting my preferences, and have switched to waterfox and icecat. Remote control of the browser is simply the last straw.

tedunangst 7y ago

> For the other groups we are developing a patch to Firefox that will install the new certificate once people update. This was released as a “dot release” so people will get it — and probably have already — through the ordinary update channel. If you have a downstream build, you’ll need to wait for your build maintainer to update. Why not link to the xpi that can be installed now?

sciurus 7y ago
FYI that should read "we have developed" instead of "we are developing". Firefox 66.0.4 and later have the patch. (Disclaimer: I work for Mozilla)
yellowapple 7y ago
> Why not link to the xpi that can be installed now? This is the crux of my remaining frustration with how Mozilla handled this issue. That XPI should've been front-and-center in all the articles that detailed the fix. And yet, instead of something like... "If you have Studies enabled, a fix should apply automatically. If it hasn't yet, or if you have Studies turned off (or are using a version which does not support Studies), you can install the hotfix add-on [here](URL to XPI)." ...pretty much all the official messaging ended up like so: "If you have Studies enabled, a fix should apply automatically. It may take up to 6 hours; please be patient and wait for it. If you don't want to (or can't) turn on Studies, you're SOL until we push out a point release (and further SOL if you're at the mercy of a Linux package maintainer or you want to use a version of Firefox that still supports XUL-based addons)." The notion that this was a deliberate ploy to get more people to turn on Studies is surely conspiracy-theorist mumbo-jumbo, but nonsense like this makes me wonder.
- callahad 7y ago
  > The notion that this was a deliberate ploy to get more people to turn on Studies is surely conspiracy-theorist mumbo-jumbo, but nonsense like this makes me wonder. This is not the case. Please see my response downthread: https://news.ycombinator.com/item?id=19872490 https://news.ycombinator.com/item?id=19872490

jeffchien 7y ago

I would've liked to see some reflection or even just acknowledgment about the fact that they intentionally disabled the "xpinstall.signatures.required" setting on Windows and OSX. I hope it's at least in the formal postmortem.

callahad 7y ago
That's covered at https://blog.mozilla.org/addons/2015/04/15/the-case-for-extension-signing/ https://blog.mozilla.org/addons/2015/04/15/the-case-for-exte..., which is linked from ekr's post. I encourage you to read the rationale as a whole, but the specific question you're asking is addressed here: > Many developers have asked why we can’t make this a runtime option or preference. There is nowhere we could store that choice on the user’s machine that these greyware apps couldn’t change and plausibly claim they were acting on behalf of the user’s “choice” not to opt-out of the light grey checkbox on page 43 of their EULA. This is not a concern about hypotheticals, we have many documented cases of add-ons disabling the mechanisms through which we inform users and give them control over their add-ons. By baking the signing requirement into the executable these programs will either have to submit to our review process or take the blatant malware step of replacing or altering Firefox. We are sure some will take that step, but it won’t be an attractive option for a Fortune 500 plugin vendor, popular download sites, or the laptop vendor involved in distributing Superfish. For the ones who do, we hope that modifying another program’s executable code is blatant enough that security software vendors will take action and stop letting these programs hide behind terms buried in their user-hostile EULAs.
- Wowfunhappy 7y ago
  Does Windows et al really not provide a mechinism for saving privileged settings in a tamper-resistant way? I frankly find that hard to believe. How does other software solve this problem? There are of course always workarounds on an open platform like Windows/Mac/Linux, but the threshold isn’t “impossible”, it’s just “as difficult as injecting into the browser’s code.” Edit: For example, what if the config file contained a checksum of the file's contents + the user's hardware? If the setting is changed by the user within Firefox, the checksum is updated and everything works—if the checksum is invalid, settings are reset to the default. Video games will occasionally do this type of thing with their config files. Modders often figure out the formula—but again, the idea here isn't to make editing the file impossible, it's to make it as difficult as injecting into the Firefox executable.

DangerousPie 7y ago

My point of view as a long-time Firefox user that cares about privacy but also knows we live in an imperfect world: It obviously sucks that this happened but I think they handled it very well. The bug was fixed so quickly that I wouldn't even have realized it had happened if it hadn't been for the thread here on HN. My extensions hadn't even been disabled yet by the time the patch came out. And pushing out the hotfix through studies followed by a new version probably ensured that a large fraction of the "average joe" userbase didn't even realize there was a problem. So obviously there are some improvements to make for the future but I think some of the criticism over the last few days has been a bit harsh. Firefox is still my preferred browser by far.

SECProto 7y ago
> The bug was fixed so quickly that I wouldn't even have realized it had happened if it hadn't been for the thread here on HN Maybe this is a timezone thing, but I was in East Asia, and I had to deal with the internet for close to 36 hrs (android) with no ublock. It was almost enough to look for a new browser (but browsers with adblock on Android are few and far between - so instead I just didn't use the internet as much for a day or two.). Part of that delay was play store being slow to push it, as I recall seeing the binaries somewhere a while sooner.
- yellowapple 7y ago
  One option would've been to use Nightly and set xpinstall.signatures.required = false in about:config. That's exactly what I did.
  - SECProto 7y ago
    I view using nightly to be as much of a barrier to entry that I might as well switch to a totally different browser. If the issue had lasted longer, I would have found (someone on the internet who had found) a solution like that.
    - yellowapple 7y ago

Faark 7y ago

> In theory, fixing a problem like this looks simple: make a new, valid certificate and republish every add-on with that certificate. Unfortunately, [..] I'd expect addon usage to follow a pareto distribution. Thus resigning the most important ones would have helped a lot of users. Why didn't they start going this route anyway? Not enough manpower for this to not diverted resources from the other more important fixes?

callahad 7y ago
I expect the full postmortem will dive into this in more detail, but one reason mentioned in ekr's post is that many add-ons are distributed independently, outside of AMO. So while we could re-sign those, we're not in a place to re-distribute them. Instead, "users would have to manually update any add-ons that they had installed from other sources, which would be very inconvenient."
sciurus 7y ago
I think this is covered in the blog post, but I'll take another stab at explaining it from my perspective: Republishing was one of the options we were investigating early on. However, the problem is that it only fixes things once you check for and install the updated version signed with the new certificate. Firefox would still have disabled your installed version that had an expired certificate. Firefox checks for addon updates every 24 hours, but it checks in with Normandy every 6 hours. Thus once we had stopgap fixes shipping to users via Normandy, and a proper fix of a new Firefox release in progress, republishing addons wasn't necessary. (Disclaimer: I work for Mozilla)
- Faark 7y ago
  Thanks for your response. An update being deactivated doesn't trigger an update check for said addon? Well, complexity of a graceful update attempt for a not loadable addon probably outweighs the benefit of such rare cases. But the main reason I had a hard time with you guys discarding this path is probably addon stats like [0]. The spike in downloads made me think it could have helped at least some users. But on second thought, it might also have been caused by extensions getting re-enabled post fix. 2m downloads at 4m DAU makes me guess this was caused by something automatic. [0] https://addons.mozilla.org/en-US/firefox/addon/ublock-origin/statistics/?last=30 https://addons.mozilla.org/en-US/firefox/addon/ublock-origin...

gpm 7y ago

THIS IS WRONG: Preserved to make below reply not seem out of context. > Second, we immediately pushed a hotfix which suppressed re-validating the signatures on add-ons. Wait, that's not right is it!? The only hotfix I'm aware of (and I was following this pretty closely, as you might guess by my dozens of comments on the topic) was the addon installed via the studies system. That addon didn't suppress the re-validation of signatures, it installed a new certificate and then triggered the re-validation of signatures immediately. It left the validation that happens on a 24 hour cycle alone.

KwanEsq 7y ago
There was a hotfix before the one that installed a new cert: > hotfix-reset-xpi-verification-timestamp-1548973•Complete > This study sets app.update.lastUpdateTime.xpi-signature-verification to 1556945257.
- gpm 7y ago
  Oh, you're completely right. Oops.

ohGoodie 7y ago

Obviously this is a bike shedding thread, and it will hang around for about 12 hours, and get hundreds of replies. And nothing will change, despite all the exasperation.

lllr_finger 7y ago

It might not have helped here, but updating certs every several years always leads to problems from my experience - people move on, processes get lost or outdated, etc. I'll accept the yearly annoyance to avoid those issues every time.

Someone1234 7y ago
Agreed. My initial reaction when Let's Encrypt had you re-issue every 90 days was negative, but I was wrong. Very wrong. A 90 day re-issue forces you to have working re-issue infrastructure and procedures, and therefore you're less likely to get stung by an accidental expiration. Long expirations are a trap, a very easy trap to fall into.
- drtillberg 7y ago
  Why not update long-duration keys every 90 days? That way you're never close to expiration, ever, best of both worlds.
  - cyphar 7y ago
    Because if you make it optional then nobody does it (except you) and then you're back to square one. By mandating a 90-day expiry, LetsEncrypt forced people to automate the process -- and everyone is on the same page. It should be noted that most LetsEncrypt tools will renew a certificate when it is 30 days from expiry, so if you run the renew script every week (or day) you're also never close to expiry.
    - bscphil 7y ago
      >By mandating a 90-day expiry, LetsEncrypt forced people to automate the process -- and everyone is on the same page. Ironically, I had the opposite problem. I used to be on top of things like cert expirations, but now I just let certbot do everything. The problem (at least in my case) was that even though certbot updated the cert on time, it doesn't restart / reload nginx so that it picks up the new cert. My site was up for the full ~30 days between the renewal and the expiration of the old cert. So my site went down because of letsencrypt's cert renewal policy. (I now have a script set up that reload's nginx's configuration whenever the certificate is updated.)

ghostly_s 7y ago

If anyone's wondering if this answers the Actual Question of why the cert was allowed to expired, don't waste your time, it doesn't. I guess implicitly that's a "social detail"?

Someone1234 7y ago
What are you after? This clearly says they screwed up and that internal procedures need to be changed so it cannot recur. If this was in fact an oversight/accident, what other kind of explanation can they give? > This was due to an error on our end: we let one of the certificates used to sign add-ons expire And: > We clearly need to adjust our processes both to make this and similar incidents it less likely to happen and to make them easier to fix.
- tedunangst 7y ago
  Well, they could tell us what the previous process was. Was the cert renewer on vacation, or was the wrong date entered on the calendar, or? How was it supposed to work? I think a lot of people are curious about that.
  - bpfrh 7y ago
    They said they will make a formal post-mortem investigation as well as make a post about what they will change. I think the information will be contained in one of the two posts?
    - tedunangst 7y ago
      Sure? But someone basically said there's nothing more to be learned or discussed.
  - c0vfefe 7y ago
    It's understandable to be curious about that, I am myself. But I don't think they owe us that. It might be sensitive information, anyway.
7y ago

Tsubasachan 7y ago

My heartfelt condolences to everyone who has had to browse the internet without an ad blocker. Nobody deserves that.

sinsterizme 7y ago
If nothing, this outage reminded me how valuable my adblocker really is. I got interrupted three times while watching a podcast, incredibly obnoxious.
driverdan 7y ago
This is why you should have multi-tier defenses. An ad blocker in the browser, blackholed domains in your hosts file, and use DNS servers that also blackholes ad domains.
- BuckRogers 7y ago
  I used to do all 3 in the past, 4 with domain blocking done in router configuration, and am no longer a fan. It made troubleshooting any issue that may pop up too much of a hassle. Now I just use single-source blocking at the application layer. When Firefox disabled extensions, I simply used Edgium until they had it fixed.
- cyborgx7 7y ago
  I forgot I had the hosts file, and was surprised by how little ads I was seeing when the add-on was disabled, until I realised the reason.

charliebrownau 7y ago

If Firefox are willing to break all addons to force everyone to stop using GAB dissenter addon What else are they willing to do as a "WOKE" company

shanxS 7y ago

Certificate expiration, if only we can figure way to see it coming. Oh wait, metrics!

keyle 7y ago

So my first thought was that a lot of ads would have been printed during that 24 hrs? Anyone notice a blip in revenues of sorts? I mean that's the most crucial add-on to ever work: ad blockers. When I visit a site with ad-blocker off I feel compelled to clear all caches and cookies and go have a shower.

djsumdog 7y ago
None of the ad companies are going to publicly talk about that data. They might have seen a blip, but I doubt it was anything major. Firefox doesn't have the market share it once did either.

toyg 7y ago

> users should be able to opt-in to updates (including hot-fixes) but opt out of everything else Finally some good news. This is what I suggested in one of the previous threads: there should be a delivery channel for important updates, and a channel for experiments/telemetry/whatnot. Some other HNer said it was an unrealistic expectation "because manpower". Guess what, it isn't. This is how things should always be.

mook 7y ago
They did have that capability (see https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-ons/Process https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-ons/Pr... ), at least the linked repository had commits in 2016. I can't tell from a 30 second search why that no longer works, just that it is a replacement of a similar previous capability.
- sciurus 7y ago
  This started before my time at Mozilla, but I think several different systems grew out of the "go faster" initiative. Of them I believe the fastest and most flexible means of shipping changes is Normandy, which is primarily used for our shield studies. Like the blog post says, we need to revisit this. (Disclosure: I work for Mozilla)

drtillberg 7y ago

The Firefox update required an administrative login in my windows system at work, which I don't have have. Normal updates haven't required that. So far I've just left it broken and it keeps prompting me for an administrative login on launch. The article doesn't explain why elevated privileges are required to apply the update.

kbrosnan 7y ago
Does the current user have rights to write to the Firefox install location? If not then elevation is required to overwrite the files.
- silversconfused 7y ago
  If normal updates do not require admin perms then it must not have been installed in a system location, and they must be doing something system-level with this particular update.
- hcs 7y ago
  The current user may not need to have those rights if the maintenance service is installed, which allows the updater to run with higher privileges. I don't know of any reason why the recent updates would be any different, though. There may be an unrelated issue that broke "silent" updates for drtillberg. (I work on the Firefox updater and am a Mozilla employee)

jsjohnst 7y ago

What I don’t see mentioned anywhere is why the heck was this a last minute scramble? Don’t they know when their certificates are going to expire!?

cheeze 7y ago
Probably not. They probably know when most of their certs schedule but likely weren't monitoring this intermediate code signing cert (which is different than a TLS/webserver leaf/intermediate cert). It seems super simple to do, but in practice IMO is harder than it seems. Most major cloud providers have been hit by at least one cert expiry causing an outage in the past year... Hell, likely in the past month. This doesn't surprise me at all, certs are hard.

throw7 7y ago

i lost all my custom multi-account containers after getting a version that has the cert fixes (firefox is still broken on fedora; i had to download the testing version today to get it back working). either today's engineers are sub-standard or the foxes rule the henhouse.

floatingatoll 7y ago
If you disable the addon, restart Firefox, and enable the addon, do your custom containers get erased again?
- lysp 7y ago
  It's my understand that they'd be deleted. Current bug for issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1549204 https://bugzilla.mozilla.org/show_bug.cgi?id=1549204 This was discovered after the expiring caused a similar process to kill data.

sohkamyung 7y ago

Related to this: Mozilla has deleted Telemetry data for those users who enabled Telemetry to get the hot-fix [1] [1] https://twitter.com/firefox/status/1126593558490693632 https://twitter.com/firefox/status/1126593558490693632

deleted 7y ago
[deleted]
spectramax 7y ago
I know Mozilla is not your typical for-profit organization, but it is so nice to see "We failed and we are sorry." It felt good. This post-mortem read like a genuine story of what happened and how they solved the problem. No lawyer language. No bullshit. I love Mozilla.
driverdan 7y ago
Thanks for sharing this. I didn't know they were doing that. This helps restore some of my faith.
sohkamyung 7y ago
Edit: a error on my part. Mozilla is deleting ALL Telemetry data collected during that time period, not just from certain users. From the post linked from the tweet [1] >In order to respect our users’ potential intentions as much as possible, based on our current set up, we will be deleting all of our source Telemetry and Studies data for our entire user population collected between 2019-05-04T11:00:00Z and 2019-05-11T11:00:00Z. [1] https://blog.mozilla.org/blog/2019/05/09/what-we-do-when-things-go-wrong/ https://blog.mozilla.org/blog/2019/05/09/what-we-do-when-thi...

ekianjo 7y ago

I love how they paint the picture that certificates "unfortunately expired" as if it were an act of god or something. Surely one cannot see it coming! No mention at all why nobody was checking certificates expiry.

lifthrasiir 7y ago
The post seems to imply that it was a simple overlook (which is frequent when every such thing is not formally tracked). I agree that it is hard. > We’ll be running a formal post-mortem next week and will publish the list of changes we intend to make, but in the meantime here are my initial thoughts about what we need to do. First, we should have a much better way of tracking the status of everything in Firefox that is a potential time bomb and making sure that we don’t find ourselves in a situation where one goes off unexpectedly. We’re still working out the details here, but at minimum we need to inventory everything of this nature.
- ekianjo 7y ago
  > a simple overlook I'm sorry but when your business is to enable 30 000 extensions to work your business is also to check that certificates that enable such extensions don't fail tomorrow. That's the core of one's job, not even a side project or something.
  - lifthrasiir 7y ago
    I don't claim to know what actually happened, but one possible cause is that the HSM (required for issuing and renewing intermediate certificates, mentioned in the post) might require a right person and right schedule to operate. The use of an HSM means that you don't normally use it to issue certificates and you should only touch it a few times a year. As a result there may have been only a few people with an effective knowledge of the HSM; when they can't be scheduled for some various reasons, well, it may be forgotten. While it of course sucks, something like this happens anywhere anytime.

carmate383 7y ago

Hate on Google / Chrome / Proprietary / Lock-In / Whatever buzzword (tm) you want, but this would _never_ happen in Chrome. And it's the reason why Firefox will die to Chrome. BRING ON THE IN-DENIAL DOWNVOTES.

dingo_bat 7y ago
That's because they actually care about users. Mozilla only cares about all its millions of inconsequential hobby projects.

some13 7y ago

I switched to falkon and hope to never go back to firefox.

James_Owens_69 7y ago

Yeah, no Dissenter App is still banned because muh Nazis.

kwccoin 7y ago

Do they follow basic PKI best practice. Do they actually know (not after the fact) the certification path validation algorithm. It shall be auto. Is Firefox use the normal PKI authentication mechanism. Their reaction is like this is a surprise and even signing intermediate cert as the first step and instead of talking about bypass or hack the whole PKI trust chain. Based on some of the comments here, I think one has to understand that it is not just timestamp and validity. The checking of PKI is per transaction and on a continuous basis. It is NOT just based on signing but also based on CRL (cert. revocation list) which is also key. I read the blog a few time. I feel frightened not enlightened. It seems they are not on the ball. A minor mistake (forget to renewal cert. like O2 (not sure but heard same issue)) gave a lot of lights on issues. Do they have CPS even ... :-) or :-(((

zelpo 7y ago

No problem for me. The only add-on I use and recommend is Pocket. Pocket helps me keep my stuff organized, when im on the Go.

sara7262 7y ago

Who is Satan and why there is evil Best video about this with proofs https://youtu.be/AWNM1-xYI5E https://youtu.be/AWNM1-xYI5E Funny monkey using phone in video Don't think just you can use phone See how he is searching and watching own monkey videos on phone http://bit.ly/2DS4p2R http://bit.ly/2DS4p2R See what happened when a lion was trying to kill man http://bit.ly/2H2zsKc http://bit.ly/2H2zsKc Best video with proofs Which showing there is creator of this universe https://youtu.be/j5HHl6-r3VU https://youtu.be/j5HHl6-r3VU Dogs are meeting their owners after a long time see in this video the reaction of dogs and try to control your cry http://bit.ly/2JmjuNw http://bit.ly/2JmjuNw Funny cats are teasing babies in video just try to control your laugh I am sure you can't http://bit.ly/2LJu58n http://bit.ly/2LJu58n

jrochkind1 7y ago

Interestingly, it seems like it's not just that they didn't realize the certificate expire date was zooming past; they didn't even realize that particular certificate expiring was a thing. If they had noticed 6 months ago it was about to expire -- they still would have had to figure out what to do about it, there wasn't an actual documented procedure in place to swap in a new non-expired cert without disruption to FF users with existing signed add-ons installed. If they had known the expire date was upcoming, they just would have had more time to figure out what to do about it in a leisurely fashion. I wonder if the engineers who implemented the original system had a clear idea of what would be done when the cert expire approached and it just never got documented (or FF accidentally changed to make it no longer applicable?), or if they just figured they'd figure that out years later when the date approached.

PudgePacket 7y ago

> Note that this is the same logic we use for validating TLS certificates, so it’s relatively well understood code that we were able to leverage. Don't know whether to be happy or scared that the TLS validation code is "relatively well understood" :D ! I assume it's just a sub-optimal choice of phrasing.

josteink 7y ago
I’m scared that they (still?) confuse regular HTTPS TLS-validation with code-signing. These are two entirely different domains which follows entirely different rules. Current Firefox behaviour is still broken, even after the “fix”.

josteink 7y ago

This is basic code-signing and “time-stamping” is not mentioned once in the post-mortem, despite lack of that was exactly what caused this issue in the first place. That’s not inspiring a lot of confidence, to be honest.

known 7y ago

Catalog of classic Firefox add-ons created before WebExtensions apocalypse https://github.com/JustOff/ca-archive https://github.com/JustOff/ca-archive

jokoon 7y ago

I hope this was not a government attempt to snoop and that mozilla did not miss any detail...

cyborgx7 7y ago

> We clearly need to adjust our processes both to make this and similar incidents it less likely to happen and to make them easier to fix. I knew it. They are going to use THEIR fuck-up to justify why they need even more remote control and access to firefox installed on users machines.

koliber 7y ago

I don't understand one thing. How were they able to generate a new certificate with the same subject, a different expiration date, and the same signature. I assume I missed something in reading this.

cyborgx7 7y ago
It's a new Date but still the same key. So the signature of the root certificate for the intermediate certificate is different, but the signature the intermediate key generates are the same.

pcvarmint 7y ago

Why is no-one talking about the Dissenter ban? Isn't that what prompted this whole fiasco? [0][1][2] 0. https://elgan.com/blog/why-mozilla-and-google-are-wrong-to-ban-gabs-dissenter-extension https://elgan.com/blog/why-mozilla-and-google-are-wrong-to-b... 1. https://discourse.mozilla.org/t/the-removal-of-the-dissenter-extention/38140/13 https://discourse.mozilla.org/t/the-removal-of-the-dissenter... 2. https://www.youtube.com/watch?v=f0Cc8RpqH1g https://www.youtube.com/watch?v=f0Cc8RpqH1g

johnchristopher 7y ago

> Even on Monday, we still had some users who hadn’t picked up either the hotfix or the dot release, which clearly isn’t ideal. My kubuntu boxes had the patch wednesday, not before.