36 ms·

All extensions disabled due to expiration of intermediate signing cert

• phyzome 8y ago
  Still working for me in my main profile in Firefox 60.6.1esr. I could even install an update to HTTPS Everywhere! When I opened my alt profile, extensions still worked, but I can't install updates—including for HTTPS Everywhere. What's up with that?
• xvector 8y ago
  Looks like all extensions have been disabled for all Firefox users. I think this fail-closed behavior is more of a security issue than the one it is trying to solve. All of my security add-ons - Privacy Badger, NoScript, Decentraleyes, and many more were disabled. Even worse, it happened without notice to the user. One moment I was browsing the internet (just barely) secured by these add-ons, and the next moment, all of them disappeared (without warning) and I only noticed when I saw my password manager was missing.
  • dontbenebby 8y ago
    Mine still work. I tried to set my clock to two days ago to avoid it and promptly got errors on every HTTPS site I visited. Damned if you do, damned if you don't :/
  • stevenwliao 8y ago
    If it failed open, anyone unlucky enough to update their extensions could end up having a malicious version installed. It also would have taken longer to notice.
    • tedunangst 8y ago
      Updating with an expired cert doesn't automatically result in compromise.
      • Thorrez 8y ago
        Yes, but what's the point of cert expiration? Is it safe to have certs that never expire? I believe there is a security benefit to expiration. Expiration is useless if it's never enforced. Probably the correct behavior is to have some sort of semi-annoying popup when it expires, and then only a week later do the full blocking. You need to strike the right balance of making it annoying enough that it can't be ignored by everyone (otherwise you just have the exact same problem, just delayed a week) and that fear of it happening is a sufficient motivator to stop people lazily relying on the grace period, but also not too annoying that it makes a lot of people quit. You also want to avoid permission fatigue.
    • Causality1 8y ago
      So why not just disable extension updates instead of disabling the extensions themselves?
      • anfilt 8y ago
        because that would make too much sense.
      • ssadler 8y ago
        Presumably because how would it differentiate between a legit "already installed" extension with a signature that cannot be verified, and an extension installed by malware that also cannot be verified?
        • Causality1 8y ago
          Personally I despise the idea of the software already on my pc being dependent on signatures stored on a remote server. I installed it and Mozilla can fuck right off. It's my responsibility to police what software is on my computer, not theirs.
          • cesarb 8y ago
            According to https://news.ycombinator.com/item?id=19824520 https://news.ycombinator.com/item?id=19824520 the signatures are on the extensions themselves, not on a remote server.
        • Thorrez 8y ago
          Browsers can only protect against malicious websites and malicious extensions. They can't protect against malware. Even without any cert problems, malware on your machine can modify the browser executable/process to insert whatever code it wants. With this reduced threat model, it's easy to simply keep existing pre-installed extensions available, and disable updates. Your only problem is if a pre-installed extension is malicious or has a vulnerability, it will remain.
        • josteink 8y ago
    • xvector 8y ago
      This disables NoScript on Tor Browser. That's much worse than the slim chance of a malicious extension being installed.
    • swalladge 8y ago
      It should fail 'locked'. continue to allow installed addons to work, notify the user of issue, disable any updates without explicit request by the user.
  • meruru 8y ago
    A warning yellow bar appeared for me below the URL bar.
  • deleted 8y ago
    [deleted]
  • vladus2000 8y ago
    It wasn't quite all for me, it left 3 of the 20ish I have installed.
  • jillesvangurp 8y ago
    It's a great argument against centrally controlled walled gardens. Basically breaking or compromising a single certificate has a widespread impact. Even Tor browser is impacted. For most of us this is a temporary inconvenience but there are people whose personal security depends on some of the extensions that just stopped working. On a positive note, it's been a while since I browsed without a lot of extensions. Ads are still annoying and I noticed some extensions apparently had more of a performance impact than you'd hope.
• MrEldritch 8y ago
  As a temporary fix, go to about:debugging, and click "load temporary addon", then paste in the download link of the missing add-on. Then just try and not restart Firefox until they fix the broken cert.
  • deleted 8y ago
    [deleted]
  • hoekit 8y ago
    This works. On a desktop, you can reload installed addons from the firefox profile folder >> extensions.
    • Animats 8y ago
      Yes, that seems to work on desktop. Thanks.
      • sasdsd 8y ago
        wtf
  • Renaud 8y ago
    That's most likely the easiest way to temporarily fix the issue until it is resolved.
• MrEldritch 8y ago
  This is a goddamned disaster. I'm just thankful that I use an offline password manager, but even still ... I like FF, don't get me wrong, but this is going to absolutely fucking destroy user trust in Mozilla. This kind of incompetence, on a browser scale, is breathtaking.
  • dontbenebby 8y ago
    >This is a goddamned disaster. I'm just thankful that I use an offline password manager I'm not sure this cert is used with the PW manager?
    • chimeracoder 8y ago
      > I'm not sure this cert is used with the PW manager? Firefox password storage isn't even encrypted by default, last I checked.
      • deleted 8y ago
        [deleted]
    • vatueil 8y ago
      I think MrEldritch is referring not to Firefox's built-in password manager, but third-party password managers such as Bitwarden, KeePass, and LastPass. Those rely on add-ons for browser integration.
      • MrEldritch 8y ago
        That's what I meant, yes. (I didn't even know that KeePass could be integrated with the browser, I've just been manually copy-pasting)
        • deleted 8y ago
          [deleted]
        • vatueil 8y ago
• vatueil 8y ago
  Active discussion here: https://news.ycombinator.com/item?id=19823465 https://news.ycombinator.com/item?id=19823465
  • xvector 8y ago
    Thanks, missed this thread.
  • Deimorz 8y ago
    Even though that's active and has more votes, it's ranked very low (#39 as of right now, and this post is now #1 on the site). I think HN penalizes non-link posts (or people are flagging it because they think it's just someone asking for tech support).
    • vatueil 8y ago
      Looks like you're right. I saw the other discussion first and it still has more comments for now, but this one is ranked higher. Perhaps the threads could be merged or something.
• deleted 8y ago
  [deleted]
• needle0 8y ago
  I’ll still keep using Firefox since I recognize the importance of browser diversity and the hazards of a Chrome monoculture (that and vertical tabs), but, yikes. Still, this type of oversight seems all too common even in large companies. I remember several cases from Fortune 500 companies in the past few years alone. What would be a good way to automate checking for them? Has anyone developed a tool designed specifically to avoid certificate expiry disasters?
  • wbl 8y ago
    We scan our codebase for anything that looks like a cert and send emails when it gets close. Might not have helped here if it was an intermediate owned by a CA. There but for the grace of God go I.
    • adtac 8y ago
      Why do you have certificates in your code to begin with?
      • justinclift 8y ago
        If you have your own CA for whatever reason, it's common to distribute the root and intermediate certs with your code so things can resolve. You don't ship the signing keys with the certs, as that would be bad. ;)
        • justinclift 8y ago
          s/resolve/validate/
    • lvh 8y ago
      If you want to get rid of those and they're public certs: odds are they're in Certificate Transparency logs and you can monitor them from there.
      • tialaramex 8y ago
• userbinator 8y ago
  I'm not familiar with Firefox extensions (and have pretty much stayed away from the stuff ever since they started making it "mandatory"...) but shouldn't the expiration only mean new signatures won't be valid, yet signatures made before expiration should remain so? At least that's how I understand things like Windows' driver signing works (when that was first introduced, I was quite scared that it would mean perfectly working drivers could just stop working due to the expiration, and asked... but apparently no one at Mozilla asked this question.) Edit: wow, downvotes? Care to explain what I'm missing?
  • MrEldritch 8y ago
    This same behavior is how certs usually work. Stuff with expired certs just does not run after the expiration date; that's because the cert tells you what server to ask for authentication, and if you have an old cert, there's no way to be sure that the original issuer is still the one in control of that domain.
    • userbinator 8y ago
      You're referring to things like HTTPS and contacting a remote server, and thus your reasoning there makes sense. I'm referring to traditional code signing, which I assume Firefox extensions are more similar to --- the goal being to ensure that some data has not changed since it was signed, and only the validity of the certificate at the time the data was signed is meaningful; even after the certificate expires, a signature created when it was valid still asserts that the data it signed has not changed.
  • rndgermandude 8y ago
    In short, yes, they should have implemented timestamping for their code signatures like most other code signing systems do. Without timestamping the expired cert always would have caused problems, even if it was replaced early and correctly: Every add-on would still need to be signed again with the new replacement certificate and shipped to all users. It's not as easy as just replacing the certificate on some server. Well, this is still what has to happen: replace the certificate, ship that new certificate[1], re-sign every add-on, ship every add-on to every user. Now, in order to ship new versions of the add-ons, you probably will have to bump the add-on version numbers as well. Which can have further unintended consequences. [1] Incorrect, see blow; it is my understanding that the certificate in question is baked into the browser itself, with no way to push updates just for the certificate remotely other than shipping an entire new Firefox build. Well 6 new builds: esr, stable, dev, beta, nightly, unbranded. Gonna be a fun night for a lot of mozilla folks... Well, a night is not gonna be enough... I might be wrong tho, and misunderstood something. EDIT I was wrong (https://news.ycombinator.com/item?id=19824520 https://news.ycombinator.com/item?id=19824520), the expired cert is not baked into the browser, just into the add-on package files. No need for new Firefox binaries, after all. Still, they have to resign all add-ons and ship new versions.
• deleted 8y ago
  [deleted]
• driverdan 8y ago
  First they force code signing on everyone without a way to disable it then they break it. This is an extreme level of incompetence I didn't expect from Mozilla. They'd better have the best post mortum ever, possibly with someone being fired.
  • cjbprime 8y ago
    > They'd better have the best post mortum ever, possibly with someone being fired. Arguably these two goals are incompatible. :)
    • AnIdiotOnTheNet 8y ago
      People are generally not inclined to be truthful about their mistakes if they expect to be punished for them. Its how problems keep getting covered up until they become catastrophes.
      • dev_dull 8y ago
        I guess it depends on if it was an honest mistake or gross negligence. I don’t think people should be fired for mistakes. I also don’t think everyone should be trusted with important tasks.
  • steve19 8y ago
    Why does someone need to be fired? Does some blood spilled really make it better? Have some compassion.
    • driverdan 8y ago
      I'm generally not a fan of firing people for making mistakes. This one is so monumental it may require it though. This breaks most FF installations.
      • mrep 8y ago
        Can you link to your linkedin profile so we all know who to dox next time you make a mistake? If you have a twitter, please add that as twitter works even better for mobs. Thanks :)
• SimeVidas 8y ago
  My extensions are still running. I even restarted Firefox a few moments ago. So it’s not everyone?
  • kam 8y ago
    All three of my devices on three different operating systems still have their add-ons (for now), so it's definitely not universal.
  • gpm 8y ago
    My extensions are also still running. Comments on the bugzilla bug restricted so adding details here and figuring that if this is useful someone can forward it to the right people $ date Fri May 3 22:45:22 EDT 2019 $ date --utc Sat May 4 02:41:47 UTC 2019 $ firefox --version # Installed from arch repositories Mozilla Firefox 66.0.3 about:config xpi.signatures.required true app.update.lastUpdateTime.xpi-signature-verification 1556920447 extension.update.enabled true 1556920447 is unix timestamp Fri May 3 21:54:07 UTC 2019. Edit: I think I know why. It checks the signatures daily, and the timing works out so it hasn't checked since the cert expired for me. Just luck, it will break within the next 21 hours for everyone. From the source code: const XPI_SIGNATURE_CHECK_PERIOD = 24 * 60 * 60; [...] timerManager.registerTimer("xpi-signature-verification", () => { XPIDatabase.verifySignatures(); }, XPI_SIGNATURE_CHECK_PERIOD);
    • gpm 8y ago
      Copying a potential workaround here from my lobste.rs comment, not really tested obviously If it hasn’t broken yet for you, I think (but I’m not very much not sure) setting that preference to 1556940100 should keep it working until 24 hours from now. And if you keep updating that value every 23 hours to the output of date '+%s' until it is fixed via a firefox update it should keep working forever. I think you need to restart the browser as well after updating the preference for the above idea to work.
  • chupasaurus 8y ago
    Definitely. Everything works on my Debian Buster while Windows machine wasn't so lucky.
• Causality1 8y ago
  So that's why I've been struggling for the last hour. It's crazy that it won't even let me I stall extensions from xpi file, even with extension signature checking disabled.
• jaequery 8y ago
  Sure, anyone one can make mistake. We have seen big companies make stupid mistakes too. But this is Mozilla we are talking about. How this slipped by is beyond me.
• bitbang 8y ago
  I like the alias name assigned to it: armagadd-on-2.0 Temporary work around till the cert gets fixed: set "xpinstall.signatures.required" to false
  • driverdan 8y ago
    That doesn't work unless you're on dev or nightly.
  • dblohm7 8y ago
    That only works on Nightly and Developer Edition.
  • bitbang 8y ago
    Guess it only works on the release version if you compile it yourself. (Which is the case for me)
• Wowfunhappy 8y ago
  This is why users need to be in control of their own computers. Why can't I tell my copy of Firefox to ignore the certificate? Why can't I sign my own extensions? Mistakes happen, it's okay. But users should be empowered to work around them.
  • userbinator 8y ago
    It's always felt somewhat unsettling that, although the Internet is decentralised, a small number of large organisations have a surprisingly large amount of control over the software which the majority of users view sites with. I don't like this concentration of power --- even if you think Mozilla is benevolent, it's not immune to making mistakes; and the more power it has, the bigger the consequences. Before the forced code signing, before the automatic updates, Mozilla or any other organisation's mistakes would not have such dramatic effects; now, they have the power to basically break almost all their userbase nearly instantly, and that is what worries me the most.
  • ehsankia 8y ago
    > Why can't I tell my copy of Firefox to ignore the certificate? Why can't I sign my own extensions? The issue is that if you leave any sort of lever that reduces security, it will be abused by bad actors. This is why browsers are having ever decreasing ways to bypass security and have full access. It is annoying, but at the end of the day, protecting 99.999% of the users trumps what us power users want.
    • userbinator 8y ago
      protecting 99.999% of the users It is horribly paternalistic to advocate for keeping users ignorant, unlearning, and --- dare I say it --- easily manipulated. I will refrain from mentioning again that infamous Franklin quote. I am frankly very fucking pissed off by this authoritarian walled-garden trend, and vehemently oppose anyone who helps this industry put the nooses around the necks of others as well as their own.
      • macintux 8y ago
• Silhouette 8y ago
  I just ran into this. All my extensions were immediately turned off in the middle of a browsing session, from UI conveniences to important safeguards, with no apparent option to turn them back on again. Warning about something that can't be verified is one thing, but automatically shutting things off -- particularly things that could affect security and privacy -- is a Windows 10 level of unacceptable interference.
• electrotype 8y ago
  Newbie question: why can't they just renew the certificate, like in 5 minutes?
  • crehn 8y ago
    They would have to reissue the intermediate certificate, as well as all dependent certificates.
    • cjbprime 8y ago
      And there are often bootstrapping problems where e.g. the push that distributes the new intermediate cert to clients is rejected because of the same expiry issue.
  • rndgermandude 8y ago
    A new certificate can be generated if you have to in a couple of minutes, sure. Of course, you probably defined some procedures to do it properly and securely that require more time. Then the issue becomes: how to get the new certificate to a few hundreds million users? If it was a certificate on some server, just replace it there, done. Client software will just pick it up. But not here. A copy of the certificate is shipped in every add-on package file. Oops. Now you have to re-sign all add-ons with the new certificate. And get those resigned files to the users. Essentially it works like this (which is a slightly modified jar/apk signing mechanism): - An add-on package is a zip file and other than the actual files there is also a list of known-good hashes of those files in a file called "manifest.mf" in the META-INF folder - Then there is a file "META-INF/mozilla.sf" giving hashes of "manifest.sf" - And finally, there is "META-INF/mozilla.rsa", which is a DER-encoded pkcs7 signature and two certificates. The signature verifies "mozilla.sf" was not tampered with and still is the same as when it was signed by mozilla. Which in turn verifies the known-good hashes are still proper. - The signature is made with a generated certificate, the first one included in "mozilla.rsa". E.g. "CN=uBlock0@raymondhill.net" in case of uBlock. - The "CN=uBlock0@raymondhill.net" certificate was issued by an intermediate certificate "CN=signingca1.addons.mozilla.org". This "CN=signingca1.addons.mozilla.org" is the second certificate in mozilla.rsa. It says "Validity Not After : May 4 00:09:46 2019 GMT". Oops. This is where the chain breaks now! - "CN:signingca1.addons.mozilla.org" was issued by "CN=root-ca-production-amo". This root certificate is baked straight into the browser and not part of mozilla.rsa. Therefore, it is not enough to issue another intermediate certificate (e.g. "CN=signingca-number-two.addons.mozilla.org"), but you have to actually generate a new "CN=uBlock0@raymondhill.net" (or whatever) signed by this new certificate, put those two certificates and a new signature of "mozilla.sf" based on those new certificates into a new mozilla.rsa FOR EACH add-on and ship updated add-on files. PS: Try it yourself... Extract some addon package (it's a zip file). Then: openssl pkcs7 -in META-INF/mozilla.rsa -inform DER -print
• weavejester 8y ago
  There's a workaround that involves going to about:config and setting xpinstall.signatures.required to false. However, if you're running the Stable or Beta version, it will only work under Linux. On Windows and MacOS you'll need to download Nightly or the Developer Edition. To fix this on MacOS I did the following: 1. Downloaded and installed Firefox Nightly 2. Ran /Applications/Firefox\ Nightly.app/Contents/MacOS/firefox-bin --profilemanager 3. Changed the profile to "default" so my normal Firefox profile would be used 4. Started up Firefox Nightly, opened about:config, then set xpinstall.signatures.required to false Not sure if it's a good idea to use my default profile in Nightly. It might be a wiser idea to copy it instead.
  • mirimir 8y ago
    Thank you! Saved me tons of ultimately pointless thrashing.
  • phyzome 8y ago
    This worked for me on Firefox 60.6.1esr on Debian 9 Linux—changing the setting instantly restored my addons.
    • vphantom 8y ago
      No go for me on Firefox 59.0 / Debian 64bit. I even restarted Firefox but they're all still "Legacy Extensions". :(
      • phyzome 8y ago
        Legacy Extensions is different.
    • cattitude 8y ago
      BINGO... X-ring. I OWE you, dude.
  • deleted 8y ago
    [deleted]
  • 8y ago
• cnst 8y ago
  Is it perhaps a good time to remind folks that the same thing could happen to all your "secure" HTTPS websites that are completely unavailable via HTTP, where the only thing served over HTTP are the 301 Moved redirects, even for sites that don't collect any user information at all, and only serve static and public content, which really hardly benefit from the mandatory encryption? Or is HTTPS / LetsEncrypt too big to fail? HTTPS still always a good choice? I see…
  • swiley 8y ago
    But it's easy to override broken https certificates. Worst case you have trust on first contact style security. This is just plain bad.
• swiley 8y ago
  DAMN! That's quite something! For a piece of open source software you really have very little control with firefox. It really sucks that the alternatives are worse. This, likely for almost all of their users, creates more of a security problem than signature checking actually solves. For me noscript no longer works which is (IMO) a critically important extension (between mozilla taking away the disable javascript button and spector.)
  • megous 8y ago
    OTOH, it's opensource, and you can re-compile it in like 20-30 minutes with whatever changes you desire. So you have the control. You can probably even add some hacked up support for multiple signing certificates (and add yours there), if you tried. It's just that it's more work than having what you want implmeneted and maintained by others.
• 333c 8y ago
  Can someone explain exactly what went wrong? I don't think I quite understand, but 7/9 of my extensions have been disabled.
  • deleted 8y ago
    [deleted]
  • lvh 8y ago
    Firefox requires extensions installed via their "store" be signed with a certificate to make sure they're actually from there. That certificate has an expiry date. It expired, so now all of its signatures are invalid -- and Firefox no longer trusts the associated extensions.
    • sfink 8y ago
      A minor added detail is that it wasn't the leaf certificate that expired. I've heard that that would have been handled properly. It was an intermediate cert, and I guess that possibly wasn't fully taken into account? (This is 3rd hand knowledge and speculation, note.)
      • lvh 8y ago
        That is accurate :)
• mukundmr 8y ago
  They have acknowledged the defect and are working on a fix. While this is a severe impact, I am still with Firefox. The are enough alternative browsers to tide over the problem for now. The fact that alternatives exist is the reason why we should support projects like Firefox.
  • otakucode 8y ago
    Just curious, should we expect that the fix (issuing a new signing cert and re-signing all the addons and whatnot) will result in the addons being automatically updated and re-enabled? They certainly seem to have streamlined disabling the addons, I wonder if it is equally simple from a users perspective to bring them back. Also now wondering just how hard their network/CDN is going to get slammed when those new re-signed addons go live and every user automatically redownloads them.
    • sjagoe 8y ago
      It looks like they have a hotfix to automatically re-enable the addons: https://twitter.com/mozamo/status/1124627930301255680 https://twitter.com/mozamo/status/1124627930301255680
  • notatoad 8y ago
    >I am still with Firefox that's kinda the problem. there's plenty of reasons to be "with" firefox still, but you shouldn't need reasons other than it's the best browser. when it starts requiring loyalty to be a user, that's a big problem.
    • reikonomusha 8y ago
      Don’t you think this is awfully dramatic?
    • ShinTakuya 8y ago
      For me, it is the best browser. Yes, this is a big fuck up, but it's not like this has caused me material harm. It's easy for me to switch over to Chrome until this is fixed, and I doubt the same mistake will be repeated in Mozilla. I expect perfection from plane and car manufacturers, and I pay for that. My browser, I can live with an occasional hiccup.
• terrycody 8y ago
  adblocker seemed stop working, no idea why, anyone know how to fix them or just wait?!
  • tapanjk 8y ago
    This helped me discover Firefox's Content Blocking setting, which is set to Standard by default, but now I set it to Strict. Works better than an ad block! Preferences > Privacy and Security > Strict
• hexo 8y ago
  This almost turned me into raging monster. How dare you mozilla disabling adblock? now i want an UNSEE function :D
• tzs 8y ago
  I'm a bit confused. I thought that the way signing works in general is that the signer issues a certificate for the thing being signed (domain, code, whatever) that contains identifying information for the thing signed (host name for an SSL certificate, checksum of the code for a code signing certificate), the valid from and valid to dates of that certificate, and assorted other information, and either a reference to or a copy of the signer's certificate, and it signs the whole issued certificate with the signer's certificate. Someone checking the signed thing is supposed to consider it validly signed if: 1. The date is in the valid range for the signed thing's certificate, 2. A check of the signature of that certificate against the signing certificate passes, 3. The signing certificate is recognized as being from an issuer considered trusted by the checker, 4. Neither the signed thing's certificate nor the signing certificate have been revoked, and 5. The signing took place during the valid date range of the signing certificate. Note there is no "the date of the check is in the valid date range of the signing certificate". A signing certificate expiring should not invalidate things signed by it. It should just prevent signing anything else with it. So why is a signing certificate expiring for Firefox breaking already signed extensions? Shouldn't it just be stopping new versions of extensions from being signed?
  • geofft 8y ago
    If a signing certificate expires and is stolen, it can backdate signatures. On the theory that expiration is useful because either people keep poorer track of key material over time or algorithms get weaker over time (which is not an unassailable theory, but it's a coherent model), you want expiration to prevent future use just as if it were revocation. Because you can't trust the date of a possibly-forged signature, you have to check the current date. The model you're suggesting is closer to the "timestamping" one commonly used in code signing (IIRC Windows and Mac both do this) where a third party that's particularly trusted to handle key material well long-term gives you a second signature over the message "I saw this signature at this time" (effectively they are analogous to a notary or witness for real-world signatures). Then you can trust that signatures from expired signing certs were actually made in the past, and not by an attacker who got hold of the key. That is, without timestamping you have no proof of #5 in your list. (I suppose you could do this now for the SSL PKI with Certificate Transparency logs.... it isn't exactly what they were built for but it's probably sound.)
• brainfeed0 8y ago
  If they can't get this right, what hope is there of browser security?
  • lvh 8y ago
    That's silly. Browser security is not one function. It's not even one team.
  • userbinator 8y ago
    It could be said they decided to make things a bit too secure this time.
• mirimir 8y ago
  Anyone know when they expect to fix this? I basically can't (OK, won't) browse anything except HN until they do.
• deleted 8y ago
  [deleted]
• m3kw9 8y ago
  That’s a bad sign on what they do with security
• hnaccy 8y ago
  I never liked change that extensions had to be signed by mozilla. Why do my personal extensions need to be hooked into some third party service that can go out at anytime?
  • lvh 8y ago
    The developer edition allows that just fine.
    • superkuh 8y ago
      Developer edition is effectively aurora/alpha. It is buggy compared to release. And yes, I do mean that. If it were not it'd be 'release'. Asking people to either give up control of their software (ie, walled garden release versions) or use buggy and insecure software Dev/Nightly/etc is not acceptable. It's why I switched to a freedom respecting Firefox fork as soon as they announced walled garden extension signing in Firefox 37.
      • lvh 8y ago
        Sure: debranded versions ("freedom respecting") will also do that. You say walled garden, I see what random WebExtensions people install on their work laptops and think "yeah maybe someone policing this thing isn't the worst thing". But most importantly: it sounds like it's not actually a problem for you?
• jtl999 8y ago
  Running Firefox 66 from Ubuntu repositories on Ubuntu 18.04 and all my extensions are enabled. Does it only occur after a restart or?
  • phyzome 8y ago
    I believe it happens the next time Firefox goes to check for addon updates. You may want to proactively set xpinstall.signatures.required=false which... I think might work for 66 on Linux? It worked for 60.
    • jtl999 8y ago
      Ah, checking `app.update.lastUpdateTime.xpi-signature-verification` shows that it hasn't checked since "yesterday" :/ jtl@laptop-linux:~$ TZ=UTC date --date="@1556919381" Fri May 3 21:36:21 UTC 2019
• ggm 8y ago
  I'm Interested in a proper root cause analysis and mitigation report.
• deleted 8y ago
  [deleted]
• deleted 8y ago
  [deleted]
• crehn 8y ago
  On a slightly different note, is there some curated collection of serious incidents like this somewhere? Something we could refer to when discussing possible pitfalls?
  • cesarb 8y ago
    I believe https://github.com/danluu/post-mortems https://github.com/danluu/post-mortems is close to what you want.
  • M0 8y ago
    I've seen a (then) current version of Glassfish (java EE server) refuse to start because of an expired internal certificate a couple of years ago. So here is one example: https://stackoverflow.com/questions/18248020/certificate-has-expired-in-log-by-starting-glassfish-3-1-2 https://stackoverflow.com/questions/18248020/certificate-has...
  • programd 8y ago
    Risks Digest https://catless.ncl.ac.uk/Risks/ https://catless.ncl.ac.uk/Risks/
• M0 8y ago
  They don't use cryptographic timestamps with their signatures ? The certificate might now be invalid, but the signatures were done at a time when it was valid...
  • fluidcruft 8y ago
    The problem is that "time" is fungible and can be forged. The date on a signature doesn't really mean anything.
    • userbinator 8y ago
      Emphasis on cryptographic timestamps.
    • altfredd 8y ago
      This is a very bizarre justification for an obvious bug. Code-signing does not work that way anywhere else — neither in Android, nor on iOS, Windows or any other common platform. There is a possibility that Mozilla implemented their backwards code-signing model on purpose — for example, it allows them to oust unwanted extensions without explicitly recalling their certificates. But personally I think that they just didn't give the matter enough thought.
• a3n 8y ago
  Advertisements. My eyes, my eyes!
• dazmax 8y ago
  Ah, so I need not to restart Firefox while they're still enabled.
• dvdgsng 8y ago
  Is Tor Browser also affected?
  • Chirael 8y ago
    Looks like it. I had the same yellow "One or more installed add-ons cannot be verified and have been disabled" banner in the Tor browser as in Firefox, the NoScript extension icon is missing, and I went to two different "do I have Javascript enabled" sites and they both said JS is enabled.
• spicytunacone 8y ago
  Amusingly, my surviving extensions are for WAVE, Print Preview and other accessibility tools. Thank you Mozilla, you really do care!
• mirimir 8y ago
  So why is this taking so long to fix? From https://github.com/mozilla/addons/issues/978 https://github.com/mozilla/addons/issues/978 > diox commented 2 hours ago > I'm locking this like I did in #851 because no new information is being added. We're aware and we're working on it. I mean, two hours? WTF.
  • vladus2000 8y ago
    I am going to guess its more than just getting the new cert. They need to get the new cert, they need to resign all addons and then they some how need a way to force everyones browser to re-enable them. This could go on for quite some time if they have no plan in place to handle this sort of thing. And if they were good at planning, this would not have happened in the first place. At least give us a button or something for "I don't give a shit if it isn't signed, enable it"
  • hu3 8y ago
    It was Friday night on most parts of the Western hemisphere. I'm guessing it took some time to get the right people back to work and assess the situation.
    • mirimir 8y ago
      Yes. I was upset. Sorry. I ought to know to stop when I'm upset.
      • hu3 8y ago
        Nah don't worry. We're all upset and a bit disappointed.
• interfixus 8y ago
  This is highly ungood, of course. The failure to notice an upcoming expiration is terrible in itself, but I suppose we can sort of almost empathize with how such shit may happen. The failure mode is inexcusable. Someone somewhere sometime must once have made a decision, "this will be the right way to fail", or - worse - developed the addon certification system and deciding it didn't need no stinkin' graceful degradation route in case of disaster. Still, it's no worse a calamity than most everyone else presents to the world from time to time. I shall stick to my Firefox, partly influenced by the complete absense of any viable alternative out there. By the way: Writing this a few hours after midnight, May the fourth, local time. Every single one of my 33 Firefox addons is active and working just fine. Wating to see. Breaking out my Waterfox, just in case. [Edit: May fifth -> fourth]
• Chirael 8y ago
  Just discovered the same message in the Tor browser, and it seems that NoScript got disabled. So people running Tor are a lot more vulnerable right now. Also, wow, the web has a ton of ads. I've been running uBlock origin so long I forgot how bad it had gotten :(
  • hnaccy 8y ago
    Yikes.
  • thekyle 8y ago
    That's really bad because it means it also effects Firefox ESR which is what lots of large enterprise users have installed.
  • deleted 8y ago
    [deleted]
  • mirimir 8y ago
    That was predictable.
    • mirimir 8y ago
      Well, it was. I mean, it's Firefox! But see https://trac.torproject.org/projects/tor/ticket/30388 https://trac.torproject.org/projects/tor/ticket/30388 for the same temporary fix as at https://news.ycombinator.com/item?id=19823928 https://news.ycombinator.com/item?id=19823928
  • mrep 8y ago
    > Also, wow, the web has a ton of ads. I've been running uBlock origin so long I forgot how bad it had gotten :( Try turning it off. I got rid of ublock after arstechnica complained about a lot of their users blocking ads years ago and it honestly isn't that bad. Every once in a while I do back out of a page for maxing out one of my cpu cores but otherwise, nothing ever bad happens. With ads: either it takes me half a second to tell I'm not interested in an ad, or I actually am interested and i follow the ad because I am interested and I want to support the website. The alternative is websites charging insane amounts of money with paywalls (Wall street journal has their "best" price for 12 months at $360 a year). That is horrible because it means only rich people can pay for high quality news as ads are one of the most progressive forms of payment (rich people ads are way more valuable than poor peoples and yet everyone gets the same quality services/news with the ad model despite their income/net worth).
• anfilt 8y ago
  Still beats using chrome.
  • mav3rick 8y ago
    Have fun defending this to your friends who switched overnight to Chrome again.
• Causality1 8y ago
  It happened hours ago. Why in the blue blazes of hell isn't this fixed yet?
• deleted 8y ago
  [deleted]
• deleted 8y ago
  [deleted]
• deleted 8y ago
  [deleted]
• donpdonp 8y ago
  "One or more installed addons cannot be verified and have been disabled" just showed up at the top of all my firefox tabs. #sadtrombone https://i.imgur.com/hUe7Nyn.png https://i.imgur.com/hUe7Nyn.png
• rachelbythebay 8y ago
  5 years since Heartbleed, yeah? Was this cert one that was pulled into sync due to being reissued?
• ww520 8y ago
  Not sure what kind business processes are practiced in Mozilla. Some organizations have the notation of recurring tasks as part of their business processes. Recurring tasks are just like bug reports except they are created and assigned automatically to task owners on a schedule, such as every month, every quarter, and every year. The goal of recurring task is to get people's attention to review and perform tasks the happen periodically. It could be as simple as reviewing it and marking it done. They will show up as part of the bug report to the assignees, so they can at one place see all the bugs, feature requests, tasks, and recurring tasks. Cert renewing would fall under the recurring task category.
  • hu3 8y ago
    Like a reminder that gets automatically added to the taskboard? I like it! Will talk about this with my team lead!
  • former_mozzer 8y ago
    Who at Mozilla is responsible for cert renewal? Is it Mozilla IT or is it the Firefox org? That question has never been answered, and those who asked were often reprimanded. And this is far from the first certificate renewal problem.
    • ww520 8y ago
      Reprimanded for raising the question? That's quite dysfunctional. Sounds like that's an ownership problem.
• verisimilitudes 8y ago
  This relates to my opinions about encrypted HTTP, which is that it shouldn't be mandatory. If you have a well-designed system that only works with encryption, then sure, but this idea of using the same mistaken systems as the WWW clearly doesn't work well. I've never seen a Tor Hidden Service fail because of something expiring. Much of this nonsense about encrypting everything, without reason and excuse, is to protect advertisements from being modified. That this hit Tor Browser and disabled NoScript is damning, but I already disable JavaScript in about:config and I'm not even using a version of Firefox this new, anyway. I can't tell if my opinion of Mozilla is lower or if it can't get lower.
  • ascorbic 8y ago
    This situation isn't about encryption, it's about code signing.
  • sfink 8y ago
    Requiring https is a different situation and much more defensible in my mind. It's way too easy to rewrite the web pages of everyone using library or coffeeshop wifi and thereby hack/phish a lot of people's browsers.
• gpm 8y ago
  To re-enable all disabled non-system addons you can do the following. I am not responsible if this fucks up your install: Open the browser console by hitting ctrl-shift-j Copy and paste the following code, hit enter. Until mozilla fixes the problem you will need to redo this once every 24 hours: // Re-enable *all* extensions async function set_addons_as_signed() { Components.utils.import("resource://gre/modules/addons/XPIDatabase.jsm"); Components.utils.import("resource://gre/modules/AddonManager.jsm"); let addons = await XPIDatabase.getAddonList(a => true); for (let addon of addons) { // The add-on might have vanished, we'll catch that on the next startup if (!addon._sourceBundle.exists()) continue; if( addon.signedState != AddonManager.SIGNEDSTATE_UNKNOWN ) continue; addon.signedState = AddonManager.SIGNEDSTATE_NOT_REQUIRED; AddonManagerPrivate.callAddonListeners("onPropertyChanged", addon.wrapper, ["signedState"]); await XPIDatabase.updateAddonDisabledState(addon); } XPIDatabase.saveChanges(); } set_addons_as_signed(); Edit: Cleanup up code slightly...
  • userbinator 8y ago
    I wonder how long until the "security vultures" come upon this workaround and stop it from working... would be ironic if that happened sooner than the expired cert getting fixed.
    • floatingatoll 8y ago
      Seems unlikely. If you’re willing and able to run code like the above, sourced from a random comment on the Internet, there’s no amount of security vulture that’s going to protect you from skillfully making your Internet experience unsafe for yourself.
      • bostik 8y ago
        Isn't that how most programming, security or not, works these days anyway? Joe Random Developer googles for a problem, hits SO, tries a couple of the different proposed snippets and keeps the one that happens to work. (For given values of "work".) This would be a great spot to end the post with a "</snark>", but sadly that'd be lying. Up until ~2 years ago the most common solution to requests between different subdomains subdomains failing was... "just use CORS: *"
• deleted 8y ago
  [deleted]
• deleted 8y ago
  [deleted]
• deleted 8y ago
  [deleted]
• rhizome 8y ago
  I tell you what: Grammarly is making a mint off this right now.
• gtdawg 8y ago
  Interesting coincidence, my Ubuntu 12.04 Let's Encrypt certbox-auto updated in March 2019 to a version which no longer runs on Ubuntu 12.04 without major surgery. https://community.letsencrypt.org/t/pip-error-with-certbot-auto/88200/4 https://community.letsencrypt.org/t/pip-error-with-certbot-a... I wonder if this bit Mozilla and caused this issue.
  • jacquesm 8y ago
    I can beat that. I had a Let's Encrypt cert renewal that upgraded python and itself and borked the system it was running on.
• deleted 8y ago
  [deleted]
• measlytwerp 8y ago
  Here is a temporary workaround: https://www.reddit.com/r/firefox/comments/bkcjoa/all_of_my_addons_got_disabled_and_they_are_all/emggvbx/ https://www.reddit.com/r/firefox/comments/bkcjoa/all_of_my_a...
• Animats 8y ago
  It's pathetic to see the attitude demonstrated by Mozilla support on this. diox commented 4 hours ago I'm locking this like I did in #851 because no new information is being added. We're aware and we're working on it. This conversation has been locked as spam and limited to collaborators.[1] Bug 1548973 (armagadd-on-2.0) All extensions disabled due to expiration of intermediate signing cert NEW Unassigned (Needinfo from 3 people) Kevin Brosnan [:kbrosnan] We have confirmed this issue. Extra comments about this being broken will not advance this bug to being fixed.[2] Mozilla just left their entire user base unprotected against ads, trackers, and some hostile code. Then they insult their users. Undoing the damage is hard. First, they have to update their signing certificate. Then they have to re-sign all the add-ons. Then users have to reload all the addons. Then, something users won't do - remove all the tracking cookies, etc. that slipped in while Firefox was broken. [1] https://github.com/mozilla/addons/issues/978 https://github.com/mozilla/addons/issues/978 [2] https://bugzilla.mozilla.org/show_bug.cgi?id=1548973 https://bugzilla.mozilla.org/show_bug.cgi?id=1548973
  • gpm 8y ago
    > Then users have to reload all the addons I'm pretty sure Mozilla will implement a fix in a way that users only have to update their browser, not do anything to all their addons.
    • hu3 8y ago
      I'm curious about how an update will be able to differentiate tracking cookies from legit ones?
  • bartc 8y ago
    They should be able to obtain a new certificate based on the same private/public key, in which case I don’t think any add-ons would need to be updated.
    • rndgermandude 8y ago
      The problem with this approach is that the expired certificate is part of the add-on package files (META-INF/mozilla.rsa; DER encoded PKCS7), not something that you can just swap out on some server. You have to replace the certificate in the add-on packages with the new cert, even if the new one reuses the keys of the old one. At which point you need to ship new add-on package files to users anyway, so key reuse or not makes no difference anymore.
• yardstick 8y ago
  In addition to the immediate fix, what needs to happen here and in general anywhere a certificate is used, is the browser should display an informational banner that the certificate is due to expire soon. I’d suggest start warning at T-7 days left. That way even if the business messed up, they would have a heads up from users to fix it before d-day when everything stops working. This includes website certs and addon signing certs and any intermediaries.
• whatshisface 8y ago
  Tomorrow (or whenever this gets fixed), ad companies are going to have some great data about what the world would look like if adblock didn't exist. I really home someome does a blog post about it. Yikes, I hope it doesn't play out like a shark smelling chum.
  • floatingatoll 8y ago
    Firefox has anti-tracking features that are enough to freak out reCaptcha regardless of adblockers, so that might not be as good for them as it seems. Be careful not to confuse hiding ads from view with preventing user tracking.
  • kelnos 8y ago
    Given that Firefox's market share has dropped below 10%, I doubt it will make much of a difference.
    • whatshisface 8y ago
      Yes but there's enough sample size for them to multiply the change by ten to get what would happen if Chrome lost adblock as well.
• kjasbndkjads 8y ago
  Embarrassing and amateurish, what a fuckup. Didn't expect Mozilla to be this incompetent. It's also Friday night, a lot of people won't be sleeping tonight.
• hex12648430 8y ago
  I'm curious to see what kind of impact on ad revenue we'll see in the next couple of days.
• fnrslvr 8y ago
  Mozilla doesn't seem to have communicated the issue well. I could imagine a lot of unsavvy users have tried some wild things in an attempt to fix the problem, and maybe made a mess in the process. Doesn't Mozilla have a mechanism for blasting out a message to all Firefox browsers? Also I have a Firefox account, why haven't I been inboxed about this? Otherwise I'm not bothered. I won't be switching as long as this gets resolved within the next few days.
  • witten 8y ago
    Even if there's no way to blast all Firefox users, there are blue links to standard Mozilla-hosted "help" web pages within the Firefox add-ons config, links that a non-insignificant number of confused users will probably click. Those web pages could easily be updated with info about the cert expiry snafu.
  • saulrh 8y ago
    > Doesn't Mozilla have a mechanism for blasting out a message to all Firefox browsers? The cynical side of me says that it must not have this feature because if it did I'd have seen someone complaining about the browser "phoning home" or "forcing Mozilla's opinions into my eyeballs".
    • Karunamon 8y ago
      ...unlike the mothership breaking all addons, and the browser being designed intentionally in a way that prevents me from working around the breakage? This isn't just a petty snipe borne out of annoyance. Ads being the malware vector that they are, and the degree of tracking and data mining out there, all of those countermeasures being turned off overnight is an exposure that should be treated with the same degree of seriousness as PII breach at a company you have an account with. God forbid you use FoxyProxy or Tor Browser or something else that masks your connection source - this could have legitimate, real-life consequences if you don't notice the change.
      • saulrh 8y ago
        On the one hand, I understand the point you're trying to make. On the other hand, I'm going to be honest, I have trouble reading your post without thinking things like "If you are literally trusting your life to FoxyProxy, you might want to rethink your entire internet safety strategy." Another favorite was "Defense in depth." I've had dozens of different experiences where my extensions silently and unexpectedly malfunctioned. Configurations getting erased, new extension releases with breakage or different behaviors, maximum compatible versions in the manifest, new permissions, botched keystrokes in the extensions page, profile corruption, incompatibilities between extensions, internal bugs that cause them to crash-loop without doing anything, the works. Like, I run a ton of addons, some of which are a bit esoteric and a couple that I compile from head every few days, but even taking into account my outlier-sized surface area it's a bit silly. I'm not going to claim that this isn't a problem. That said, blaming Mozilla for a life-threatening failure of FoxyProxy, of all things, is like blaming Cessna because a journalist flew one of their planes into a combat zone. There just wasn't any way it was going to end well.
• indygreg2 8y ago
  Patches to workaround the issue were just committed. Those patches were tracked at https://bugzilla.mozilla.org/show_bug.cgi?id=1549010 https://bugzilla.mozilla.org/show_bug.cgi?id=1549010. At the time of writing this comment, that bug is not chained up to the bug linked by this post.
• homero 8y ago
  Does this happen when you open firefox? Forgetting what i need to reenable and what i had disabled is going to make me mad. If i just don't open ff will i be fine?
• HaoZeke 8y ago
  Honestly, I just have to wonder what the Mozilla devs do. Don't they use extensions? Didn't they notice what was going on? Why does it increasingly feel like a more random Google Chrome-Lite paired with Apple's 'best intentions'?
  • minitech 8y ago
    This is a certificate that expired, not some insufficiently-tested change someone deployed. Mozilla devs had their extensions disabled at the same time as everyone else.
    • teddyh 8y ago
      Well, most of the actual developers probably run Daily and have xpinstall.signatures.required = false, and are therefore not impacted quite as severely.
• rb666 8y ago
  Wow this is really bad, breaking millions of people's workflow in one go. Fixed for now by switching to Firefox Nightly and disabling signing.
• pritambaral 8y ago
  Run this in your Browser Console[1] to delay signature checking for a day: function set_xpi_sign_time_now() { const {Services} = ChromeUtils.import("resource://gre/modules/Services.jsm"); const now = (new Date()).getTime() / 1000; Services.prefs.setIntPref('app.update.lastUpdateTime.xpi-signature-verification', now); } set_xpi_sign_time_now(); EDIT: Changed `Components.utils.import` to `ChromeUtils.import` because apparently Beta and Nightly versions have removed the former, while the latter was introduced in 60. This does the equivalent of setting in about:config the time of last signature verification to the current time. By default, Firefox re-checks signatures in 24 hours (or so I read somewhere here). I like the temporary effect of this, compared to the permanent disabling of signature verification suggested elsewhere. ---- 1: https://developer.mozilla.org/en-US/docs/Tools/Browser_Console https://developer.mozilla.org/en-US/docs/Tools/Browser_Conso...
  • nodja 8y ago
    Can you set this to the future and get say a week of buffer time for the fix to get pushed? edit: nope, you cannot:(
    • pritambaral 8y ago
      1. I don't think that'd be necessary; I believe Mozilla will fix it in a day, or two at most (PS: they're currently testing a fix); and 2. I don't know if there's sanity-checking code in Firefox to ignore times in the future.
      • nodja 8y ago
        I tried it and it was set to 0 when I restarted firefox (and the extensions were gone).
    • rampant_ai 8y ago
      I tried essentially the same thing but via about:config. All it did was make it happen immediately on next restart. Also if it hasn't happened to you yet, make a backup of your profile right now in case it wipes out your addons data as some have reported.
• nydel 8y ago
  i’ve a strong emotional reaction to this. it feels like a failure on all of us. will be fighting toward a more optimistic interpretation of such a snafu.
• Adamantcheese 8y ago
  I'm surprised nobody's mentioned addon debugging if you really need an extension working. All of the extension packages are in <your profile>/extensions folder and you can load them for the duration of your session by turning addon debugging on in about:debugging and loading them in. This should be fixed fast enough that this fix will be good enough, just don't close your browser.
• former_mozzer 8y ago
  Will Mozilla IT finally be held accountable? Whether or not the direct cause of the expired cert is IT failure, the fact is that a healthy IT would have prevented this. Take a look at Mozilla IT leadership. Take a look at all the people who have left in the last few months. For a year people tried to bring attention to the IT leadership disaster, and every person who did that was penalized for it and left Mozilla. A disaster like this has been brewing for a long time, and Moz leaders didn’t listen to the canaries in the coalmine. It’s a very sad day for all the great people at Mozilla who work so hard only to see Mozilla IT let them down.
• MikusR 8y ago
  Expired certificates are not Mr. Robot ads that have to be pushed in timely manner.
• Ghost_of_USSR 8y ago
  A positive side effect: my 2 years old FF installation stopped freezing constantly.
• Grue3 8y ago
  What kind of idiot thought that the add-ons I have personally installed on my browser need to have a capability to be remotely disabled despite literally nothing being changed. This is absolutely inexcusable. I want to see everyone being responsible for this "verified add-ons" fiasco fired from the team (after they roll it back of course).
  • swalladge 8y ago
    Exactly. If me/firefox has verified the signature (or approved the download) when downloading or updating the addon, that should be all that's necessary. Why does firefox have to check signatures constantly?
    • egwor 8y ago
      in case it was revoked? Seems fairly reasonable approach
      • TeMPOraL 8y ago
        Disabling with no option of user override is not reasonable. It only creates unnecessary dependency on third parties. When product recalls happen, the manufacturer isn't taking your product away by force.
    • hinkley 8y ago
      I worked on a code signing system a number of years ago and it was surprising the degree to which I had to rearrange the logic and the tool chain in order to get high code coverage on all the failure modes. IIRC one of the tools wouldn’t let me work with expired certs. I can’t recall now whether I fixed that or made carts that expire in ten seconds and just waited it out. Anyway, a number of people weren’t even sure why I was going through the trouble. It’s easier to get something wrong than it should be (super obtuse APIs) and you don’t always get enough support or pushback to get everything absolutely right.
  • founderling 8y ago
    remotely disabled Were they really remotely disabled? That would mean somebody out there pushed a button and made your add-ons go poof. As I understand it, the browser checks the certificate of add-ons at some point (on startup? on an interval?) and only uses signed ones. And since signatures are date restricted, previously valid signatures can become invalid. I'm not 100% sure if this really is the mechanism. Would be interesting to hear from someone in the know.
• s9w 8y ago
  What a spectacular failure... it's hours later, and still not working. The only thing firefox has going for itself stopped working. Already one family member and one friend switched to chrome.
  • 59nadir 8y ago
    Just switch for a day. You're making this out to be a much bigger issue than it has the capacity to be.
    • gspetr 8y ago
      NoScript and uBlock Origin have stopped working. This opens up a LOT of attack surface for malicious hackers. If this isn't a critical security issue, what is?
      • envolt 8y ago
        The Container extension is no longer working. I'm logged out of mostly everything. There are few sites I don't want to open without container.
        • inawarminister 8y ago
          Exactly my use case. Already installed chromium-ungoogled and am using a judicious amount of profiles to emulate this. Still not as nice as what the Container extension do, unfortunately.
        • envolt 8y ago
          Update - Container data is lost post addons recovery (I installed Nightly build). This is ridiculous. My Firefox usage will be so unproductive for few days. I had around 6-8 containers for different purpose, and somehow I'm habitual to using shortcut to launch a container and open whatever I'm supposed to (e.g I've access to 3 different AWS account, and I tend to press shortcut key to launch the relevant container tab)
    • deleted 8y ago
• nullc 8y ago
  Karma for blocking that gab commenting extension...
• jamewatson 8y ago
  Duplicate of this bug
• former_mozzer 8y ago
  There have been major organizational problems at Mozilla for a long time that precipitated this. Many of us saw something like this coming, saw gaps and unclear responsibilities, reported these gaps and confusions up the chain, and were reprimanded and financially penalized for asking the tough questions. The questions were never answered, and we all quit, were fired, or lost motivation as a result. This is a tech problem, yes. Cert renewal has bitten everyone in a high profile way (apple, google, and ms have all had renewal-related outages in recent years). But this was preventable at Mozilla. Ask a Mozillian about IT and Cloud Sevices, and what their respective responsibilities are. Ask Mozilla’s VP of IT- who is responsible for cert renewal? Ask Mozilla leadership- why are people afraid to ask questions?
  • antihero 8y ago
    If you're going to quit anyway, and not saying this is a good idea, what prevents you literally going and yelling at these idiots demanding answers as to why they are not doing their jobs?
    • former_mozzer 8y ago
      So many people have done this. Watch videos of past Mozilla All Hands meetings. You see the questions asked, you see the leaders dodge, and you see that soon after the askers are gone and leaders say- you have nothing to be afraid of, you don’t need to ask questions anonymously. “Executives” at 1000 person Mozilla are so distant from the workers, and middle managers listen to executives, not staff. It is so sad because Mozilla staff are smart, caring people who love the web, but they are powerless and afraid in the org today.
      • gspetr 8y ago
        It seems that to enact a change somebody would have to go public and talk with big journos to bring this dysfunctional stuff to light, much like the recent Bioware scandal.
      • user17843 8y ago
• deleted 8y ago
  [deleted]
• lucb1e 8y ago
  If everyone's add-ons are disabled, I wonder why mine are not. My computer has been running over night (coincidentally, first time in years) and my add-ons are intact. Does it take a browser restart? Or might I have a setting that prevents this from happening? My system time is correct. Edit: am on Firefox 66, Linux (Debian Buster/testing), using Firefox from Mozilla directly (not through repositories), and my internet/wifi should not have disconnected. System has been up since 2019-05-03T17:30:00Z, suspended before that.
  • dschuetz 8y ago
    Are you on nightly build?
    • lucb1e 8y ago
      No, same version as the bug report is about: 66.
  • Svip 8y ago
    I think it only checks at start up, or something? Mine has been running for days, and all my add-ons are working. As such, I've decided not to restart Firefox until news arrive that it's working again.
    • blahyawnblah 8y ago
      Mines been running for days and everything was disabled.
      • lucb1e 8y ago
        Oh, darn. Thought I found a way to avoid this. Thanks for reporting! Just a thought, might it be that you suspend (sleep mode in Windows) your system? So not restart the browser, technically, but iirc applications still receive some event when this happens. At the very least, open connections would break. Any TLS connection would re-validate the certificate. In that case, going offline would also trigger it. Have you done any of those (suspend, be offline / switch networks)?
• dwdz 8y ago
  Because of this my Firefox crashed and I lost all my open tabs. Just fking great...
• deleted 8y ago
  [deleted]
• dessant 8y ago
  I have disabled signature checks in Firefox because otherwise it is impossible to install a private extension without uploading the source code to Mozilla. This is how you allow unsigned extensions in Firefox on Arch Linux, the same files can be edited on Windows and macOS, restart the browser after changes: sudo tee /usr/lib/firefox/defaults/pref/config-prefs.js &>/dev/null <<EOF pref("general.config.obscure_value", 0); pref("general.config.filename", "config.js"); pref("general.config.sandbox_enabled", false); EOF sudo tee /usr/lib/firefox/config.js &>/dev/null <<EOF // keep this comment try { Components.utils .import('resource://gre/modules/addons/XPIDatabase.jsm', {}) .XPIDatabase['SIGNED_TYPES'].clear(); } catch (ex) { Components.utils.reportError(ex.message); } EOF This method also works for the stable version of Firefox.
  • dvfjsdhgfv 8y ago
    It's very good that it's possible, but it's literally the worst solution for most people.
  • dingo_bat 8y ago
    I feel like this should be a checkbox in settings, considering the spirit of Firefox.
• campbellmorgan 8y ago
  For anybody using Firefox Android and ublock origin who now sees ads, worth checking out the duckduckgo Android privacy browser. So far, it seems to be working beautifully so something good for me at least has come out of this mess!
• XorNot 8y ago
  So this just happened in my browser...when exactly will this be fixed?
• BuckRogers 8y ago
  I've been using Firefox since 2002, always loyal because it works as I expect a browser to work. For users like me who see the benefits of Firefox being the only browser worth losing the benefits of native browsers (Safari/Edge), the only real place we'd move to due to sufficient issues is to those native browsers. The new Chromium-based Edge[0] for someone like me, who would never use Chrome, is the only cross-platform alternative that could pull me away from Firefox. Native power usage advantages on Windows, with portability to other platforms and Chromium's speed advantages makes it a very attractive choice. I hate to see this happen to Firefox, but they don't have a lot of room for error. I've been using "ChrEdge" at work, and it's already a great browser as it is. [0]https://www.microsoftedgeinsider.com/en-us/download/ https://www.microsoftedgeinsider.com/en-us/download/
• tempodox 8y ago
  Mozilla continues the war against its users.
• en 8y ago
  The saddest thing today is to realize that I can't use Firefox without extensions on privacy/security.
• sushikokk 8y ago
  Shit, I just switched to lastpass and deleted all my saved passwords in firefox. Good timing. On windows, so the config hack is not working.
• gminialpha 8y ago
  I déinstalled most of my addons and/or extensions How can I return to my previous profile to get them back?
• bArray 8y ago
  I'm running the workaround suggested by Reddit use @MeaslyTwerp (the one after the HN solution): https://www.reddit.com/r/firefox/comments/bkcjoa/all_of_my_addons_got_disabled_and_they_are_all/ https://www.reddit.com/r/firefox/comments/bkcjoa/all_of_my_a...
• deleted 8y ago
  [deleted]
• nkkollaw 8y ago
  I wonder how it's possible that these kinds of--I assume--easily preventable problems happen at huge corporations and projects. I routinely renew certs for clients and it never happened that a website was down because of an expired cert. I would think that Apple, Microsoft, Mozilla would be more efficient than me in avoiding these kinds of fuckups? I also bet these companies make huge investments to their infrastructure so that their services have a near-100% uptime, and then they let certs expire.
• kgwxd 8y ago
  As I'm reading these comments, the Unverified addons warning popped up :) Well, I refuse to use the internet without uBlock Origin. Looks like this is going to be a productive day.
• rmbryan 8y ago
  Update: We have rolled out a partial fix for this issue. We generated a new intermediate certificate with the same name/key but an updated validity window and pushed it out to users via Normandy (this should be most users). Users who have Normandy on should see their add-ons start working over the next few hours. We are continuing to work on packaging up the new certificate for users who have Normandy disabled.
  • brador 8y ago
    What is Normandy?
    • megous 8y ago
      https://wiki.mozilla.org/Firefox/Normandy/PreferenceRollout https://wiki.mozilla.org/Firefox/Normandy/PreferenceRollout
      • chinathrow 8y ago
        So is that a backdoor into my prefs? How can I check if Normandy is active on my installation?
        • verdandi 8y ago
          Type about:config in the address bar and search for 'app.normandy.enabled' flag.
          • DoctorOetker 8y ago
            here the flag is true but the extensions are still unsupported
            • Yoric 8y ago
              It may take a little time for the partial fix to be distributed.
• abdulmuhaimin 8y ago
  its been so long since i browsed the web without adblock. Now I remember why
• jopsen 8y ago
  Wow, that's a bad slip up... I hope users stick with Firefox. I feel like this sort of thing is a risk when being vigilant about security.
• bluejay2387 8y ago
  Not happy that Firefox didn't ask before it did this. I am getting really tired of my technology trying to tell me how to do things.
• deleted 8y ago
  [deleted]
• geniium 8y ago
  This is huge. This had a huge impact on my work in the last hours. Still not fixed for me. What's going on???
• koolba 8y ago
  I'd be curious to see if there's a marked increase in net sales during the time that this snafu is ongoing just because uBlock Origin is disabled and users are forced to see more ads.
  • kgwxd 8y ago
    Maybe, if FF had a bigger market share. On top of that, I think many FF users know not to browse without it.
• 2T1Qka0rEiPr 8y ago
  > Steps to reproduce: Wait until it's past midnight on 2019-05-04 UTC. This has got to be one of my favourite bug reports ever.
• user17843 8y ago
  Is it possible that this wasn't an expired certificate but someone accidentally changed the signing process? The disabling happened right after the announcement by Mozilla to implement a new policy towards extensions. Maybe someone didn't realize their mistake, so now everyone thinks it was an old certificate.
• new_guy 8y ago
  This is absolutely going to tank Firefoxs market-share and reputation. I have a site full of tech-illiterate users and they're all uninstalling Firefox and searching for alternatives. None of them are interested in the workarounds presented, most of them even struggle to turn their computers on! Mozilla really dropped the ball here.
• akerro 8y ago
  All my setup of 24 containers has been reset... Not only it forgot which domains were assigned to which containers, it also removed my manually created containers.
• carmate383 8y ago
  Well, it was fun whilst it lasted Firefox. Sorry, but this is a deal-breaker for me. Hello Chromium again...
• v8engine 8y ago
  So let's say I'm the IT department in my company. I've already got my root cert on every employee's PC(including Firefox because they can't browse otherwise). Can I act like the Normandy endpoint and let's say remotely disable the ability to install any extension including those pesky VPN ones and also do a lot of other such things I would like, you get my drift? Am I right? Am I right? Please tell me I'm wrong.
  • fjsolwmv 8y ago
    You are wondering if an IT admin can admin machines in its network? Yes, an IT admin can admin machines in it's network.
• gordiancaesar 8y ago
  when is the fix?
• gordiancaesar 8y ago
  when might the fix be?
• deleted 8y ago
  [deleted]
• StreamBright 8y ago
  I guess there was no monitoring on cert expiry.
• Metropolix 8y ago
  I was able to temporarily work around this problem by going to "about:debugging" and using the "Load Temporary Add-On" button to load up the disabled add-ons inside of ~/.mozilla/firefox/<my_profile>/extensions/
• deleted 8y ago
  [deleted]
• kisamoto 8y ago
  So: * Started day with browser, no problems with extensions * At some point in the afternoon, all extensions disappeared * A couple of hours (cannot be more specific I am afraid), all extensions came back However I'm noticing the saved preferences of some of the extensions has gone. e.g. Password manager has kept the username however all "Multi-account containers" are now reset to factory defaults. Is anybody seeing similar behaviour ?
  • tux3 8y ago
    Yep, there's a bug open for that: https://bugzilla.mozilla.org/show_bug.cgi?id=1549013 https://bugzilla.mozilla.org/show_bug.cgi?id=1549013
• joeyh 8y ago
  Firefox extensions installed using apt are not affected by this problem AFAICS, so to get a working ad blocker, apt install xul-ext-ublock-origin
• throw2016 8y ago
  This is a shocking display of not just incompetence and bad practices but of brazen undisclosed covert control. Why should your local browser depend in this fragile way on some muckup in Mozilla HQ? And people line up to defend this? Where does it explicitly say Mozilla can disable my addons remotely? When did I give them this power? And this from a so called 'open source privacy focused' browser. This is a mockery of privacy and open source and they shouldn't trade on this goodwill to gain users. There can be no bigger security hole yet security fear mongers preach exactly this abusive model. This kind of centralized remote power is a far greater security threat that anything they keep on harping about, 'good intentions' and 'good faith' are not remotely something anyone should have to depend on. Why should Mozilla babysit my installation? Shouldn't they be using their resources to do something productive? There is something rotten in SV culture and we urgently need to think of alternatives that are not infused in this 'know it all' abusive surveillance culture as even after such an egregious abuse of peoples trust and faith all you will get is hand waving, normalization, apologism and snarky entitled comments that trivialize people's concerns and choices made on the goodwill of open source.
• deleted 8y ago
  [deleted]
• JoshuaRLi 8y ago
  I've had normandy disabled in my user.js for a while, but this was only after thoroughly perusing documentation and some firefox "hardening" projects. Point being, no end user should have to do what I did. Sane defaults, and transparency about things that should be opt-in, are sorely needed. Regardless, I still stand by Firefox, and am thankful there are chromium alternatives. The web is what it is, though I dream of a simpler one.
• Pxtl 8y ago
  I don't get why an expiring cert disables the extensions. Shouldn't the browser be checking the cert expiry date against the date the extension was installed, not against current time? As long as there's no way to manipulate the extension installation date that would be fine, wouldn't it? edit: or even why the browser is checking this at run-time. As long as it checked the cert when the extension was installed, isn't that enough?
  • Grollicus 8y ago
    If Mozilla somehow lost control of one of these signing certs (as in: it got stolen) they would put in on a revocation list. If certificates don't get re-checked, all installations between "cert got stolen" and "noticed that the cert got stolen" would keep installed & running.
• gxx 8y ago
  My password manager and every other plugin disappeared this morning first on one of my Macs and a few hours later on the other. I could not log into anything so I switched immediately to Chrome because I don't have time to fuss with workarounds. I'll return to Firefox when I hear the problem is definitely fixed.
• sara7262 8y ago
  Reality about terrorism Must watch this https://youtu.be/l_C_T1NCvDw https://youtu.be/l_C_T1NCvDw See the friendship between cat and squirrel That's so cute see how much they are taking care of each others http://bit.ly/2VdHs4E http://bit.ly/2VdHs4E Friendship between cat and dog see in video how much they are teasing each others also they are loving http://bit.ly/2WoRU66 http://bit.ly/2WoRU66
• nimbius 8y ago
  another month, another browser vendor that does something inconceivably bone-headed in "the service of users." first it was deprecating ALSA for pulseaudio, then it was pocket, then tiles and their suggestions, then that weird video/voice chat thing, and then running "studies" as if my use of the browser was some tacit acceptance of my position as a guinea pig of the internet. Today every extension I use to make the internet even remotely usable is just...deactivated? without my consent or knowledge? enough. im switching to waterfox. Icecat is even worthwhile at this point. Anything that respects my freedom and intelligence as a user.
  • jopsen 8y ago
    This appears to be a mistake. It's clearly bad that things can break this way. But this happened because a certificate expired. Mozilla didn't have to require signatures and certificates for extensions. They did so because they want to protect users. Protecting users with signature schemes, increase complexity and, thus, the risk of debacles like this.
    • thekingofh 8y ago
      People take their privacy seriously. Our addons were disabled. I was browsing for a few minutes until I saw ads and didn't realize what was going on. Unacceptable that the default to an expiring signature is to disable them completely.
      • jopsen 8y ago
        You would have said the opposite if an expired signature had caused your browser to be compromised!!! The default behavior seems desired to me. The problem was that the certificate was allowed to expire. A have ton of respect for the fact that it was enforced!
• michaelvkpdx 8y ago
  I really hope Mozilla hosts a very open and public postmortem on this. Lapsed certificates keep causing outages everywhere. We narrowly avoided one in my new org. How can we as a discipline solve this? This is a great opportunity for Mozilla to lead, to demonstrate openness, and to help all of us who love the web secure our work without risking catastrophic downtime. Thank you to all the Mozillians who worked hard to solve this. I'm sure there were a lot of ripple effects into other products outside of Firefox. Mozilla is so small for an organization that does so much. Give your closest Mozillian a hug today.
• mikeash 8y ago
  Certificates have been in common use online for maybe two decades now, if not more. This is a common failure mode and it keeps happening. Is there some fix so we don’t have to keep dealing with spontaneous failures due to expirations? Or are we doomed to suffer with this until the end of time?
  • gator-io 8y ago
    Use a certificate monitor https://letsmonitor.org https://letsmonitor.org (free)
    • yread 8y ago
      Is that site legit? Why don't they even have a HTTPS redirect?
• chaz6 8y ago
  Maybe everybody should set app.normandy.user_id to "anonymous". I am disappointed that Mozilla still has unique tracking ids embedded in Firefox.
• thekingofh 8y ago
  You've fucked up mozilla. I had no option other than to switch over to a browser that would let me install my ad blocker. The web sucks too bad for this. My addons are mine, not yours to disable and enable at will.
• overgard 8y ago
  Well, a little bit of empathy for mozilla here: I've seen a lot of IT departments that don't have any sort of great system in place for managing certs. A lot of places I worked, I always had a suspicion they were a ticking time bomb. It's not enough work that it's really anyone's full time job to manage them. Also, at larger companies, you might have divisions that do it in different ways without cohesion. And then a lot of times certificate expiration is so far in the future that the people that initially setup a certificate might have left the company and forgot to document it, etc. So that kind of thing can easily fall through the cracks. Maybe a constructive thing I'm curious about: What is considered best practice for managing certs? How do people do this in a secure way that makes sure they get renewed in a timely way?
• deleted 8y ago
  [deleted]
• semperfi 8y ago
  I did my part by sending angry emails to every @mozilla.org address and making duplicate bug reports on https://bugzilla.mozilla.org https://bugzilla.mozilla.org with multiple IPs. This bug won't be fixed until "Wednesday at the earliest," so you have a lot of time.
• masterfooo 8y ago
  After learning about Normady thing, I am literally now blocking every Mozilla, Firefox and their affiliate domains and their Ip adresses. The trust is lost. I rely on Debian repo and like other's mentioned, Normandy seems to bypass Debian oversight.
• scottcruss 8y ago
  Temporary fix without enabling feedback to Mozilla. Open the address about:config in the Tor Browser address bar At the top of the page, search for xpinstall.signatures.required Set the xpinstall.signatures.requiredentry to false by double clicking it Note: This workaround should only be used temporarily, as it disables a security feature. Please remember to set the xpinstall.signatures.requiredentry back to true again once the Tor Browser security update is applied.
• drdon75220 8y ago
  What a nightmare! ---- ALL of my extensions have disappeared and have been disabled this weekend (with ZERO warning) ---- Trying to use my Roboform password program is a nightmare---- If the problem is not 100% solved by Monday (May 6), I plan to exterminate every trace of Firefox of every computer I own ---- Shame on Firefox for placing uses in such a nightmare situation with ZERO warning and ZERO solutions ----
• happypuppy 8y ago
  Waterfox works in Windows, no issues with extensions and supposed to have a greater compatibility with legacy extensions. Fennec works in Android, no issues with extensions and has the third party trackers removed.
• deleted 8y ago
  [deleted]
• F-B-Aye 8y ago
  This problem has screwed up Firefox 53 as well so unfortunately reverting to a superior version won't fix the problem either. At the moment Waterfox is looking like the best fix.
• Addning 8y ago
  www.facebook.com/imelaos.tk https://www.facebook.com/profile.php?id=100014512247253 https://www.facebook.com/profile.php?id=100014512247253
• trollaccount155 8y ago
  Plot-twist: google is behind this .....
• pcvarmint 8y ago
  This was probably an unintended consequence of trying to ban Dissenter. [0] [1] 0. https://dissenter.com/download https://dissenter.com/download 1. https://youtube.com/watch?v=f0Cc8RpqH1g https://youtube.com/watch?v=f0Cc8RpqH1g
• GRBurst 8y ago
  And now, all my plug-in / extension settings / preferences are gone after updating...