43 ms·

Update Regarding Add-Ons in Firefox

455 points by akyuu 7y ago

peshooo 7y ago
Firefox has an interesting backdoor...
- SquareWheel 7y ago
  They also have a frontdoor: built-in automatic updates.
  - mlthoughts2018 7y ago
    This type of reply is ludicrous. An obscure feature (Normandy modifying default settings as part of studies) that requires you to actively opt out (and even then it’s still not clear if you _also_ have to go to about:config to _really_ disable it) and which can make such large scale errors as evicting all extensions is in absolutely no way comparable to the case of a user agreeing to auto-updates and fairly easily being able to disable them. It is astoundingly disingenuous to act like these things are comparable.
    - SquareWheel 7y ago
      I think you're overstating it. >It is astoundingly disingenuous to act like these things are comparable. Why aren't they comparable? In both cases, it's Mozilla pushing code to the end user. There's a different process behind both but calling one a frontdoor and one a backdoor seems apt to me. >and which can make such large scale errors as evicting all extensions Normandy was not used to disable all extensions. It was caused by a certificate expiration error completely independently. Normandy is being used instead to work around the error until a more permanent fix can be issued.
      - mlthoughts2018 7y ago
        > “Why aren't they comparable?” Because regular auto-updates are easy to understand and turn on/off. Normandy is clearly extremely hard for many users to understand, enabled by default, and hard to disable.
pmontra 7y ago
No Firefox Studies on Android. Maybe they'll release a whole new version.
- AsyncAwait 7y ago
  You can click on https://storage.googleapis.com/moz-fx-normandy-prod-addons/extensions/hotfix-update-xpi-intermediate%40mozilla.com-1.0.2-signed.xpi https://storage.googleapis.com/moz-fx-normandy-prod-addons/e... to get the hotfix on Android.
  - classichasclass 7y ago
    I tested this earlier and it works great here.
tempodox 7y ago
An article that mentions a timeframe of “the next few hours”, but doesn't have any timestamp besides a date without a timezone.
- reubenmorais 7y ago
  You can hover the date to see the full timestamp, it's 2019-05-04 07:01:35 UTC-7.
  - teej 7y ago
    Not available on all clients, notably mobile.
  - TeMPOraL 7y ago
    It's an annoying UX antipattern, the only thing worse being not adding the on-hover text at all. Displaying date and time should be the default. HN has this problem too.
- sachdevap 7y ago
  Actually, if I'm not mistaken, the complete timestamp is available when you hover over the date.
- teej 7y ago
  Poor communication with zero accountability. A few hours could be 2, it could be 12. “I (name the individual accountable) will give you an update at 12:00 PT (name a time) as an update to this post (name the communication channel) with the current status and latest information on this issue (don’t promise time to resolution, just time to info).” Simple, clear, concrete, and unambiguous. I had hoped that Firefox had better communication procedures in the event of global-impact P1 issues.
- derefr 7y ago
  If Mozilla Studies are implemented the way other "push" update systems are, then probably your browser has an ID that it hashes to get a bucket ID that it builds into a URL to check for updates, plus a cron time offset for running those checks. Then, the experiments are rolled out by walking up the bucket ID list and gradually adding the addon to said buckets. Usually, this mechanism is explained as being helpful to ensure a rollout of an experimental update can be rolled back if it's failing. That's not so much a concern in this case, I think. But this mechanism has another effect: it works as a solution to the thundering-herd problem. Every browser updating at once is bad, not just for Mozilla's servers, but for every piece of Internet infrastructure that those browsers (and their arbitrary set of addons) talk to when they update/restart. Within the time budget you have for running a rolling update, you ideally want as few machines updating concurrently as possible, just because you don't want to generate mysterious correlated traffic bursts that make NOCs paranoid.
deleted 7y ago
[deleted]
jimmaswell 7y ago
Mobile's still broken. Which is a browser Mozilla has apparently abandoned the userbase of in favor of focusing all their mobile effort on some silly pointless separate browser that caters to millenials more somehow or some other nonsense, instead of just working on the browser they already have. Really fed up with Mozilla at this point and considering switching to a fork or something.
- firethief 7y ago
  Servo was a silly pointless separate browser once
  - gpm 7y ago
    I'm not sure I would call it "pointless", they've merged and are merging a lot of the code they used servo to experiment with into firefox. Of everything mozilla has done recently, Servo is one of the things I'm most positive about.
    - jacoblambda 7y ago
      I think that was the point they were making.
    - sachdevap 7y ago
      I'm sure your parent comment was being sarcastic. The quotes were missing.
      - gpm 7y ago
        You're probably right, oops.
  - jimmaswell 7y ago
    I still hold that the multithreaded performance benefit was nowhere near worth wiping away so many of hours of developer time and ripping so many good extensions out of users' hands with no replacement for so much of the lost functionality.
jchw 7y ago
Interesting. Sadly, I imagine many users will have studies disabled since the Mr. Robot incident. I've re-enabled it but there does not appear to be a way to force it to check for updates. Guess it will show up in the next 6 hours.
- dralley 7y ago
  You can set app.normandy.first_run to true and then restart Firefox.
  - jchw 7y ago
    Thanks, that worked! Out of curiosity, how did you know that? What does this actually do?
    - bwat49 7y ago
      I think it just triggers firefox to check for any new studies on the next launch I stumbled upon this myself earlier
  - _0ffh 7y ago
    I've seen the same tip elsewhere and tried, but didn't work for me. What did (seem to) help was setting app.normandy.run_interval_seconds to a small value (21). At least just a couple of seconds after I did, all my addons came back. Edit: plugins -> addons
    - Dylan16807 7y ago
      Remember to put it back to normal after!
- akerro 7y ago
  I just checked my preferences to make sure it's still disabled.
  - jchw 7y ago
dvanwag 7y ago
Mozilla has been on the downward spiral over the last several years. They took something (i.e. Firefox) that wasn't broken and "fixed" it until it was, first by killing off XPCOM and then suffering through the misadventures of such bastard products as Firefox OS. The folks at Mozilla should really stick to what their good at and focus on an all around open source browser that people will actually WANT to use.
- asutekku 7y ago
  That doesn’t make money which kinda is required to work on Firefox.
  - chrisseaton 7y ago
    I thought they made money by sending searches to Google. A browser that people want to use means more searches and so more money to keep working on the browser. Childish stuff like TV show references, and this certificate issue, means less users, less searches, less money.
    - ChrisSD 7y ago
      They've been attempting to diversify their income for ages. A lot of people have issues with Firefox being substantially reliant on Google.
  - super_mario 7y ago
    But they have money to trow at Antifa fascists.
    - TazeTSchnitzel 7y ago
      Anti-fascist fascists?
      - super_mario 7y ago
        anti-fascist only in name, but behave exactly like fascists.
hartator 7y ago
Isn’t the issue that they’ve forgotten to renew a SSL certificate? Why not just renew it?
- Xylakant 7y ago
  It’s not an SSL certificate. It’s a certificate that’s used by the browser to validate the signature of an installed extension. It’s baked into the release.
- deleted 7y ago
  [deleted]
Grollicus 7y ago
I'm interested in the general writeup what went wrong that they missed this certificate expiring. That's a structural problem. Also why it took 6 hrs to assign P1 to the bug
- gpm 7y ago
  They closed the trees (stopped merging other code changes to prioritize this) for the bug <22 minutes after it was opened. I would assume the delay in assigning P1 is really just a result of assigning P1 not being as high priority as fixing the damn problem.
  - JamesMcGill 7y ago
    This is one of the things about this whole episode that I find baffling. Stuff like adjusting bug priorities and arranging for someone to tweet an announcement is the work of a good engineering manager. This is the right person to run interference and handle comms and deal with things outside of the critical path, like bugzilla updates.
    - dao- 7y ago
      The priority field just plays no practical role here since the bug was immediately escalated.
      - pbhjpbhj 7y ago
        If people can see it externally it definitely has an effect, "oh moz aren't taking this browser breaking bug seriously". Which is probably why the parent said it's a management issue rather than a directly technical one, per se.
    - rhizome 7y ago
      Stuff like adjusting bug priorities and arranging for someone to tweet an announcement is the work of a good engineering manager So we can come to the obvious conclusion about Mozilla, then? No good "engineering" managers? Miss one reprioritization and you're out! This is what sane people think?
gpm 7y ago
Instead of enabling studies just click on this link. It installs that specific "study" (hotfix) without installing anything else. https://storage.googleapis.com/moz-fx-normandy-prod-addons/extensions/hotfix-update-xpi-intermediate%40mozilla.com-1.0.2-signed.xpi https://storage.googleapis.com/moz-fx-normandy-prod-addons/e...
- deleted 7y ago
  [deleted]
- _JamesA_ 7y ago
  Thanks. That was easy.
- fyfy18 7y ago
  This works for Firefox on Android too.
  - RandomBacon 7y ago
    Thank you. I can also verify this works on FF for Android.
- JamesMcGill 7y ago
  Hold up there. Before people start clicking and installing random add-on links, how about linking to something official (either from a FF dev, or in a soure repository) that references this URL?
  - gpm 7y ago
    So, I just got this url from this HN comment: https://news.ycombinator.com/item?id=19825921 https://news.ycombinator.com/item?id=19825921 I'm not clear if they rehosted the XPI or if that's the original mozilla url. I'm not too worried about it either. The only reason anyone is clicking on this fine link is because firefox only lets you install addons signed by Mozilla. And since the typical signing process gives addons signed by the broken intermediary we can be pretty confident that this wasn't just signed by mozilla, but is the original study. In general caution about installing software from random links is definitely a good idea though. Edit: Looks to me like it's an original mozilla url (judging by github comments on mozilla/normandy - I haven't found an official source saying it is official due to lack of continuing to search: https://github.com/mozilla/normandy/pull/1697 https://github.com/mozilla/normandy/pull/1697)
open-source-ux 7y ago
I hate to say all these things because I use Firefox all the time, but...the communication around the add-ons issue has been poorly handled by Mozilla. I only learned of the problem by visiting HN. But what of the thousands of other users who don't visit HN? If you visit the Mozilla homepage, there is nothing to acknowledge the problem (at least at the time of writing this message). Let's try the Support page. Where is it? Scroll down to the bottom of the lengthy Mozilla homepage to the page footer to find the link. (How many visitors will make it to the bottom?) When you click through to the Support page, an easy-to-miss banner in tiny text appears at the top of the page that mentions the problem - screenshot here: https://imgur.com/a/TAHZSWa https://imgur.com/a/TAHZSWa Additionally, when the add-ons are disabled, Firefox misleading says: "These extensions do not meet current Firefox standards so they have been deactivated". This is probably a generic message but it's also an example when a generic message is misleading. Finally, poorly-named settings like "Normandy" and "studies" that give no hint of their meaning only adds to the confusion.
- ekianjo 7y ago
  That's sadly not the first time Mozilla fails to communicate appropriately about issues/changes that are pushed down to the end users. They should reshuffle some of their Marketing Resources to work on proper non-promotional communication instead, so that current users at least know what to deal with.
- notimetorelax 7y ago
  The way I see it, people might have gotten used to software break from time to time. Once software breaks it is reasonable to expect it to get fix in a couple days when it is updated. At least this was probably the experience for the majority of users, those that noticed the issue.
  - orthecreedence 7y ago
    The sad reality is Mozilla has been losing mindshare to Chrome for a long time and this will rapidly accelerate it. People don't expect things to break. They expect things to work, and when things break they get angry. I love Firefox. It's my daily driver. It will continue to be. But this is a huge fuck-up and they're probably going to pay big in usership because of it.
dingo_bat 7y ago
I asked this in the other thread but I guess there's too many comments there: Is there a project for Firefox that is analogous to Chromium for Chrome? I need a Firefox build with all the Mozilla shit ripped out. I don't trust the org that decided their certificate expiration was more important than giving users the choice to run what they want.
- Karunamon 7y ago
  Librefox isn't exactly what you described, but it's close. It's a set of configs that disable a bunch of telemetry and other unauthorized mothership connectivity and settings pushing. https://github.com/intika/Librefox https://github.com/intika/Librefox However, Librefox is only Firefox with some configuration changes. It is not a whole new build, and it wouldn't have protected you from this problem since the problematic addon cert checking is still there. Note that this would have happened even if the browser never communicated back home - this problem was triggered via an unwitting time bomb of sorts, not because Mozilla actively took an action that inadvertently broke something.
  - dingo_bat 7y ago
    > Note that this would have happened even if the browser never communicated back home - this problem was triggered via an unwitting time bomb of sorts, not because Mozilla actively took an action that inadvertently broke something. I was never worried about Mozilla's telemetry. I happily enabled all feedback/telemetry options because I genuinely wanted them to fix any problems. I understand that this problem is not caused due to phoning home. The decision process that led them to make it impossible to load an extension without it being signed by them is problematic. It means that I don't trust them with all my debugging data. I don't know what other time bombs are hiding inside the code. What would happen if Mozilla ceased to exist tomorrow? Will existing firefox installs get bricked after some time? Even big bad Microsoft allows you to install unsigned hardware drivers if you dismiss the scary warning.
- lugg 7y ago
  Upvoting because this is a really good question, why the hell can't I get a warning and click a button to enable the thing anyway?
deleted 7y ago
[deleted]
kreetx 7y ago
Would have disabling automatic updates have prevented this entire issue?
- mfjordvald 7y ago
  Don't think so, from what I understand, the problem was the intermediate certificate expired, it would have expired regardless if there were no automatic updates.
  - Karunamon 7y ago
    Even on completely isolated distributions like Tor Browser or enterprise ESR installs. The only way you avoided this is if you were running Nightly or Developer or the normal one on Linux, and you disabled signature checks.
    - kreetx 7y ago
      Yup. Am on macOS and Nightly and still got hit with the isdue (luckily the fix was already out). Guess we'll see a post-mortem soon and get to know how did this even came to be.
no-razors-here 7y ago
At this point, I have to figure Mozilla has been infiltrated by saboteurs. It’s maliciousness masquerading as incompetance. The inverse of Hanlon’s Razor, to be sure.
Tharkun 7y ago
I have a bunch of privacy-enhancing addons installed, which have now all been disabled. If I hadn't read HN this morning, I wouldn't even have known why. Until now, I had no idea that it was even possible to remotely disable my addons. And now Mozilla are saying that the "fix" is to allow them to install & run "studies" on my machine? What are they smoking? I'm having a hard time trusting a company that randomly & remotely disabled all my addons, regardless of the cause.
- zamadatix 7y ago
  Not initiated by user != remotely initiated. And agreed no the fix side, they should have posted how to do it yourself as well.
- sgift 7y ago
  No one remotely disabled anything. There's a certificate deployed with Firefox. The certificate Firefox used to check addons was only valid till yesterday. So, when the browser started next time it couldn't validate the addons and disabled them. That all happened locally.
  - overgard 7y ago
    You could say it was remotely disabled by design. What other piece of software randomly just breaks because of the calendar date? I can boot up almost any 20 year old piece of Windows software and it'll work fine, it might not make sense in the current world but it won't go "2019? Fuck off!"
    - Xylakant 7y ago
      And the reason you can install 20 year old windows software without caring about code signing certs is that 20 years ago nobody bothered to sign code.
      - overgard 7y ago
        Not every piece of code needs to be signed. Should my ancient copy of Doom 2 stop working because it's not with the times? Or a level editor for it? Or an old turboC compiler? Some software lives a LONG time and it's fine, and it's up to the user whether that software is still useful to them or not. Seriously how many posts do we see on hacker news about like "We rebuilt this ancient machine from the 1970s to learn about it." People care about computing history. Not everyone, but there's no reason to force your software to break because the calendar rolls over. Remember Y2K? Things often live a LONG time. People are still actively writing Fortran and COBOL. The short-sightedness of this is amazing, as is the condescending "we know what's best for you" security argument.
ncmncm 7y ago
Can anybody confirm thst Mozilla is scrubbing replies on the linked page that mention about:config and toggling "xpinstall.signatures.required" ? I find it suspicious that no replies there mention it.
- zamadatix 7y ago
  I found 5 replies mentioning it, the earliest on page 3. It's more likely it wasn't mentioned as often as it only works for the minority of the install base: "The Nightly and Developer Edition versions of Firefox have a preference to disable signature enforcement. There are also be special unbranded versions of Release and Beta that have this preference, so that add-on developers can work on their add-ons without having to sign every build. To disable signature checks, you will need to set the xpinstall.signatures.required preference to "false"." https://wiki.mozilla.org/Add-ons/Extension_Signing https://wiki.mozilla.org/Add-ons/Extension_Signing
  - eikenberry 7y ago
    ESR release in Debian also has this option available. I think the option is generally available with Firefox installed from any Linux distro.
dehrmann 7y ago
The study pushed, but all my installed plugins are listed as "unsupported" and says "<plugin> could not be verified for use in Firefox and has been disabled."
Causality1 7y ago
So the studies update has hilariously enabled my essential legacy extensions while leaving my more modern WebExtensions still disabled. Way to go, Mozilla.
erikpukinskis 7y ago
This is why you always have one of your employees keep their clock 30 days in the future. Preferably someone who doesn't go to meetings and installd updates.
gshdg 7y ago
Huh, doing this re-enabled my add-ons but nothing is listed under about:studies.
overgard 7y ago
I know Firefox isn't being malicious, but ugh, this seems like the worst possible PR move for this, optics wise. "Hey so uh, we accidentally broke your browser, so you need to opt-in to becoming a guinney pig. But don't worry! You probably were already opted in anyway and just didn't realize it! Also it might take six hours to work."
- floatingatoll 7y ago
  They only point out the opt-in instructions for the few people that voluntarily opt out of Shield studies and wish to get the fix sooner. Most Firefox users have that checkbox enabled by default, and so most Firefox users received the fix within 0-6 hours of the blog post's publication. HN readers often take special care to prevent Mozilla from updating Firefox, but that in no way represents the wider population of either all addons users or all Firefox users.
  - toyg 7y ago
    This has nothing to do with "studies" and should be pushed as a general fix. There is just no excuse. What if this had been a security incident?
    - floatingatoll 7y ago
      Then it seems it’s a good thing they had a system in place that could deliver the quick fix in less than a day, on a Friday evening, when shipping a normal fix could have — is — taking longer to ship than that quick fix did.
      - toyg 7y ago
        > it’s a good thing they had a system in place that could deliver the quick fix No, it's not. "Studies" is not a security-related mechanism and it didn't exist in the past, when fixes were rolled out very quickly anyway for security reasons. "Studies" should not be relied on to be a fix-delivery mechanism, because it just isn't. This is not even about privacy, it's simple good engineering sense.
wool_gather 7y ago
Another workaround if you don't want to enable "studies" is to manually re-load the add-ons in Debug Mode. I don't know the full consequences of this, but Firefox seems to be behaving normally having done it. Go to about:debugging from the address bar. Right at the top is a button to "Load Temporary Add-on", with a checkbox "Enable add-on debugging". (On a Mac, the add-ons are in ~/Library/Application Support/Firefox/Profiles/«ID».default/extensions (assuming that you have only a single profile).) They should stay enabled until Firefox relaunches.
lostmsu 7y ago
Sadly, this removed my settings for multi-account containers extension :(
- lostmsu 7y ago
  Just heads up here. I was able to restore it partially on Windows using https://www.shadowexplorer.com/downloads.html https://www.shadowexplorer.com/downloads.html (which is, btw, a great tool!) You'd be looking for a file C:\Users\YOUR_USER_NAME\AppData\Roaming\Mozilla\Firefox\Profiles\YOUR_PROFILE\containers.json and also ...\YOUR_PROFILE\browser-extension-data\@testpilot-containers
  - ruricolist 7y ago
    Thanks, you just saved me a headache.
furgooswft13 7y ago
Granted I'm using Nightly (and previously disabled extension signing in about:config), but now all my themes are disabled, even the default one apparently, though that is what it is using. Cannot be re-enabled. When do I get my dark theme back? Also...my default search engine is now Amazon.com?? WTF is going on. EDIT: Also my only search engine. Heck of a job Mozilla.
- lugg 7y ago
  Bezos and his strategy to name the company something starting with A so it always comes first finally pays off.
  - furgooswft13 7y ago
    Now I understand why Google renamed itself Alphabet. One step ahead, literally.
    - sydney6 7y ago
      Have you been able to restore your search providers? Really a kick in the nuts, this one..
      - furgooswft13 7y ago
        Nope. Maybe I should restart Nightly again but...I just added Google back, and it was not as easy as I'd expect. The Firefox "add search engines" page is a mess and straight Google search was like 3 pages down. Also I found another dark theme that is somewhat similar to what I had (but not official Mozilla) and installed that. All the default ones are still disabled.
        sydney6 7y ago
        Thanks.. Same here on every machine.. The hotfix update has been installed and add-on functionality is indeed recovered, but still i only have "amazon.com" as a pre-installed search provider and have been unable to restore them to their default settings without creating a completely new profile. Besides, i would be rather hesitant to install search providers from that list, imho.
pmlnr 7y ago
about:config xpinstall.signatures.required => false Yes, this will void your warranty.
vesinisa 7y ago
For me (repository firefox on ArchLinux), the temporary fix was setting devtools.chrome.enabled = true in about:config, and running this small JavaScript snippet in Chrome DevTools (Ctrl+Shift+J): https://wiki.archlinux.org/index.php/Firefox#Firefox_disables_all_extensions_due_to_Mozilla_not_renewing_their_certificates https://wiki.archlinux.org/index.php/Firefox#Firefox_disable... AFAIK, this will enable all the disabled add-ons until the next check, which is in 24 hours. This will be hopefully enough time for Mozilla to release a stable channel update, instead of the "Studies hack". At least for me, fiddling with the Studies settings had no effect; the about:studies page remained empty regardless of what I did. I've also seen multiple reports from people who got the Studies hack working that the fix actually failed to address the issue properly.
- nathanaldensr 7y ago
  It appears to take several minutes for the studies to be downloaded and enabled. The process worked for me on two separate Windows 10 machines.
- deleted 7y ago
  [deleted]
- eikenberry 7y ago
  On Linux you should also have the option of disabling the cert signature check. Go into about:config and set `xpinstall.signatures.required` to false. Just remember to set it back to true once this is fixed.
LifeLiverTransp 7y ago
Best chrome add there ever was.
deleted 7y ago
[deleted]
gnud 7y ago
I'm not gonna bother with 'studies' or manual workaround - I'm just going to wait for an update. In the meantime I'm enjoying trying out Vivaldi[1] - really reminds me of opera 3/4, that I loved. 1: https://vivaldi.com https://vivaldi.com
- Causality1 7y ago
  Will Vivaldi let me put the tabs on bottom where they belong?
  - jobigoud 7y ago
    What? Tabs belong on the side. In a tree. ;-)
  - keeganwitt 7y ago
    Yes, Vivaldi supports tabs on your choice of any of the 4 sides of the window.
- ripdog 7y ago
  Hmm, I'm not sure switching to a closed source browser is in any way an upgrade... Especially when you could switch to the unbranded/nightly firefox builds, disable addon signature checking and continue using the only independent FOSS browser remaining.
  - gnud 7y ago
    Vivaldi is only slightly more closed source than Chrome. Anyway, used to be Firefox had a lot more going for it than being open source. Now that's really the only thing left I can think of.
the8472 7y ago
At the minimum they should add a testsuite that runs at least a month into the future to catch these kinds of things. There was a similar issue[0] a few years ago that was only caught a month in advance. Even better would be to set things up to only do a verify on install instead on every startup. [0] https://bugzilla.mozilla.org/show_bug.cgi?id=1267318 https://bugzilla.mozilla.org/show_bug.cgi?id=1267318
- Mindless2112 7y ago
  > Even better would be to set things up to only do a verify on install instead on every startup. That would defeat the purpose of verification: "Add-on signing in Firefox helps protect against browser hijackers and other malware by making it harder for them to be installed." [1] And it's not just malware that was doing that. Microsoft force-installed the ".NET Framework Assistant" into Firefox on Windows, and you had to edit the registry to remove it. [2] If I recall correctly, AVG and Logitech were also among the list of offenders. [1] https://support.mozilla.org/en-US/kb/add-on-signing-in-firefox https://support.mozilla.org/en-US/kb/add-on-signing-in-firef... [2] https://support.microsoft.com/en-us/help/963707/how-to-remove-the-net-framework-assistant-for-firefox https://support.microsoft.com/en-us/help/963707/how-to-remov...
  - deogeo 7y ago
    Could you explain why verifying on every startup, instead of just on install, is necessary? The page you linked doesn't mention it. Edit: Let me amend my question - why is it necessary for the certificates to expire? If a plugin is signed by Mozilla, why wouldn't it be trusted once it gets old?
    - Xylakant 7y ago
      There’s no guarantee that a (malicious or undesirable) extension was added via FF, so check on install could be circumvented by any of the crapware installers we all love.
    - makomk 7y ago
      How do you propose Firefox should tell whether the unsigned, shady add-on was installed by the user, or by some other dodgy app messing with Firefox's files whilst it wasn't running to make it look like the user installed the add-on?
pbhjpbhj 7y ago
On Android I get this: >We rolled out a hotfix that re-enables affected add-ons. The fix will be automatically applied in the background within the next few hours. For more details, please check out the update at https://support.mozilla.org/en-US/kb/add-ons-failing-install-firefox https://support.mozilla.org/en-US/kb/add-ons-failing-install... Which is like "we did something we shouldn't have causing unauthorised changes to your computer, so we're going to make unauthorised changes to fix it". Quite telling is that this is supposed to protect us from other developers. On the add-on screen "Enable" is greyed out, there's no "Enable even though Mozilla doesn't like it". The UX is just like the "fuck you this computer isn't yours it belongs to Microsoft and we'll do what we like with it" that I thought I'd left in the past decades ago. It's not your computer Mozilla, you fucked it up, you don't get to mess around with it without asking the owner. My understanding is that this is literally illegal in the UK. Mozilla barely had any trust left to burn IMO but they sure went all out.
- morpheuskafka 7y ago
  As I understand it, you agree to the terms of Studies as part of the ToS agreed to on installation. You can disable it later. And--while it was a ridiculous mistake--they didn't make any "unauthorized changes" to your computer. They just let a certificate expire and your computer, running the same code it always had, stopped trusting it.
  - admax88q 7y ago
    Hiding behind ToS is ridiculous. Nobody reads them and a moral company should never assume that because its in the ToS they actually have informed consent.
    - MichaelBravo 7y ago
      TOS agreements should be illegal in the US, especially those that try to circumvent our 'Freedoms of Speech'.
      - 7y ago
akersten 7y ago
Very curious how the decision to use the Studies program happened. Why not just roll the version early and include the fix in the new version - isn't Firefox an evergreen browser now? Maybe there is extra bureaucracy to roll a new version, or the hotfixers didn't have permission to do it. Either of which I can understand, being that they made the fix late on a Friday night - so huge kudos to those who worked hard to get this fix started. Seems like a good case study for lessons learned here, I'm eagerly anticipating the postmortem and follow-ups.
- mccr8 7y ago
  Doing a full release takes a lot longer, so they presumably decided to use a faster method where possible. https://hacks.mozilla.org/2018/03/shipping-a-security-update-of-firefox-in-less-than-a-day/ https://hacks.mozilla.org/2018/03/shipping-a-security-update...
argd678 7y ago
Certificate expirations show up on causes of outage lists so frequently, yet little has been done to address the underlying issues of how PKI works to address it. The core issue of time limits on trust and no specs and requirements on how to handle the most common case of saying “yes, this is still trusted” is oddly absent. Could we have for example a publicly verifiable ledger that can be used to verify a cert chain with not only a defined workflow to answer if a cert is still trusted but also a requirement for the workflow to be fully implemented? Seems quite doable, vs sort of hacks of auto-renew which are hit and miss depending on the CA. In other words, when do we fix the sport rather than the players here?
DireLizard 7y ago
Assuming I understand it correctly the reason the addons stopped working is because an intermediate certificate they were signed with expired, question is - when and why are already installed and running addons are being revalidated every time (and when exactly ?), does this serve any user-aligned purpose or is just means for mozilla to exercise control over addon publishers ?
NicoJuicy 7y ago
Install https://storage.googleapis.com/moz-fx-normandy-prod-addons/extensions/hotfix-update-xpi-intermediate%40mozilla.com-1.0.2-signed.xpi https://storage.googleapis.com/moz-fx-normandy-prod-addons/e... Restart browser Fixed
mrb 7y ago
It's incredibly strange that this blog post doesn't even link to the bug description https://bugzilla.mozilla.org/show_bug.cgi?id=1548973 https://bugzilla.mozilla.org/show_bug.cgi?id=1548973 Imagine someone stumbling upon this post and trying to find more technical information... Poor communication.
tianheyang 7y ago
If this hadn't happened, I probably wouldn't have discovered Brave. Noticeably faster and the ad-blocking is built in. Thanks Mozilla
firefox1234 7y ago
It seems my mistrust in Mozilla is not misplaced. Every time, I install Firefox, I open about:config and search for all http[s] / [s]ftp links and set them all to random strings. My installed extensions still work.
clairity 7y ago
hmm, i don't seem to have been affected by this bug somehow (my extensions are all still working). i turn off as much phoning home as i can (including turning studies off) and block connections to *.services.mozilla.com any idea why i might not be affected? it may help others who might want to retain control of their firefox browser (chromium-based browsers being non-sequiturs).
- kermitismyhero 7y ago
  It was the same for me. Firefox 66.0.3 on Xubuntu 19.04. I also turn off a lot of phone-home behaviour. And I never had any problems with extensions getting disabled. I have no idea why I wasn't affected. But frankly, "not being affected" isn't good enough for me. Even if I dodged this bullet I might not dodge the next. I'm looking for an alternative browser. Falkon has been interesting so far. It's a little bare-bones in many ways, but at least it's immunue to any future Mozilla screwups.
  - Gene_Parmesan 7y ago
    One screwup is enough for you to jump ship? Or have there been others? (I'm genuinely curious, not trying to be aggressive.) I personally don't think all this talk of "Bye-bye firefox" is quite fair (I'm referring to a number of comments on this page, not yours specifically). In my opinion (you may disagree), Mozilla is one of our best and strongest allies in the fight for a fair internet. Their values matter because they still have the user base to back them up. If all of us technical-minded folk jump ship to smaller, boutique browsers at every bug and gaffe, leading to a sort of browser balkanization, then Firefox loses its strength and those smaller browsers aren't impactful enough to be able to resist Chrome/Chromium. But, maybe your choice of browser is purely utilitarian, and that's totally fine of course.
    - chappi42 7y ago
      It's not one screw up. I love the sync functionality and Rust/Syncro is great. But on the other side it feels like 'they' take constantly power functionality away while the shallow and shiny homepage claims: "More power to you". But maybe I'm just still sour because of the Eich ousting?
Chardok 7y ago
The most frustrating part of this for me is there is no (relatively) easy way to override this behavior. Its fine to disable the addons, but please allow me to "understand the risks" and continue against Firefox's recommendations. The feeling of no control over my web browser was why I left Chrome in the first place.
watermelon0 7y ago
IMHO it seems problematic, that they can remotely push code changes, including replacement of trusted certificate, and bypass package managers. I don't expect software to (significantly?) change during runtime, outside of what was packaged, signed, distributed and installed as part of apt/yum/pacman/etc. I understand (not that I like or agree with) that some apps are just embedded web browsers, and load everything externally, and that Firefox extensions are in the end just some JS/CSS/HTML loaded outside of system's package manager. However, extensions have limited API they can interact with, and you need to allow permissions for each extension. Having Mozilla owned extension, that can modify core functionality, seems a bit scary.
- benbristow 7y ago
  If you didn't have browsers auto updating no-one would update them manually, meaning bad news for web developers wanting to take advantage of newer features.
  - HNthrow22 7y ago
    I understand the appeal of that for developers but it comes at the cost of users agency and control of their own system, I've been very annoyed with even simple UI changes in firefox updates as I simply didn't ask or want any such change. Reading other comments here it's clear I'm a dying breed of old and stubborn users that prefers full control and agency over my own system. Making it easier for web developers to implement new features is absolutely not a tradeoff I'd make willingly at the cost of my systems consistency and reliability. Also the reason I use firefox is because of all the major browsers vendors they seem the most aligned with those values although this seems to be changing more and more every year.
    - damnyou 7y ago
      The incentive structures of society (capitalism, if you're so inclined, but I don't think this is unique to capitalism) are incompatible with your wishes.
    - benbristow 7y ago
c64ec30650ea3c4 7y ago
I'm shocked and suprised to find out that mozilla is using EXPIRING certificates for this. It requires them to continuously take action to prevent all addons from breaking, which will eventually fail (like it did). Firefox has a pretty robust update system and everyone is used to frequent updates. Why don't they instead have a revocation system built into updates? That way they would have to take action to disable malicious addons, and the good ones could go on working forever. Is there something about this idea that is so much worse than what happened today?
- user17843 7y ago
  You are right. It doesn't make any sense to use certificates for this kind of stuff. If an extension turns out to be malicious, you simply deactivate it in the store, and then proactively deactivate the existing installs. This is how Chrome is doing it. But having a certificate does offer Mozilla the feeling of absolute control, which seems to be of primary importance for them nowadays. This is probably the reason release and beta users are not even allowed to deactivate signing in the about:config settings.
martin_a 7y ago
So, I left Chrome for all the b's they were doing with/to the web. Now Mozilla is fcking it up, too. Which browser to choose now?
- userbinator 7y ago
  IE6. No, I'm only half-serious. ;-) Remember when the Web was mostly about sharing information, browsers didn't silently auto-update nor break in the process of doing so, organisations didn't add invasive "telemetry" to everything, and things would mostly stay working because the pace of change was generally much slower? Now that the "keep pushing it forward and breaking things" trendchasers seem to have gotten their way, instead we have the constant churn of web development, increasingly bloated sites and JS annoyances, browsers becoming more complex and fragile than OSs, dumbed-down UIs and taking control away from the user --- yes, that includes Mozilla who got to where they are today for their "user freedom respecting" position, and the repulsively ignorant "newer is always better" mentality that's infected even search engines like Google. Maybe I'm just being overly nostalgic, but incidents like these really put things into perspective.
  - martin_a 7y ago
    I understand what you mean and I think I feel quite the same. "The web" has just become so... "strange" in the way everything works and we take care of it or however you'd like to call it. Often it's just broken with full intention to do just to push some new shiny technology on us. And I'd really like if it wasn't that way.
pleasecalllater 7y ago
This one will be emotional as this destroyed some of my today's work. F you Mozilla. I lost all my tabs opened in other containers. The containers don't work too, so I cannot reopen them. This bug has been known for 3 years, and you did nothing to fix it. You get so much money, and what you do is basically provide a pathetic software (thunderbird) and a nice browser (which you just stopped from working) and you show me banners asking for more money. You should be ashamed. 3 years. And no, I'm not going to listen things like "this is open source, you are free to fix it". I will just go and switch to another browser. I need a browser which works, not a one which suddenly decides that my stuff should be broken because all the developers and managers have been ignoring a critical issue for a couple of years. :( I know, this will be flagged, and removed. I don't care, I just need to get all my tabs in containers back. I have never thought that a browser can just close my tabs because a certificate expired.
- jchw 7y ago
  Hey, just FYI, there are some good solutions in this thread to get your problem fixed ASAP. This one should work. https://news.ycombinator.com/item?id=19827302 https://news.ycombinator.com/item?id=19827302
  - pleasecalllater 7y ago
    No. JUST A HUGE NOOOO. My reply is here https://news.ycombinator.com/item?id=19828472 https://news.ycombinator.com/item?id=19828472
    - jchw 7y ago
      We’re literally in this mess because Firefox requires digital signatures on this kind of thing, please don’t make FUD posts. This is much better than disabling the very same safe guard, signature checking, that prevents you from running arbitrary code in the first place.
      - OrgNet 7y ago
MichaelBravo 7y ago
Mozilla, what ever the fuck your trying to fix, at least fix the add blockers asap & let us get tat fix before you work for hours on the rest of whatever the helm your doing. My uBlock has been good on my android for years, there is no reason it shouldn't be good now. Your reason for blocking as I've seen is that other develoers may have hacked your shit or done such but if you block our use of our ad blockers then we can just easily get infected all the more. Fix it now please & let us get those working then continue to fix whatever the hell else your see as problematic.
- ripdog 7y ago
  First, take a deep breath and calm down. No one category of addon is more affected by this than any other, and Mozilla has no way of 'fixing ad blockers' before anything else. Once the signing certificate is replaced, all addons will work. Simply turn on studies and allow it to install the new cert, and you'll be on your way.
deleted 7y ago
[deleted]
userbinator 7y ago
Mozilla decided to make signing mandatory, then screwed up and now they're trying to fix it by making use of a feature that basically allows them to remotely execute code on all their users silently? I checked Mozilla's main site again, and it still has this ironic statement in its description tag (it's been there for many years): https://www.mozilla.org/en-US/firefox/new/ https://www.mozilla.org/en-US/firefox/new/ Firefox is created by a global non- profit dedicated to putting individuals in control online. ...I guess it's more dedicated to putting Mozilla in control now. Something about this whole incident brings up a point that just feels very wrong to me --- it's not a Google or Microsoft, but the fact that Mozilla also seems to have this large amount of control over its users is unsettling.
coffekaesque 7y ago
How long until heads roll? There's something really wrong with the organization. And I thought it was only their marketing/pr that was bad. > We can't afford to lose Mozilla and Firefox. https://news.ycombinator.com/item?id=18800360 https://news.ycombinator.com/item?id=18800360
- user17843 7y ago
  nowadays they seem to make it a hobby to make negative headlines at least once every quarter. I fear there will be no negative repercussions for the leadership. Basically, the management set their own salaries, the entire work force gets a 40% yearly bonus, and they have no one from the outside to report to. On top of all of this, the money flows regardless of what anyone is doing. (While there is a yearly loss of 10% of their users, the past deal with Verizon made them very rich, so they can go like this for years). Revenue has been only going up, despite a loss of absolute users. So this explains why they continue to do bad things even though outside observes can not understand - during the last 5 years losing users did not impact their financials in any meaningful way. While people were complaining and users leaving the product, revenue was increasing. They do take care of their employees with lots of benefits and other stuff, so as an employee you don't want to risk all that with speaking up against your superior. Over the years they have created a company culture where there are endless number of small teams doing irrelevant stuff, with absurd hierarchies, with some people doing no work at all. With 16 people in the upper management, there's also fragmentetion of decision making going on. It's all a bit headless. Due to the complicated hierarchies in the company everyone is content with doing just enough to not make life harder for anyone else - suggest to change things fundamentally and actually work on delivering a great product and you will not get very far.
  - jdashg 7y ago
    This is categorically untrue, and unhelpful.
    - user17843 7y ago
      some things I wrote I can not prove, that is right. I would love to revise my negative opinion in light of better evidence.
RenRav 7y ago
>Your Firefox extensions are all disabled? That's a bug! Is this supposed to affect everything installed? I'm running Firefox 56.0.2 and this only affected addons which I had already disabled, all the other addons are fine... still, am I forced to update to fix this bullshit?
gorbachev 7y ago
This isn't even working reliably. The hotfix study is showing up completed, not active, on my Firefox at home. No idea why, there's no information about how to reactivate it. No, re-installing it didn't help.
tiredwired 7y ago
I got the update and containers are working again but, it seems like it forgot my old container configuration and I had to set them all up again.
Metricon 7y ago
From ghacks (comment section): https://www.ghacks.net/2019/05/04/your-firefox-extensions-are-all-disabled-thats-a-bug/ https://www.ghacks.net/2019/05/04/your-firefox-extensions-ar... This should allow the extensions to work until the next check (Verified locally): 1) Shut down Firefox 2) Open extensions.json (located by about:profile -> Root Directory) 3) Replace all instances of “appDisabled”:false to “appDisabled”:true 4) Replace all instances of “signedState”:-1 to “signedState”:2 5) Save and close extensions.json 6) Start Firefox 7) Close Firefox 8) Open extensions.json 9) Replace all instances of “appDisabled”:true to “appDisabled”:false 10) Start Firefox 11) Disable and re-enable all extensions in about:addons
- svnpenn 7y ago
  it works!
huhtenberg 7y ago
Can we take a moment and consider the side effects? This is a once in a lifetime chance for Google & Co. to get a glimpse of all those sly fuckers hiding behind adblockers. This effectively uncloaked a very specific subset of Internet users and exposed them to the very companies that they've been actively trying to avoid. Not just those who avoid Chrome, but those who take extra steps to explicitly evade the tracking. Surely Mozilla, the privacy advocate, must understand the impact of this fuck up, and yet the offered "fix" doesn't even mention a one-click .xpi install, but rather asks to enable a mechanism that, if left enabled, will grant unnecessary control to Mozilla over people installs. This ain't right.
- ne0sapi3n 7y ago
  >Surely Mozilla, the privacy advocate
  - calvertdw 7y ago
    The post is still great despite that weakness.
- overgard 7y ago
  FWIW I think it's pretty easy to test if someone is using an adblocker anyway. (I see sites do the "It looks like you're using an adblocker" thing all the time). I don't know if there's any realistic way to entirely hide that.
  - tedivm 7y ago
    That's a bit different though- generally speaking those messages show up when the javascript tracker can't talk to the server it's communicating with. Even though it's "detecting" the adblock it isn't able to send information back from the client about it.
    - overgard 7y ago
      > Even though it's "detecting" the adblock it isn't able to send information back from the client about it. Sure they can, they can just send back a resource request. It could even be for like an image with a query string attached with it, it doesn't have to be an ajax request necessarily.
felixfbecker 7y ago
I think Firefox needs to stop this add-on signing and review madness. The web is OPEN. It's not a walled-garden Apple App Store. Yes, extensions run arbitrary JavaScript code. So does any webpage you go to, and nobody from Mozilla reviewed all that JavaScript either. How are extensions any different? Chrome is doing just fine without all this non-sense process and policy.
- ripdog 7y ago
  Extensions have dramatically more access to powerful APIs to affect the browser. They can be used to perform a great variety of annoying, intrusive or downright malicious actions which a website is incapable of. Of course they have much more stringent policy. Chrome implements mandatory addon signatures as well, and only Google can sign them.
  - user17843 7y ago
    I doubt google has certificates that run out automatically. Rather, the best way is with each signing to include signing the date and not allow the certificate to expire retroactively.
    - ripdog 7y ago
      All signing certificates expire. They must. It's a fundamental part of the security model, because otherwise a malicious actor could take an old, comprimised cert and inject it into Firefox, allowing them to run malicious 'signed' addons. This attack would work the exact same way in Chrome, so Chrome will expire it's certificates too.
      - speedplane 7y ago
        > All signing certificates expire. They must. It's a fundamental part of the security model Is it really necessary? Aren't there other possible ways of invalidating a certificate other than its date?
      - user17843 7y ago
makecheck 7y ago
From a UX point of view I don’t know why tools don’t have these two features: 1. “Warning, a critical method for verifying authenticity is set to expire in X days. Please visit <Y> to update now.” 2. “A critical verification certificate has expired; while you should immediately go to <Z> to obtain an update, you may defer authentication for up to 5 more days.” ...or in other words, why can’t tools cut us some slack on either side of a deadline? Security for most things is not going to fall apart just by giving people a little room to deal with issues on their own schedule.
- chronogram 7y ago
  That would be assuming this would ever happen, which clearly Mozilla didn’t expect, wrongly. It’s a shame, but eh! Mistakes were made. Precautions will hopefully be made.
wetpaws 7y ago
I'm trying to be angry at Mozilla for messing up, but cant force myself to do it. They ended up to be that adorkable kid that spilled the paint all over the carpet and you just sigh and start cleaning up the mess.
nullc 7y ago
I wonder if there is someone out there in the middle of the ocean with a browser extension based communication and navagation system which is dead in the water? It sounds to me that the real headline here is that every copy of firefox out there was timebombed and we only noticed because someone forgot to elongate the fuse.
- javagram 7y ago
  Could other code signing systems like macOS gatekeeper also be vulnerable to problems like this? IMO this seems like just plain bad design. The Firefox addon certificate should never have had an expiry date. If they ever needed to revoke it, they could distribute an updated version of the browser with the previous intermediate explicitly marked as revoked.
- zimmund 7y ago
  The browser itself continued working fine. Are you aware of any life-depending extension? Leaving this particular issue aside, your hypothetical "browser extension for people in the middle of the ocean" was doomed from its inception if it was designed to run as a browser extension (though it opens the door for an interesting discussion about similar scenarios that are happenning, like pilots relying on ipads)
  - Dylan16807 7y ago
    > your hypothetical "browser extension for people in the middle of the ocean" was doomed from its inception if it was designed to run as a browser extension Why? You haven't backed up that statement at all. Especially before they killed XUL it was easy to make a non-doomed app that runs as a browser extension, and it's still plenty possible. No (non-demo) program should brick itself if it can't connect home.
  - kwak 7y ago
    No but what about people who use password managers, that were locked out of not being able to access bank accounts, credit cards, and reddit.
  - 7y ago
Insanity 7y ago
Funny, I only just noticed my extensions were gone because of advertisements on youtube. I just assumed I somehow messed up my browser and started looking around the settings. A banner displaying why they silently updated FF and disabled the addons would have been nice as well :)
lewiscollard 7y ago
> Please note: The fix does not apply to Firefox ESR or Firefox for Android. Damn it, Firefox, because it supports extensions, i.e. uBlock Origin, is the only usable browser on Android :(
levlaz 7y ago
If anyone is running firefox-esr on debian (I am), there is a separate discussion happening here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=928415 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=928415
chenshuiluke 7y ago
Am I the only one whose addons weren't affected at all?
zwaps 7y ago
I am a bit salty that this reset my default search to Google of course, just an option away. Still :<
- mk89 7y ago
  I am also extremely pissed. Still, I was searching for alternatives on Android. It's ridiculous that no other browser allows extensions, not even Chrome itself. Firefox is way more advanced on this. That's one more reason that is very hard to say goodbye to them...
deleted 7y ago
[deleted]
philipwhiuk 7y ago
Patching security bugs using Shield is utterly stupid. What would happen if they found a Zero Day - would they use the same method?
stubish 7y ago
Good to see Mozilla thinking out of the box and getting a fix out as quick as possible. It would take a lot of time to get an emergency build out of standard channels, because there are so many of them.
rodorgas 7y ago
I don't like the fact that I can't run an extension that wasn't signed by Mozilla. This behaviour lacks freedom.
aswan 7y ago
(disclosure: I am a Mozilla employee but not commenting in any official capacity) "Give me control over what code I run on my computer" (meaning "provide a switch to disable the requirement that extensions be signed") keeps coming up over and over. And perhaps it hasn't been clearly stated but the problem is this: if there's a switch that a user can flip, the browser has to record the state of that switch somewhere (presumably on disk). If such a switch becomes available, we'll quickly be flooded with malware that flips that switch without users' consent. At that point, there's no way to tell the difference between savvy users making an informed choice to enable unsigned extensions and malware doing it behind their backs. The browser can do various things to obscure the way that setting is stored, but ultimately any method the browser uses to read and write the state of that switch is something that other software can easily mimic. This is not a theoretical concern, a modern web browser target is an irresistible target for all sorts of get-rich-quick scammers -- if you don't experience this day-to-day its due in no small part to the fact that browser vendors among others are constantly working to keep the bad guys at bay. But make no mistake: the bad guys are out there and they quickly find and exploit any opportunities that are available to them. So as to the problem of how to let users disable signing but ensure that they have made a conscious decision to do so, there is a stark tradeoff here: giving the most savvy users that switch necessarily makes other users less safe. The solution that Firefox has opted for here is to handle this tradeoff differently on different channels. The release channel (aka the stable channel, or the thing you get by default when you download Firefox) is intended for a very wide audience, and so it handles this tradeoff by favoring safety for all users regardless of their level of technical knowledge. The developer edition and nightly channels are intended for more technically savvy users and they handle this tradeoff differently; specifically they do provide a switch for disabling extension signing. If there are other (practical and effective) ways to solve this problem of determining true user intent, I (and I'm sure many many others) would be very interested in hearing about them. In the mean time, using the mass-market versus developer-focused channels as a signal for users' preferences on the risk-configurability continuum seems like a reasonable way to handle this.
- greendestiny_re 7y ago
  My gripes with the switch paradigm are that it: a) isn't transparent b) doesn't empower the user c) isn't easily modifiable a), b) and c) are the exact opposites of what open source software is meant to stand for. Firefox is slowly losing its unique position of being an amazing open source browser in favor of what seems to me a negligible increase in user security. In my mind, Mozilla is wasting time on micromanaging user risk instead of actually innovating. To put it this way, every time I go out biking, I can get hit by a car. It is a known and well understood risk, one that I have to consider whenever making a turn. However, riding a bike also provides chances to go faster, meet new people and so on. Should Firefox aim to reduce my risk of being hit by a car? No, because I get to choose the level of risk in my life, not Mozilla.
spevman 7y ago
Well, I'm about to make a lot of people happy with this info. I was researching this today as I'm using FF 56.0.2 and found the solution on this discussion thread. Leave it to an end user to do the job the professionals either failed or refused to due. It worked for me on 3 different machines. Go to this link and follow the instructions detailed: http://bit.ly/2DUiOLN http://bit.ly/2DUiOLN

455 points by akyuu 7y ago

peshooo 7y ago

Firefox has an interesting backdoor...

SquareWheel 7y ago
They also have a frontdoor: built-in automatic updates.
- mlthoughts2018 7y ago
  This type of reply is ludicrous. An obscure feature (Normandy modifying default settings as part of studies) that requires you to actively opt out (and even then it’s still not clear if you _also_ have to go to about:config to _really_ disable it) and which can make such large scale errors as evicting all extensions is in absolutely no way comparable to the case of a user agreeing to auto-updates and fairly easily being able to disable them. It is astoundingly disingenuous to act like these things are comparable.
  - SquareWheel 7y ago
    I think you're overstating it. >It is astoundingly disingenuous to act like these things are comparable. Why aren't they comparable? In both cases, it's Mozilla pushing code to the end user. There's a different process behind both but calling one a frontdoor and one a backdoor seems apt to me. >and which can make such large scale errors as evicting all extensions Normandy was not used to disable all extensions. It was caused by a certificate expiration error completely independently. Normandy is being used instead to work around the error until a more permanent fix can be issued.
    - mlthoughts2018 7y ago
      > “Why aren't they comparable?” Because regular auto-updates are easy to understand and turn on/off. Normandy is clearly extremely hard for many users to understand, enabled by default, and hard to disable.

pmontra 7y ago

No Firefox Studies on Android. Maybe they'll release a whole new version.

AsyncAwait 7y ago
You can click on https://storage.googleapis.com/moz-fx-normandy-prod-addons/extensions/hotfix-update-xpi-intermediate%40mozilla.com-1.0.2-signed.xpi https://storage.googleapis.com/moz-fx-normandy-prod-addons/e... to get the hotfix on Android.
- classichasclass 7y ago
  I tested this earlier and it works great here.

tempodox 7y ago

An article that mentions a timeframe of “the next few hours”, but doesn't have any timestamp besides a date without a timezone.

reubenmorais 7y ago
You can hover the date to see the full timestamp, it's 2019-05-04 07:01:35 UTC-7.
- teej 7y ago
  Not available on all clients, notably mobile.
- TeMPOraL 7y ago
  It's an annoying UX antipattern, the only thing worse being not adding the on-hover text at all. Displaying date and time should be the default. HN has this problem too.
sachdevap 7y ago
Actually, if I'm not mistaken, the complete timestamp is available when you hover over the date.
teej 7y ago
Poor communication with zero accountability. A few hours could be 2, it could be 12. “I (name the individual accountable) will give you an update at 12:00 PT (name a time) as an update to this post (name the communication channel) with the current status and latest information on this issue (don’t promise time to resolution, just time to info).” Simple, clear, concrete, and unambiguous. I had hoped that Firefox had better communication procedures in the event of global-impact P1 issues.
derefr 7y ago
If Mozilla Studies are implemented the way other "push" update systems are, then probably your browser has an ID that it hashes to get a bucket ID that it builds into a URL to check for updates, plus a cron time offset for running those checks. Then, the experiments are rolled out by walking up the bucket ID list and gradually adding the addon to said buckets. Usually, this mechanism is explained as being helpful to ensure a rollout of an experimental update can be rolled back if it's failing. That's not so much a concern in this case, I think. But this mechanism has another effect: it works as a solution to the thundering-herd problem. Every browser updating at once is bad, not just for Mozilla's servers, but for every piece of Internet infrastructure that those browsers (and their arbitrary set of addons) talk to when they update/restart. Within the time budget you have for running a rolling update, you ideally want as few machines updating concurrently as possible, just because you don't want to generate mysterious correlated traffic bursts that make NOCs paranoid.

deleted 7y ago

[deleted]

jimmaswell 7y ago

Mobile's still broken. Which is a browser Mozilla has apparently abandoned the userbase of in favor of focusing all their mobile effort on some silly pointless separate browser that caters to millenials more somehow or some other nonsense, instead of just working on the browser they already have. Really fed up with Mozilla at this point and considering switching to a fork or something.

firethief 7y ago
Servo was a silly pointless separate browser once
- gpm 7y ago
  I'm not sure I would call it "pointless", they've merged and are merging a lot of the code they used servo to experiment with into firefox. Of everything mozilla has done recently, Servo is one of the things I'm most positive about.
  - jacoblambda 7y ago
    I think that was the point they were making.
  - sachdevap 7y ago
    I'm sure your parent comment was being sarcastic. The quotes were missing.
    - gpm 7y ago
      You're probably right, oops.
- jimmaswell 7y ago
  I still hold that the multithreaded performance benefit was nowhere near worth wiping away so many of hours of developer time and ripping so many good extensions out of users' hands with no replacement for so much of the lost functionality.

jchw 7y ago

Interesting. Sadly, I imagine many users will have studies disabled since the Mr. Robot incident. I've re-enabled it but there does not appear to be a way to force it to check for updates. Guess it will show up in the next 6 hours.

dralley 7y ago
You can set app.normandy.first_run to true and then restart Firefox.
- jchw 7y ago
  Thanks, that worked! Out of curiosity, how did you know that? What does this actually do?
  - bwat49 7y ago
    I think it just triggers firefox to check for any new studies on the next launch I stumbled upon this myself earlier
- _0ffh 7y ago
  I've seen the same tip elsewhere and tried, but didn't work for me. What did (seem to) help was setting app.normandy.run_interval_seconds to a small value (21). At least just a couple of seconds after I did, all my addons came back. Edit: plugins -> addons
  - Dylan16807 7y ago
    Remember to put it back to normal after!
akerro 7y ago
I just checked my preferences to make sure it's still disabled.
- jchw 7y ago

dvanwag 7y ago

Mozilla has been on the downward spiral over the last several years. They took something (i.e. Firefox) that wasn't broken and "fixed" it until it was, first by killing off XPCOM and then suffering through the misadventures of such bastard products as Firefox OS. The folks at Mozilla should really stick to what their good at and focus on an all around open source browser that people will actually WANT to use.

asutekku 7y ago
That doesn’t make money which kinda is required to work on Firefox.
- chrisseaton 7y ago
  I thought they made money by sending searches to Google. A browser that people want to use means more searches and so more money to keep working on the browser. Childish stuff like TV show references, and this certificate issue, means less users, less searches, less money.
  - ChrisSD 7y ago
    They've been attempting to diversify their income for ages. A lot of people have issues with Firefox being substantially reliant on Google.
- super_mario 7y ago
  But they have money to trow at Antifa fascists.
  - TazeTSchnitzel 7y ago
    Anti-fascist fascists?
    - super_mario 7y ago
      anti-fascist only in name, but behave exactly like fascists.

hartator 7y ago

Isn’t the issue that they’ve forgotten to renew a SSL certificate? Why not just renew it?

Xylakant 7y ago
It’s not an SSL certificate. It’s a certificate that’s used by the browser to validate the signature of an installed extension. It’s baked into the release.
deleted 7y ago
[deleted]

Grollicus 7y ago

I'm interested in the general writeup what went wrong that they missed this certificate expiring. That's a structural problem. Also why it took 6 hrs to assign P1 to the bug

gpm 7y ago
They closed the trees (stopped merging other code changes to prioritize this) for the bug <22 minutes after it was opened. I would assume the delay in assigning P1 is really just a result of assigning P1 not being as high priority as fixing the damn problem.
- JamesMcGill 7y ago
  This is one of the things about this whole episode that I find baffling. Stuff like adjusting bug priorities and arranging for someone to tweet an announcement is the work of a good engineering manager. This is the right person to run interference and handle comms and deal with things outside of the critical path, like bugzilla updates.
  - dao- 7y ago
    The priority field just plays no practical role here since the bug was immediately escalated.
    - pbhjpbhj 7y ago
      If people can see it externally it definitely has an effect, "oh moz aren't taking this browser breaking bug seriously". Which is probably why the parent said it's a management issue rather than a directly technical one, per se.
  - rhizome 7y ago
    Stuff like adjusting bug priorities and arranging for someone to tweet an announcement is the work of a good engineering manager So we can come to the obvious conclusion about Mozilla, then? No good "engineering" managers? Miss one reprioritization and you're out! This is what sane people think?

gpm 7y ago

Instead of enabling studies just click on this link. It installs that specific "study" (hotfix) without installing anything else. https://storage.googleapis.com/moz-fx-normandy-prod-addons/extensions/hotfix-update-xpi-intermediate%40mozilla.com-1.0.2-signed.xpi https://storage.googleapis.com/moz-fx-normandy-prod-addons/e...

deleted 7y ago
[deleted]
_JamesA_ 7y ago
Thanks. That was easy.
fyfy18 7y ago
This works for Firefox on Android too.
- RandomBacon 7y ago
  Thank you. I can also verify this works on FF for Android.
JamesMcGill 7y ago
Hold up there. Before people start clicking and installing random add-on links, how about linking to something official (either from a FF dev, or in a soure repository) that references this URL?
- gpm 7y ago
  So, I just got this url from this HN comment: https://news.ycombinator.com/item?id=19825921 https://news.ycombinator.com/item?id=19825921 I'm not clear if they rehosted the XPI or if that's the original mozilla url. I'm not too worried about it either. The only reason anyone is clicking on this fine link is because firefox only lets you install addons signed by Mozilla. And since the typical signing process gives addons signed by the broken intermediary we can be pretty confident that this wasn't just signed by mozilla, but is the original study. In general caution about installing software from random links is definitely a good idea though. Edit: Looks to me like it's an original mozilla url (judging by github comments on mozilla/normandy - I haven't found an official source saying it is official due to lack of continuing to search: https://github.com/mozilla/normandy/pull/1697 https://github.com/mozilla/normandy/pull/1697)

open-source-ux 7y ago

I hate to say all these things because I use Firefox all the time, but...the communication around the add-ons issue has been poorly handled by Mozilla. I only learned of the problem by visiting HN. But what of the thousands of other users who don't visit HN? If you visit the Mozilla homepage, there is nothing to acknowledge the problem (at least at the time of writing this message). Let's try the Support page. Where is it? Scroll down to the bottom of the lengthy Mozilla homepage to the page footer to find the link. (How many visitors will make it to the bottom?) When you click through to the Support page, an easy-to-miss banner in tiny text appears at the top of the page that mentions the problem - screenshot here: https://imgur.com/a/TAHZSWa https://imgur.com/a/TAHZSWa Additionally, when the add-ons are disabled, Firefox misleading says: "These extensions do not meet current Firefox standards so they have been deactivated". This is probably a generic message but it's also an example when a generic message is misleading. Finally, poorly-named settings like "Normandy" and "studies" that give no hint of their meaning only adds to the confusion.

ekianjo 7y ago
That's sadly not the first time Mozilla fails to communicate appropriately about issues/changes that are pushed down to the end users. They should reshuffle some of their Marketing Resources to work on proper non-promotional communication instead, so that current users at least know what to deal with.
notimetorelax 7y ago
The way I see it, people might have gotten used to software break from time to time. Once software breaks it is reasonable to expect it to get fix in a couple days when it is updated. At least this was probably the experience for the majority of users, those that noticed the issue.
- orthecreedence 7y ago
  The sad reality is Mozilla has been losing mindshare to Chrome for a long time and this will rapidly accelerate it. People don't expect things to break. They expect things to work, and when things break they get angry. I love Firefox. It's my daily driver. It will continue to be. But this is a huge fuck-up and they're probably going to pay big in usership because of it.

dingo_bat 7y ago

I asked this in the other thread but I guess there's too many comments there: Is there a project for Firefox that is analogous to Chromium for Chrome? I need a Firefox build with all the Mozilla shit ripped out. I don't trust the org that decided their certificate expiration was more important than giving users the choice to run what they want.

Karunamon 7y ago
Librefox isn't exactly what you described, but it's close. It's a set of configs that disable a bunch of telemetry and other unauthorized mothership connectivity and settings pushing. https://github.com/intika/Librefox https://github.com/intika/Librefox However, Librefox is only Firefox with some configuration changes. It is not a whole new build, and it wouldn't have protected you from this problem since the problematic addon cert checking is still there. Note that this would have happened even if the browser never communicated back home - this problem was triggered via an unwitting time bomb of sorts, not because Mozilla actively took an action that inadvertently broke something.
- dingo_bat 7y ago
  > Note that this would have happened even if the browser never communicated back home - this problem was triggered via an unwitting time bomb of sorts, not because Mozilla actively took an action that inadvertently broke something. I was never worried about Mozilla's telemetry. I happily enabled all feedback/telemetry options because I genuinely wanted them to fix any problems. I understand that this problem is not caused due to phoning home. The decision process that led them to make it impossible to load an extension without it being signed by them is problematic. It means that I don't trust them with all my debugging data. I don't know what other time bombs are hiding inside the code. What would happen if Mozilla ceased to exist tomorrow? Will existing firefox installs get bricked after some time? Even big bad Microsoft allows you to install unsigned hardware drivers if you dismiss the scary warning.
lugg 7y ago
Upvoting because this is a really good question, why the hell can't I get a warning and click a button to enable the thing anyway?

deleted 7y ago

[deleted]

kreetx 7y ago

Would have disabling automatic updates have prevented this entire issue?

mfjordvald 7y ago
Don't think so, from what I understand, the problem was the intermediate certificate expired, it would have expired regardless if there were no automatic updates.
- Karunamon 7y ago
  Even on completely isolated distributions like Tor Browser or enterprise ESR installs. The only way you avoided this is if you were running Nightly or Developer or the normal one on Linux, and you disabled signature checks.
  - kreetx 7y ago
    Yup. Am on macOS and Nightly and still got hit with the isdue (luckily the fix was already out). Guess we'll see a post-mortem soon and get to know how did this even came to be.

no-razors-here 7y ago

At this point, I have to figure Mozilla has been infiltrated by saboteurs. It’s maliciousness masquerading as incompetance. The inverse of Hanlon’s Razor, to be sure.

Tharkun 7y ago

I have a bunch of privacy-enhancing addons installed, which have now all been disabled. If I hadn't read HN this morning, I wouldn't even have known why. Until now, I had no idea that it was even possible to remotely disable my addons. And now Mozilla are saying that the "fix" is to allow them to install & run "studies" on my machine? What are they smoking? I'm having a hard time trusting a company that randomly & remotely disabled all my addons, regardless of the cause.

zamadatix 7y ago
Not initiated by user != remotely initiated. And agreed no the fix side, they should have posted how to do it yourself as well.
sgift 7y ago
No one remotely disabled anything. There's a certificate deployed with Firefox. The certificate Firefox used to check addons was only valid till yesterday. So, when the browser started next time it couldn't validate the addons and disabled them. That all happened locally.
- overgard 7y ago
  You could say it was remotely disabled by design. What other piece of software randomly just breaks because of the calendar date? I can boot up almost any 20 year old piece of Windows software and it'll work fine, it might not make sense in the current world but it won't go "2019? Fuck off!"
  - Xylakant 7y ago
    And the reason you can install 20 year old windows software without caring about code signing certs is that 20 years ago nobody bothered to sign code.
    - overgard 7y ago
      Not every piece of code needs to be signed. Should my ancient copy of Doom 2 stop working because it's not with the times? Or a level editor for it? Or an old turboC compiler? Some software lives a LONG time and it's fine, and it's up to the user whether that software is still useful to them or not. Seriously how many posts do we see on hacker news about like "We rebuilt this ancient machine from the 1970s to learn about it." People care about computing history. Not everyone, but there's no reason to force your software to break because the calendar rolls over. Remember Y2K? Things often live a LONG time. People are still actively writing Fortran and COBOL. The short-sightedness of this is amazing, as is the condescending "we know what's best for you" security argument.

ncmncm 7y ago

Can anybody confirm thst Mozilla is scrubbing replies on the linked page that mention about:config and toggling "xpinstall.signatures.required" ? I find it suspicious that no replies there mention it.

zamadatix 7y ago
I found 5 replies mentioning it, the earliest on page 3. It's more likely it wasn't mentioned as often as it only works for the minority of the install base: "The Nightly and Developer Edition versions of Firefox have a preference to disable signature enforcement. There are also be special unbranded versions of Release and Beta that have this preference, so that add-on developers can work on their add-ons without having to sign every build. To disable signature checks, you will need to set the xpinstall.signatures.required preference to "false"." https://wiki.mozilla.org/Add-ons/Extension_Signing https://wiki.mozilla.org/Add-ons/Extension_Signing
- eikenberry 7y ago
  ESR release in Debian also has this option available. I think the option is generally available with Firefox installed from any Linux distro.

dehrmann 7y ago

The study pushed, but all my installed plugins are listed as "unsupported" and says "<plugin> could not be verified for use in Firefox and has been disabled."

Causality1 7y ago

So the studies update has hilariously enabled my essential legacy extensions while leaving my more modern WebExtensions still disabled. Way to go, Mozilla.

erikpukinskis 7y ago

This is why you always have one of your employees keep their clock 30 days in the future. Preferably someone who doesn't go to meetings and installd updates.

gshdg 7y ago

Huh, doing this re-enabled my add-ons but nothing is listed under about:studies.

overgard 7y ago

I know Firefox isn't being malicious, but ugh, this seems like the worst possible PR move for this, optics wise. "Hey so uh, we accidentally broke your browser, so you need to opt-in to becoming a guinney pig. But don't worry! You probably were already opted in anyway and just didn't realize it! Also it might take six hours to work."

floatingatoll 7y ago
They only point out the opt-in instructions for the few people that voluntarily opt out of Shield studies and wish to get the fix sooner. Most Firefox users have that checkbox enabled by default, and so most Firefox users received the fix within 0-6 hours of the blog post's publication. HN readers often take special care to prevent Mozilla from updating Firefox, but that in no way represents the wider population of either all addons users or all Firefox users.
- toyg 7y ago
  This has nothing to do with "studies" and should be pushed as a general fix. There is just no excuse. What if this had been a security incident?
  - floatingatoll 7y ago
    Then it seems it’s a good thing they had a system in place that could deliver the quick fix in less than a day, on a Friday evening, when shipping a normal fix could have — is — taking longer to ship than that quick fix did.
    - toyg 7y ago
      > it’s a good thing they had a system in place that could deliver the quick fix No, it's not. "Studies" is not a security-related mechanism and it didn't exist in the past, when fixes were rolled out very quickly anyway for security reasons. "Studies" should not be relied on to be a fix-delivery mechanism, because it just isn't. This is not even about privacy, it's simple good engineering sense.

wool_gather 7y ago

Another workaround if you don't want to enable "studies" is to manually re-load the add-ons in Debug Mode. I don't know the full consequences of this, but Firefox seems to be behaving normally having done it. Go to about:debugging from the address bar. Right at the top is a button to "Load Temporary Add-on", with a checkbox "Enable add-on debugging". (On a Mac, the add-ons are in ~/Library/Application Support/Firefox/Profiles/«ID».default/extensions (assuming that you have only a single profile).) They should stay enabled until Firefox relaunches.

lostmsu 7y ago

Sadly, this removed my settings for multi-account containers extension :(

lostmsu 7y ago
Just heads up here. I was able to restore it partially on Windows using https://www.shadowexplorer.com/downloads.html https://www.shadowexplorer.com/downloads.html (which is, btw, a great tool!) You'd be looking for a file C:\Users\YOUR_USER_NAME\AppData\Roaming\Mozilla\Firefox\Profiles\YOUR_PROFILE\containers.json and also ...\YOUR_PROFILE\browser-extension-data\@testpilot-containers
- ruricolist 7y ago
  Thanks, you just saved me a headache.

furgooswft13 7y ago

Granted I'm using Nightly (and previously disabled extension signing in about:config), but now all my themes are disabled, even the default one apparently, though that is what it is using. Cannot be re-enabled. When do I get my dark theme back? Also...my default search engine is now Amazon.com?? WTF is going on. EDIT: Also my only search engine. Heck of a job Mozilla.

lugg 7y ago
Bezos and his strategy to name the company something starting with A so it always comes first finally pays off.
- furgooswft13 7y ago
  Now I understand why Google renamed itself Alphabet. One step ahead, literally.
  - sydney6 7y ago
    Have you been able to restore your search providers? Really a kick in the nuts, this one..
    - furgooswft13 7y ago
      Nope. Maybe I should restart Nightly again but...I just added Google back, and it was not as easy as I'd expect. The Firefox "add search engines" page is a mess and straight Google search was like 3 pages down. Also I found another dark theme that is somewhat similar to what I had (but not official Mozilla) and installed that. All the default ones are still disabled.
      - sydney6 7y ago
        Thanks.. Same here on every machine.. The hotfix update has been installed and add-on functionality is indeed recovered, but still i only have "amazon.com" as a pre-installed search provider and have been unable to restore them to their default settings without creating a completely new profile. Besides, i would be rather hesitant to install search providers from that list, imho.

pmlnr 7y ago

about:config xpinstall.signatures.required => false Yes, this will void your warranty.

vesinisa 7y ago

For me (repository firefox on ArchLinux), the temporary fix was setting devtools.chrome.enabled = true in about:config, and running this small JavaScript snippet in Chrome DevTools (Ctrl+Shift+J): https://wiki.archlinux.org/index.php/Firefox#Firefox_disables_all_extensions_due_to_Mozilla_not_renewing_their_certificates https://wiki.archlinux.org/index.php/Firefox#Firefox_disable... AFAIK, this will enable all the disabled add-ons until the next check, which is in 24 hours. This will be hopefully enough time for Mozilla to release a stable channel update, instead of the "Studies hack". At least for me, fiddling with the Studies settings had no effect; the about:studies page remained empty regardless of what I did. I've also seen multiple reports from people who got the Studies hack working that the fix actually failed to address the issue properly.

nathanaldensr 7y ago
It appears to take several minutes for the studies to be downloaded and enabled. The process worked for me on two separate Windows 10 machines.
deleted 7y ago
[deleted]
eikenberry 7y ago
On Linux you should also have the option of disabling the cert signature check. Go into about:config and set `xpinstall.signatures.required` to false. Just remember to set it back to true once this is fixed.

LifeLiverTransp 7y ago

Best chrome add there ever was.

deleted 7y ago

[deleted]

gnud 7y ago

I'm not gonna bother with 'studies' or manual workaround - I'm just going to wait for an update. In the meantime I'm enjoying trying out Vivaldi[1] - really reminds me of opera 3/4, that I loved. 1: https://vivaldi.com https://vivaldi.com

Causality1 7y ago
Will Vivaldi let me put the tabs on bottom where they belong?
- jobigoud 7y ago
  What? Tabs belong on the side. In a tree. ;-)
- keeganwitt 7y ago
  Yes, Vivaldi supports tabs on your choice of any of the 4 sides of the window.
ripdog 7y ago
Hmm, I'm not sure switching to a closed source browser is in any way an upgrade... Especially when you could switch to the unbranded/nightly firefox builds, disable addon signature checking and continue using the only independent FOSS browser remaining.
- gnud 7y ago
  Vivaldi is only slightly more closed source than Chrome. Anyway, used to be Firefox had a lot more going for it than being open source. Now that's really the only thing left I can think of.

the8472 7y ago

At the minimum they should add a testsuite that runs at least a month into the future to catch these kinds of things. There was a similar issue[0] a few years ago that was only caught a month in advance. Even better would be to set things up to only do a verify on install instead on every startup. [0] https://bugzilla.mozilla.org/show_bug.cgi?id=1267318 https://bugzilla.mozilla.org/show_bug.cgi?id=1267318

Mindless2112 7y ago
> Even better would be to set things up to only do a verify on install instead on every startup. That would defeat the purpose of verification: "Add-on signing in Firefox helps protect against browser hijackers and other malware by making it harder for them to be installed." [1] And it's not just malware that was doing that. Microsoft force-installed the ".NET Framework Assistant" into Firefox on Windows, and you had to edit the registry to remove it. [2] If I recall correctly, AVG and Logitech were also among the list of offenders. [1] https://support.mozilla.org/en-US/kb/add-on-signing-in-firefox https://support.mozilla.org/en-US/kb/add-on-signing-in-firef... [2] https://support.microsoft.com/en-us/help/963707/how-to-remove-the-net-framework-assistant-for-firefox https://support.microsoft.com/en-us/help/963707/how-to-remov...
- deogeo 7y ago
  Could you explain why verifying on every startup, instead of just on install, is necessary? The page you linked doesn't mention it. Edit: Let me amend my question - why is it necessary for the certificates to expire? If a plugin is signed by Mozilla, why wouldn't it be trusted once it gets old?
  - Xylakant 7y ago
    There’s no guarantee that a (malicious or undesirable) extension was added via FF, so check on install could be circumvented by any of the crapware installers we all love.
  - makomk 7y ago
    How do you propose Firefox should tell whether the unsigned, shady add-on was installed by the user, or by some other dodgy app messing with Firefox's files whilst it wasn't running to make it look like the user installed the add-on?

pbhjpbhj 7y ago

On Android I get this: >We rolled out a hotfix that re-enables affected add-ons. The fix will be automatically applied in the background within the next few hours. For more details, please check out the update at https://support.mozilla.org/en-US/kb/add-ons-failing-install-firefox https://support.mozilla.org/en-US/kb/add-ons-failing-install... Which is like "we did something we shouldn't have causing unauthorised changes to your computer, so we're going to make unauthorised changes to fix it". Quite telling is that this is supposed to protect us from other developers. On the add-on screen "Enable" is greyed out, there's no "Enable even though Mozilla doesn't like it". The UX is just like the "fuck you this computer isn't yours it belongs to Microsoft and we'll do what we like with it" that I thought I'd left in the past decades ago. It's not your computer Mozilla, you fucked it up, you don't get to mess around with it without asking the owner. My understanding is that this is literally illegal in the UK. Mozilla barely had any trust left to burn IMO but they sure went all out.

morpheuskafka 7y ago
As I understand it, you agree to the terms of Studies as part of the ToS agreed to on installation. You can disable it later. And--while it was a ridiculous mistake--they didn't make any "unauthorized changes" to your computer. They just let a certificate expire and your computer, running the same code it always had, stopped trusting it.
- admax88q 7y ago
  Hiding behind ToS is ridiculous. Nobody reads them and a moral company should never assume that because its in the ToS they actually have informed consent.
  - MichaelBravo 7y ago
    TOS agreements should be illegal in the US, especially those that try to circumvent our 'Freedoms of Speech'.
    - 7y ago

akersten 7y ago

Very curious how the decision to use the Studies program happened. Why not just roll the version early and include the fix in the new version - isn't Firefox an evergreen browser now? Maybe there is extra bureaucracy to roll a new version, or the hotfixers didn't have permission to do it. Either of which I can understand, being that they made the fix late on a Friday night - so huge kudos to those who worked hard to get this fix started. Seems like a good case study for lessons learned here, I'm eagerly anticipating the postmortem and follow-ups.

mccr8 7y ago
Doing a full release takes a lot longer, so they presumably decided to use a faster method where possible. https://hacks.mozilla.org/2018/03/shipping-a-security-update-of-firefox-in-less-than-a-day/ https://hacks.mozilla.org/2018/03/shipping-a-security-update...

argd678 7y ago

Certificate expirations show up on causes of outage lists so frequently, yet little has been done to address the underlying issues of how PKI works to address it. The core issue of time limits on trust and no specs and requirements on how to handle the most common case of saying “yes, this is still trusted” is oddly absent. Could we have for example a publicly verifiable ledger that can be used to verify a cert chain with not only a defined workflow to answer if a cert is still trusted but also a requirement for the workflow to be fully implemented? Seems quite doable, vs sort of hacks of auto-renew which are hit and miss depending on the CA. In other words, when do we fix the sport rather than the players here?

DireLizard 7y ago

Assuming I understand it correctly the reason the addons stopped working is because an intermediate certificate they were signed with expired, question is - when and why are already installed and running addons are being revalidated every time (and when exactly ?), does this serve any user-aligned purpose or is just means for mozilla to exercise control over addon publishers ?

NicoJuicy 7y ago

Install https://storage.googleapis.com/moz-fx-normandy-prod-addons/extensions/hotfix-update-xpi-intermediate%40mozilla.com-1.0.2-signed.xpi https://storage.googleapis.com/moz-fx-normandy-prod-addons/e... Restart browser Fixed

mrb 7y ago

It's incredibly strange that this blog post doesn't even link to the bug description https://bugzilla.mozilla.org/show_bug.cgi?id=1548973 https://bugzilla.mozilla.org/show_bug.cgi?id=1548973 Imagine someone stumbling upon this post and trying to find more technical information... Poor communication.

tianheyang 7y ago

If this hadn't happened, I probably wouldn't have discovered Brave. Noticeably faster and the ad-blocking is built in. Thanks Mozilla

firefox1234 7y ago

It seems my mistrust in Mozilla is not misplaced. Every time, I install Firefox, I open about:config and search for all http[s] / [s]ftp links and set them all to random strings. My installed extensions still work.

clairity 7y ago

hmm, i don't seem to have been affected by this bug somehow (my extensions are all still working). i turn off as much phoning home as i can (including turning studies off) and block connections to *.services.mozilla.com any idea why i might not be affected? it may help others who might want to retain control of their firefox browser (chromium-based browsers being non-sequiturs).

kermitismyhero 7y ago
It was the same for me. Firefox 66.0.3 on Xubuntu 19.04. I also turn off a lot of phone-home behaviour. And I never had any problems with extensions getting disabled. I have no idea why I wasn't affected. But frankly, "not being affected" isn't good enough for me. Even if I dodged this bullet I might not dodge the next. I'm looking for an alternative browser. Falkon has been interesting so far. It's a little bare-bones in many ways, but at least it's immunue to any future Mozilla screwups.
- Gene_Parmesan 7y ago
  One screwup is enough for you to jump ship? Or have there been others? (I'm genuinely curious, not trying to be aggressive.) I personally don't think all this talk of "Bye-bye firefox" is quite fair (I'm referring to a number of comments on this page, not yours specifically). In my opinion (you may disagree), Mozilla is one of our best and strongest allies in the fight for a fair internet. Their values matter because they still have the user base to back them up. If all of us technical-minded folk jump ship to smaller, boutique browsers at every bug and gaffe, leading to a sort of browser balkanization, then Firefox loses its strength and those smaller browsers aren't impactful enough to be able to resist Chrome/Chromium. But, maybe your choice of browser is purely utilitarian, and that's totally fine of course.
  - chappi42 7y ago
    It's not one screw up. I love the sync functionality and Rust/Syncro is great. But on the other side it feels like 'they' take constantly power functionality away while the shallow and shiny homepage claims: "More power to you". But maybe I'm just still sour because of the Eich ousting?

Chardok 7y ago

The most frustrating part of this for me is there is no (relatively) easy way to override this behavior. Its fine to disable the addons, but please allow me to "understand the risks" and continue against Firefox's recommendations. The feeling of no control over my web browser was why I left Chrome in the first place.

watermelon0 7y ago

IMHO it seems problematic, that they can remotely push code changes, including replacement of trusted certificate, and bypass package managers. I don't expect software to (significantly?) change during runtime, outside of what was packaged, signed, distributed and installed as part of apt/yum/pacman/etc. I understand (not that I like or agree with) that some apps are just embedded web browsers, and load everything externally, and that Firefox extensions are in the end just some JS/CSS/HTML loaded outside of system's package manager. However, extensions have limited API they can interact with, and you need to allow permissions for each extension. Having Mozilla owned extension, that can modify core functionality, seems a bit scary.

benbristow 7y ago
If you didn't have browsers auto updating no-one would update them manually, meaning bad news for web developers wanting to take advantage of newer features.
- HNthrow22 7y ago
  I understand the appeal of that for developers but it comes at the cost of users agency and control of their own system, I've been very annoyed with even simple UI changes in firefox updates as I simply didn't ask or want any such change. Reading other comments here it's clear I'm a dying breed of old and stubborn users that prefers full control and agency over my own system. Making it easier for web developers to implement new features is absolutely not a tradeoff I'd make willingly at the cost of my systems consistency and reliability. Also the reason I use firefox is because of all the major browsers vendors they seem the most aligned with those values although this seems to be changing more and more every year.
  - damnyou 7y ago
    The incentive structures of society (capitalism, if you're so inclined, but I don't think this is unique to capitalism) are incompatible with your wishes.
  - benbristow 7y ago

c64ec30650ea3c4 7y ago

I'm shocked and suprised to find out that mozilla is using EXPIRING certificates for this. It requires them to continuously take action to prevent all addons from breaking, which will eventually fail (like it did). Firefox has a pretty robust update system and everyone is used to frequent updates. Why don't they instead have a revocation system built into updates? That way they would have to take action to disable malicious addons, and the good ones could go on working forever. Is there something about this idea that is so much worse than what happened today?

user17843 7y ago
You are right. It doesn't make any sense to use certificates for this kind of stuff. If an extension turns out to be malicious, you simply deactivate it in the store, and then proactively deactivate the existing installs. This is how Chrome is doing it. But having a certificate does offer Mozilla the feeling of absolute control, which seems to be of primary importance for them nowadays. This is probably the reason release and beta users are not even allowed to deactivate signing in the about:config settings.

martin_a 7y ago

So, I left Chrome for all the b's they were doing with/to the web. Now Mozilla is fcking it up, too. Which browser to choose now?

userbinator 7y ago
IE6. No, I'm only half-serious. ;-) Remember when the Web was mostly about sharing information, browsers didn't silently auto-update nor break in the process of doing so, organisations didn't add invasive "telemetry" to everything, and things would mostly stay working because the pace of change was generally much slower? Now that the "keep pushing it forward and breaking things" trendchasers seem to have gotten their way, instead we have the constant churn of web development, increasingly bloated sites and JS annoyances, browsers becoming more complex and fragile than OSs, dumbed-down UIs and taking control away from the user --- yes, that includes Mozilla who got to where they are today for their "user freedom respecting" position, and the repulsively ignorant "newer is always better" mentality that's infected even search engines like Google. Maybe I'm just being overly nostalgic, but incidents like these really put things into perspective.
- martin_a 7y ago
  I understand what you mean and I think I feel quite the same. "The web" has just become so... "strange" in the way everything works and we take care of it or however you'd like to call it. Often it's just broken with full intention to do just to push some new shiny technology on us. And I'd really like if it wasn't that way.

pleasecalllater 7y ago

This one will be emotional as this destroyed some of my today's work. F you Mozilla. I lost all my tabs opened in other containers. The containers don't work too, so I cannot reopen them. This bug has been known for 3 years, and you did nothing to fix it. You get so much money, and what you do is basically provide a pathetic software (thunderbird) and a nice browser (which you just stopped from working) and you show me banners asking for more money. You should be ashamed. 3 years. And no, I'm not going to listen things like "this is open source, you are free to fix it". I will just go and switch to another browser. I need a browser which works, not a one which suddenly decides that my stuff should be broken because all the developers and managers have been ignoring a critical issue for a couple of years. :( I know, this will be flagged, and removed. I don't care, I just need to get all my tabs in containers back. I have never thought that a browser can just close my tabs because a certificate expired.

jchw 7y ago
Hey, just FYI, there are some good solutions in this thread to get your problem fixed ASAP. This one should work. https://news.ycombinator.com/item?id=19827302 https://news.ycombinator.com/item?id=19827302
- pleasecalllater 7y ago
  No. JUST A HUGE NOOOO. My reply is here https://news.ycombinator.com/item?id=19828472 https://news.ycombinator.com/item?id=19828472
  - jchw 7y ago
    We’re literally in this mess because Firefox requires digital signatures on this kind of thing, please don’t make FUD posts. This is much better than disabling the very same safe guard, signature checking, that prevents you from running arbitrary code in the first place.
    - OrgNet 7y ago

MichaelBravo 7y ago

Mozilla, what ever the fuck your trying to fix, at least fix the add blockers asap & let us get tat fix before you work for hours on the rest of whatever the helm your doing. My uBlock has been good on my android for years, there is no reason it shouldn't be good now. Your reason for blocking as I've seen is that other develoers may have hacked your shit or done such but if you block our use of our ad blockers then we can just easily get infected all the more. Fix it now please & let us get those working then continue to fix whatever the hell else your see as problematic.

ripdog 7y ago
First, take a deep breath and calm down. No one category of addon is more affected by this than any other, and Mozilla has no way of 'fixing ad blockers' before anything else. Once the signing certificate is replaced, all addons will work. Simply turn on studies and allow it to install the new cert, and you'll be on your way.

deleted 7y ago

[deleted]

userbinator 7y ago

Mozilla decided to make signing mandatory, then screwed up and now they're trying to fix it by making use of a feature that basically allows them to remotely execute code on all their users silently? I checked Mozilla's main site again, and it still has this ironic statement in its description tag (it's been there for many years): https://www.mozilla.org/en-US/firefox/new/ https://www.mozilla.org/en-US/firefox/new/ Firefox is created by a global non- profit dedicated to putting individuals in control online. ...I guess it's more dedicated to putting Mozilla in control now. Something about this whole incident brings up a point that just feels very wrong to me --- it's not a Google or Microsoft, but the fact that Mozilla also seems to have this large amount of control over its users is unsettling.

coffekaesque 7y ago

How long until heads roll? There's something really wrong with the organization. And I thought it was only their marketing/pr that was bad. > We can't afford to lose Mozilla and Firefox. https://news.ycombinator.com/item?id=18800360 https://news.ycombinator.com/item?id=18800360

user17843 7y ago
nowadays they seem to make it a hobby to make negative headlines at least once every quarter. I fear there will be no negative repercussions for the leadership. Basically, the management set their own salaries, the entire work force gets a 40% yearly bonus, and they have no one from the outside to report to. On top of all of this, the money flows regardless of what anyone is doing. (While there is a yearly loss of 10% of their users, the past deal with Verizon made them very rich, so they can go like this for years). Revenue has been only going up, despite a loss of absolute users. So this explains why they continue to do bad things even though outside observes can not understand - during the last 5 years losing users did not impact their financials in any meaningful way. While people were complaining and users leaving the product, revenue was increasing. They do take care of their employees with lots of benefits and other stuff, so as an employee you don't want to risk all that with speaking up against your superior. Over the years they have created a company culture where there are endless number of small teams doing irrelevant stuff, with absurd hierarchies, with some people doing no work at all. With 16 people in the upper management, there's also fragmentetion of decision making going on. It's all a bit headless. Due to the complicated hierarchies in the company everyone is content with doing just enough to not make life harder for anyone else - suggest to change things fundamentally and actually work on delivering a great product and you will not get very far.
- jdashg 7y ago
  This is categorically untrue, and unhelpful.
  - user17843 7y ago
    some things I wrote I can not prove, that is right. I would love to revise my negative opinion in light of better evidence.

RenRav 7y ago

>Your Firefox extensions are all disabled? That's a bug! Is this supposed to affect everything installed? I'm running Firefox 56.0.2 and this only affected addons which I had already disabled, all the other addons are fine... still, am I forced to update to fix this bullshit?

gorbachev 7y ago

This isn't even working reliably. The hotfix study is showing up completed, not active, on my Firefox at home. No idea why, there's no information about how to reactivate it. No, re-installing it didn't help.

tiredwired 7y ago

I got the update and containers are working again but, it seems like it forgot my old container configuration and I had to set them all up again.

Metricon 7y ago

From ghacks (comment section): https://www.ghacks.net/2019/05/04/your-firefox-extensions-are-all-disabled-thats-a-bug/ https://www.ghacks.net/2019/05/04/your-firefox-extensions-ar... This should allow the extensions to work until the next check (Verified locally): 1) Shut down Firefox 2) Open extensions.json (located by about:profile -> Root Directory) 3) Replace all instances of “appDisabled”:false to “appDisabled”:true 4) Replace all instances of “signedState”:-1 to “signedState”:2 5) Save and close extensions.json 6) Start Firefox 7) Close Firefox 8) Open extensions.json 9) Replace all instances of “appDisabled”:true to “appDisabled”:false 10) Start Firefox 11) Disable and re-enable all extensions in about:addons

svnpenn 7y ago
it works!

huhtenberg 7y ago

Can we take a moment and consider the side effects? This is a once in a lifetime chance for Google & Co. to get a glimpse of all those sly fuckers hiding behind adblockers. This effectively uncloaked a very specific subset of Internet users and exposed them to the very companies that they've been actively trying to avoid. Not just those who avoid Chrome, but those who take extra steps to explicitly evade the tracking. Surely Mozilla, the privacy advocate, must understand the impact of this fuck up, and yet the offered "fix" doesn't even mention a one-click .xpi install, but rather asks to enable a mechanism that, if left enabled, will grant unnecessary control to Mozilla over people installs. This ain't right.

ne0sapi3n 7y ago
>Surely Mozilla, the privacy advocate
- calvertdw 7y ago
  The post is still great despite that weakness.
overgard 7y ago
FWIW I think it's pretty easy to test if someone is using an adblocker anyway. (I see sites do the "It looks like you're using an adblocker" thing all the time). I don't know if there's any realistic way to entirely hide that.
- tedivm 7y ago
  That's a bit different though- generally speaking those messages show up when the javascript tracker can't talk to the server it's communicating with. Even though it's "detecting" the adblock it isn't able to send information back from the client about it.
  - overgard 7y ago
    > Even though it's "detecting" the adblock it isn't able to send information back from the client about it. Sure they can, they can just send back a resource request. It could even be for like an image with a query string attached with it, it doesn't have to be an ajax request necessarily.

felixfbecker 7y ago

I think Firefox needs to stop this add-on signing and review madness. The web is OPEN. It's not a walled-garden Apple App Store. Yes, extensions run arbitrary JavaScript code. So does any webpage you go to, and nobody from Mozilla reviewed all that JavaScript either. How are extensions any different? Chrome is doing just fine without all this non-sense process and policy.

ripdog 7y ago
Extensions have dramatically more access to powerful APIs to affect the browser. They can be used to perform a great variety of annoying, intrusive or downright malicious actions which a website is incapable of. Of course they have much more stringent policy. Chrome implements mandatory addon signatures as well, and only Google can sign them.
- user17843 7y ago
  I doubt google has certificates that run out automatically. Rather, the best way is with each signing to include signing the date and not allow the certificate to expire retroactively.
  - ripdog 7y ago
    All signing certificates expire. They must. It's a fundamental part of the security model, because otherwise a malicious actor could take an old, comprimised cert and inject it into Firefox, allowing them to run malicious 'signed' addons. This attack would work the exact same way in Chrome, so Chrome will expire it's certificates too.
    - speedplane 7y ago
      > All signing certificates expire. They must. It's a fundamental part of the security model Is it really necessary? Aren't there other possible ways of invalidating a certificate other than its date?
    - user17843 7y ago

makecheck 7y ago

From a UX point of view I don’t know why tools don’t have these two features: 1. “Warning, a critical method for verifying authenticity is set to expire in X days. Please visit <Y> to update now.” 2. “A critical verification certificate has expired; while you should immediately go to <Z> to obtain an update, you may defer authentication for up to 5 more days.” ...or in other words, why can’t tools cut us some slack on either side of a deadline? Security for most things is not going to fall apart just by giving people a little room to deal with issues on their own schedule.

chronogram 7y ago
That would be assuming this would ever happen, which clearly Mozilla didn’t expect, wrongly. It’s a shame, but eh! Mistakes were made. Precautions will hopefully be made.

wetpaws 7y ago

I'm trying to be angry at Mozilla for messing up, but cant force myself to do it. They ended up to be that adorkable kid that spilled the paint all over the carpet and you just sigh and start cleaning up the mess.

nullc 7y ago

I wonder if there is someone out there in the middle of the ocean with a browser extension based communication and navagation system which is dead in the water? It sounds to me that the real headline here is that every copy of firefox out there was timebombed and we only noticed because someone forgot to elongate the fuse.

javagram 7y ago
Could other code signing systems like macOS gatekeeper also be vulnerable to problems like this? IMO this seems like just plain bad design. The Firefox addon certificate should never have had an expiry date. If they ever needed to revoke it, they could distribute an updated version of the browser with the previous intermediate explicitly marked as revoked.
zimmund 7y ago
The browser itself continued working fine. Are you aware of any life-depending extension? Leaving this particular issue aside, your hypothetical "browser extension for people in the middle of the ocean" was doomed from its inception if it was designed to run as a browser extension (though it opens the door for an interesting discussion about similar scenarios that are happenning, like pilots relying on ipads)
- Dylan16807 7y ago
  > your hypothetical "browser extension for people in the middle of the ocean" was doomed from its inception if it was designed to run as a browser extension Why? You haven't backed up that statement at all. Especially before they killed XUL it was easy to make a non-doomed app that runs as a browser extension, and it's still plenty possible. No (non-demo) program should brick itself if it can't connect home.
- kwak 7y ago
  No but what about people who use password managers, that were locked out of not being able to access bank accounts, credit cards, and reddit.
- 7y ago

Insanity 7y ago

Funny, I only just noticed my extensions were gone because of advertisements on youtube. I just assumed I somehow messed up my browser and started looking around the settings. A banner displaying why they silently updated FF and disabled the addons would have been nice as well :)

lewiscollard 7y ago

> Please note: The fix does not apply to Firefox ESR or Firefox for Android. Damn it, Firefox, because it supports extensions, i.e. uBlock Origin, is the only usable browser on Android :(

levlaz 7y ago

If anyone is running firefox-esr on debian (I am), there is a separate discussion happening here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=928415 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=928415

chenshuiluke 7y ago

Am I the only one whose addons weren't affected at all?

zwaps 7y ago

I am a bit salty that this reset my default search to Google of course, just an option away. Still :<

mk89 7y ago
I am also extremely pissed. Still, I was searching for alternatives on Android. It's ridiculous that no other browser allows extensions, not even Chrome itself. Firefox is way more advanced on this. That's one more reason that is very hard to say goodbye to them...

deleted 7y ago

[deleted]

philipwhiuk 7y ago

Patching security bugs using Shield is utterly stupid. What would happen if they found a Zero Day - would they use the same method?

stubish 7y ago

Good to see Mozilla thinking out of the box and getting a fix out as quick as possible. It would take a lot of time to get an emergency build out of standard channels, because there are so many of them.

rodorgas 7y ago

I don't like the fact that I can't run an extension that wasn't signed by Mozilla. This behaviour lacks freedom.

aswan 7y ago

(disclosure: I am a Mozilla employee but not commenting in any official capacity) "Give me control over what code I run on my computer" (meaning "provide a switch to disable the requirement that extensions be signed") keeps coming up over and over. And perhaps it hasn't been clearly stated but the problem is this: if there's a switch that a user can flip, the browser has to record the state of that switch somewhere (presumably on disk). If such a switch becomes available, we'll quickly be flooded with malware that flips that switch without users' consent. At that point, there's no way to tell the difference between savvy users making an informed choice to enable unsigned extensions and malware doing it behind their backs. The browser can do various things to obscure the way that setting is stored, but ultimately any method the browser uses to read and write the state of that switch is something that other software can easily mimic. This is not a theoretical concern, a modern web browser target is an irresistible target for all sorts of get-rich-quick scammers -- if you don't experience this day-to-day its due in no small part to the fact that browser vendors among others are constantly working to keep the bad guys at bay. But make no mistake: the bad guys are out there and they quickly find and exploit any opportunities that are available to them. So as to the problem of how to let users disable signing but ensure that they have made a conscious decision to do so, there is a stark tradeoff here: giving the most savvy users that switch necessarily makes other users less safe. The solution that Firefox has opted for here is to handle this tradeoff differently on different channels. The release channel (aka the stable channel, or the thing you get by default when you download Firefox) is intended for a very wide audience, and so it handles this tradeoff by favoring safety for all users regardless of their level of technical knowledge. The developer edition and nightly channels are intended for more technically savvy users and they handle this tradeoff differently; specifically they do provide a switch for disabling extension signing. If there are other (practical and effective) ways to solve this problem of determining true user intent, I (and I'm sure many many others) would be very interested in hearing about them. In the mean time, using the mass-market versus developer-focused channels as a signal for users' preferences on the risk-configurability continuum seems like a reasonable way to handle this.

greendestiny_re 7y ago
My gripes with the switch paradigm are that it: a) isn't transparent b) doesn't empower the user c) isn't easily modifiable a), b) and c) are the exact opposites of what open source software is meant to stand for. Firefox is slowly losing its unique position of being an amazing open source browser in favor of what seems to me a negligible increase in user security. In my mind, Mozilla is wasting time on micromanaging user risk instead of actually innovating. To put it this way, every time I go out biking, I can get hit by a car. It is a known and well understood risk, one that I have to consider whenever making a turn. However, riding a bike also provides chances to go faster, meet new people and so on. Should Firefox aim to reduce my risk of being hit by a car? No, because I get to choose the level of risk in my life, not Mozilla.

spevman 7y ago

Well, I'm about to make a lot of people happy with this info. I was researching this today as I'm using FF 56.0.2 and found the solution on this discussion thread. Leave it to an end user to do the job the professionals either failed or refused to due. It worked for me on 3 different machines. Go to this link and follow the instructions detailed: http://bit.ly/2DUiOLN http://bit.ly/2DUiOLN