16 ms·

Retrospective and technical details on the recent Firefox outage

323 points by feross 5y ago

galosh 5y ago
That was an extremely high risk change on GCP's part, reminds me of the App Engine days when you'd wake up to find a totally healthy program spamming 500s because they'd make a breaking change without any announcement. It's shocking they're still pulling stuff like this in 2022
- agilob 5y ago
  It reminds me how YouTube enforced new codec with a few days notice knowing that FireFox doesn't support it, so FF couldn't play most YT videos for over a week.
  - faeyanpiraat 5y ago
    Did yt back up or ff updated to fix issue?
    - agilob 5y ago
      Firefox got the codecs working in the next release
  - TheGoddessInari 5y ago
    > enforced new codec YouTube turned off flash player as the default in 2015, and VP9 was supported in Firefox at the same time. YouTube still serves h.264, vp9, and av1. I was trying to figure out what this could be referring to.
    - agilob 5y ago
      Not all codecs were ported at the same time, and then not enabled by default, and when enabled by default it was platform dependent, for others where it worked FF was eating all possible CPU resources and videos were glitching. I remember this as I was using Debian and I was active on /r/firefox, where this [1] link was posted 10 times every day [1] https://www.youtube.com/html5 https://www.youtube.com/html5
- kevingadd 5y ago
  Feels like an especially severe version of the consistent Google pattern of only testing stuff on Chrome, so new updates/features ship in a way that is some degree of broken on Firefox/Safari. For a significant amount of time YouTube had bad performance on Firefox because they chose to use Web Components by default with a horrible polyfill instead of using the old (still working!) html5 version that ran great.
  - loulouxiv 5y ago
    Putting in place an infrastructure to test this kind on changes on the 5-10 most popular browser would be, I think, very cheap for a company like Google. I can't help thinking these may be deliberate moves to eat the little market shares of Chrome concurrents. I remember reading here on HN an article written by an ex-Mozilla insider relating the dissonance between the "friendly" Mozilla-Google employees exchanges and the year-long track record of very oddly recurrent "unfortunate mistakes" from Google degrading the Firefox compatibility.
    - magicalist 5y ago
      > Putting in place an infrastructure to test this kind on changes on the 5-10 most popular browser would be, I think, very cheap for a company like Google. The problem wasn't some web server, it's the Firefox backend services running on GCP.
    - loulouxiv 5y ago
      I am trying to find the link, but for the moment I only find comment making, I think, references to it : https://news.ycombinator.com/item?id=19815348 https://news.ycombinator.com/item?id=19815348 https://news.ycombinator.com/item?id=28495546 https://news.ycombinator.com/item?id=28495546 Does anybody here remember enough keywords to find it out ? Edit: I guess it was this Twitter thread https://mobile.twitter.com/johnath/status/1116871231792455686 https://mobile.twitter.com/johnath/status/111687123179245568... Edit 2: The associated HN thread https://news.ycombinator.com/item?id=19662852 https://news.ycombinator.com/item?id=19662852
    - deleted 5y ago
      [deleted]
- charcircuit 5y ago
  Would a warning have even helped that much? Since HTTP/3 was expected to be working there wouldn't be a cause to worry.
  - Semaphor 5y ago
    > Would a warning have even helped that much? Since HTTP/3 was expected to be working there wouldn't be a cause to worry. It might (as mentioned in TFA) have made them think to run some extra tests, which could have caught the bug. But it also would have made the response faster, as they would have known what changed far sooner.
  - 0xbadcafebee 5y ago
    The key word there is "expected". When you do Operations for a living, the only thing you can expect is the unexpected. That's why even after you think you've tested a change, you carefully and slowly roll it out a bit at a time, monitoring golden metrics so you can detect a problem, stop the roll-out, and roll back. It sounds like somebody just flipped a giant switch and never checked error rates, connection metrics, anything. Check out this graph: https://hacks.mozilla.org/files/2022/01/crashes-foxstuck2-2048x946.png https://hacks.mozilla.org/files/2022/01/crashes-foxstuck2-20... Think maybe that would indicate somebody needs to roll back the last change? The problem here, as usual, is a disconnect between stakeholders. Google has this service (it seems like the load balancer for their customer?) it wants to change for one reason or another. The customers may or may not have planned for the change Google is making. Google makes the change, but it isn't a stakeholder of the customer (they basically don't care what happens to the customer). So there is no direct feedback loop for the customer to tell Google something is wrong. If Google was at risk of losing business from its customers going down, it would have a strong relationship with those customers and have a way to quickly help diagnose problems and roll back changes if needed. This is a great lesson for all customers to take away: don't depend on people who you don't have a close relationship with.
- brabel 5y ago
  The client, Firefox, said it supported HTTP/3 though. Otherwise it wouldn't get to use that. I don't think that's as bad as you try to make it... if the client says it supports something then it breaks when it uses it, it's the fault of the client, not the server.
  - alisonkisk 5y ago
  - throwaway984393 5y ago
    No SRE in the world that is halfway decent at their job would think that way. You never make assumptions about any kind of change, much less a global change to a completely different protocol. Doesn't matter whose fault it is. You just don't introduce any change that has a chance of unexpected behavior without rigorous testing, and you roll it out g r a d u a l l y, and you stop when error rates increase. Google literally wrote the books on SRE. For them to not know better is absurd.
- notyourday 5y ago
  > That was an extremely high risk change on GCP's part, reminds me of the App Engine days when you'd wake up to find a totally healthy program spamming 500s because they'd make a breaking change without any announcement. It's shocking they're still pulling stuff like this in 2022 reply Lay with the dogs, wake up with the fleas. Google is a shitty company producing shitty products. When you select to do business with Google you select to do business with a shitty company producing shitty products and treating its customers like shit. Hence I fail to understand the Surprised Pikachu face when something like this happens.
0x002A 5y ago
"Because all network requests go through one socket thread, this loop blocked any further network communication and made Firefox unresponsive, unable to load web content." Why side functionality (telemetry) of a tool uses only one network thread and can block any network communication ?
- floatingatoll 5y ago
  Per elsethread, supposedly any site could have triggered the crash, not just ”side functionality”: https://news.ycombinator.com/item?id=30175916 https://news.ycombinator.com/item?id=30175916
  - 0x002A 5y ago
    Yes that's another issue. But as a design pattern shouldn't we design our products to do their core functionality as much as independent from any anomalies that can happen? This is almost akin to me if Tesla rolls out an update and the car decides to pull over to the curb to do the update, while you are driving to your job or worse to hospital with an emergency. My theory is there should be at least one health enterprise using firefox as their only browser for business functionality out in the wild.
    - Yoric 5y ago
      In theory, that's how it happens in Firefox. But when you have a bug in the core of the product (the network stack), there isn't much that the rest of the product can do to isolate from it.
      - 0x002A 5y ago
        Yes that's what you get when you have one thread for all network communications. The network stack did not fail, only the sole network thread got stuck. From the write up I understand if there was another thread for communications firefox would only fail to communicate with telemetry service but firefox would be able to function as users needed.
ghostly_s 5y ago
Went into this with an indignant "failures in telemetry should NEVER bring down core functionality!" feeling, left with a more nuanced understanding of the fault and impressed and reassured by the mitigation steps being taken. That's a great post-mortem.
- nathansherburn 5y ago
  I felt a similar way when it first happened. I was fuming "an update should never break your core functionality across all devices!" only to find that it wasn't actually the latest update but rather code that had been there for ages. It's hard to hate something once you truly understand it, I guess.
- minerva23 5y ago
  Anyone know why telemetry is sent from the main process rather than having a separate reporter process?
  - rockdoe 5y ago
    It would have made no difference: the hang was in the networking code & process, not the telemetry code (and whatever process that is in). Anything that sent a message with the "bad" header type would have provoked the hang. As to why, dunno, presumably it's extra effort for an unclear gain. Telemetry code wouldn't be parsing hostile input etc. And it doesn't stop bugs like this either.
  - marcan_42 5y ago
    That would be significant extra engineering effort without much clear benefit. It's always possible to look back and say "if only they'd done that", but this bug was a freak coincidence and it's not possible to foresee things like that, nor is it worth the engineering effort to try to avoid them with hammers as big as "move all telemetry to another process". You'd need a much stronger reason than "what if telemetry happens to trigger a longstanding bug in the network stack" to decide to go with that. In this case, that the problem was triggered by telemetry was a coincidence; it would've been triggered by other bits of coden in the future as they moved to Rust.
deleted 5y ago
[deleted]
blackbear_ 5y ago
- kevingadd 5y ago
  Does Firefox force updates on your configuration? On Windows it's opt-in for me. I know Android will forcibly update any app (including Firefox) while you're using it, but you can shut that off system-wide.
  - joshgev 5y ago
    The default is that it forces a restart when a new update has been downloaded, which has been a frustration for me as apparently it has been for the parent. Apparently this can be changed by requesting that FF only install updates with explicit consent[1]. I'd think the best way would be to install anything that's available locally when the browser starts without forcing a restart. [1] https://superuser.com/questions/1451210/how-can-i-make-firefox-stop-forcing-me-to-restart-my-browser https://superuser.com/questions/1451210/how-can-i-make-firef...
    - Vinnl 5y ago
      Are you using Linux, by any chance? IIRC, the issue here is that the update is done by your package manager, changing Firefox's files out from under them. If you use the direct download from Mozilla, it shouldn't be as disruptive. I still hope a fix or workaround to this can be found, but knowing why something is the cause makes it easier to accept, at least for me :)
  - Nextgrid 5y ago
    What pisses me off is less about the update itself (they're typically unnoticeable) but the constant nags and "what's new" crap that opens up after the update. Firefox is more hostile than a lot of paid, proprietary software in this regard.
    - brimble 5y ago
      This. That crap is significantly more disruptive and irritating that the actual paid ads in old-school free Opera (the largest feature-comparable browser when FF/Phoenix/Firebird first launched).
deleted 5y ago
[deleted]
luciusdomitius 5y ago
- Traubenfuchs 5y ago
  Previous discussion: https://news.ycombinator.com/item?id=24563698 https://news.ycombinator.com/item?id=24563698
- hackerfromthefu 5y ago
  She's probably earned another raise if her actual reason for appointment is to drive chrome adoption while soaking up community engagement to prevent another open source browser competitor, then she's succeeding very well!
  - luciusdomitius 5y ago
    https://www.androidheadlines.com/2020/08/mozilla-firefox-google-search https://www.androidheadlines.com/2020/08/mozilla-firefox-goo... Well. Seems that it cannot be ruled out.
    - deleted 5y ago
      [deleted]
- deleted 5y ago
  [deleted]
- ricardobayes 5y ago
  It's very interesting to see how rapidly FF fell out of favor for some reason. Went from poster child to pariah in what seemed like no time.
  - luciusdomitius 5y ago
    Well. The actual lead dev got kicked out as Mozilla Foundation chair and got replaced by some SJWs over his support for banning abortions, same-sex marriages or something similar.
drekipus 5y ago
- intothemild 5y ago
  I don't get this. The issue had nothing to do with the language used.
  - account42 5y ago
    It did have something to do with there being two languages used though, resulting in two different ways of excercising the network stack whith less testing for each. So its not a bug caused by Rust, but it is caused by the addition of Rust in Firefox.
- kevingadd 5y ago
  This isn't specifically caused by rust, it's a second-system problem where they had two different ways to touch the http system and one was broken. It could have easily been the other way around with only rust being correct.
  - drekipus 5y ago
    Yes, I know that, I'm trying to make the joke stand out but it's hard to do
    - milliams 5y ago
      https://en.wikipedia.org/wiki/Poe%27s_law https://en.wikipedia.org/wiki/Poe%27s_law strikes again.
- evilpie 5y ago
  The actual bug was in the C++ code handling the Content-Length header (https://hg.mozilla.org/integration/autoland/rev/48c0e854be623e654f1a0814d22502ca5c618422 https://hg.mozilla.org/integration/autoland/rev/48c0e854be62...) and the Rust code just happened to be first to trigger it.
sirl1on 5y ago
Why does my browser need connectivity to some internal services? I am fine with offering opt-in service integration (Firefox Sync, Pocket, ...) but is there a reason why Firefox needs internal infrastructure to do the one thing it is supposed to do, browsing the web? I can only think of DNS over HTTP, but AFAIK that is also opt-in, right? Man, I love Firefox and used it since it was called Firebird (with a small gap when Chrome was shiny and new and Firefox a slow RAM hog). But I really resent the Mozilla Foundation, they seem to be interested in everything but browser development. To be fair, (ab)using the browser as application runtime brought us so much complexity that developing and maintaining a secure browser as free software spare time project isn't feasible anymore.
- lmm 5y ago
  > Why does my browser need connectivity to some internal services? I am fine with offering opt-in service integration (Firefox Sync, Pocket, ...) but is there a reason why Firefox needs internal infrastructure to do the one thing it is supposed to do, browsing the web? If you read the whole post, the connection was explicitly for telemetry (and so you could avoid the issue by turning off telemetry), and it blocked other connections because the request went into an infinite loop rather than failing outright.
  - nanis 5y ago
    > If you read the whole post, ... (and so you could avoid the issue by turning off telemetry) Speaking of reading the whole post: >> users who disabled Telemetry would see this problem resolved even though the problem is not related to Telemetry functionality itself and could have been triggered otherwise. This does not make sense to me: >> Without the header, the request was determined by the Necko code to be complete, This is written as if it makes sense to treat a request as "complete" when it's missing a content length header. Huh?!
    - mozdeco 5y ago
      At this point, the code relied on the Content-Length header being present because the higher-level API was supposed to add it. The field that is supposed to be populated by Content-Length (mRequestBodyLenRemaining) is pre-initialized to 0.
ricardobayes 5y ago
96 by far is the worst release, probably even in the history of FF.
- danuker 5y ago
  What can we do about it? Use an older version? Security problems. Use a simpler browser? Many sites will stop working. Maybe it's for the best to avoid using sites that use complicated JS/CSS/HTML. But will still need it for say, government sites to pay taxes.
  - iqanq 5y ago
    Everybody should use Firefox ESR. It's stuck at Firefox 91 and you receive security updates, but you get no ads, your settings do not reset...
    - Semaphor 5y ago
      I’m up-to-date and have neither ads, nor reset settings.
  - userbinator 5y ago
    You mean "Security" problems... There's a lot of FUD and paranoia out there; 99% of exploits need JS and even those which don't technically need it, are almost always obfuscated using JS. Leave JS off by default (there are extensions to do that) and don't turn it on unless you really do trust the site to run arbitrary code on your computer, and you're unlikely to encounter any problems.
- FeepingCreature 5y ago
  57 was worse.
shantara 5y ago
It's incredible to see a normalization of the term "outage" applied to a software running on a local hardware.
- deleted 5y ago
  [deleted]
pdenton 5y ago
I haven't noticed anything and I use Firefox every day. Nor have any of my clients where I deployed Firefox called. Is this because I always disable data collection in settings?
- ainar-g 5y ago
  Yes, the bug was in the Telemetry code. I'm not sure if it's on by default, but it's probably better to disable it for any large-scale deployments. Both to prevent things like this and to make sure that things that should be disabled by default actually are.
  - pdenton 5y ago
    Ah, that explains it. Thank you. I recommend https://ffprofile.com/ https://ffprofile.com/ which was posted earlier here on HN. Makes it easier to deploy Firefox with saner defaults. You know I moved from Netscape 3.0 Gold to later versions, to Mozilla, to Phoenix, Firebird, and then Firefox. I tried other browsers but it's always a subpar experience for me. My only gripe is that they kept changing the UI.
    - floatingatoll 5y ago
      (FYI: The bug was not in the telemetry code, it was in the http3 code; see comments further up in the post for details.)
- Ygg2 5y ago
  It's not a telemetry bug. It's a networking bug. Avoiding it basically comes down to being extremely lucky.
- mccr8 5y ago
  It was broken for a couple of hours, and late at night in the Americas. You might have been asleep for the entire duration of the outage.
mrweasel 5y ago
It's a nice write up, but the "Lessons learned" is terrible, it appears as if they didn't learn anything and view the whole thing as an infrastructure problem. As others pointed out, why does Firefox even need to communicate with Mozilla services? Sure, telemetry needs to feed data back, if enabled, but if that fails why does it need to stop the browser from working? Shouldn't the lesson learned be: The telemetry functionality in Firefox has a bug, where an infrastructure outage at Mozilla can "break" the browser. The fix isn't in infrastructure, the fix has to be in the code that communicates back, it should fail gracefully. It's not a problem if telemetry fail, either cache locally and just drop the data, it's honestly not important. I'm sorry, I get that it's interesting how and why all this failed, but Mozilla makes it seem like they don't get what the root of the problem is.
- floatingatoll 5y ago
  Per other comments here, supposedly the same issue would have occurred with a variety of non-telemetry tasks as well. One of them is indicated to be certificate-related, which suggests to me that CRL lookups could have triggered it, though I don’t know for sure. It ended up being that the issue occurred first with telemetry rather than with certificate stuff or who knows what else. But the flaw wasn’t in telemetry code, so focusing on telemetry wouldn’t have prevented this at all.
- progbits 5y ago
  Exactly. I love Firefox and use it as my primary browser but this is the wrong conclusion. Don't make your telemetry backend more reliable. Instead break it on purpose several times a day. That way a similar bug in the browser will not make it past dev channel.
  - Yoric 5y ago
    As mentioned in a few places, the telemetry backend was not the culprit. The network stack was. The network stack is already pretty heavily tested and fuzzed, but obviously that didn't catch this specific bug.
    - 5y ago
brabel 5y ago
A lot of intelligent sounding words to explain a trivial bug: their network stack was using case-sensitive header names, which anyone doing anything remotely related to HTTP knows is a mistake. We all make mistakes, but don't try to make it sound more grandiose ("a combination of multiple factors blah balh blah") than it is.
- viraptor 5y ago
  Http2 and 3 are case sensitive at when encoding the headers: > As in HTTP/2, characters in field names MUST be converted to lowercase prior to their encoding. A request or response containing uppercase characters in field names MUST be treated as malformed https://quicwg.org/base-drafts/draft-ietf-quic-http.html#section-4.1.1-2 https://quicwg.org/base-drafts/draft-ietf-quic-http.html#sec...
- matsemann 5y ago
  > which anyone doing anything remotely related to HTTP knows is a mistake. Only if you haven't updated your knowledge since HTTP/1.1 days...
denton-scratch 5y ago
> there were many contributing factors working together Looks like one factor, to my eyes: telemetry. I have telemetry disabled. But if you're going to default to "telemetry on", and then silently send data to sites that aren't in the address-bar, then it's your responsibility not to "break the web". You can't blame it on rust, or necko, or viaduct, or google.
- Liquid_Fire 5y ago
  > This is why users who disabled Telemetry would see this problem resolved even though the problem is not related to Telemetry functionality itself and could have been triggered otherwise. It's pretty clear from the article that this was a bug, and telemetry requests failing was not intended to break the rest of the browser. If Google had rolled out this change a year from now, it could probably have broken something other than just telemetry (e.g. maybe update checks, or certificate management) and your browser would still have been broken even with telemetry disabled.
  - hypertele-Xii 5y ago
    > It's pretty clear from the article that this was a bug, and telemetry requests failing was not intended to break the rest of the browser. If telemetry didn't exist as core part of the browser, nothing would've broken. Therefore, the telemetry itself is the direct cause of the bug. It was, at best, poorly handled and too deeply integrated into the browser's core function.
    - matsemann 5y ago
      The browser would have broken the next time a malformed http3 request happened. Read the quote again.
  - pessimizer 5y ago
    > telemetry requests failing was not intended to break the rest of the browser. You can't find a lighter straw man than that. Find me somebody who said that Firefox intended to break the browser if telemetry failed.
truth_revealer 5y ago
[flagged]
- dismalpedigree 5y ago
  In standard conspiracy theory fashion, short on who and why. Two parts that are critical. People talk. The more outrageous the conspiracy, the more people need to be involved. Someone always talks. Then lets understand how these people stand to benefit. Nobody is going to extraordinary lengths just to burn it all down. I love me a good conspiracy theory and playing “what-if”, but the things i take action on are the ones which pass my two criteria above.
  - truth_revealer 5y ago
ankurdhama 5y ago
Not sure how Firefox code is structured but it is weird that a particular HTTP3 request would hang the entire network stack and you cannot perform any HTTP 2/1 requests.
- mozdeco 5y ago
  All requests go through one socket thread, no matter which HTTP version. I am not a Necko engineer, but since requests can be upgraded, an HTTP/1 request could switch to HTTP/2 and if there was a separation by protocol, the request would have to be "moved" to a different thread. So I'm not sure that would work easily.
- marcan_42 5y ago
  Is nobody reading the article? It was an infinite loop. Not a blocked request. A bug. A logic flaw. Something that wouldn't normally happen. It was broken code. You know what happens when you put a while(1); in the middle of the nginx codebase? The whole server process hangs. This is normal in an async design. We don't write software to be magically resilient against freak bugs, especially not something like a browser that is not intended to be used in life-critical applications.
kragen 5y ago
It should be impossible for the phrase "the recent Firefox outage" to make sense. Has there ever been an "outage" of linear algebra? Of Linux? Of mitochondria? Of Bitcoin? It is critically important that we not introduce new single points of failure into the systems that our civilization depends on, and that we remove the ones that already exist. If it can happen by accident, it can happen on purpose.
- ainar-g 5y ago
  While I agree with the SPOF point (heh), the header should really read “the recent Firefox Telemetry outage”. It's kind of like a website causing your browser or a tab of the browser to be unresponsive by introducing an infinite busy loop. Except the “tab” was invisible.
  - detaro 5y ago
    The user impact (Firefox not working) is more important than the technical reason, and thus should be in the title.
  - mozdeco 5y ago
    The infinite busy loop in this case was not the tab no (neither visible or invisible). The loop was directly in the network stack, as stated in the post, not in the caller.
  - kragen 5y ago
    The problem isn't that there was a telemetry outage; the problem is that the telemetry outage caused a Firefox outage, which should not be a thing. Firefox needs to be more robust than that.
    - dralley 5y ago
      That's not what happened though. Telemetry uses the same networking stack as everything else, and the busy loop was in the networking stack.
deng 5y ago
I'd claim that no other company is criticized as harshly as Mozilla around here. The amount of blame that is assigned to the Firefox team is staggering. To me, this is a perfectly valid write-up with a good lessons learned. They have written it in a very diplomatic way, but to me, it is absolutely clear that Google screwed up here. How can you make such a change to a default behavior of critical infrastructure unannounced? That's just reckless towards your customers, and solidifies my belief to stay away from GCP. If they had properly announced the change, even if the Firefox team hadn't then tested beforehand, at least the DevOps team would have put one and one together and just changed back to HTTP/2 and the outage would have lasted maybe 10 minutes. Instead, they frantically went through their git log to see what in the code base might have triggered this bug. Everyone who has been in such a position knows how incredibly stressful this is. I'd be absolutely livid at Google in their position. That it took two hours to fix this is clearly their fault.
- radmuzom 5y ago
  - deleted 5y ago
    [deleted]
  - jrochkind1 5y ago
    it's probably not the same people, even though you see both opinions on HN.
- moeris 5y ago
  I agree that Google is at fault here for failing Firefox. But Firefox is guilty of failing its users. Why should the functioning of a browser be dependent on telemetry working? It sounds like if there is high enough latency in their telemetry, or if request for telemetry start failing, it's possible for that to disrupt using the network stack at all. They have a massive design flaw, and they didn't even mention that in the article. Maybe they have good reasons for designing a single point of failure that relies on a cloud provider, but it's not clear what those might be since they don't address it.
hngpt2 5y ago
The recent outage for users of Mozilla’s Firefox web browser was caused by a Firefox update, the source code for which was inadvertently pushed to the Mozilla Add-ons Repository. That repository is not publicly available, so it was not possible for users to update their installations. Fortunately, a bug was introduced in the Firefox update which prevented the update from being installed. The bug was fixed when the update was rolled back. Users who updated earlier were protected. The outage could have been worse, though, because the bug that was introduced in the code for the Firefox update was in a feature of Firefox itself that allows users to block certain updates until a later time. In the case of Firefox, that feature was used to block the update that caused the outage. If a similar situation occurred in the future, users would not have a way to block the update that causes the outage. That feature is available in other web browsers, but it is not as advanced or robust.
- deleted 5y ago
  [deleted]
causi 5y ago
God I hate the modern web. Why the hell does my browser need more than periodic contact with any server other than my DNS provider and the host of the website I'm connecting to?
- MauranKilom 5y ago
  What does "more than periodic" mean? But to answer your question: For example, to inform you that an update is available.
  - account42 5y ago
    Updates and associated checks should not be done by each application itself.
    - Karunamon 5y ago
      Hard disagree. Having that in the hands of the OS is a separation of concerns issue. I want my OS people focusing on OS stuff. The versions of installed apps is up to the developers of those apps, doubly so on something like a browser that updates rapidly.
      - pessimizer 5y ago
        No, the versions of installed apps should be up to me. And my OS should help me manage that.
  - causi 5y ago
    "Need" is the key phrase. If it wants to check for updates, do telemetry, fiddle around, whatever, that's fine by me. It should not shit the bed if it has to go a few hours without doing so, however. For example, you recall a couple of years ago when Mozilla screwed up their certificates and every Firefox extension was simultaneously disabled. That should not have happened. That should have resulted in a pop-up box that said "The security of your extensions cannot be verified. Using them at this time could be highly dangerous. Disable them to browse safely? Yes/No" Instead Mozilla just blanket turned them all off, which had the potential for getting people killed as they suddenly found themselves not protected by VPN/Tor/NoScript/etc extensions.
cosmin800 5y ago
jacquesm 5y ago
There was no 'Firefox Outage' because Firefox is not a service. There was a bug, and a production issue with a service that Firefox users were involuntarily opted into. Lesson learned: do not opt your users into services without their consent.
- Klonoar 5y ago
  The article outright explains how it wasn’t limited to Telemetry and could be triggered otherwise.
  - jacquesm 5y ago
    It could have been, but it wasn't. See, if I had opted out of this junk, which I wasn't because it was enabled without my consent I would not have experienced that particular problem (but others would have) and I would have been able to save myself a couple of hours of debugging. So yes, it wasn't limited to Telemetry, but no I had not seen the bug in practice until that very moment.
perryizgr8 5y ago
When you read the whole account, everything has a justifiable reason and the entire thing is very rational. But if you look at the 10000ft view, if an app hangs and refuses to work at all due to nasty/unexpected input coming over the network, that is just a bad bug. These sorts of bugs should have been caught earlier. It shouldn't matter if the entire internet is sending bad responses, Firefox should still handle it gracefully. It does not need this big of an explanation. It's just a silly bug, they can do better by improving their testing. The lady doth protest too much.
- rockdoe 5y ago
  I think the length of the explanation is due to the impact this had. Most Firefox bugs don't cause all Firefoxes everywhere to stop working simultaneously. That's quite the WTF and it's hard to understand how it is even possible without the explanation.
IAmNotAFix 5y ago
> all network requests go through one socket thread Looks like the crucial issue to me. The SPOF which enables something as absurd as "Firefox Outage".
- rockdoe 5y ago
  This isn't atypical for high performance network software, AFAIK. Single-thread with aio.

323 points by feross 5y ago

That was an extremely high risk change on GCP's part, reminds me of the App Engine days when you'd wake up to find a totally healthy program spamming 500s because they'd make a breaking change without any announcement. It's shocking they're still pulling stuff like this in 2022

agilob 5y ago
It reminds me how YouTube enforced new codec with a few days notice knowing that FireFox doesn't support it, so FF couldn't play most YT videos for over a week.
- faeyanpiraat 5y ago
  Did yt back up or ff updated to fix issue?
  - agilob 5y ago
    Firefox got the codecs working in the next release
- TheGoddessInari 5y ago
  > enforced new codec YouTube turned off flash player as the default in 2015, and VP9 was supported in Firefox at the same time. YouTube still serves h.264, vp9, and av1. I was trying to figure out what this could be referring to.
  - agilob 5y ago
    Not all codecs were ported at the same time, and then not enabled by default, and when enabled by default it was platform dependent, for others where it worked FF was eating all possible CPU resources and videos were glitching. I remember this as I was using Debian and I was active on /r/firefox, where this [1] link was posted 10 times every day [1] https://www.youtube.com/html5 https://www.youtube.com/html5
kevingadd 5y ago
Feels like an especially severe version of the consistent Google pattern of only testing stuff on Chrome, so new updates/features ship in a way that is some degree of broken on Firefox/Safari. For a significant amount of time YouTube had bad performance on Firefox because they chose to use Web Components by default with a horrible polyfill instead of using the old (still working!) html5 version that ran great.
- loulouxiv 5y ago
  Putting in place an infrastructure to test this kind on changes on the 5-10 most popular browser would be, I think, very cheap for a company like Google. I can't help thinking these may be deliberate moves to eat the little market shares of Chrome concurrents. I remember reading here on HN an article written by an ex-Mozilla insider relating the dissonance between the "friendly" Mozilla-Google employees exchanges and the year-long track record of very oddly recurrent "unfortunate mistakes" from Google degrading the Firefox compatibility.
  - magicalist 5y ago
    > Putting in place an infrastructure to test this kind on changes on the 5-10 most popular browser would be, I think, very cheap for a company like Google. The problem wasn't some web server, it's the Firefox backend services running on GCP.
  - loulouxiv 5y ago
    I am trying to find the link, but for the moment I only find comment making, I think, references to it : https://news.ycombinator.com/item?id=19815348 https://news.ycombinator.com/item?id=19815348 https://news.ycombinator.com/item?id=28495546 https://news.ycombinator.com/item?id=28495546 Does anybody here remember enough keywords to find it out ? Edit: I guess it was this Twitter thread https://mobile.twitter.com/johnath/status/1116871231792455686 https://mobile.twitter.com/johnath/status/111687123179245568... Edit 2: The associated HN thread https://news.ycombinator.com/item?id=19662852 https://news.ycombinator.com/item?id=19662852
  - deleted 5y ago
    [deleted]
charcircuit 5y ago
Would a warning have even helped that much? Since HTTP/3 was expected to be working there wouldn't be a cause to worry.
- Semaphor 5y ago
  > Would a warning have even helped that much? Since HTTP/3 was expected to be working there wouldn't be a cause to worry. It might (as mentioned in TFA) have made them think to run some extra tests, which could have caught the bug. But it also would have made the response faster, as they would have known what changed far sooner.
- 0xbadcafebee 5y ago
  The key word there is "expected". When you do Operations for a living, the only thing you can expect is the unexpected. That's why even after you think you've tested a change, you carefully and slowly roll it out a bit at a time, monitoring golden metrics so you can detect a problem, stop the roll-out, and roll back. It sounds like somebody just flipped a giant switch and never checked error rates, connection metrics, anything. Check out this graph: https://hacks.mozilla.org/files/2022/01/crashes-foxstuck2-2048x946.png https://hacks.mozilla.org/files/2022/01/crashes-foxstuck2-20... Think maybe that would indicate somebody needs to roll back the last change? The problem here, as usual, is a disconnect between stakeholders. Google has this service (it seems like the load balancer for their customer?) it wants to change for one reason or another. The customers may or may not have planned for the change Google is making. Google makes the change, but it isn't a stakeholder of the customer (they basically don't care what happens to the customer). So there is no direct feedback loop for the customer to tell Google something is wrong. If Google was at risk of losing business from its customers going down, it would have a strong relationship with those customers and have a way to quickly help diagnose problems and roll back changes if needed. This is a great lesson for all customers to take away: don't depend on people who you don't have a close relationship with.
brabel 5y ago
The client, Firefox, said it supported HTTP/3 though. Otherwise it wouldn't get to use that. I don't think that's as bad as you try to make it... if the client says it supports something then it breaks when it uses it, it's the fault of the client, not the server.
- alisonkisk 5y ago
- throwaway984393 5y ago
  No SRE in the world that is halfway decent at their job would think that way. You never make assumptions about any kind of change, much less a global change to a completely different protocol. Doesn't matter whose fault it is. You just don't introduce any change that has a chance of unexpected behavior without rigorous testing, and you roll it out g r a d u a l l y, and you stop when error rates increase. Google literally wrote the books on SRE. For them to not know better is absurd.
notyourday 5y ago
> That was an extremely high risk change on GCP's part, reminds me of the App Engine days when you'd wake up to find a totally healthy program spamming 500s because they'd make a breaking change without any announcement. It's shocking they're still pulling stuff like this in 2022 reply Lay with the dogs, wake up with the fleas. Google is a shitty company producing shitty products. When you select to do business with Google you select to do business with a shitty company producing shitty products and treating its customers like shit. Hence I fail to understand the Surprised Pikachu face when something like this happens.

0x002A 5y ago

"Because all network requests go through one socket thread, this loop blocked any further network communication and made Firefox unresponsive, unable to load web content." Why side functionality (telemetry) of a tool uses only one network thread and can block any network communication ?

floatingatoll 5y ago
Per elsethread, supposedly any site could have triggered the crash, not just ”side functionality”: https://news.ycombinator.com/item?id=30175916 https://news.ycombinator.com/item?id=30175916
- 0x002A 5y ago
  Yes that's another issue. But as a design pattern shouldn't we design our products to do their core functionality as much as independent from any anomalies that can happen? This is almost akin to me if Tesla rolls out an update and the car decides to pull over to the curb to do the update, while you are driving to your job or worse to hospital with an emergency. My theory is there should be at least one health enterprise using firefox as their only browser for business functionality out in the wild.
  - Yoric 5y ago
    In theory, that's how it happens in Firefox. But when you have a bug in the core of the product (the network stack), there isn't much that the rest of the product can do to isolate from it.
    - 0x002A 5y ago
      Yes that's what you get when you have one thread for all network communications. The network stack did not fail, only the sole network thread got stuck. From the write up I understand if there was another thread for communications firefox would only fail to communicate with telemetry service but firefox would be able to function as users needed.

ghostly_s 5y ago

Went into this with an indignant "failures in telemetry should NEVER bring down core functionality!" feeling, left with a more nuanced understanding of the fault and impressed and reassured by the mitigation steps being taken. That's a great post-mortem.

nathansherburn 5y ago
I felt a similar way when it first happened. I was fuming "an update should never break your core functionality across all devices!" only to find that it wasn't actually the latest update but rather code that had been there for ages. It's hard to hate something once you truly understand it, I guess.
minerva23 5y ago
Anyone know why telemetry is sent from the main process rather than having a separate reporter process?
- rockdoe 5y ago
  It would have made no difference: the hang was in the networking code & process, not the telemetry code (and whatever process that is in). Anything that sent a message with the "bad" header type would have provoked the hang. As to why, dunno, presumably it's extra effort for an unclear gain. Telemetry code wouldn't be parsing hostile input etc. And it doesn't stop bugs like this either.
- marcan_42 5y ago
  That would be significant extra engineering effort without much clear benefit. It's always possible to look back and say "if only they'd done that", but this bug was a freak coincidence and it's not possible to foresee things like that, nor is it worth the engineering effort to try to avoid them with hammers as big as "move all telemetry to another process". You'd need a much stronger reason than "what if telemetry happens to trigger a longstanding bug in the network stack" to decide to go with that. In this case, that the problem was triggered by telemetry was a coincidence; it would've been triggered by other bits of coden in the future as they moved to Rust.

deleted 5y ago

[deleted]

blackbear_ 5y ago

kevingadd 5y ago
Does Firefox force updates on your configuration? On Windows it's opt-in for me. I know Android will forcibly update any app (including Firefox) while you're using it, but you can shut that off system-wide.
- joshgev 5y ago
  The default is that it forces a restart when a new update has been downloaded, which has been a frustration for me as apparently it has been for the parent. Apparently this can be changed by requesting that FF only install updates with explicit consent[1]. I'd think the best way would be to install anything that's available locally when the browser starts without forcing a restart. [1] https://superuser.com/questions/1451210/how-can-i-make-firefox-stop-forcing-me-to-restart-my-browser https://superuser.com/questions/1451210/how-can-i-make-firef...
  - Vinnl 5y ago
    Are you using Linux, by any chance? IIRC, the issue here is that the update is done by your package manager, changing Firefox's files out from under them. If you use the direct download from Mozilla, it shouldn't be as disruptive. I still hope a fix or workaround to this can be found, but knowing why something is the cause makes it easier to accept, at least for me :)
- Nextgrid 5y ago
  What pisses me off is less about the update itself (they're typically unnoticeable) but the constant nags and "what's new" crap that opens up after the update. Firefox is more hostile than a lot of paid, proprietary software in this regard.
  - brimble 5y ago
    This. That crap is significantly more disruptive and irritating that the actual paid ads in old-school free Opera (the largest feature-comparable browser when FF/Phoenix/Firebird first launched).

deleted 5y ago

[deleted]

luciusdomitius 5y ago

Traubenfuchs 5y ago
Previous discussion: https://news.ycombinator.com/item?id=24563698 https://news.ycombinator.com/item?id=24563698
hackerfromthefu 5y ago
She's probably earned another raise if her actual reason for appointment is to drive chrome adoption while soaking up community engagement to prevent another open source browser competitor, then she's succeeding very well!
- luciusdomitius 5y ago
  https://www.androidheadlines.com/2020/08/mozilla-firefox-google-search https://www.androidheadlines.com/2020/08/mozilla-firefox-goo... Well. Seems that it cannot be ruled out.
  - deleted 5y ago
    [deleted]
deleted 5y ago
[deleted]
ricardobayes 5y ago
It's very interesting to see how rapidly FF fell out of favor for some reason. Went from poster child to pariah in what seemed like no time.
- luciusdomitius 5y ago
  Well. The actual lead dev got kicked out as Mozilla Foundation chair and got replaced by some SJWs over his support for banning abortions, same-sex marriages or something similar.

drekipus 5y ago

intothemild 5y ago
I don't get this. The issue had nothing to do with the language used.
- account42 5y ago
  It did have something to do with there being two languages used though, resulting in two different ways of excercising the network stack whith less testing for each. So its not a bug caused by Rust, but it is caused by the addition of Rust in Firefox.
kevingadd 5y ago
This isn't specifically caused by rust, it's a second-system problem where they had two different ways to touch the http system and one was broken. It could have easily been the other way around with only rust being correct.
- drekipus 5y ago
  Yes, I know that, I'm trying to make the joke stand out but it's hard to do
  - milliams 5y ago
    https://en.wikipedia.org/wiki/Poe%27s_law https://en.wikipedia.org/wiki/Poe%27s_law strikes again.
evilpie 5y ago
The actual bug was in the C++ code handling the Content-Length header (https://hg.mozilla.org/integration/autoland/rev/48c0e854be623e654f1a0814d22502ca5c618422 https://hg.mozilla.org/integration/autoland/rev/48c0e854be62...) and the Rust code just happened to be first to trigger it.

sirl1on 5y ago

Why does my browser need connectivity to some internal services? I am fine with offering opt-in service integration (Firefox Sync, Pocket, ...) but is there a reason why Firefox needs internal infrastructure to do the one thing it is supposed to do, browsing the web? I can only think of DNS over HTTP, but AFAIK that is also opt-in, right? Man, I love Firefox and used it since it was called Firebird (with a small gap when Chrome was shiny and new and Firefox a slow RAM hog). But I really resent the Mozilla Foundation, they seem to be interested in everything but browser development. To be fair, (ab)using the browser as application runtime brought us so much complexity that developing and maintaining a secure browser as free software spare time project isn't feasible anymore.

lmm 5y ago
> Why does my browser need connectivity to some internal services? I am fine with offering opt-in service integration (Firefox Sync, Pocket, ...) but is there a reason why Firefox needs internal infrastructure to do the one thing it is supposed to do, browsing the web? If you read the whole post, the connection was explicitly for telemetry (and so you could avoid the issue by turning off telemetry), and it blocked other connections because the request went into an infinite loop rather than failing outright.
- nanis 5y ago
  > If you read the whole post, ... (and so you could avoid the issue by turning off telemetry) Speaking of reading the whole post: >> users who disabled Telemetry would see this problem resolved even though the problem is not related to Telemetry functionality itself and could have been triggered otherwise. This does not make sense to me: >> Without the header, the request was determined by the Necko code to be complete, This is written as if it makes sense to treat a request as "complete" when it's missing a content length header. Huh?!
  - mozdeco 5y ago
    At this point, the code relied on the Content-Length header being present because the higher-level API was supposed to add it. The field that is supposed to be populated by Content-Length (mRequestBodyLenRemaining) is pre-initialized to 0.

ricardobayes 5y ago

96 by far is the worst release, probably even in the history of FF.

danuker 5y ago
What can we do about it? Use an older version? Security problems. Use a simpler browser? Many sites will stop working. Maybe it's for the best to avoid using sites that use complicated JS/CSS/HTML. But will still need it for say, government sites to pay taxes.
- iqanq 5y ago
  Everybody should use Firefox ESR. It's stuck at Firefox 91 and you receive security updates, but you get no ads, your settings do not reset...
  - Semaphor 5y ago
    I’m up-to-date and have neither ads, nor reset settings.
- userbinator 5y ago
  You mean "Security" problems... There's a lot of FUD and paranoia out there; 99% of exploits need JS and even those which don't technically need it, are almost always obfuscated using JS. Leave JS off by default (there are extensions to do that) and don't turn it on unless you really do trust the site to run arbitrary code on your computer, and you're unlikely to encounter any problems.
FeepingCreature 5y ago
57 was worse.

shantara 5y ago

It's incredible to see a normalization of the term "outage" applied to a software running on a local hardware.

deleted 5y ago
[deleted]

pdenton 5y ago

I haven't noticed anything and I use Firefox every day. Nor have any of my clients where I deployed Firefox called. Is this because I always disable data collection in settings?

ainar-g 5y ago
Yes, the bug was in the Telemetry code. I'm not sure if it's on by default, but it's probably better to disable it for any large-scale deployments. Both to prevent things like this and to make sure that things that should be disabled by default actually are.
- pdenton 5y ago
  Ah, that explains it. Thank you. I recommend https://ffprofile.com/ https://ffprofile.com/ which was posted earlier here on HN. Makes it easier to deploy Firefox with saner defaults. You know I moved from Netscape 3.0 Gold to later versions, to Mozilla, to Phoenix, Firebird, and then Firefox. I tried other browsers but it's always a subpar experience for me. My only gripe is that they kept changing the UI.
  - floatingatoll 5y ago
    (FYI: The bug was not in the telemetry code, it was in the http3 code; see comments further up in the post for details.)
Ygg2 5y ago
It's not a telemetry bug. It's a networking bug. Avoiding it basically comes down to being extremely lucky.
mccr8 5y ago
It was broken for a couple of hours, and late at night in the Americas. You might have been asleep for the entire duration of the outage.

mrweasel 5y ago

It's a nice write up, but the "Lessons learned" is terrible, it appears as if they didn't learn anything and view the whole thing as an infrastructure problem. As others pointed out, why does Firefox even need to communicate with Mozilla services? Sure, telemetry needs to feed data back, if enabled, but if that fails why does it need to stop the browser from working? Shouldn't the lesson learned be: The telemetry functionality in Firefox has a bug, where an infrastructure outage at Mozilla can "break" the browser. The fix isn't in infrastructure, the fix has to be in the code that communicates back, it should fail gracefully. It's not a problem if telemetry fail, either cache locally and just drop the data, it's honestly not important. I'm sorry, I get that it's interesting how and why all this failed, but Mozilla makes it seem like they don't get what the root of the problem is.

floatingatoll 5y ago
Per other comments here, supposedly the same issue would have occurred with a variety of non-telemetry tasks as well. One of them is indicated to be certificate-related, which suggests to me that CRL lookups could have triggered it, though I don’t know for sure. It ended up being that the issue occurred first with telemetry rather than with certificate stuff or who knows what else. But the flaw wasn’t in telemetry code, so focusing on telemetry wouldn’t have prevented this at all.
progbits 5y ago
Exactly. I love Firefox and use it as my primary browser but this is the wrong conclusion. Don't make your telemetry backend more reliable. Instead break it on purpose several times a day. That way a similar bug in the browser will not make it past dev channel.
- Yoric 5y ago
  As mentioned in a few places, the telemetry backend was not the culprit. The network stack was. The network stack is already pretty heavily tested and fuzzed, but obviously that didn't catch this specific bug.
  - 5y ago

brabel 5y ago

A lot of intelligent sounding words to explain a trivial bug: their network stack was using case-sensitive header names, which anyone doing anything remotely related to HTTP knows is a mistake. We all make mistakes, but don't try to make it sound more grandiose ("a combination of multiple factors blah balh blah") than it is.

viraptor 5y ago
Http2 and 3 are case sensitive at when encoding the headers: > As in HTTP/2, characters in field names MUST be converted to lowercase prior to their encoding. A request or response containing uppercase characters in field names MUST be treated as malformed https://quicwg.org/base-drafts/draft-ietf-quic-http.html#section-4.1.1-2 https://quicwg.org/base-drafts/draft-ietf-quic-http.html#sec...
matsemann 5y ago
> which anyone doing anything remotely related to HTTP knows is a mistake. Only if you haven't updated your knowledge since HTTP/1.1 days...

denton-scratch 5y ago

> there were many contributing factors working together Looks like one factor, to my eyes: telemetry. I have telemetry disabled. But if you're going to default to "telemetry on", and then silently send data to sites that aren't in the address-bar, then it's your responsibility not to "break the web". You can't blame it on rust, or necko, or viaduct, or google.

Liquid_Fire 5y ago
> This is why users who disabled Telemetry would see this problem resolved even though the problem is not related to Telemetry functionality itself and could have been triggered otherwise. It's pretty clear from the article that this was a bug, and telemetry requests failing was not intended to break the rest of the browser. If Google had rolled out this change a year from now, it could probably have broken something other than just telemetry (e.g. maybe update checks, or certificate management) and your browser would still have been broken even with telemetry disabled.
- hypertele-Xii 5y ago
  > It's pretty clear from the article that this was a bug, and telemetry requests failing was not intended to break the rest of the browser. If telemetry didn't exist as core part of the browser, nothing would've broken. Therefore, the telemetry itself is the direct cause of the bug. It was, at best, poorly handled and too deeply integrated into the browser's core function.
  - matsemann 5y ago
    The browser would have broken the next time a malformed http3 request happened. Read the quote again.
- pessimizer 5y ago
  > telemetry requests failing was not intended to break the rest of the browser. You can't find a lighter straw man than that. Find me somebody who said that Firefox intended to break the browser if telemetry failed.

truth_revealer 5y ago

[flagged]

dismalpedigree 5y ago
In standard conspiracy theory fashion, short on who and why. Two parts that are critical. People talk. The more outrageous the conspiracy, the more people need to be involved. Someone always talks. Then lets understand how these people stand to benefit. Nobody is going to extraordinary lengths just to burn it all down. I love me a good conspiracy theory and playing “what-if”, but the things i take action on are the ones which pass my two criteria above.
- truth_revealer 5y ago

ankurdhama 5y ago

Not sure how Firefox code is structured but it is weird that a particular HTTP3 request would hang the entire network stack and you cannot perform any HTTP 2/1 requests.

mozdeco 5y ago
All requests go through one socket thread, no matter which HTTP version. I am not a Necko engineer, but since requests can be upgraded, an HTTP/1 request could switch to HTTP/2 and if there was a separation by protocol, the request would have to be "moved" to a different thread. So I'm not sure that would work easily.
marcan_42 5y ago
Is nobody reading the article? It was an infinite loop. Not a blocked request. A bug. A logic flaw. Something that wouldn't normally happen. It was broken code. You know what happens when you put a while(1); in the middle of the nginx codebase? The whole server process hangs. This is normal in an async design. We don't write software to be magically resilient against freak bugs, especially not something like a browser that is not intended to be used in life-critical applications.

kragen 5y ago

It should be impossible for the phrase "the recent Firefox outage" to make sense. Has there ever been an "outage" of linear algebra? Of Linux? Of mitochondria? Of Bitcoin? It is critically important that we not introduce new single points of failure into the systems that our civilization depends on, and that we remove the ones that already exist. If it can happen by accident, it can happen on purpose.

ainar-g 5y ago
While I agree with the SPOF point (heh), the header should really read “the recent Firefox Telemetry outage”. It's kind of like a website causing your browser or a tab of the browser to be unresponsive by introducing an infinite busy loop. Except the “tab” was invisible.
- detaro 5y ago
  The user impact (Firefox not working) is more important than the technical reason, and thus should be in the title.
- mozdeco 5y ago
  The infinite busy loop in this case was not the tab no (neither visible or invisible). The loop was directly in the network stack, as stated in the post, not in the caller.
- kragen 5y ago
  The problem isn't that there was a telemetry outage; the problem is that the telemetry outage caused a Firefox outage, which should not be a thing. Firefox needs to be more robust than that.
  - dralley 5y ago
    That's not what happened though. Telemetry uses the same networking stack as everything else, and the busy loop was in the networking stack.

deng 5y ago

I'd claim that no other company is criticized as harshly as Mozilla around here. The amount of blame that is assigned to the Firefox team is staggering. To me, this is a perfectly valid write-up with a good lessons learned. They have written it in a very diplomatic way, but to me, it is absolutely clear that Google screwed up here. How can you make such a change to a default behavior of critical infrastructure unannounced? That's just reckless towards your customers, and solidifies my belief to stay away from GCP. If they had properly announced the change, even if the Firefox team hadn't then tested beforehand, at least the DevOps team would have put one and one together and just changed back to HTTP/2 and the outage would have lasted maybe 10 minutes. Instead, they frantically went through their git log to see what in the code base might have triggered this bug. Everyone who has been in such a position knows how incredibly stressful this is. I'd be absolutely livid at Google in their position. That it took two hours to fix this is clearly their fault.

radmuzom 5y ago
- deleted 5y ago
  [deleted]
- jrochkind1 5y ago
  it's probably not the same people, even though you see both opinions on HN.
moeris 5y ago
I agree that Google is at fault here for failing Firefox. But Firefox is guilty of failing its users. Why should the functioning of a browser be dependent on telemetry working? It sounds like if there is high enough latency in their telemetry, or if request for telemetry start failing, it's possible for that to disrupt using the network stack at all. They have a massive design flaw, and they didn't even mention that in the article. Maybe they have good reasons for designing a single point of failure that relies on a cloud provider, but it's not clear what those might be since they don't address it.

hngpt2 5y ago

The recent outage for users of Mozilla’s Firefox web browser was caused by a Firefox update, the source code for which was inadvertently pushed to the Mozilla Add-ons Repository. That repository is not publicly available, so it was not possible for users to update their installations. Fortunately, a bug was introduced in the Firefox update which prevented the update from being installed. The bug was fixed when the update was rolled back. Users who updated earlier were protected. The outage could have been worse, though, because the bug that was introduced in the code for the Firefox update was in a feature of Firefox itself that allows users to block certain updates until a later time. In the case of Firefox, that feature was used to block the update that caused the outage. If a similar situation occurred in the future, users would not have a way to block the update that causes the outage. That feature is available in other web browsers, but it is not as advanced or robust.

deleted 5y ago
[deleted]

causi 5y ago

God I hate the modern web. Why the hell does my browser need more than periodic contact with any server other than my DNS provider and the host of the website I'm connecting to?

MauranKilom 5y ago
What does "more than periodic" mean? But to answer your question: For example, to inform you that an update is available.
- account42 5y ago
  Updates and associated checks should not be done by each application itself.
  - Karunamon 5y ago
    Hard disagree. Having that in the hands of the OS is a separation of concerns issue. I want my OS people focusing on OS stuff. The versions of installed apps is up to the developers of those apps, doubly so on something like a browser that updates rapidly.
    - pessimizer 5y ago
      No, the versions of installed apps should be up to me. And my OS should help me manage that.
- causi 5y ago
  "Need" is the key phrase. If it wants to check for updates, do telemetry, fiddle around, whatever, that's fine by me. It should not shit the bed if it has to go a few hours without doing so, however. For example, you recall a couple of years ago when Mozilla screwed up their certificates and every Firefox extension was simultaneously disabled. That should not have happened. That should have resulted in a pop-up box that said "The security of your extensions cannot be verified. Using them at this time could be highly dangerous. Disable them to browse safely? Yes/No" Instead Mozilla just blanket turned them all off, which had the potential for getting people killed as they suddenly found themselves not protected by VPN/Tor/NoScript/etc extensions.

cosmin800 5y ago

jacquesm 5y ago

There was no 'Firefox Outage' because Firefox is not a service. There was a bug, and a production issue with a service that Firefox users were involuntarily opted into. Lesson learned: do not opt your users into services without their consent.

Klonoar 5y ago
The article outright explains how it wasn’t limited to Telemetry and could be triggered otherwise.
- jacquesm 5y ago
  It could have been, but it wasn't. See, if I had opted out of this junk, which I wasn't because it was enabled without my consent I would not have experienced that particular problem (but others would have) and I would have been able to save myself a couple of hours of debugging. So yes, it wasn't limited to Telemetry, but no I had not seen the bug in practice until that very moment.

perryizgr8 5y ago

When you read the whole account, everything has a justifiable reason and the entire thing is very rational. But if you look at the 10000ft view, if an app hangs and refuses to work at all due to nasty/unexpected input coming over the network, that is just a bad bug. These sorts of bugs should have been caught earlier. It shouldn't matter if the entire internet is sending bad responses, Firefox should still handle it gracefully. It does not need this big of an explanation. It's just a silly bug, they can do better by improving their testing. The lady doth protest too much.

rockdoe 5y ago
I think the length of the explanation is due to the impact this had. Most Firefox bugs don't cause all Firefoxes everywhere to stop working simultaneously. That's quite the WTF and it's hard to understand how it is even possible without the explanation.

IAmNotAFix 5y ago

> all network requests go through one socket thread Looks like the crucial issue to me. The SPOF which enables something as absurd as "Firefox Outage".

rockdoe 5y ago
This isn't atypical for high performance network software, AFAIK. Single-thread with aio.