8 ms·

I'd claim that no other company is criticized as harshly as Mozilla around here. The amount of blame that is assigned to the Firefox team is staggering. To me,

by deng 5y ago

I'd claim that no other company is criticized as harshly as Mozilla around here. The amount of blame that is assigned to the Firefox team is staggering. To me, this is a perfectly valid write-up with a good lessons learned. They have written it in a very diplomatic way, but to me, it is absolutely clear that Google screwed up here. How can you make such a change to a default behavior of critical infrastructure unannounced? That's just reckless towards your customers, and solidifies my belief to stay away from GCP. If they had properly announced the change, even if the Firefox team hadn't then tested beforehand, at least the DevOps team would have put one and one together and just changed back to HTTP/2 and the outage would have lasted maybe 10 minutes. Instead, they frantically went through their git log to see what in the code base might have triggered this bug. Everyone who has been in such a position knows how incredibly stressful this is. I'd be absolutely livid at Google in their position. That it took two hours to fix this is clearly their fault.

radmuzom 5y ago
- deleted 5y ago
  [deleted]
- jrochkind1 5y ago
  it's probably not the same people, even though you see both opinions on HN.
moeris 5y ago
I agree that Google is at fault here for failing Firefox. But Firefox is guilty of failing its users. Why should the functioning of a browser be dependent on telemetry working? It sounds like if there is high enough latency in their telemetry, or if request for telemetry start failing, it's possible for that to disrupt using the network stack at all. They have a massive design flaw, and they didn't even mention that in the article. Maybe they have good reasons for designing a single point of failure that relies on a cloud provider, but it's not clear what those might be since they don't address it.
- calcifer 5y ago
  > Why should the functioning of a browser be dependent on telemetry working? It isn't. The bug was in the networking stack, and it just happened to be triggered by a GCP change which effected the telemetry service. Firefox having telemetry has nothing to do with the issue here.
  - dralley 5y ago
    Thus proving OP's point. "The amount of blame that is assigned to the Firefox team is staggering"
  - barrkel 5y ago
    That's not quite right. A single socket thread does all the requests and telemetry is multiplexed with user traffic. If telemetry is different in some way to other network traffic, then it's always possible for it to cause problems with user traffic. Telemetry is different to user traffic - it's less important! - but of course any in-process QoS would still create a point of interaction with user traffic.
  - LeifCarrotson 5y ago
    How do you reach that conclusion? From the article: > It just so happens that Telemetry is currently the only Rust-based component in Firefox Desktop that uses the [viaduct/Necko] network stack and adds a Content-Length header. This is why users who disabled Telemetry would see this problem resolved ... The article contradicts your conclusion. If Firefox did not have telemetry, the bug would have had no impact, and users would not have suffered an outage. > ...even though the problem is not related to Telemetry functionality itself and could have been triggered otherwise. And then the article contradicts itself and agrees with you using some heavy-duty doublethink. Sure, if there were hypothetically other Rust services using the buggy network stack, they'd also have hit the bug: BUT THERE ARE NONE. The bug was in code which is only running because it's used by the telemetry services, so even though it might be in a different semantic layer it's the fault of the browser trying to send telemetry. As a user, I place very low (often negative) importance on the tools I use collecting telemetry data, or on protecting DRM content, or on checking licensing status. They should focus on doing the job I'm trying to do with them on my computerr, serving the uses of the user, rather than doing something that someone else wants them to do. Sure, I understand that debugging and quality monitoring are easier with logs and maybe with telemetry, so I can understand using a few resources in the background to serve some of that data, but it must never get in the way of actual work getting done.
magicalist 5y ago
> Instead, they frantically went through their git log to see what in the code base might have triggered this bug. This seems like you're embellishing this part to tell a story? It's not supported by the linked post, and from the bugzilla bugs it seems like it was known almost immediately that the ESR builds were affected as well and so it almost had to be an external service, they just weren't sure which one at first.
TheIronMark 5y ago
Google should not be rolling out unannounced changes, but Mozilla increased their risk by not pinning the HTTP version.
Macha 5y ago
> I'd claim that no other company is criticized as harshly as Mozilla around here. The amount of blame that is assigned to the Firefox team is staggering. Mozilla has opened themselves up here, as they market as a privacy and user respecting alternative, so when they fail to live up to their own marketing people are more annoyed while they expect random startup #456 to not care about their users privacy and have telemetry out the wahoo.
- dblohm7 5y ago
  (I used to work for Mozilla, and spent a few years on the team that at the time owned the telemetry component.) The way Mozilla does telemetry is different from how most places do. I think that the biggest issue with these discussions is that there always seems to be this assumption that there is only one way to do telemetry, it always contains super invasive PII, Mozilla's telemetry must do the same, and therefore Mozilla's telemetry is just as evil as anybody else's. Mozilla is remarkably open about how its telemetry works, beyond just being open source. Maybe this is more a problem of that information not being surfaced well, I dunno. I get that some people are philosophically opposed to telemetry no matter what, but I have seen enough cases of, "Wow, I didn't know it worked that way, and I'm actually okay with this," to know that informed users are not universally opposed to it.
  - wolverine876 5y ago
    Could you share a link?
    - dblohm7 5y ago
      I can share several! If you're running Firefox, you can go to `about:telemetry` and see what data is there. Note that some of that data might be populated even if you have telemetry turned off. Don't reach for your pitchfork quite yet: I assure you that the data isn't being sent. You can go to https://telemetry.mozilla.org https://telemetry.mozilla.org to see various analyses involving telemetry data. In the source tree, telemetry probes must be defined in one of a few places. Here are their locations: https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/Histograms.json https://searchfox.org/mozilla-central/source/toolkit/compone... https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/Scalars.yaml https://searchfox.org/mozilla-central/source/toolkit/compone... https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/Events.yaml https://searchfox.org/mozilla-central/source/toolkit/compone... https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/geckoview/streaming/metrics.yaml https://searchfox.org/mozilla-central/source/toolkit/compone...
yjftsjthsd-h 5y ago
> I'd claim that no other company is criticized as harshly as Mozilla around here. The amount of blame that is assigned to the Firefox team is staggering. They set themselves a higher standard by marketing as the good guys who fight for the user, and then made any number of moves that said users viewed as not being in their interests. Of course they get more blame. Like, Chrome has issues, but they're issues in line with being made by an adtech company; we might be unhappy at Google breaking adblockers (https://www.eff.org/deeplinks/2021/12/chrome-users-beware-manifest-v3-deceitful-and-threatening https://www.eff.org/deeplinks/2021/12/chrome-users-beware-ma...), but it's not out of character. Mozilla can say "More power to you. Mozilla puts people before profit, creating products, technologies and programs that make the internet healthier for everyone." (https://www.mozilla.org/en-US/ https://www.mozilla.org/en-US/) or they can, say, make Google the default engine ($), bake in a proprietary service (Pocket), rip out features (RIP compact theme), overrule user autonomy (Want to install an extension? Better upload it to Mozilla to get signed so they permit you to run it on your own computer!), ship a marketing extension through the "experiments" feature (https://blog.mozilla.org/en/products/firefox/update-looking-glass-add/ https://blog.mozilla.org/en/products/firefox/update-looking-...).... but not both. Either empower the user, or don't, but don't pretend to empower the user while ripping away their control.
- ygjb 5y ago
  Yep, you are correct. Each of those decisions was made over the protests of a vocal but relatively small group of users. You can't please all people all of the time, and I agree the pocket integration, and the looking glass add were mistakes, but the other items were directly related to sustainability of the project ($, eng cycles), or user safety. You can disagree with them as much as you like, but Firefox continues to support the ultimate in user control by releasing their product as open source. Roll your own build that doesn't require those features, sideload your add-ons, and/or fork the product. As a user, the average Firefox user has far more control over the browser than Chrome, Edge, or Safari users do, and have the flexibility to use one of many Firefox forks that have the same beef as you.
  - ttybird2 5y ago
Santosh83 5y ago
What happened to the famed intelligence level of Hacker News? Every single person (almost) in this thread is blaming telemetry while it was clear even when the bug was ongoing that it was unrelated to telemetry. I for example had had telemetry disabled and still hit the bug through other traffic and had to temporarily disable HTTP3 from about:config.

by deng 5y ago

radmuzom 5y ago

deleted 5y ago
[deleted]
jrochkind1 5y ago
it's probably not the same people, even though you see both opinions on HN.

moeris 5y ago

I agree that Google is at fault here for failing Firefox. But Firefox is guilty of failing its users. Why should the functioning of a browser be dependent on telemetry working? It sounds like if there is high enough latency in their telemetry, or if request for telemetry start failing, it's possible for that to disrupt using the network stack at all. They have a massive design flaw, and they didn't even mention that in the article. Maybe they have good reasons for designing a single point of failure that relies on a cloud provider, but it's not clear what those might be since they don't address it.

calcifer 5y ago
> Why should the functioning of a browser be dependent on telemetry working? It isn't. The bug was in the networking stack, and it just happened to be triggered by a GCP change which effected the telemetry service. Firefox having telemetry has nothing to do with the issue here.
- dralley 5y ago
  Thus proving OP's point. "The amount of blame that is assigned to the Firefox team is staggering"
- barrkel 5y ago
  That's not quite right. A single socket thread does all the requests and telemetry is multiplexed with user traffic. If telemetry is different in some way to other network traffic, then it's always possible for it to cause problems with user traffic. Telemetry is different to user traffic - it's less important! - but of course any in-process QoS would still create a point of interaction with user traffic.
- LeifCarrotson 5y ago
  How do you reach that conclusion? From the article: > It just so happens that Telemetry is currently the only Rust-based component in Firefox Desktop that uses the [viaduct/Necko] network stack and adds a Content-Length header. This is why users who disabled Telemetry would see this problem resolved ... The article contradicts your conclusion. If Firefox did not have telemetry, the bug would have had no impact, and users would not have suffered an outage. > ...even though the problem is not related to Telemetry functionality itself and could have been triggered otherwise. And then the article contradicts itself and agrees with you using some heavy-duty doublethink. Sure, if there were hypothetically other Rust services using the buggy network stack, they'd also have hit the bug: BUT THERE ARE NONE. The bug was in code which is only running because it's used by the telemetry services, so even though it might be in a different semantic layer it's the fault of the browser trying to send telemetry. As a user, I place very low (often negative) importance on the tools I use collecting telemetry data, or on protecting DRM content, or on checking licensing status. They should focus on doing the job I'm trying to do with them on my computerr, serving the uses of the user, rather than doing something that someone else wants them to do. Sure, I understand that debugging and quality monitoring are easier with logs and maybe with telemetry, so I can understand using a few resources in the background to serve some of that data, but it must never get in the way of actual work getting done.

magicalist 5y ago

> Instead, they frantically went through their git log to see what in the code base might have triggered this bug. This seems like you're embellishing this part to tell a story? It's not supported by the linked post, and from the bugzilla bugs it seems like it was known almost immediately that the ESR builds were affected as well and so it almost had to be an external service, they just weren't sure which one at first.

TheIronMark 5y ago

Google should not be rolling out unannounced changes, but Mozilla increased their risk by not pinning the HTTP version.

Macha 5y ago

> I'd claim that no other company is criticized as harshly as Mozilla around here. The amount of blame that is assigned to the Firefox team is staggering. Mozilla has opened themselves up here, as they market as a privacy and user respecting alternative, so when they fail to live up to their own marketing people are more annoyed while they expect random startup #456 to not care about their users privacy and have telemetry out the wahoo.

dblohm7 5y ago
(I used to work for Mozilla, and spent a few years on the team that at the time owned the telemetry component.) The way Mozilla does telemetry is different from how most places do. I think that the biggest issue with these discussions is that there always seems to be this assumption that there is only one way to do telemetry, it always contains super invasive PII, Mozilla's telemetry must do the same, and therefore Mozilla's telemetry is just as evil as anybody else's. Mozilla is remarkably open about how its telemetry works, beyond just being open source. Maybe this is more a problem of that information not being surfaced well, I dunno. I get that some people are philosophically opposed to telemetry no matter what, but I have seen enough cases of, "Wow, I didn't know it worked that way, and I'm actually okay with this," to know that informed users are not universally opposed to it.
- wolverine876 5y ago
  Could you share a link?
  - dblohm7 5y ago
    I can share several! If you're running Firefox, you can go to `about:telemetry` and see what data is there. Note that some of that data might be populated even if you have telemetry turned off. Don't reach for your pitchfork quite yet: I assure you that the data isn't being sent. You can go to https://telemetry.mozilla.org https://telemetry.mozilla.org to see various analyses involving telemetry data. In the source tree, telemetry probes must be defined in one of a few places. Here are their locations: https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/Histograms.json https://searchfox.org/mozilla-central/source/toolkit/compone... https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/Scalars.yaml https://searchfox.org/mozilla-central/source/toolkit/compone... https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/Events.yaml https://searchfox.org/mozilla-central/source/toolkit/compone... https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/geckoview/streaming/metrics.yaml https://searchfox.org/mozilla-central/source/toolkit/compone...

yjftsjthsd-h 5y ago

> I'd claim that no other company is criticized as harshly as Mozilla around here. The amount of blame that is assigned to the Firefox team is staggering. They set themselves a higher standard by marketing as the good guys who fight for the user, and then made any number of moves that said users viewed as not being in their interests. Of course they get more blame. Like, Chrome has issues, but they're issues in line with being made by an adtech company; we might be unhappy at Google breaking adblockers (https://www.eff.org/deeplinks/2021/12/chrome-users-beware-manifest-v3-deceitful-and-threatening https://www.eff.org/deeplinks/2021/12/chrome-users-beware-ma...), but it's not out of character. Mozilla can say "More power to you. Mozilla puts people before profit, creating products, technologies and programs that make the internet healthier for everyone." (https://www.mozilla.org/en-US/ https://www.mozilla.org/en-US/) or they can, say, make Google the default engine ($), bake in a proprietary service (Pocket), rip out features (RIP compact theme), overrule user autonomy (Want to install an extension? Better upload it to Mozilla to get signed so they permit you to run it on your own computer!), ship a marketing extension through the "experiments" feature (https://blog.mozilla.org/en/products/firefox/update-looking-glass-add/ https://blog.mozilla.org/en/products/firefox/update-looking-...).... but not both. Either empower the user, or don't, but don't pretend to empower the user while ripping away their control.

ygjb 5y ago
Yep, you are correct. Each of those decisions was made over the protests of a vocal but relatively small group of users. You can't please all people all of the time, and I agree the pocket integration, and the looking glass add were mistakes, but the other items were directly related to sustainability of the project ($, eng cycles), or user safety. You can disagree with them as much as you like, but Firefox continues to support the ultimate in user control by releasing their product as open source. Roll your own build that doesn't require those features, sideload your add-ons, and/or fork the product. As a user, the average Firefox user has far more control over the browser than Chrome, Edge, or Safari users do, and have the flexibility to use one of many Firefox forks that have the same beef as you.
- ttybird2 5y ago

Santosh83 5y ago

What happened to the famed intelligence level of Hacker News? Every single person (almost) in this thread is blaming telemetry while it was clear even when the bug was ongoing that it was unrelated to telemetry. I for example had had telemetry disabled and still hit the bug through other traffic and had to temporarily disable HTTP3 from about:config.