6 ms·

I agree that Google is at fault here for failing Firefox. But Firefox is guilty of failing its users. Why should the functioning of a browser be dependent on

by moeris 5y ago

I agree that Google is at fault here for failing Firefox. But Firefox is guilty of failing its users. Why should the functioning of a browser be dependent on telemetry working? It sounds like if there is high enough latency in their telemetry, or if request for telemetry start failing, it's possible for that to disrupt using the network stack at all. They have a massive design flaw, and they didn't even mention that in the article. Maybe they have good reasons for designing a single point of failure that relies on a cloud provider, but it's not clear what those might be since they don't address it.

calcifer 5y ago
> Why should the functioning of a browser be dependent on telemetry working? It isn't. The bug was in the networking stack, and it just happened to be triggered by a GCP change which effected the telemetry service. Firefox having telemetry has nothing to do with the issue here.
- dralley 5y ago
  Thus proving OP's point. "The amount of blame that is assigned to the Firefox team is staggering"
- barrkel 5y ago
  That's not quite right. A single socket thread does all the requests and telemetry is multiplexed with user traffic. If telemetry is different in some way to other network traffic, then it's always possible for it to cause problems with user traffic. Telemetry is different to user traffic - it's less important! - but of course any in-process QoS would still create a point of interaction with user traffic.
- LeifCarrotson 5y ago
  How do you reach that conclusion? From the article: > It just so happens that Telemetry is currently the only Rust-based component in Firefox Desktop that uses the [viaduct/Necko] network stack and adds a Content-Length header. This is why users who disabled Telemetry would see this problem resolved ... The article contradicts your conclusion. If Firefox did not have telemetry, the bug would have had no impact, and users would not have suffered an outage. > ...even though the problem is not related to Telemetry functionality itself and could have been triggered otherwise. And then the article contradicts itself and agrees with you using some heavy-duty doublethink. Sure, if there were hypothetically other Rust services using the buggy network stack, they'd also have hit the bug: BUT THERE ARE NONE. The bug was in code which is only running because it's used by the telemetry services, so even though it might be in a different semantic layer it's the fault of the browser trying to send telemetry. As a user, I place very low (often negative) importance on the tools I use collecting telemetry data, or on protecting DRM content, or on checking licensing status. They should focus on doing the job I'm trying to do with them on my computerr, serving the uses of the user, rather than doing something that someone else wants them to do. Sure, I understand that debugging and quality monitoring are easier with logs and maybe with telemetry, so I can understand using a few resources in the background to serve some of that data, but it must never get in the way of actual work getting done.
  - acdha 5y ago
    > The article contradicts your conclusion. If Firefox did not have telemetry, the bug would have had no impact, and users would not have suffered an outage. This is your mistake: as explained in the article, it could have affected any component. Telemetry happened to hit it first but anything using HTTP/3 with that path would have been affected. “This is why users who disabled Telemetry would see this problem resolved even though the problem is not related to Telemetry functionality itself and could have been triggered otherwise.”
    - KronisLV 5y ago
      > ...as explained in the article, it could have affected any component. Telemetry happened to hit it first but anything using HTTP/3 with that path would have been affected. Is this really relevant, though? To the users who were unable to use their browsers normally it doesn't matter that this problem could have occurred elsewhere as well, but rather that it did occur here in particular. If particular sites would break, then that could be debugged separately, but as it stands even people who'd be perfectly fine with browsing regular HTTP/1.1 or HTTP/2 sites were also now impacted, not even due to opening a site that they wanted to visit themselves, but rather some background piece of functionality. That's not to say that i think there shouldn't be telemetry in place, just that the poster is correct in saying that this wouldn't be such a high visibility issue if there was no telemetry in place and thus no HTTP/3 apart from sites the user visits.
      - acdha 5y ago
        The comment I was replying to worded in a way which was trying to attribute blame to the telemetry service. As shown in this thread, there's a certain ideological position which welcomes any attacks on telemetry and I think that's a distraction from the technical discussion about how Mozilla could better have avoided a bug in their networking libraries. Recognizing this as a bug in the network stack first triggered by Telemetry makes it clear that this is not the place to have the millionth iteration of flamewars about that service but rather questions like the design of that network loop or not having test suite of the intersection of those particular libraries.
- moeris 5y ago
  > It isn't. So you're saying that Firefox did not on fact have an outage due to a change in their telemetry servers? That's not what the article said. I understand that you mean to say that it isn't intended for networking to be taken down by telemetry. That's nonetheless what happened, and it could have been prevented by treating telemetry as a different class of traffic (not collocating it with normal requests), or by not having it, as others point out.
  - acdha 5y ago
    They're saying what is clearly explained in the article: “This is why users who disabled Telemetry would see this problem resolved even though the problem is not related to Telemetry functionality itself and could have been triggered otherwise.”
    - KronisLV 5y ago
      Yes, but the fact that telemetry is in place was the cause for the issue. > So you're saying that Firefox did not on fact have an outage due to a change in their telemetry servers? Not the telemetry code. Not the fact that it "could" happen elsewhere. But rather the fact that it was in place and in this instance happened because of it. Not that it matters that much. Regardless of the particular cause, a browser failing to work because of something changing externally is crazy (at least to me), no matter how you look at it. Edit: this is now largely a duplicate of the other comment, hmm: https://news.ycombinator.com/item?id=30179023 https://news.ycombinator.com/item?id=30179023
  - marcan_42 5y ago
    So you're saying telemetry should be handled as a separate process that has nothing to do with the rest of the browser, and treated like a hostile service? Because that's the only way you'd have avoided this. It's natural for all the network stuff that goes in inside a browser to share code. You can say what you want about telemetry (I'm not a huge fan, personally), but this was a dumb bug and it is completely unreasonable to expect some kind of adversarial design "just in case a freak bug triggers on telemetry network requests".
moeris 5y ago
Firefox also said that this switch to default was an unannounced change. But a quick Google shows that it was announced > In the coming weeks, we’ll bring HTTP/3 to more users when it's enabled by default for all Cloud CDN and HTTPS Load Balancing customers: you won't need to lift a finger for your end users to start enjoying improved performance. In their blog on June 22, 2021. [1]. It probably should have been it's own standalone message sent to users (a "this should be a no-op" email), bit to claim that it was unannounced is misleading. 1. https://cloud.google.com/blog/products/networking/cloud-cdn-and-load-balancing-support-http3 https://cloud.google.com/blog/products/networking/cloud-cdn-...
- acdha 5y ago
  That’s half a year earlier and it’s described as an opt-in change until the very end, where it’s mentioned as a default changing in a few weeks. That’s far different from what, say, AWS does proactively sending email and SNS notifications with a time range and usually listing the affected instances.
  - moeris 5y ago
    Yeah, it doesn't sound like you're disagreeing with me.
- ghusbands 5y ago
  You expect everyone to read the google cloud blog? The distinction between "unannounced" and "not usefully announced" isn't of merit. If they did not specifically make their affected customers aware of the change and when it would actually happen, it was unannounced. And caused a major outage for at least one of their customers.
- tialaramex 5y ago
  This afternoon I tried to clone a git repo which, in the morning, was highlighted as containing a useful example to start from in the work I had targeted next. The clone failed with a mysterious error. After some minutes I checked the accompanying web site. The web site failed too, but, on refresh, this time I got a holding page explaining that the service was down. So I check the overall ticket system, and I find a change ticket, for the git system, saying there is planned maintenance, at 8am for one hour. Unadvertised because hey, it's 8am, most people aren't at work at 8am and this is a regular (Wednesday 8am) maintenance slot. And I scroll down and I find that nobody remembered to actually do the task. They wrote it up, submitted, got it OK'd and then, eh, never did it. By the time the people who were supposed to do it were reminded it was 9am already. So, astoundingly, the service owner OK'd just doing it after lunch instead. That failed 8am change was actually a re-run, of a re-run, of a re-run, of an upgrade that keeps failing and definitely takes over an hour to complete. So instead of "It's fine to do this when nobody is at work and it's low risk" suddenly "It's fine to do this for 2 hours in the middle of the working day, though it'll probably fail and we have no roll back plan". That's pretty shoddy. Glad to know an "Enterprise" cloud offering is hardly better.
matsemann 5y ago
> They have a massive design flaw, and they didn't even mention that in the article. From the article: > This is why users who disabled Telemetry would see this problem resolved even though the problem is not related to Telemetry functionality itself and could have been triggered otherwise. Don't spread FUD.
- barrkel 5y ago
  Telemetry traffic is multiplexed with user traffic on a single socket thread, per article. That creates a single point of failure where telemetry can affect user traffic. Of course all network access is shared for a machine so it's not possible to not have a single point of failure, but there are different ways of slicing up the access.
  - marcan_42 5y ago
    You're grasping at straws with this argument. That it shares a thread is a technicality. I'm sure the socket management is asynchronous and telemetry wouldn't normally affect normal traffic. This was an infinite loop bug. What if it had been a memory corruption bug instead, would you be saying that telemetry needs to be a separate process, not just a separate thread? The design was reasonable. Dumb bugs can happen anyway and cause things not to work as designed. That's what happened here.
- moeris 5y ago
  I agree, they should have addressed the issue in the article so that there would be less uncertainty.
phkahler 5y ago
>> Why should the functioning of a browser be dependent on telemetry working? That was my thought after reading the start of it. Like "Oh no, Firefox has fallen into that void where their need for telemetry trumps users". Another product falling down at doing its primary function. But after reading the entire report that's just not fair at all. A bug relating to telemetry and their network stack caused failure in that networking code which affected everything. That is entirely different than software depending on telemetry to function properly. It wasn't by design that failing to phone home broke the software, it really was just a bug - a fairly obscure one. Sounds like if someone wanted they could just as easily blame the use of Rust in Firefox since some of the code involved was written in Rust. But that's not a fair or accurate conclusion either.
- account42 5y ago
  Blame Telemetry and Rust are both fair here since they added complexity that resulted in this bug slipping past testing.
wintermutestwin 5y ago
Not an answer to your question, but why does a browser use telemetry at all?

by moeris 5y ago

calcifer 5y ago

> Why should the functioning of a browser be dependent on telemetry working? It isn't. The bug was in the networking stack, and it just happened to be triggered by a GCP change which effected the telemetry service. Firefox having telemetry has nothing to do with the issue here.

dralley 5y ago
Thus proving OP's point. "The amount of blame that is assigned to the Firefox team is staggering"
barrkel 5y ago
That's not quite right. A single socket thread does all the requests and telemetry is multiplexed with user traffic. If telemetry is different in some way to other network traffic, then it's always possible for it to cause problems with user traffic. Telemetry is different to user traffic - it's less important! - but of course any in-process QoS would still create a point of interaction with user traffic.
LeifCarrotson 5y ago
How do you reach that conclusion? From the article: > It just so happens that Telemetry is currently the only Rust-based component in Firefox Desktop that uses the [viaduct/Necko] network stack and adds a Content-Length header. This is why users who disabled Telemetry would see this problem resolved ... The article contradicts your conclusion. If Firefox did not have telemetry, the bug would have had no impact, and users would not have suffered an outage. > ...even though the problem is not related to Telemetry functionality itself and could have been triggered otherwise. And then the article contradicts itself and agrees with you using some heavy-duty doublethink. Sure, if there were hypothetically other Rust services using the buggy network stack, they'd also have hit the bug: BUT THERE ARE NONE. The bug was in code which is only running because it's used by the telemetry services, so even though it might be in a different semantic layer it's the fault of the browser trying to send telemetry. As a user, I place very low (often negative) importance on the tools I use collecting telemetry data, or on protecting DRM content, or on checking licensing status. They should focus on doing the job I'm trying to do with them on my computerr, serving the uses of the user, rather than doing something that someone else wants them to do. Sure, I understand that debugging and quality monitoring are easier with logs and maybe with telemetry, so I can understand using a few resources in the background to serve some of that data, but it must never get in the way of actual work getting done.
- acdha 5y ago
  > The article contradicts your conclusion. If Firefox did not have telemetry, the bug would have had no impact, and users would not have suffered an outage. This is your mistake: as explained in the article, it could have affected any component. Telemetry happened to hit it first but anything using HTTP/3 with that path would have been affected. “This is why users who disabled Telemetry would see this problem resolved even though the problem is not related to Telemetry functionality itself and could have been triggered otherwise.”
  - KronisLV 5y ago
    > ...as explained in the article, it could have affected any component. Telemetry happened to hit it first but anything using HTTP/3 with that path would have been affected. Is this really relevant, though? To the users who were unable to use their browsers normally it doesn't matter that this problem could have occurred elsewhere as well, but rather that it did occur here in particular. If particular sites would break, then that could be debugged separately, but as it stands even people who'd be perfectly fine with browsing regular HTTP/1.1 or HTTP/2 sites were also now impacted, not even due to opening a site that they wanted to visit themselves, but rather some background piece of functionality. That's not to say that i think there shouldn't be telemetry in place, just that the poster is correct in saying that this wouldn't be such a high visibility issue if there was no telemetry in place and thus no HTTP/3 apart from sites the user visits.
    - acdha 5y ago
      The comment I was replying to worded in a way which was trying to attribute blame to the telemetry service. As shown in this thread, there's a certain ideological position which welcomes any attacks on telemetry and I think that's a distraction from the technical discussion about how Mozilla could better have avoided a bug in their networking libraries. Recognizing this as a bug in the network stack first triggered by Telemetry makes it clear that this is not the place to have the millionth iteration of flamewars about that service but rather questions like the design of that network loop or not having test suite of the intersection of those particular libraries.
moeris 5y ago
> It isn't. So you're saying that Firefox did not on fact have an outage due to a change in their telemetry servers? That's not what the article said. I understand that you mean to say that it isn't intended for networking to be taken down by telemetry. That's nonetheless what happened, and it could have been prevented by treating telemetry as a different class of traffic (not collocating it with normal requests), or by not having it, as others point out.
- acdha 5y ago
  They're saying what is clearly explained in the article: “This is why users who disabled Telemetry would see this problem resolved even though the problem is not related to Telemetry functionality itself and could have been triggered otherwise.”
  - KronisLV 5y ago
    Yes, but the fact that telemetry is in place was the cause for the issue. > So you're saying that Firefox did not on fact have an outage due to a change in their telemetry servers? Not the telemetry code. Not the fact that it "could" happen elsewhere. But rather the fact that it was in place and in this instance happened because of it. Not that it matters that much. Regardless of the particular cause, a browser failing to work because of something changing externally is crazy (at least to me), no matter how you look at it. Edit: this is now largely a duplicate of the other comment, hmm: https://news.ycombinator.com/item?id=30179023 https://news.ycombinator.com/item?id=30179023
- marcan_42 5y ago
  So you're saying telemetry should be handled as a separate process that has nothing to do with the rest of the browser, and treated like a hostile service? Because that's the only way you'd have avoided this. It's natural for all the network stuff that goes in inside a browser to share code. You can say what you want about telemetry (I'm not a huge fan, personally), but this was a dumb bug and it is completely unreasonable to expect some kind of adversarial design "just in case a freak bug triggers on telemetry network requests".

moeris 5y ago

Firefox also said that this switch to default was an unannounced change. But a quick Google shows that it was announced > In the coming weeks, we’ll bring HTTP/3 to more users when it's enabled by default for all Cloud CDN and HTTPS Load Balancing customers: you won't need to lift a finger for your end users to start enjoying improved performance. In their blog on June 22, 2021. [1]. It probably should have been it's own standalone message sent to users (a "this should be a no-op" email), bit to claim that it was unannounced is misleading. 1. https://cloud.google.com/blog/products/networking/cloud-cdn-and-load-balancing-support-http3 https://cloud.google.com/blog/products/networking/cloud-cdn-...

acdha 5y ago
That’s half a year earlier and it’s described as an opt-in change until the very end, where it’s mentioned as a default changing in a few weeks. That’s far different from what, say, AWS does proactively sending email and SNS notifications with a time range and usually listing the affected instances.
- moeris 5y ago
  Yeah, it doesn't sound like you're disagreeing with me.
ghusbands 5y ago
You expect everyone to read the google cloud blog? The distinction between "unannounced" and "not usefully announced" isn't of merit. If they did not specifically make their affected customers aware of the change and when it would actually happen, it was unannounced. And caused a major outage for at least one of their customers.
tialaramex 5y ago
This afternoon I tried to clone a git repo which, in the morning, was highlighted as containing a useful example to start from in the work I had targeted next. The clone failed with a mysterious error. After some minutes I checked the accompanying web site. The web site failed too, but, on refresh, this time I got a holding page explaining that the service was down. So I check the overall ticket system, and I find a change ticket, for the git system, saying there is planned maintenance, at 8am for one hour. Unadvertised because hey, it's 8am, most people aren't at work at 8am and this is a regular (Wednesday 8am) maintenance slot. And I scroll down and I find that nobody remembered to actually do the task. They wrote it up, submitted, got it OK'd and then, eh, never did it. By the time the people who were supposed to do it were reminded it was 9am already. So, astoundingly, the service owner OK'd just doing it after lunch instead. That failed 8am change was actually a re-run, of a re-run, of a re-run, of an upgrade that keeps failing and definitely takes over an hour to complete. So instead of "It's fine to do this when nobody is at work and it's low risk" suddenly "It's fine to do this for 2 hours in the middle of the working day, though it'll probably fail and we have no roll back plan". That's pretty shoddy. Glad to know an "Enterprise" cloud offering is hardly better.

matsemann 5y ago

> They have a massive design flaw, and they didn't even mention that in the article. From the article: > This is why users who disabled Telemetry would see this problem resolved even though the problem is not related to Telemetry functionality itself and could have been triggered otherwise. Don't spread FUD.

barrkel 5y ago
Telemetry traffic is multiplexed with user traffic on a single socket thread, per article. That creates a single point of failure where telemetry can affect user traffic. Of course all network access is shared for a machine so it's not possible to not have a single point of failure, but there are different ways of slicing up the access.
- marcan_42 5y ago
  You're grasping at straws with this argument. That it shares a thread is a technicality. I'm sure the socket management is asynchronous and telemetry wouldn't normally affect normal traffic. This was an infinite loop bug. What if it had been a memory corruption bug instead, would you be saying that telemetry needs to be a separate process, not just a separate thread? The design was reasonable. Dumb bugs can happen anyway and cause things not to work as designed. That's what happened here.
moeris 5y ago
I agree, they should have addressed the issue in the article so that there would be less uncertainty.

phkahler 5y ago

>> Why should the functioning of a browser be dependent on telemetry working? That was my thought after reading the start of it. Like "Oh no, Firefox has fallen into that void where their need for telemetry trumps users". Another product falling down at doing its primary function. But after reading the entire report that's just not fair at all. A bug relating to telemetry and their network stack caused failure in that networking code which affected everything. That is entirely different than software depending on telemetry to function properly. It wasn't by design that failing to phone home broke the software, it really was just a bug - a fairly obscure one. Sounds like if someone wanted they could just as easily blame the use of Rust in Firefox since some of the code involved was written in Rust. But that's not a fair or accurate conclusion either.

account42 5y ago
Blame Telemetry and Rust are both fair here since they added complexity that resulted in this bug slipping past testing.

wintermutestwin 5y ago

Not an answer to your question, but why does a browser use telemetry at all?