4 ms·

While I agree with the SPOF point (heh), the header should really read “the recent Firefox Telemetry outage”. It's kind of like a website causing your browser

While I agree with the SPOF point (heh), the header should really read “the recent Firefox Telemetry outage”. It's kind of like a website causing your browser or a tab of the browser to be unresponsive by introducing an infinite busy loop. Except the “tab” was invisible.

detaro 5y ago
The user impact (Firefox not working) is more important than the technical reason, and thus should be in the title.
mozdeco 5y ago
The infinite busy loop in this case was not the tab no (neither visible or invisible). The loop was directly in the network stack, as stated in the post, not in the caller.
kragen 5y ago
The problem isn't that there was a telemetry outage; the problem is that the telemetry outage caused a Firefox outage, which should not be a thing. Firefox needs to be more robust than that.
- dralley 5y ago
  That's not what happened though. Telemetry uses the same networking stack as everything else, and the busy loop was in the networking stack.
  - barrkel 5y ago
    ... due to code in Telemetry being different and triggering different code paths in the network stack. There is a why here, and it includes Telemetry mixed traffic as a potential culprit. There are reasons to unify traffic (proxy support, QoS and whatnot) but unification of the user and Telemetry streams isn't without risk, as has been shown.
    - cxr 5y ago
      > unification of the user and Telemetry streams isn't without risk, as has been shown A constant refrain over the last 10 years or so of Mozilla's descent while trying to justify the removal of features from Firefox has been that not doing so unnecessarily bloats the surface area of the codebase, and specifically that this increases the chance of vulnerabilities and defects. Will the same argument be applied here, now with a case in hand, to justify the removal of telemetry, too?
    - marcan_42 5y ago
      It's the same app. I don't get why you're replying to every thread trying to somehow argue that sharing a thread for all network code is a bad thing and telemetry needs to be a special snowflake that gets a different thread. The networking code had an infinite loop bug. It was triggered by telemetry, but it could've been anything. Telemetry getting its own network thread wouldn't have magically made it impossible for it to cause problems. Bugs happen, and sometimes make things interact in weird ways.
  - kragen 5y ago
    First you say "that's not what happened" and then you explain how that is what happened.
  - cxr 5y ago
    > That's not what happened though What do you mean that's not what happened? Was there a "Firefox outage" or not? Are you disputing claims that the engineering team made about Firefox becoming unusable for users "for close to two hours"?
    - dralley 5y ago
      > about this telemetry bug That's my point. A "telemetry bug" didn't make Firefox unusable, a networking bug that was triggered by a telemetry bug did. But it could just as easily have been triggered by anything else.
      - cxr 5y ago
        First, you're going anachronistic. They didn't write "telemetry bug". Secondly, "cause" doesn't automatically mean "root cause". (That's the entire reason we distinguish between the two by qualifying the latter to begin with.) It's perfectly reasonable to say "A caused B" even if the root cause lies elsewhere, with C. Thirdly, none of this matters. It has no impact on the point being made by the person you responded to, which—to repeat—is that: > It should be impossible for the phrase "the recent Firefox outage" to make sense.
        dralley 5y ago
        >It should be impossible for the phrase "the recent Firefox outage" to make sense. It makes perfect sense a world where half the internet is going through Google / Cloudflare / Amazon /Akamai servers or some combination of the above, and they decide to roll out brand-spanking-new protocols to half of the internet at once. Sometimes that's going to break clients. I don't like that world very much, but it's the one we live in.
floatingatoll 5y ago
This is wrong on two points, based on the other discussion and the post itself. There was no Telemetry outage. An HTTP3 response header’s case was changed by a third party without notice. Telemetry continued working, other than the case change causing a bug. There was an http3 infinite loop bug in Firefox that crashed all networking. Many different things could have triggered the bug once it was introduced. Telemetry happened to be the first thing to do so, but not due to any faults in Telemetry’s code or implementation.

detaro 5y ago

The user impact (Firefox not working) is more important than the technical reason, and thus should be in the title.

mozdeco 5y ago

The infinite busy loop in this case was not the tab no (neither visible or invisible). The loop was directly in the network stack, as stated in the post, not in the caller.

kragen 5y ago

The problem isn't that there was a telemetry outage; the problem is that the telemetry outage caused a Firefox outage, which should not be a thing. Firefox needs to be more robust than that.

dralley 5y ago
That's not what happened though. Telemetry uses the same networking stack as everything else, and the busy loop was in the networking stack.
- barrkel 5y ago
  ... due to code in Telemetry being different and triggering different code paths in the network stack. There is a why here, and it includes Telemetry mixed traffic as a potential culprit. There are reasons to unify traffic (proxy support, QoS and whatnot) but unification of the user and Telemetry streams isn't without risk, as has been shown.
  - cxr 5y ago
    > unification of the user and Telemetry streams isn't without risk, as has been shown A constant refrain over the last 10 years or so of Mozilla's descent while trying to justify the removal of features from Firefox has been that not doing so unnecessarily bloats the surface area of the codebase, and specifically that this increases the chance of vulnerabilities and defects. Will the same argument be applied here, now with a case in hand, to justify the removal of telemetry, too?
  - marcan_42 5y ago
    It's the same app. I don't get why you're replying to every thread trying to somehow argue that sharing a thread for all network code is a bad thing and telemetry needs to be a special snowflake that gets a different thread. The networking code had an infinite loop bug. It was triggered by telemetry, but it could've been anything. Telemetry getting its own network thread wouldn't have magically made it impossible for it to cause problems. Bugs happen, and sometimes make things interact in weird ways.
- kragen 5y ago
  First you say "that's not what happened" and then you explain how that is what happened.
- cxr 5y ago
  > That's not what happened though What do you mean that's not what happened? Was there a "Firefox outage" or not? Are you disputing claims that the engineering team made about Firefox becoming unusable for users "for close to two hours"?
  - dralley 5y ago
    > about this telemetry bug That's my point. A "telemetry bug" didn't make Firefox unusable, a networking bug that was triggered by a telemetry bug did. But it could just as easily have been triggered by anything else.
    - cxr 5y ago
      First, you're going anachronistic. They didn't write "telemetry bug". Secondly, "cause" doesn't automatically mean "root cause". (That's the entire reason we distinguish between the two by qualifying the latter to begin with.) It's perfectly reasonable to say "A caused B" even if the root cause lies elsewhere, with C. Thirdly, none of this matters. It has no impact on the point being made by the person you responded to, which—to repeat—is that: > It should be impossible for the phrase "the recent Firefox outage" to make sense.
      - dralley 5y ago
        >It should be impossible for the phrase "the recent Firefox outage" to make sense. It makes perfect sense a world where half the internet is going through Google / Cloudflare / Amazon /Akamai servers or some combination of the above, and they decide to roll out brand-spanking-new protocols to half of the internet at once. Sometimes that's going to break clients. I don't like that world very much, but it's the one we live in.

floatingatoll 5y ago

This is wrong on two points, based on the other discussion and the post itself. There was no Telemetry outage. An HTTP3 response header’s case was changed by a third party without notice. Telemetry continued working, other than the case change causing a bug. There was an http3 infinite loop bug in Firefox that crashed all networking. Many different things could have triggered the bug once it was introduced. Telemetry happened to be the first thing to do so, but not due to any faults in Telemetry’s code or implementation.