3 ms·

Went into this with an indignant "failures in telemetry should NEVER bring down core functionality!" feeling, left with a more nuanced understanding of the faul

Went into this with an indignant "failures in telemetry should NEVER bring down core functionality!" feeling, left with a more nuanced understanding of the fault and impressed and reassured by the mitigation steps being taken. That's a great post-mortem.

nathansherburn 5y ago
I felt a similar way when it first happened. I was fuming "an update should never break your core functionality across all devices!" only to find that it wasn't actually the latest update but rather code that had been there for ages. It's hard to hate something once you truly understand it, I guess.
minerva23 5y ago
Anyone know why telemetry is sent from the main process rather than having a separate reporter process?
- rockdoe 5y ago
  It would have made no difference: the hang was in the networking code & process, not the telemetry code (and whatever process that is in). Anything that sent a message with the "bad" header type would have provoked the hang. As to why, dunno, presumably it's extra effort for an unclear gain. Telemetry code wouldn't be parsing hostile input etc. And it doesn't stop bugs like this either.
- marcan_42 5y ago
  That would be significant extra engineering effort without much clear benefit. It's always possible to look back and say "if only they'd done that", but this bug was a freak coincidence and it's not possible to foresee things like that, nor is it worth the engineering effort to try to avoid them with hammers as big as "move all telemetry to another process". You'd need a much stronger reason than "what if telemetry happens to trigger a longstanding bug in the network stack" to decide to go with that. In this case, that the problem was triggered by telemetry was a coincidence; it would've been triggered by other bits of coden in the future as they moved to Rust.
marcan_42 5y ago
Unfortunately, a good fraction of the commenters here don't seem to be doing the same. There's a whole pile of people throwing around "failures in telemetry should NEVER bring down core functionality!"...

nathansherburn 5y ago

I felt a similar way when it first happened. I was fuming "an update should never break your core functionality across all devices!" only to find that it wasn't actually the latest update but rather code that had been there for ages. It's hard to hate something once you truly understand it, I guess.

minerva23 5y ago

Anyone know why telemetry is sent from the main process rather than having a separate reporter process?

rockdoe 5y ago
It would have made no difference: the hang was in the networking code & process, not the telemetry code (and whatever process that is in). Anything that sent a message with the "bad" header type would have provoked the hang. As to why, dunno, presumably it's extra effort for an unclear gain. Telemetry code wouldn't be parsing hostile input etc. And it doesn't stop bugs like this either.
marcan_42 5y ago
That would be significant extra engineering effort without much clear benefit. It's always possible to look back and say "if only they'd done that", but this bug was a freak coincidence and it's not possible to foresee things like that, nor is it worth the engineering effort to try to avoid them with hammers as big as "move all telemetry to another process". You'd need a much stronger reason than "what if telemetry happens to trigger a longstanding bug in the network stack" to decide to go with that. In this case, that the problem was triggered by telemetry was a coincidence; it would've been triggered by other bits of coden in the future as they moved to Rust.

marcan_42 5y ago

Unfortunately, a good fraction of the commenters here don't seem to be doing the same. There's a whole pile of people throwing around "failures in telemetry should NEVER bring down core functionality!"...