3 ms·

You can manage to have things without telemetry, but having telemetry is incredibly useful. I think the example of "how much of our user base actually uses thes

You can manage to have things without telemetry, but having telemetry is incredibly useful. I think the example of "how much of our user base actually uses these features" is a very good one, specially in a compiler where maintaining old features could be adding a lot of complexity to the code base. And, as they also explain, a lot of bugs and undesired behaviors are things that the users won't know they have to report and just accept as part of the normal behavior. Things like cache misses, slowed down compilation times in certain situations, sporadic crashes... All of those things could be improved if the developers knew about them.

estebank 3y ago
A common concern I see is people being worried that the reaction to having "how much of our user base actually uses these features" answered will be to remove the feature entirely, when in practice it is more of "we think implementation A of the feature is no longer in use and we have migrated everyone to implementation B, is that the case?" or "we have implementation A produce user visible effects, while we have implementation B run parallel to it to detect divergences, have any been detected in the wild in the past X months?". In Rust we've had large migrations like that (the "new" borrow checker comes to mind) and we had a really long periods of time where they were tested against the latest crate versions on crates.io until crater came back clean, and even then we had bug reports about regressions in the wild only after it was released on stable. For me personally there's one big blind spot when testing only against published code, no matter how big the corpus is: humans are excellent fuzzers and the malformed code they write and try to compile is hard to replicate. Having visibility into uncommon cases that are only visible on users machines would be incredibly useful. An example of this could be the "botched" 1.52.0 Rust release[1], where invalid incremental compilations were changed from silent to visible Internal Compiler Errors in nightly for several releases to the point where the team felt all of the outstanding incr comp bugs related to them had been addressed, but when turned on in stable immediately hit users in the real world, making it necessary to do an emergency dot-release reverting that change. If we had telemetry on stable compilers, instead of turning on the silent to ICE change we could have added a metric for the silent error and know ahead of time that the feature wasn't ready for prime time. With more telemetry we could have known what was causing this. Snooping over a user's shoulder while they hit the case could make identifying the bug trivial, but of course then that user would then turn around and ask me in unfriendly terms "who are you and how did you get in?" Arguments can be made against even implementing telemetry, and they can be compelling enough to elect against it, but shouting down the conversation from even happening is not helpful. [1]: https://blog.rust-lang.org/2021/05/10/Rust-1.52.1.html https://blog.rust-lang.org/2021/05/10/Rust-1.52.1.html
- JohnFen 3y ago
  > A common concern I see is people being worried that the reaction to having "how much of our user base actually uses these features" answered will be to remove the feature entirely That's probably because it's happened in real life quite a lot.
VancouverMan 3y ago
Among the telemetry-collecting applications and websites I use, or have used, I can't think of a single case where the software obviously improved due to this data being collected. In fact, it seems to be the opposite in some cases. Firefox and Windows, for example, have generally become significantly worse for me over time, despite the telemetry that they're collecting. In the "best" case, software like Visual Studio Code and Homebrew have merely remained mediocre. I've seen much better results from developers who base decisions on feedback and bug reports that have been manually submitted by users, rather than trying to make assumptions based on automatically-collected telemetry data.
- gjulianm 3y ago
  Well, precisely the case for telemetry is to make improvements that aren’t obvious to the users. If they were obvious people would see them and send bug reports. I have used telemetry for mobile apps in the past and now at work to monitor performance of certain software we deploy in client data centers, and there’s a lot of things I notice in the telemetry that users wouldn’t find and report. For example, I remember I was able to fix an issue in a mobile app because I noticed startup times increasing each time users opened the app, and I could fix the problematic cache quickly. I bet most users didn’t really notice. Same at my current job, we’re able to detect slowdowns and processing bottlenecks that for the users just show as subtly erroneous data. Do they notice those fixes? Nope, but that’s the point, I want to be able to fix things before they notice and report them.
chaxor 3y ago
Telemetry is incredibly useful for ensuring that people don't use the product. Many likely be doing a lot of switching away from Go intentionally, due to this.

estebank 3y ago

A common concern I see is people being worried that the reaction to having "how much of our user base actually uses these features" answered will be to remove the feature entirely, when in practice it is more of "we think implementation A of the feature is no longer in use and we have migrated everyone to implementation B, is that the case?" or "we have implementation A produce user visible effects, while we have implementation B run parallel to it to detect divergences, have any been detected in the wild in the past X months?". In Rust we've had large migrations like that (the "new" borrow checker comes to mind) and we had a really long periods of time where they were tested against the latest crate versions on crates.io until crater came back clean, and even then we had bug reports about regressions in the wild only after it was released on stable. For me personally there's one big blind spot when testing only against published code, no matter how big the corpus is: humans are excellent fuzzers and the malformed code they write and try to compile is hard to replicate. Having visibility into uncommon cases that are only visible on users machines would be incredibly useful. An example of this could be the "botched" 1.52.0 Rust release[1], where invalid incremental compilations were changed from silent to visible Internal Compiler Errors in nightly for several releases to the point where the team felt all of the outstanding incr comp bugs related to them had been addressed, but when turned on in stable immediately hit users in the real world, making it necessary to do an emergency dot-release reverting that change. If we had telemetry on stable compilers, instead of turning on the silent to ICE change we could have added a metric for the silent error and know ahead of time that the feature wasn't ready for prime time. With more telemetry we could have known what was causing this. Snooping over a user's shoulder while they hit the case could make identifying the bug trivial, but of course then that user would then turn around and ask me in unfriendly terms "who are you and how did you get in?" Arguments can be made against even implementing telemetry, and they can be compelling enough to elect against it, but shouting down the conversation from even happening is not helpful. [1]: https://blog.rust-lang.org/2021/05/10/Rust-1.52.1.html https://blog.rust-lang.org/2021/05/10/Rust-1.52.1.html

JohnFen 3y ago
> A common concern I see is people being worried that the reaction to having "how much of our user base actually uses these features" answered will be to remove the feature entirely That's probably because it's happened in real life quite a lot.

VancouverMan 3y ago

Among the telemetry-collecting applications and websites I use, or have used, I can't think of a single case where the software obviously improved due to this data being collected. In fact, it seems to be the opposite in some cases. Firefox and Windows, for example, have generally become significantly worse for me over time, despite the telemetry that they're collecting. In the "best" case, software like Visual Studio Code and Homebrew have merely remained mediocre. I've seen much better results from developers who base decisions on feedback and bug reports that have been manually submitted by users, rather than trying to make assumptions based on automatically-collected telemetry data.

gjulianm 3y ago
Well, precisely the case for telemetry is to make improvements that aren’t obvious to the users. If they were obvious people would see them and send bug reports. I have used telemetry for mobile apps in the past and now at work to monitor performance of certain software we deploy in client data centers, and there’s a lot of things I notice in the telemetry that users wouldn’t find and report. For example, I remember I was able to fix an issue in a mobile app because I noticed startup times increasing each time users opened the app, and I could fix the problematic cache quickly. I bet most users didn’t really notice. Same at my current job, we’re able to detect slowdowns and processing bottlenecks that for the users just show as subtly erroneous data. Do they notice those fixes? Nope, but that’s the point, I want to be able to fix things before they notice and report them.

chaxor 3y ago

Telemetry is incredibly useful for ensuring that people don't use the product. Many likely be doing a lot of switching away from Go intentionally, due to this.