3 ms·

If Telemetry is valuable, it's largely valuable to the degree it's representative. Opt-in sacrifices this and even a well optimized opt-in process won't prevent

by sf_rob 3y ago

If Telemetry is valuable, it's largely valuable to the degree it's representative. Opt-in sacrifices this and even a well optimized opt-in process won't prevent that. This is not to say that Telemetry should be opt-out, just that you place other justifications ahead of the usefulness of Telemetry (which is fine).

quickthrower2 3y ago
These days I reckon running an LLM over the storybook discord and reddits would yield more valuable data. How big peoples projects are on average doesn’t matter. 100 people asking the same question does.
- asoneth 3y ago
  I completely agree that qualitative feedback is the most valuable kind of product input. However, quantitative metrics like telemetry are used to answer different kinds of questions. > How big peoples projects are on average doesn’t matter. To give concrete examples, at my current company we do not collect telemetry and have regularly encountered problems where some assumption about customer usage (the number of items in the largest project, the size of the largest document, the smallest browser window, the oldest browser version, the usage of a deprecated feature) turns out to be incorrect, sometimes wildly so, and we only find out after a customer finally complains that they've been struggling with something for months. If we had telemetry we could have known about these issues well ahead of time and in many cases avoided the problem in the first place. We still choose not to have telemetry for many of the reasons presented in other comments, but we should at least be honest that it can provide immense value to the product team.
denton-scratch 3y ago
> Opt-in sacrifices this You're right. Opt-in results in a survey population that is self-selected, and therefore largely meaningless. But compulsory participation in surveys is deeply objectionable; just because it's easy doesn't make it OK. So you can't have representative telemetry that is also optional (for the user). So firms should stop trying to use telemetry, unless they're OK with telemetry data from a narrow pool of users, e.g. users of Beta versions of software.

by sf_rob 3y ago

quickthrower2 3y ago

These days I reckon running an LLM over the storybook discord and reddits would yield more valuable data. How big peoples projects are on average doesn’t matter. 100 people asking the same question does.

asoneth 3y ago
I completely agree that qualitative feedback is the most valuable kind of product input. However, quantitative metrics like telemetry are used to answer different kinds of questions. > How big peoples projects are on average doesn’t matter. To give concrete examples, at my current company we do not collect telemetry and have regularly encountered problems where some assumption about customer usage (the number of items in the largest project, the size of the largest document, the smallest browser window, the oldest browser version, the usage of a deprecated feature) turns out to be incorrect, sometimes wildly so, and we only find out after a customer finally complains that they've been struggling with something for months. If we had telemetry we could have known about these issues well ahead of time and in many cases avoided the problem in the first place. We still choose not to have telemetry for many of the reasons presented in other comments, but we should at least be honest that it can provide immense value to the product team.

denton-scratch 3y ago

> Opt-in sacrifices this You're right. Opt-in results in a survey population that is self-selected, and therefore largely meaningless. But compulsory participation in surveys is deeply objectionable; just because it's easy doesn't make it OK. So you can't have representative telemetry that is also optional (for the user). So firms should stop trying to use telemetry, unless they're OK with telemetry data from a narrow pool of users, e.g. users of Beta versions of software.