3 ms·

> You need a good volume of data it is called testing. _Testing_. But of course, testing sucks and it's expensive.

by hulitu 1y ago

zbentley 1y ago
No, sorry. Testing answers “does the feature work?”. Usage telemetry answers questions like “was the feature a good idea?” and “are enough users successfully using the feature to justify the cost of creating/maintaining it?”. Those are not questions for which pre-release testing can provide answers. I’m not weighing in on opt-in vs opt-out, or on anonymization. Just saying that testing doesn’t cover this niche. (Separately, I think you’re largely wrong about testing as well: crash dump collection is about finding issues that pre-release testing wouldn’t find at any price. For things like OSes especially, the permutation space of hardware * software * user behavior is too large. While I’m sure a few companies use crash reporting as a crutch to support anemic QA programs, I do not think that many do.)
- hulitu 1y ago
  > Usage telemetry answers questions like “was the feature a good idea?” You mean you implement something even if nobody asked for it ? Wow. The project really has some spare budget.
  - alkonaut 1y ago
    That people ask for it doesn’t make it a good idea. Even if it’s a good idea and people asked for it that doesn’t mean people used it because they might not know about it. Is the feature prominent or intuitive enough? Should it be described in some newsletter or documentation? You will never know without actually asking enough users (which is a large sample). And there is a simple way of ”asking” this.
  - zbentley 1y ago
    https://en.wikipedia.org/wiki/XY_problem https://en.wikipedia.org/wiki/XY_problem https://xkcd.com/1172/ https://xkcd.com/1172/
alkonaut 1y ago
Testing? You can’t say how your users use your software through testing. Not by surveys/panels/interviews either. But yes: alternatives are also morr expensive (which means it’s expensive for the end user). Users pay one way or another.

> You need a good volume of data it is called testing. _Testing_. But of course, testing sucks and it's expensive.

by hulitu 1y ago

> You need a good volume of data it is called testing. _Testing_. But of course, testing sucks and it's expensive.

No, sorry. Testing answers “does the feature work?”. Usage telemetry answers questions like “was the feature a good idea?” and “are enough users successfully using the feature to justify the cost of creating/maintaining it?”. Those are not questions for which pre-release testing can provide answers. I’m not weighing in on opt-in vs opt-out, or on anonymization. Just saying that testing doesn’t cover this niche. (Separately, I think you’re largely wrong about testing as well: crash dump collection is about finding issues that pre-release testing wouldn’t find at any price. For things like OSes especially, the permutation space of hardware * software * user behavior is too large. While I’m sure a few companies use crash reporting as a crutch to support anemic QA programs, I do not think that many do.)

hulitu 1y ago
> Usage telemetry answers questions like “was the feature a good idea?” You mean you implement something even if nobody asked for it ? Wow. The project really has some spare budget.
- alkonaut 1y ago
  That people ask for it doesn’t make it a good idea. Even if it’s a good idea and people asked for it that doesn’t mean people used it because they might not know about it. Is the feature prominent or intuitive enough? Should it be described in some newsletter or documentation? You will never know without actually asking enough users (which is a large sample). And there is a simple way of ”asking” this.
- zbentley 1y ago
  https://en.wikipedia.org/wiki/XY_problem https://en.wikipedia.org/wiki/XY_problem https://xkcd.com/1172/ https://xkcd.com/1172/

alkonaut 1y ago

Testing? You can’t say how your users use your software through testing. Not by surveys/panels/interviews either. But yes: alternatives are also morr expensive (which means it’s expensive for the end user). Users pay one way or another.