6 ms·

That is simply not true. Both matter, but telemetry is something you can trust far more than user feedback. Want to know how much a functionality is used, or ho

That is simply not true. Both matter, but telemetry is something you can trust far more than user feedback. Want to know how much a functionality is used, or how long a user spends doing a particular action - telemetry is the answer. Compared to direct user feedback where a user might use some functionality twice a year, but rank it high as something that needs to be improved because it's sort of backwards. Instead of improving the performance and flow of the functionality they use hundreds of times daily, because they're reasonably happy with how it works. It's not that you don't want to fix the former, but it's obviously not business critical to perform often, and it's not considered high priority yet.

TeMPOraL 7y ago
> Both matter, but telemetry is something you can trust far more than user feedback. Disagree. User feedback, however biased or incoherent, is a direct representation of real people with real issues. Telemetry is a stream of context-free data from which you need to infer the hidden user feedback. Telemetry data seems to me to be much more prone to finding in it whatever it is that you want to find, or all the other failure modes that happen when people who analyze it have more skills in statistical tools than in doing science. > Compared to direct user feedback where a user might use some functionality twice a year, but rank it high as something that needs to be improved because it's sort of backwards. Instead of improving the performance and flow of the functionality they use hundreds of times daily, because they're reasonably happy with how it works. And all I want, all my life when using software, is for companies to fucking listen to that. If I say I'm reasonably happy with things I spend 95% of my time, then I am reasonably happy. I want you to fix the 5%-time case. There's no direct relationship between how often you use a feature and how critical it is. To give a simplified example, when I'm using 3D Studio Max, 99% of my time is spent modelling, and only 1% (or less) is spent on saving the file. And yet, without that feature, I wouldn't even buy the program. This is of course a silly example, but most software has plenty of very important features executed very rarely (and some that are executed rarely because the experience sucks). Relying only on naive telemetry analysis is going to lead you to misprioritizing work on improving the product.
- samoa42 7y ago
  > And all I want, all my life when using software, is for companies to fucking listen to that. If I say I'm reasonably happy with things I spend 95% of my time, then I am reasonably happy. I want you to fix the 5%-time case. soo much this!
  - pilif 7y ago
    Though you have to be careful not to fall into the https://xkcd.com/1172/ https://xkcd.com/1172/ trap. Sometimes a user might think they need feature A to make task B easier when in reality instead of doing B they actually want to do C but they never thought of it that way. If you implement A, you take on maintanance burden when C would be way easier to implement and potentially much more useful for the majority of your users.
    - TeMPOraL 7y ago
      The way to tell whether the user needs feature A or C is by talking with them, and watching them use your product in context of their overall work. Telemetry does neither, and will not reveal to you that the user needs C instead of A. On the contrary, it's likely to make you double down on A, because that's what shows up in the data.
- hvidgaard 7y ago
  > Relying only on naive telemetry analysis is going to lead you to misprioritizing work on improving the product. Which is exactly why I said both matters.
- SkyBelow 7y ago
  >If I say I'm reasonably happy with things I spend 95% of my time, then I am reasonably happy. Isn't there a decent amount of psychology research showing this isn't the case. Not with happiness with a service in particular, but that we poorly report on our internal state and often are biased by factors that impact our self reporting but not our choices. What people say and what people do do not always align and thus it is possible for there to be a gap between what people want and what people say they want. If you try to meet the latter you can end up losing out to someone who tries to meet the former. Now, that is research on average people (and particularly average college students), so when dealing with software developers talking about software issues it may not apply.
samoa42 7y ago
> Telemetry is a lazy, paternalistic way of improving user experience > That is simply not true... but telemetry is something you can trust far more than user feedback nutin said
beojan 7y ago
You've just stated exactly why you shouldn't trust telemetry. You improve the parts that are good enough already and not the parts that are so bad users avoid them.
- hvidgaard 7y ago
  I didn't want to write an essay, so let me give you an example instead. We've had a customers insisting that a particular feature was very important for them, however telemetry showed that they used that particular feature last time more than 26 months ago, and they discarded the resulting document anyway. Drilling down revealed that everyone thought that everybody else was using it sometimes, while the one user that had a need for it actually had a different and far more efficient workflow. My point is that my department have telemetry, the support team and PO have user feedback. If those two versions of the truth does not match up, we discuss it, often involving some users. Neither is perfect, and if you have to choose, obviously listening to the users seems preferable. Just remember that if Henry Ford build what people asked for, he would have made faster horses (quote is probably not even from him, but it gets the point across). To improve and make innovation you need many things, but especially empirical (telemetry), observational (watch the users), and anecdotal (listen to the users) evidence. Only with that you can make an informed decision.
NullPrefix 7y ago
Frequency of usage and importance may not necessarily correlate. You can drive far by only using gas pedal and take the foot off it to decelerate. Does it mean that brake pedal is not important, because usage telemetry lists it as "rarely used, user does not know what he wants" ?

TeMPOraL 7y ago

> Both matter, but telemetry is something you can trust far more than user feedback. Disagree. User feedback, however biased or incoherent, is a direct representation of real people with real issues. Telemetry is a stream of context-free data from which you need to infer the hidden user feedback. Telemetry data seems to me to be much more prone to finding in it whatever it is that you want to find, or all the other failure modes that happen when people who analyze it have more skills in statistical tools than in doing science. > Compared to direct user feedback where a user might use some functionality twice a year, but rank it high as something that needs to be improved because it's sort of backwards. Instead of improving the performance and flow of the functionality they use hundreds of times daily, because they're reasonably happy with how it works. And all I want, all my life when using software, is for companies to fucking listen to that. If I say I'm reasonably happy with things I spend 95% of my time, then I am reasonably happy. I want you to fix the 5%-time case. There's no direct relationship between how often you use a feature and how critical it is. To give a simplified example, when I'm using 3D Studio Max, 99% of my time is spent modelling, and only 1% (or less) is spent on saving the file. And yet, without that feature, I wouldn't even buy the program. This is of course a silly example, but most software has plenty of very important features executed very rarely (and some that are executed rarely because the experience sucks). Relying only on naive telemetry analysis is going to lead you to misprioritizing work on improving the product.

samoa42 7y ago
> And all I want, all my life when using software, is for companies to fucking listen to that. If I say I'm reasonably happy with things I spend 95% of my time, then I am reasonably happy. I want you to fix the 5%-time case. soo much this!
- pilif 7y ago
  Though you have to be careful not to fall into the https://xkcd.com/1172/ https://xkcd.com/1172/ trap. Sometimes a user might think they need feature A to make task B easier when in reality instead of doing B they actually want to do C but they never thought of it that way. If you implement A, you take on maintanance burden when C would be way easier to implement and potentially much more useful for the majority of your users.
  - TeMPOraL 7y ago
    The way to tell whether the user needs feature A or C is by talking with them, and watching them use your product in context of their overall work. Telemetry does neither, and will not reveal to you that the user needs C instead of A. On the contrary, it's likely to make you double down on A, because that's what shows up in the data.
hvidgaard 7y ago
> Relying only on naive telemetry analysis is going to lead you to misprioritizing work on improving the product. Which is exactly why I said both matters.
SkyBelow 7y ago
>If I say I'm reasonably happy with things I spend 95% of my time, then I am reasonably happy. Isn't there a decent amount of psychology research showing this isn't the case. Not with happiness with a service in particular, but that we poorly report on our internal state and often are biased by factors that impact our self reporting but not our choices. What people say and what people do do not always align and thus it is possible for there to be a gap between what people want and what people say they want. If you try to meet the latter you can end up losing out to someone who tries to meet the former. Now, that is research on average people (and particularly average college students), so when dealing with software developers talking about software issues it may not apply.

samoa42 7y ago

> Telemetry is a lazy, paternalistic way of improving user experience > That is simply not true... but telemetry is something you can trust far more than user feedback nutin said

beojan 7y ago

You've just stated exactly why you shouldn't trust telemetry. You improve the parts that are good enough already and not the parts that are so bad users avoid them.

hvidgaard 7y ago
I didn't want to write an essay, so let me give you an example instead. We've had a customers insisting that a particular feature was very important for them, however telemetry showed that they used that particular feature last time more than 26 months ago, and they discarded the resulting document anyway. Drilling down revealed that everyone thought that everybody else was using it sometimes, while the one user that had a need for it actually had a different and far more efficient workflow. My point is that my department have telemetry, the support team and PO have user feedback. If those two versions of the truth does not match up, we discuss it, often involving some users. Neither is perfect, and if you have to choose, obviously listening to the users seems preferable. Just remember that if Henry Ford build what people asked for, he would have made faster horses (quote is probably not even from him, but it gets the point across). To improve and make innovation you need many things, but especially empirical (telemetry), observational (watch the users), and anecdotal (listen to the users) evidence. Only with that you can make an informed decision.

NullPrefix 7y ago

Frequency of usage and importance may not necessarily correlate. You can drive far by only using gas pedal and take the foot off it to decelerate. Does it mean that brake pedal is not important, because usage telemetry lists it as "rarely used, user does not know what he wants" ?