4 ms·

> The real answer to your question: ask and listen. People will gladly tell you. Do not just take the data otherwise you end up with a set of poorly selected me

by mbesto 6y ago

> The real answer to your question: ask and listen. People will gladly tell you. Do not just take the data otherwise you end up with a set of poorly selected metrics which do not represent user opinion and a lot of pissed off customers who don't want to or can't tell you due to legislation and data protection. And likewise simply listening to a vocal minority via "ask and listen" is not a silver bullet. So, you're both right, you need both to make informed decisions.

m0xte 6y ago
See the comment about user panels. Select a random portion of your userbase and ask them. Talk to your account managers. Communicate between them. This is software 101. I've built and supported software with 80k end users and did that effectively single-handedly. I'm sure a large corp can do the same if it sacrifices a bit of bottom line...
- mbesto 6y ago
  > Select a random portion of your userbase and ask them. This sounds great in theory - harder to do in practice. Often what ends up happening is the only people who will share their time with you are the ones who want something specifically changed for them. Thus my point, it's effective, but it's not a silver bullet. > I've built and supported software with 80k end users and did that effectively single-handedly. And plenty of businesses have used Google Analytics, Mixpanel, etc. combined with the aforementioned technique. TL;DR - The two strategies are not mutually exclusive.
  - wtetzner 6y ago
    > This sounds great in theory - harder to do in practice. Sure, but lots of things are hard. That doesn't mean we should all be happy about software phoning home without the consent of the user.
  - TeMPOraL 6y ago
    Problem is, the latter strategy is rarely used (telemetry is cheaper!), and as far as I can see the consequences, telemetry data is hard to use correctly. In particular, it's prone to become a mirror of your design, creating a feedback loop. Telemetry will show people will use more of the things you've exposed more, and less of the things you've exposed less, so if you take that at face value, then you're just amplifying your own initial design bias.
  - reaperducer 6y ago
    This sounds great in theory - harder to do in practice And therein lies the root of "telemetry" — the SV bubble's lack of interest, lack of effort, and fear of interfacing with the wetware. Telemetry is easy. Talking to people is hard. Too bad.
- kevingadd 6y ago
  Finding representative portions of your userbase that will actually talk to you is pretty difficult. I've worked on a couple different products with millions of end users and we frequently discovered a subset of our userbase was having big problems and we simply weren't hearing from them.
  - coribuci 6y ago
    > Finding representative portions of your userbase that will actually talk to you is pretty difficult. Finding developers that will listen to users is even more difficult. And Microsoft with its telemetry is one of the worse. They just collect user data and ignore every bug report. > I've worked on a couple different products with millions of end users and we frequently discovered a subset of our userbase was having big problems and we simply weren't hearing from them. How can people report bugs or express dissatisfaction with your program ?
jbay808 6y ago
What does listening to telemetry even look like? The user spent an hour fiddling with settings. Is that because they love the new settings toolbox? Or is it because they were very frustrated with it and couldn't find what they were looking for?
- guitarbill 6y ago
  This could also be done by reading their forums, and reaching out to people. For example, I have an XBox One controller. It used to pair fine via Bluetool. It still pairs fine with my Mac. Other stuff still pairs fine with my PC. But it just won't work after a Windows update. What is telemetry going to tell them that they don't already know from the forum? Maybe the scope, but it's fuzzy. Some users might give up after one or two tries. Some users might be using the "Add hardware" box several times in a row for different reasons. Telemetry isn't a magic insights thing. It's difficult to get right, and to draw the right conclusions. One thing's for sure, telemetry's cheaper than QA-ing updates properly.
  - m0xte 6y ago
    I think your last point probably nailed it.
  - kevingadd 6y ago
    Telemetry is generally a key component for quality assurance. While a paid tester can file bugs about "it crashes" or "it takes 10 seconds to load", a regression taking load times from 100ms to 200ms will be very hard (if not impossible) for a tester to notice and file a bug about. It will only show up in your telemetry. You could argue that telemetry should then only exist in your beta channel or testing builds, and some developers do that. It's silly to argue everything can be caught by your QA team, that is simply not true for online services. In the past projects I've worked on have had long-standing bugs that took weeks of ongoing effort between both our paid QA staff and customers to finally identify reproduction steps, at which point we were able to examine telemetry for those reproductions and fix the problem.
  - Arainach 6y ago

by mbesto 6y ago

m0xte 6y ago

See the comment about user panels. Select a random portion of your userbase and ask them. Talk to your account managers. Communicate between them. This is software 101. I've built and supported software with 80k end users and did that effectively single-handedly. I'm sure a large corp can do the same if it sacrifices a bit of bottom line...

mbesto 6y ago
> Select a random portion of your userbase and ask them. This sounds great in theory - harder to do in practice. Often what ends up happening is the only people who will share their time with you are the ones who want something specifically changed for them. Thus my point, it's effective, but it's not a silver bullet. > I've built and supported software with 80k end users and did that effectively single-handedly. And plenty of businesses have used Google Analytics, Mixpanel, etc. combined with the aforementioned technique. TL;DR - The two strategies are not mutually exclusive.
- wtetzner 6y ago
  > This sounds great in theory - harder to do in practice. Sure, but lots of things are hard. That doesn't mean we should all be happy about software phoning home without the consent of the user.
- TeMPOraL 6y ago
  Problem is, the latter strategy is rarely used (telemetry is cheaper!), and as far as I can see the consequences, telemetry data is hard to use correctly. In particular, it's prone to become a mirror of your design, creating a feedback loop. Telemetry will show people will use more of the things you've exposed more, and less of the things you've exposed less, so if you take that at face value, then you're just amplifying your own initial design bias.
- reaperducer 6y ago
  This sounds great in theory - harder to do in practice And therein lies the root of "telemetry" — the SV bubble's lack of interest, lack of effort, and fear of interfacing with the wetware. Telemetry is easy. Talking to people is hard. Too bad.
kevingadd 6y ago
Finding representative portions of your userbase that will actually talk to you is pretty difficult. I've worked on a couple different products with millions of end users and we frequently discovered a subset of our userbase was having big problems and we simply weren't hearing from them.
- coribuci 6y ago
  > Finding representative portions of your userbase that will actually talk to you is pretty difficult. Finding developers that will listen to users is even more difficult. And Microsoft with its telemetry is one of the worse. They just collect user data and ignore every bug report. > I've worked on a couple different products with millions of end users and we frequently discovered a subset of our userbase was having big problems and we simply weren't hearing from them. How can people report bugs or express dissatisfaction with your program ?

jbay808 6y ago

What does listening to telemetry even look like? The user spent an hour fiddling with settings. Is that because they love the new settings toolbox? Or is it because they were very frustrated with it and couldn't find what they were looking for?

guitarbill 6y ago
This could also be done by reading their forums, and reaching out to people. For example, I have an XBox One controller. It used to pair fine via Bluetool. It still pairs fine with my Mac. Other stuff still pairs fine with my PC. But it just won't work after a Windows update. What is telemetry going to tell them that they don't already know from the forum? Maybe the scope, but it's fuzzy. Some users might give up after one or two tries. Some users might be using the "Add hardware" box several times in a row for different reasons. Telemetry isn't a magic insights thing. It's difficult to get right, and to draw the right conclusions. One thing's for sure, telemetry's cheaper than QA-ing updates properly.
- m0xte 6y ago
  I think your last point probably nailed it.
- kevingadd 6y ago
  Telemetry is generally a key component for quality assurance. While a paid tester can file bugs about "it crashes" or "it takes 10 seconds to load", a regression taking load times from 100ms to 200ms will be very hard (if not impossible) for a tester to notice and file a bug about. It will only show up in your telemetry. You could argue that telemetry should then only exist in your beta channel or testing builds, and some developers do that. It's silly to argue everything can be caught by your QA team, that is simply not true for online services. In the past projects I've worked on have had long-standing bugs that took weeks of ongoing effort between both our paid QA staff and customers to finally identify reproduction steps, at which point we were able to examine telemetry for those reproductions and fix the problem.
- Arainach 6y ago