4 ms·

Talking to users can be cheap and easy - easier today with the ease of running remote usability sessions for example. Telemetry is the lazy, unthinking approac

Talking to users can be cheap and easy - easier today with the ease of running remote usability sessions for example. Telemetry is the lazy, unthinking approach developers typically take. They think the telemetry data tells you what users are doing and how. But it doesn't really. It does not tell you how easy, difficult or frustrating the user finds using an app. It does not tell you whether the user accomplished their goal or gave up in frustration. If anyone still thinks running a usability study (to spot usability problems in an app) is too complicated, read this 20+ year old article from Jakob Nielsen (of NNGroup): Why You Only Need to Test with 5 Users: (published in 2000) https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ https://www.nngroup.com/articles/why-you-only-need-to-test-w...

lifthrasiir 5y ago
Tantacrul, the project lead for both MuseScore and Audacity, is a professional UI/UX designer and has once conducted a significant first-time user testing for Dorico [1], so I'm pretty sure that he already knows what you are saying. Given his latest video about Audacity [2] though, I guess that Audacity has a very diverse user base and it might be hard to do the user testing while accounting for that diversity (the article you've linked specifically mention this issue). [1] https://www.youtube.com/watch?v=S-3wEC6Fj_8 https://www.youtube.com/watch?v=S-3wEC6Fj_8 [2] https://www.youtube.com/watch?v=RMWNvwLiXIQ https://www.youtube.com/watch?v=RMWNvwLiXIQ (includes an interview with original Audacity developers)
- fartcannon 5y ago
  So he's already flubbed this once before. How many flubs until we can assume malice? Im getting a Miguel de Icaza vibe from him. Wolf in sheeps clothing.
  - fartcannon 5y ago
    It's even worse than I imagined. He's refusing to say who owns and how they came to own Audacity 'for fear of getting it wrong'... Something shady is going on for sure and this guy is their PR wall.
    - ergot_vacation 5y ago
      And he's made it clear repeatedly now that he's not actually in control of the project or empowered to make any significant decisions on it, despite what his youtube video implied. He appears to be just a glorified UI/UX contractor, albeit one with a huge youtube audience. Whoever IS in control clearly has no understanding of the community or history of Audacity, and no interest in gaining that understanding, given that he apparently had to beg them just to get them to step away from sabotaging the software within a week of buying it.
  - y2bd 5y ago
    >So he’s already flubbed this once before To my understanding he has never worked for Dorico in any capacity. He did the usability testing independently for the sake of his video essay, not in order to actually guide work.
hutzlibu 5y ago
"It does not tell you how easy, difficult or frustrating the user finds using an app. It does not tell you whether the user accomplished their goal or gave up in frustration." You actually can, it just takes a more sophisticated analysis. For example by tracking mousemovemt you can quite easily tell wheter the user finds what he wants or gets more and more stressed (faster and more undirected mouse movement) But I agree that you should only do so when explicitly testing your design with consenting (and paid) testers and not your regular users. They will indeed be happy, if they find a form where they can give you precise feedback. (depending of course on the software - you will get better feedback from CAD software users, than from the typical online game)
- bavell 5y ago
  > quite easily tell wheter the user finds what he wants or gets more and more stressed (faster and more undirected mouse movement) Could be just me but I don't make frantic mouse movements when I'm stressed or can't find something. Also sometimes I just let the mouse drift around as I scroll or read that probably looks like undirected movement. My point being that I think these metrics have a lot of inherent noise. With mountains of data you hopefully see the averages shake out but my gut tells me you shouldn't put too much weight on the analysis of these kinds of metrics and they should be supported by other sources of user feedback with better SNR.
542458 5y ago
User testing and analytics are completely different tools and their purposes should not be conflated. Analytics are for quantitative analysis of the entire UI. For example, if I want to see what tools are and are not being used, only observing 5 workflows would be massively insufficient. Sure, it doesn’t tell me how people feel - but that’s what user testing is for. Analytics can tell me if nobody is using the shiny new button that we put into the menu (at which point I’d do user testing to examine why this is happening), or that many users make heavy use of an item buried three levels deep in sub menus that we thought wasn’t important (but maybe turns out to be essential to some specialized workflow).
- aYsY4dDQ2NrcNzA 5y ago
  We were told to use the term “usability testing” since after all were weren’t studying the users.

lifthrasiir 5y ago

Tantacrul, the project lead for both MuseScore and Audacity, is a professional UI/UX designer and has once conducted a significant first-time user testing for Dorico [1], so I'm pretty sure that he already knows what you are saying. Given his latest video about Audacity [2] though, I guess that Audacity has a very diverse user base and it might be hard to do the user testing while accounting for that diversity (the article you've linked specifically mention this issue). [1] https://www.youtube.com/watch?v=S-3wEC6Fj_8 https://www.youtube.com/watch?v=S-3wEC6Fj_8 [2] https://www.youtube.com/watch?v=RMWNvwLiXIQ https://www.youtube.com/watch?v=RMWNvwLiXIQ (includes an interview with original Audacity developers)

fartcannon 5y ago
So he's already flubbed this once before. How many flubs until we can assume malice? Im getting a Miguel de Icaza vibe from him. Wolf in sheeps clothing.
- fartcannon 5y ago
  It's even worse than I imagined. He's refusing to say who owns and how they came to own Audacity 'for fear of getting it wrong'... Something shady is going on for sure and this guy is their PR wall.
  - ergot_vacation 5y ago
    And he's made it clear repeatedly now that he's not actually in control of the project or empowered to make any significant decisions on it, despite what his youtube video implied. He appears to be just a glorified UI/UX contractor, albeit one with a huge youtube audience. Whoever IS in control clearly has no understanding of the community or history of Audacity, and no interest in gaining that understanding, given that he apparently had to beg them just to get them to step away from sabotaging the software within a week of buying it.
- y2bd 5y ago
  >So he’s already flubbed this once before To my understanding he has never worked for Dorico in any capacity. He did the usability testing independently for the sake of his video essay, not in order to actually guide work.

hutzlibu 5y ago

"It does not tell you how easy, difficult or frustrating the user finds using an app. It does not tell you whether the user accomplished their goal or gave up in frustration." You actually can, it just takes a more sophisticated analysis. For example by tracking mousemovemt you can quite easily tell wheter the user finds what he wants or gets more and more stressed (faster and more undirected mouse movement) But I agree that you should only do so when explicitly testing your design with consenting (and paid) testers and not your regular users. They will indeed be happy, if they find a form where they can give you precise feedback. (depending of course on the software - you will get better feedback from CAD software users, than from the typical online game)

bavell 5y ago
> quite easily tell wheter the user finds what he wants or gets more and more stressed (faster and more undirected mouse movement) Could be just me but I don't make frantic mouse movements when I'm stressed or can't find something. Also sometimes I just let the mouse drift around as I scroll or read that probably looks like undirected movement. My point being that I think these metrics have a lot of inherent noise. With mountains of data you hopefully see the averages shake out but my gut tells me you shouldn't put too much weight on the analysis of these kinds of metrics and they should be supported by other sources of user feedback with better SNR.

542458 5y ago

User testing and analytics are completely different tools and their purposes should not be conflated. Analytics are for quantitative analysis of the entire UI. For example, if I want to see what tools are and are not being used, only observing 5 workflows would be massively insufficient. Sure, it doesn’t tell me how people feel - but that’s what user testing is for. Analytics can tell me if nobody is using the shiny new button that we put into the menu (at which point I’d do user testing to examine why this is happening), or that many users make heavy use of an item buried three levels deep in sub menus that we thought wasn’t important (but maybe turns out to be essential to some specialized workflow).

aYsY4dDQ2NrcNzA 5y ago
We were told to use the term “usability testing” since after all were weren’t studying the users.