2 ms·

Good UX designers would also never conduct a user study with a sample size of 3.

Vinnl 7y ago
While 3 is a bit on the low end, user studies are not required to have large sample sizes. You're not trying to estimate what percentage of users are running into a particular issue, but only which issues many users run into. Whether it affects 10% or 30% of users does not really influence whether it should be fixed. More on this: https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ https://www.nngroup.com/articles/why-you-only-need-to-test-w...
- pembrook 7y ago
  And this is why the field of design gets shit on so much. Jakob Nielsen might have a PhD, but he’s much better at PR than he is at conducting actual research. It’s mind boggling to me that people will still quote his opinion pieces 20 years later as if there was any actual science behind them. There’s a few other fields that happen to conduct a lot of studies around human behavior. It turns out, there’s actually an established process for calculating the confidence level of your findings: https://www.calculator.net/sample-size-calculator.html https://www.calculator.net/sample-size-calculator.html Is 3 or 5 better than nothing? Sure! But don’t trick yourself into thinking your findings actually mean anything because a Danish dude drew a graph 20 years ago.
  - Vinnl 7y ago
    So again, a large margin of error and a low confidence level are fine, because it doesn't matter if in reality, 80% of your users run into something rather than the 60% that came out of user testing. In fact, we don't care about the specific numbers at all. All we want to know is what things are unclear to many users. Often, you test a design on five users, and all five of them miss the same thing - that gives you plenty of confidence that you'll want to try something else there. User testing is not a statistical test, and it's not a way to learn about human behaviour (so indeed, in that sense it doesn't "mean" anything); it's to give insight into your designs. I'd highly recommend everyone to take part in conducting user research sometime, as it will quickly become obvious just how easy it is to miss "obvious" things in something you're closely involved with, and how you need to have just few people interact with it to find out which.
nkrisc 7y ago
Anecdotal, but I've observed quite a few in-person usability studies. After the the third or fourth person you've usually discovered all your're going to with your current test script and stimuli. You probably could design a study that would continue to uncover new issues after many tens of people, but the test would probably be two hours long and would be unreliable based on that fact alone. The NN Group article linked by the sibling comment holds true in my experience.

While 3 is a bit on the low end, user studies are not required to have large sample sizes. You're not trying to estimate what percentage of users are running into a particular issue, but only which issues many users run into. Whether it affects 10% or 30% of users does not really influence whether it should be fixed. More on this: https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ https://www.nngroup.com/articles/why-you-only-need-to-test-w...

pembrook 7y ago
And this is why the field of design gets shit on so much. Jakob Nielsen might have a PhD, but he’s much better at PR than he is at conducting actual research. It’s mind boggling to me that people will still quote his opinion pieces 20 years later as if there was any actual science behind them. There’s a few other fields that happen to conduct a lot of studies around human behavior. It turns out, there’s actually an established process for calculating the confidence level of your findings: https://www.calculator.net/sample-size-calculator.html https://www.calculator.net/sample-size-calculator.html Is 3 or 5 better than nothing? Sure! But don’t trick yourself into thinking your findings actually mean anything because a Danish dude drew a graph 20 years ago.
- Vinnl 7y ago
  So again, a large margin of error and a low confidence level are fine, because it doesn't matter if in reality, 80% of your users run into something rather than the 60% that came out of user testing. In fact, we don't care about the specific numbers at all. All we want to know is what things are unclear to many users. Often, you test a design on five users, and all five of them miss the same thing - that gives you plenty of confidence that you'll want to try something else there. User testing is not a statistical test, and it's not a way to learn about human behaviour (so indeed, in that sense it doesn't "mean" anything); it's to give insight into your designs. I'd highly recommend everyone to take part in conducting user research sometime, as it will quickly become obvious just how easy it is to miss "obvious" things in something you're closely involved with, and how you need to have just few people interact with it to find out which.

nkrisc 7y ago

Anecdotal, but I've observed quite a few in-person usability studies. After the the third or fourth person you've usually discovered all your're going to with your current test script and stimuli. You probably could design a study that would continue to uncover new issues after many tens of people, but the test would probably be two hours long and would be unreliable based on that fact alone. The NN Group article linked by the sibling comment holds true in my experience.