4 ms·

Ah, this old chestnut. First of all, I love the fact that Nielsen thinks you can quantify "usability problems found" as being on a measurable scale. To think

Ah, this old chestnut. First of all, I love the fact that Nielsen thinks you can quantify "usability problems found" as being on a measurable scale. To think you can identify "100%" of usability problems with 15 users has always cracked me up. I completely agree with his suggestions of small groups of users and iterative design. However the problem with his approach is it's like a shotgun - he admits, you get several users to make sure you find the non-overlapping areas. What I found is that while it's good running sessions with groups of 'naive' users, a few targeted, repeated sessions with individual users that match personas being designed for is incredibly useful, and produces results that are surprisingly universal. Not only does the usability engineer discover new things about users each time someone comes in, each time the user comes in they find something new too. Repeated tests with individuals is a method that is rarely covered except in perhaps user-centered and participatory design approaches.

olavk 18y ago
Why can't you quantify usability problems found? Isn't it just a matter of counting? More users yields diminishing returns in number of problems found, and with 15 users you basically find all problems that you will find using any larger number of users. That sounds sensible to me. What am I missing?
- timcederman 18y ago
  Firstly, with the simplistic view that there are a finite number of problems - how do you know that you've reached 100%? Isn't the point of getting more people in because you don't know about all the problems? So how can you ever say "I've found 100%"? Next problem is how do you define a universal usability issue? What might cause problems for one person is a feature for the next. I suppose you could argue you've at least identified a concern, but even then you still need a very large n to say with significant certainty you've found most problems. But it's only ever certainty, unless it's the world's simplest interface.
- donal 18y ago
  The issue is with false negatives (not finding real problems). It is hard to say that 100% of issues were found, because how can you quantify the number of problems that weren't found? The claim of this article that you can somehow quantify the percentage of usability problems is rather absurd. How can you quantify the total number of usability issues in a program a priori? Also, I don't think that graph ever reaches 100%, I think it approaches 100% (but then I'm not a math-whiz, so I will gladly accept correction). They've successfully performed statistical sleight of hand. The assumption that you can quantify the total number of defects is untenable, but they leave that part out of their article and just show the nice charts and math to provide strength to their broken hypothesis. I think this is unfortunate because I think that there is probably some great insights in this article and the intent is good. Usability and Human-Centered Computing/Human-Computer Interaction has a tendency to suffer from this type of "pseudo-science" of using fuzzy statistics to present great sounding finding. I was made aware of this trend by the following article: Wayne D. Gray and Marilyn C. Salzman, "Damaged merchandise? A review of experiments that compare usability evaluation methods", Human-Computer Interaction, vol. 13, pp. 203-261, 1998.
  - olavk 18y ago
    Well the idea of a usability problem that is never found sounds pretty metaphysical to me. I mean - if no one actually experience the problem, then how is it a problem? Also, how can a problem experienced by a user be a false positive? Nielsen just claims that if you test with 15 users, you can be pretty sure you have found all problems you will ever find - you most likely won't find any new problems by using an additional 15 or 50 tests. (But obviously you cannot be 100% sure.) But whether his numbers are based on sound research I wont judge.
    - donal 18y ago
      The problem isn't with Nielsen's recommendations, it is his methods. He presents his findings as being based on scientific research, but unfortunately his methods are deceitful. Have you ever seen the infomercial for "Dual Action Cleanse" with Klee Irwin? Irwin and Nielsen use very similar methods. The make claims that pass the common sense test, then they use something resembling science to "prove" that they are spouting fact. They are both trying to sell you something, Nielsen has just found a much more profitable customer. I imagine selling super-laxatives to homebodies and the sleep deprived isn't anywhere as lucrative to selling consulting services to businessmen. The problem with Nielsen is his "research" is dangerous to the field of HCI and usability. Anybody that uses his "findings" to support their own research is building a house on a broken foundation. For his business clients this "academic navel gazing" doesn't matter, they probably see the results they were looking for (since he already told them exactly what to look for). Irwin's customers probably get the results they were after too, and whether or not the FDA monitors the claims made by "herbal supplements" probably doesn't bother them much either.

olavk 18y ago

Why can't you quantify usability problems found? Isn't it just a matter of counting? More users yields diminishing returns in number of problems found, and with 15 users you basically find all problems that you will find using any larger number of users. That sounds sensible to me. What am I missing?

timcederman 18y ago
Firstly, with the simplistic view that there are a finite number of problems - how do you know that you've reached 100%? Isn't the point of getting more people in because you don't know about all the problems? So how can you ever say "I've found 100%"? Next problem is how do you define a universal usability issue? What might cause problems for one person is a feature for the next. I suppose you could argue you've at least identified a concern, but even then you still need a very large n to say with significant certainty you've found most problems. But it's only ever certainty, unless it's the world's simplest interface.
donal 18y ago
The issue is with false negatives (not finding real problems). It is hard to say that 100% of issues were found, because how can you quantify the number of problems that weren't found? The claim of this article that you can somehow quantify the percentage of usability problems is rather absurd. How can you quantify the total number of usability issues in a program a priori? Also, I don't think that graph ever reaches 100%, I think it approaches 100% (but then I'm not a math-whiz, so I will gladly accept correction). They've successfully performed statistical sleight of hand. The assumption that you can quantify the total number of defects is untenable, but they leave that part out of their article and just show the nice charts and math to provide strength to their broken hypothesis. I think this is unfortunate because I think that there is probably some great insights in this article and the intent is good. Usability and Human-Centered Computing/Human-Computer Interaction has a tendency to suffer from this type of "pseudo-science" of using fuzzy statistics to present great sounding finding. I was made aware of this trend by the following article: Wayne D. Gray and Marilyn C. Salzman, "Damaged merchandise? A review of experiments that compare usability evaluation methods", Human-Computer Interaction, vol. 13, pp. 203-261, 1998.
- olavk 18y ago
  Well the idea of a usability problem that is never found sounds pretty metaphysical to me. I mean - if no one actually experience the problem, then how is it a problem? Also, how can a problem experienced by a user be a false positive? Nielsen just claims that if you test with 15 users, you can be pretty sure you have found all problems you will ever find - you most likely won't find any new problems by using an additional 15 or 50 tests. (But obviously you cannot be 100% sure.) But whether his numbers are based on sound research I wont judge.
  - donal 18y ago
    The problem isn't with Nielsen's recommendations, it is his methods. He presents his findings as being based on scientific research, but unfortunately his methods are deceitful. Have you ever seen the infomercial for "Dual Action Cleanse" with Klee Irwin? Irwin and Nielsen use very similar methods. The make claims that pass the common sense test, then they use something resembling science to "prove" that they are spouting fact. They are both trying to sell you something, Nielsen has just found a much more profitable customer. I imagine selling super-laxatives to homebodies and the sleep deprived isn't anywhere as lucrative to selling consulting services to businessmen. The problem with Nielsen is his "research" is dangerous to the field of HCI and usability. Anybody that uses his "findings" to support their own research is building a house on a broken foundation. For his business clients this "academic navel gazing" doesn't matter, they probably see the results they were looking for (since he already told them exactly what to look for). Irwin's customers probably get the results they were after too, and whether or not the FDA monitors the claims made by "herbal supplements" probably doesn't bother them much either.