3 ms·

Not many people are needed to test _specific_ problems with _specific_ environments/programs/features. 24 of 26 people not being able to figure out how to sign

Not many people are needed to test _specific_ problems with _specific_ environments/programs/features. 24 of 26 people not being able to figure out how to sign up for hacker news would be a clearly identified problem. Drawing overarching conclusions about the entire ipad ecosystem from 16 out of 50 million users on 26 apps out of ~100,000 and providing no methodology, no data? That's not enough to distill into "ipad usability" for an entire year. He'd have far more credibility if he simply said "I don't like this feature, and after asking several people, I know I'm not the only one. Here's why I don't like it, and here's what I think can be done about it."

ugh 15y ago
He sits them in front of computer and looks what they are doing. That would be a terrible way of finding specific problems. It’s a great way of searching for problems. I suggest you look into the method Nielsen actually uses.
- efsavage 15y ago
  I know how he does it, that's my point. Would you be able to make useful, high-level recommendations to the auto industry after riding in the back seat of 26 drivers for an hour or so? Do you think it matters what kind of car you're driving? What kind of roads you're driving on? What time of day it is? What state or country you're in? Of course it does. If your conclusion is that "driving on the PCH in a convertible is fun" then, no, you don't need many participants or any scientific rigor. But if you're say things like "the roads aren't wide enough" or "radios are too loud" or "signs aren't big enough" then you're going to need to back that up with some context and some data. My beef here is that people who don't want to really think about this stuff but want to seem like they do will read articles like his, which draw conclusions of the latter type described above, and spout them off as authoritative science, because that's how it's presented by Jakob Nielsen, Ph. D. This confuses clients and teams and ends up having people defend their decisions against a ghost of a misinterpretion of a supposed rule, and ultimately yields an inferior product. I've seen his articles quoted verbatim by designers and clients alike, almost always solely to justify their own opinion, but with the weight of his authority.
  - ugh 15y ago
    Would you be able to make useful, high-level recommendations to the auto industry after riding in the back seat of 26 drivers for an hour or so? I certainly do believe that to be possible, yes.
    - efsavage 15y ago
      Interesting, any ideas what those might be? I'm genuinely curious because I actually can't think of any.

ugh 15y ago

He sits them in front of computer and looks what they are doing. That would be a terrible way of finding specific problems. It’s a great way of searching for problems. I suggest you look into the method Nielsen actually uses.

efsavage 15y ago
I know how he does it, that's my point. Would you be able to make useful, high-level recommendations to the auto industry after riding in the back seat of 26 drivers for an hour or so? Do you think it matters what kind of car you're driving? What kind of roads you're driving on? What time of day it is? What state or country you're in? Of course it does. If your conclusion is that "driving on the PCH in a convertible is fun" then, no, you don't need many participants or any scientific rigor. But if you're say things like "the roads aren't wide enough" or "radios are too loud" or "signs aren't big enough" then you're going to need to back that up with some context and some data. My beef here is that people who don't want to really think about this stuff but want to seem like they do will read articles like his, which draw conclusions of the latter type described above, and spout them off as authoritative science, because that's how it's presented by Jakob Nielsen, Ph. D. This confuses clients and teams and ends up having people defend their decisions against a ghost of a misinterpretion of a supposed rule, and ultimately yields an inferior product. I've seen his articles quoted verbatim by designers and clients alike, almost always solely to justify their own opinion, but with the weight of his authority.
- ugh 15y ago
  Would you be able to make useful, high-level recommendations to the auto industry after riding in the back seat of 26 drivers for an hour or so? I certainly do believe that to be possible, yes.
  - efsavage 15y ago
    Interesting, any ideas what those might be? I'm genuinely curious because I actually can't think of any.