6 ms·

+1 but several of your points are incorrect or need caveats. Your 1st point isn't correct, in that you will learn interesting things from 1m users that you won

+1 but several of your points are incorrect or need caveats. Your 1st point isn't correct, in that you will learn interesting things from 1m users that you won't from 15. The thing is that the 15 will tell you why, whereas the 1m won't unless you ask precisely the right question (which is a problem in itself). It basically takes experience here... (And this is something an AI may eventually become very good at.) Your 3rd and 4th point aren't correct, in that given some sampling error you may very well find that what 15 people don't like 1m will. I'm in full agreement with you in principle, in the sense that I firmly hold that far too many start-ups do quantitative without qualitative and end up with the wrong conclusions; but you can't just wave away quantitative data like that.

baxtr 7y ago
Re 1st point: OP was referring specifically to usability learnings
- nightcracker 7y ago
  I'd argue that usability also must include accessibility and I highly doubt that in your 15 user sample set you will cover the deaf, the (color) blind, the dyslexic, the paralyzed, the epileptic and various other groups of people with special accessibility requirements.
  - shkkmo 7y ago
    Usability should include accessibility when possible, but there is no must here. There are lots of tools that need usability testing that have a sufficiently small team and user-base to make accessibility a waste of time and resources. Indeed, on teams and products of any size there is always a decision to be made about which accessibility requirements will be met and to what degree. I would make the separate argument that what you need with accessibility requirements is separate testing expirements for each one. You don't need 100 testers to to make sure you have some special accessibility requirements, e.g. you need 15 normal vision testers and 15 impared vision testers.
    - deleted 7y ago
      [deleted]
  - cwyers 7y ago
    Or, for an American developer, what are the odds that your 15-user sample will find usability problems having to do with your icons only making sense to someone in the US? There's more to localization than just translation. Or if you have a product that does voice recognition, what range of accents are you covering with those 15 users?
    - adrianratnapala 7y ago
      That's OK, we nowadays use icons that don't make sense to anyone at all. I mean how would you know that three horizontal lines was a "Hamburger menu", and if you do know that, why would you want a Hamburger?
      - eddieh 7y ago
        Actually the icon looks like three menu items. It is a pretty good icon for signaling that it will open a menu. The hamburger nomenclature seems to be an ex pos facto name for something that rightfully should be called the menu icon.
        eitland 7y ago
        I can only speak for myself, but for me who grew up with C64, Windows 3.1, 95, 98, Linux, Windows XP more Linux etc that icon still didn't make sense until I got it explained. Then again, the magnifykng glass icon for search didn't make sense either until I read the docs but at least back then people made docs and kept the UI stable enough so that it made sense to learn it. I still remember fondly being good with OLE (Object Linking and Embedding) in a Microsoft Works.
        collyw 7y ago
        The old days seemed so much more intuitive when there was a crappy low res icon with text underneath.
        eitland 7y ago
visarga 7y ago
> but you can't just wave away quantitative data like that quantity has a quality of its own

baxtr 7y ago

Re 1st point: OP was referring specifically to usability learnings

nightcracker 7y ago
I'd argue that usability also must include accessibility and I highly doubt that in your 15 user sample set you will cover the deaf, the (color) blind, the dyslexic, the paralyzed, the epileptic and various other groups of people with special accessibility requirements.
- shkkmo 7y ago
  Usability should include accessibility when possible, but there is no must here. There are lots of tools that need usability testing that have a sufficiently small team and user-base to make accessibility a waste of time and resources. Indeed, on teams and products of any size there is always a decision to be made about which accessibility requirements will be met and to what degree. I would make the separate argument that what you need with accessibility requirements is separate testing expirements for each one. You don't need 100 testers to to make sure you have some special accessibility requirements, e.g. you need 15 normal vision testers and 15 impared vision testers.
  - deleted 7y ago
    [deleted]
- cwyers 7y ago
  Or, for an American developer, what are the odds that your 15-user sample will find usability problems having to do with your icons only making sense to someone in the US? There's more to localization than just translation. Or if you have a product that does voice recognition, what range of accents are you covering with those 15 users?
  - adrianratnapala 7y ago
    That's OK, we nowadays use icons that don't make sense to anyone at all. I mean how would you know that three horizontal lines was a "Hamburger menu", and if you do know that, why would you want a Hamburger?
    - eddieh 7y ago
      Actually the icon looks like three menu items. It is a pretty good icon for signaling that it will open a menu. The hamburger nomenclature seems to be an ex pos facto name for something that rightfully should be called the menu icon.
      - eitland 7y ago
        I can only speak for myself, but for me who grew up with C64, Windows 3.1, 95, 98, Linux, Windows XP more Linux etc that icon still didn't make sense until I got it explained. Then again, the magnifykng glass icon for search didn't make sense either until I read the docs but at least back then people made docs and kept the UI stable enough so that it made sense to learn it. I still remember fondly being good with OLE (Object Linking and Embedding) in a Microsoft Works.
        collyw 7y ago
        The old days seemed so much more intuitive when there was a crappy low res icon with text underneath.
        eitland 7y ago

visarga 7y ago

> but you can't just wave away quantitative data like that quantity has a quality of its own