3 ms·

The 5 user Nielsen test referred to in this article is quite inaccurate. If ( and that's a very big if) the users are IID, then yes, 5 users is all you need. Bu

by dxbydt 7y ago

The 5 user Nielsen test referred to in this article is quite inaccurate. If ( and that's a very big if) the users are IID, then yes, 5 users is all you need. But your users from Russia aren't the users from USA aren't the users from China etc. Even if your userbase belongs to single country, there's differences between CA user vs TX user vs NY user etc. Further, the analysis isn't static in time! So you as a single user will be a different person tomorrow because you are more familiar with the software, or your mood is better, or your worktable/mind is less cluttered so you can pay more attention etc. In other words, world isn't multinomial coins with fixed head probabilities. Here the coins are people & the probabilities change over time. So the only sufficiency statistics are order stats. Hence logs. Nielsen adds a massive caveat "The formula only holds for comparable users who will be using the site in fairly similar ways" - its possible to find such people in very homogenous groups. Like if you have a GRE saas & target all only the white college kids taking the GRE, hopefully 5 white kids is enough. Now you bring in black & brown & hispanic & chinese & so forth...maybe 5 of each. Or maybe you want to separate by sexes, so 10 of each...it gets complicated very soon, which is why its much simpler to just log everything/everybody.

zer00eyz 7y ago
> The 5 user Nielsen test referred to in this article is quite inaccurate > "The formula only holds for comparable users who will be using the site in fairly similar ways" This is the article where Neilson breaks down what is being tested, and why it is statistically relevant. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ https://www.nngroup.com/articles/why-you-only-need-to-test-w... Neilson is looking to solve HCI(1) and Human Factors(2) issues - and most of these are byproducts of having to have a deep (insider) understanding of a product and that bubbling up into your UI. You are going to catch a lot of errors that fit the adage "can't see the forest for the trees". Having sat through a LOT Of these tests, you will pick out user frustrations, and reasons for product abandonment that would likely be NON apparent in a log. Your examples of US/China, TX vs CA and GRE with race and class MIGHT be relevant but it is going to depend a whole lot more on what your building. The problem is there are other means and places where these issues might manifest, and again use testing would tell you a lot. If we were to build a VR game that used a chopstick like interface, and test it only in china, we would likely think that we had a good product. If we find out later that "this isn't selling in America" then testing in that demographic group would quickly give us the insight that people lack the muscle memory to use this intuitively. There isn't any log in the known universe that would give us that clue, and "test here" can (and likely would) be gleaned by other means. When you get past HCI and Human factors log data can be useful, and be a contra-indicator of the results of formal testing. Given a choice between A and B in a formal setting may give you one set of results even with a large sample size, but real world behavior turns out to be very different. This is akin to people slowing down when they see a police car - but driving fast when one isn't present or kids acting differently because they know someone is watching. We aren't discussing UI and UI interactions were now discussing human behavior, and preference. I can't tell you how many times I have seen the non preferred solution be the winning one in an A/B test, but I would generally bet against what the group likes and pick the most garish solution as the winner. These behavioral types of tests can only really be driven by logs, by people being themselves and "feeling" unmonitored, and accurate demographic (to your point) slicing and sorting. En mass people are far more predictable than they would like to believe. Were delving into something more along the lines of Asimovs Psychohistory(3) as I don't think these sorts of statistically predicable behaviors have been given a formal name. 1. https://en.wikipedia.org/wiki/Human–computer_interaction https://en.wikipedia.org/wiki/Human–computer_interaction 2. https://en.wikipedia.org/wiki/Human_factors_and_ergonomics https://en.wikipedia.org/wiki/Human_factors_and_ergonomics 3. https://en.wikipedia.org/wiki/Psychohistory_(fictional) https://en.wikipedia.org/wiki/Psychohistory_(fictional)

by dxbydt 7y ago

zer00eyz 7y ago

> The 5 user Nielsen test referred to in this article is quite inaccurate > "The formula only holds for comparable users who will be using the site in fairly similar ways" This is the article where Neilson breaks down what is being tested, and why it is statistically relevant. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ https://www.nngroup.com/articles/why-you-only-need-to-test-w... Neilson is looking to solve HCI(1) and Human Factors(2) issues - and most of these are byproducts of having to have a deep (insider) understanding of a product and that bubbling up into your UI. You are going to catch a lot of errors that fit the adage "can't see the forest for the trees". Having sat through a LOT Of these tests, you will pick out user frustrations, and reasons for product abandonment that would likely be NON apparent in a log. Your examples of US/China, TX vs CA and GRE with race and class MIGHT be relevant but it is going to depend a whole lot more on what your building. The problem is there are other means and places where these issues might manifest, and again use testing would tell you a lot. If we were to build a VR game that used a chopstick like interface, and test it only in china, we would likely think that we had a good product. If we find out later that "this isn't selling in America" then testing in that demographic group would quickly give us the insight that people lack the muscle memory to use this intuitively. There isn't any log in the known universe that would give us that clue, and "test here" can (and likely would) be gleaned by other means. When you get past HCI and Human factors log data can be useful, and be a contra-indicator of the results of formal testing. Given a choice between A and B in a formal setting may give you one set of results even with a large sample size, but real world behavior turns out to be very different. This is akin to people slowing down when they see a police car - but driving fast when one isn't present or kids acting differently because they know someone is watching. We aren't discussing UI and UI interactions were now discussing human behavior, and preference. I can't tell you how many times I have seen the non preferred solution be the winning one in an A/B test, but I would generally bet against what the group likes and pick the most garish solution as the winner. These behavioral types of tests can only really be driven by logs, by people being themselves and "feeling" unmonitored, and accurate demographic (to your point) slicing and sorting. En mass people are far more predictable than they would like to believe. Were delving into something more along the lines of Asimovs Psychohistory(3) as I don't think these sorts of statistically predicable behaviors have been given a formal name. 1. https://en.wikipedia.org/wiki/Human–computer_interaction https://en.wikipedia.org/wiki/Human–computer_interaction 2. https://en.wikipedia.org/wiki/Human_factors_and_ergonomics https://en.wikipedia.org/wiki/Human_factors_and_ergonomics 3. https://en.wikipedia.org/wiki/Psychohistory_(fictional) https://en.wikipedia.org/wiki/Psychohistory_(fictional)