Well, I'm not a statistician, but since nobody seems to be answering, I'll give it a shot. Choosing the statistical method is slightly tricky; we have binary d

Well, I'm not a statistician, but since nobody seems to be answering, I'll give it a shot. Choosing the statistical method is slightly tricky; we have binary data (which is a bit different to the methods most people are used to). Guidance from measuringusability.com [1] tells us that our scenario is very similar to their "example 2", essentially where we have two groups with simple totals (1/12 and 11/12). Their advice is to use a N-1 two proportion test [2] for this. Plugging in the values 1/12 and 11/12 here, we get a 99.994% probability that the distributions are different. This is because although the sample sizes are relatively small, 1/12 and 11/12 are very far apart. Slightly surprising, nevertheless. A more reasonable comparison would be 5/12 vs 8/12, and in this case the same test yields a 77.108% probability that they are different (i.e. not particularly significant). [1] http://www.measuringusability.com/blog/what-test.php http://www.measuringusability.com/blog/what-test.php [2] http://www.measuringusability.com/ab-calc.php http://www.measuringusability.com/ab-calc.php