True, but if you get enough people to do it repeatedly theoretically you could also model accuracy as a random variable. So then you get an overall population e

True, but if you get enough people to do it repeatedly theoretically you could also model accuracy as a random variable. So then you get an overall population estimate of accuracy as well as a model of the heterogeneity among individuals. Still, yes, with this small of a sample and minimal replication, there's issues. But they're not super serious about this scientifically, it's really for entertainment. Taking this a step further, I think to really test the simple discernment hypothesis, they should have the tasters sort the samples into two groups without labeling. The current approach sort of assumes that not only can they distinguish between the samples, but can also identify which is which. I think those are two different, separable issues. Where your concerns become more important for me personally are in ratings of foods etc.; I often wonder how much small cultures develop around these things and influence the ratings. It's interesting to me how much ratings can differ sometimes, and I suspect it's related to what you're mentioning.