I've only read the first two chapters of Jaynes' book, and bits and pieces of various other books. (One of the late chapters in http://uncertainty.stat.cmu.edu/

I've only read the first two chapters of Jaynes' book, and bits and pieces of various other books. (One of the late chapters in http://uncertainty.stat.cmu.edu/ http://uncertainty.stat.cmu.edu/ is a great reference for outlining the problems with significance and hypothesis testing.) Anyway, to me the problem described above sounds like it can generally be phrased as the common problem of ranking user-rated products on an e-commerce page. (A good Bayesian write-up is here, with simple code at the end: http://masanjin.net/blog/how-to-rank-products-based-on-user-input http://masanjin.net/blog/how-to-rank-products-based-on-user-...) At least if one assumes that treatments are applied more-or-less the same, and sets of objects amongst a treatment are more-or-less the same. There's also the question of how rankings change when both treatments are presented to a volunteer instead of just one. I interpreted the problem as just showing the OLD sets or just showing the NEW sets, never showing the original sets pre-treatment. How does sanskritabelt solve the problem, I wonder? Set an arbitrary number of volunteers for each group to say "this is the point we can stop collecting data", assume normalcy of ratings, take the means and use some test to see if the difference is significant or not? Really the problem of the problem is defining what is meant by 'better'. Is 'better' a higher score? (Which scoring function do you use -- mean, median, most 5-stars, least 1-stars, something else that takes into account the number of ratings per rating bucket?) Does it have to be higher by a certain amount (how meaningful are deltas in your scoring function)? Once it's defined what is meant by "A is better than B", then you can go about the business of computing the likelihood of your data given "A is better than B" to fulfill the RHS of Bayes' theorem.