Maybe I was unclear. To be explicit, the experimental setup was: * Draw a sample of objects (in this case, an 'object' was a set of text documents) * Apply th

Maybe I was unclear. To be explicit, the experimental setup was: * Draw a sample of objects (in this case, an 'object' was a set of text documents) * Apply the OLD and NEW treatments to each sampled object * Show the results of OLD and NEW to volunteers (double blind, of course), get an evaluation. And so the data I get is: (object 1, SCORE OLD 1, SCORE NEW 1) (object 2, SCORE OLD 2, SCORE NEW 2) etc So again: * I don't have any prior knowledge of how the scores are going to be distributed * The primary reason I expected OLD to be 'better' than NEW is because OLD had theory behind it, and NEW was an ad hoc tweak of a previously rejected treatment * The very initial text corpus was fixed * The objects were selected from the initial corpus through a multistep process, at least one of those steps was a stochastic optimization of an objective function * One of the treatments was also obtained through a constrained optimization * The initial text corpus wasn't all that large, and the samples were pretty small So what I did was look at all observations for which OLD SCORE was different than NEW SCORE. Say there were n of them, and say that for p of those OLD<NEW. If NEW wasn't producing results that scored better than OLD, we would expect p/n to be less than 0.5, up to some uncertainty from the random sample. This is just a straight up test of a binomial parameter, so I computed confidence intervals and found that 0.5 was several standard deviations below the observed p/n, from which I concluded that NEW was better than OLD and we should switch. (as described here: http://en.wikipedia.org/wiki/Sign_test http://en.wikipedia.org/wiki/Sign_test) In a more responsible analysis I could have done something more sophisticated and modeled the scores as a sum of object effect, tester effect, treatment effect, fixed term, noise, etc. Given the magnitude of the sign test score, I didn't feel that was necessary.