3 ms·
Given the fact that he had a null result, I don't think this criticism could possibly be valid.
by javajosh 13y ago
Given the fact that he had a null result, I don't think this criticism could possibly be valid.
- Someone 13y agoMost importantly, the criticism of a _design_ is independent of the results of the experiment. Secondly, he may have wanted a null result.
- gwern 13y ago> Most importantly, the criticism of a _design_ is independent of the results of the experiment. Flaws can go in any direction, and make a design conservative or optimistic. In this case, the identified flaw does not undermine the actual result of the experiment - though it would have undermined the opposite (hypothetical) result of positive effects.
- Someone 13y agoIf you design an experiment in such a way that outcome A can be explained away by "yes, but the design is flawed", while outcome B cannot, I don't think it is fair to do the experiment.
- gwern 13y agoExperiments don't have to be 'fair'; your 'fairness' is not required for tests to be informative. Imagine an IQ test which spits out a binary yes/no result based on whether you're in the top 10% or not. This is a useful, meaningful test, which is 'unfair' in your naive sense. Issues of information content, discrimination, false positives and negatives - it's more complex than saying 'it is fair to do the experiment'. In this case, I got a reliable negative, rather than a less reliable positive.