7 ms·
I also find the methodology lacking. "This suggests the following design: a randomized dose on day 1, followed by days 2 and 3 off, then on day 4, drinking the
by Someone 13y ago
I also find the methodology lacking.
"This suggests the following design: a randomized dose on day 1, followed by days 2 and 3 off, then on day 4, drinking the second container; on day 7, examine the containers recording whether active/placebo and finally, starting over as day 1 of a new pair of 3-day blocks."
That means that, during the experiment, the subject can learn correlations between the test and the control ("So, I tend to go to the bathroom in the middle of the night in the test.")
If the subject has some (conscious or unconscious) prejudice for the result, he can use that information to adjust his self-reporting (again, consciously or unconsciously). For example, a subject may report a silly thought as a creative one instead of as a stupid one.
Also, given that the period of the experiment is exactly a week, I also would control for 'LSD is the first thing exactly as often as the test"
- javajosh 13y agoGiven the fact that he had a null result, I don't think this criticism could possibly be valid.
- Someone 13y agoMost importantly, the criticism of a _design_ is independent of the results of the experiment. Secondly, he may have wanted a null result.
- gwern 13y ago> Most importantly, the criticism of a _design_ is independent of the results of the experiment. Flaws can go in any direction, and make a design conservative or optimistic. In this case, the identified flaw does not undermine the actual result of the experiment - though it would have undermined the opposite (hypothetical) result of positive effects.
- Someone 13y agoIf you design an experiment in such a way that outcome A can be explained away by "yes, but the design is flawed", while outcome B cannot, I don't think it is fair to do the experiment.
- gwern 13y agoExperiments don't have to be 'fair'; your 'fairness' is not required for tests to be informative. Imagine an IQ test which spits out a binary yes/no result based on whether you're in the top 10% or not. This is a useful, meaningful test, which is 'unfair' in your naive sense. Issues of information content, discrimination, false positives and negatives - it's more complex than saying 'it is fair to do the experiment'. In this case, I got a reliable negative, rather than a less reliable positive.
- gwern 13y ago>For example, a subject may report a silly thought as a creative one instead of as a stupid one. You would have to assume that I was subconsciously also lying in the predictions, to hide my foreknowledge, and how would I be able to precisely affect the sleep or Mnemosyne scores? > Also, given that the period of the experiment is exactly a week, I also would control for 'LSD is the first thing exactly as often as the test" ? 3-day blocks imply a 6-day cycle, not 'exactly a week'. And I'm not sure what you mean by 'control' either.
- Someone 13y agoI must have interpreted the "finally" in "on day 7, examine the containers recording whether active/placebo and finally, starting over as day 1 of a new pair of 3-day blocks." And control: see http://thesocietypages.org/socimages/2013/01/12/what-does-it-mean-to-control-for-something/ http://thesocietypages.org/socimages/2013/01/12/what-does-it...
- gwern 13y ago> And control: see http://thesocietypages.org/socimages/2013/01/12/what-does-it.. http://thesocietypages.org/socimages/2013/01/12/what-does-it.... Yes, I'm aware of the statistical concept, and do controlling in a few other essays. My point was: 'control' is not an instantly obvious application. What am I controlling and how? Am I modeling it as a separate additive categorical variable? Quantitative variable? An interaction? (All different in R's formula notation, because they're different - +, *, :, and |) When you say > I also would control for 'LSD is the first thing exactly as often as the test" I don't even know what I would be controlling here, since I don't know what the order is supposed, what the 'test' is, and so on. You may think your suggestion is clear, but to me it's as clear as mud.
- Someone 13y agoIf your cycle is 7 days (which it isn't, but which I initially understood it to be), you will want to make sure that, with an experiment running for 2N weeks, you take the LSD-spiked water exactly N times on Monday and N times on Thursday, rather than relying on luck to get close to that. One way to do that is to make 2N packages containing a spiked and a non-spiked sample, with one sample marked 'take me first', and, in N cases, have the LSD-spiked sample be that sample. Of course, if you do that, it is even more important to not inform the subject about the history of he experiment before it is over.