4 ms·

He really needs a positive control, i.e. something he knows for sure improves his sleep in order to quantify the effect. Say his data shows a small effect, and

He really needs a positive control, i.e. something he knows for sure improves his sleep in order to quantify the effect. Say his data shows a small effect, and it is even statistically significant. He still won't have learned anything without having some reference for the magnitude of the change.

gwern 15y ago
> He still won't have learned anything without having some reference for the magnitude of the change. I don't think that's true. This is what the concept of 'effect size' (http://en.wikipedia.org/wiki/Effect_size http://en.wikipedia.org/wiki/Effect_size) was developed for, quantifying how big a divergence we're talking about. I didn't calculate it here because I literally just starting using R a few days ago and I don't know how. A quick and dirty way is to look at the difference in the 2 means and compare it to what is the 'normal range' (intuitively something like a standard deviation or two), which in this case, my ZQ ranges normally from the mid-80s to the very low 100s (a range of ~20 points), and the difference in ZQ means is 4 points. So that's 1/5 the normal variability - that's nothing to sneeze at! (By a strange coincidence, this is almost the same sort of numbers as for grades in class, so we could analogize: not taking vitamin D at night would improve my sleep grades by one letter. When I was in school, I would not have turned my nose up at such an intervention.) So, your comment is wrong as stated, I think. But as it happens, I do have something which could serve as your 'positive control': my previous (unblinded non-randomized) experiment with melatonin over half a year or so, which found a shift in mean ZQs of ~8 points. I was really impressed with melatonin when I first started (as have many people been), so to find vitamin D is half as hurtful as melatonin was helpful... I had no plans to take vitamin D in the evening before, but I sure as heck don't now!
- deleted 15y ago
  [deleted]

gwern 15y ago

> He still won't have learned anything without having some reference for the magnitude of the change. I don't think that's true. This is what the concept of 'effect size' (http://en.wikipedia.org/wiki/Effect_size http://en.wikipedia.org/wiki/Effect_size) was developed for, quantifying how big a divergence we're talking about. I didn't calculate it here because I literally just starting using R a few days ago and I don't know how. A quick and dirty way is to look at the difference in the 2 means and compare it to what is the 'normal range' (intuitively something like a standard deviation or two), which in this case, my ZQ ranges normally from the mid-80s to the very low 100s (a range of ~20 points), and the difference in ZQ means is 4 points. So that's 1/5 the normal variability - that's nothing to sneeze at! (By a strange coincidence, this is almost the same sort of numbers as for grades in class, so we could analogize: not taking vitamin D at night would improve my sleep grades by one letter. When I was in school, I would not have turned my nose up at such an intervention.) So, your comment is wrong as stated, I think. But as it happens, I do have something which could serve as your 'positive control': my previous (unblinded non-randomized) experiment with melatonin over half a year or so, which found a shift in mean ZQs of ~8 points. I was really impressed with melatonin when I first started (as have many people been), so to find vitamin D is half as hurtful as melatonin was helpful... I had no plans to take vitamin D in the evening before, but I sure as heck don't now!

deleted 15y ago
[deleted]