Methodology varies with the terrain. Studies of human genetics are limited to observation, as experimentation is unethical. Studies of history -- archeology,

Methodology varies with the terrain. Studies of human genetics are limited to observation, as experimentation is unethical. Studies of history -- archeology, evolution -- don't have a lot going on in the repeatability department. Studies on topics of opinion -- advertising effectiveness, political surveys -- necessarily contend with a host of sampling biases and confounding variables. Not everything is physics. And I don't think it's intellectually optimal to flag everything that can't be physics as simple nonsense. I'd rather acknowledge and talk about methodological weaknesses when we discuss how persuasive the experiment is. I mean, it's not like an experiment that does everything "right" is always perfectly persuasive either. Reality is complicated; sometimes you were asking the wrong question. Most things are neither ironclad nor nonsense, but fall somewhere in the middle. So, two things about subjective measurement. First, just because something's subjective doesn't mean it's totally unreliable. You can fool yourself about small things; large things are harder. Suppose you're trying to demonstrate the presence of a material by the color of a flame. If you're trying to distinguish between pale green and pale pale green, better break out the spectrometer. But calling whether it's yellow or blue? Qualia will suffice. Applied to this case, I think I would rely on someone to be able to answer the question, "Are you well-rested or not?" with a "yes", "no", or "sorta". On the other hand, a 99 vs a 98 percent measurement -- as done here -- is pretty darn iffy. The second thing about subjective measurements is that, when you're after subjective results, that's the right sort of measure. For example, if someone says, "When I spend time with friends, I feel like I do better on math tests the next day," you might rightly respond, "We should be measuring the scores, not your feelings. It's possible your friends are making you feel competent when you really aren't." But if someone says, "When I spend time with friends, I feel happier," does it even make sense to say, "It's possible your friends are making you feel happier when you really aren't?" The best you can do is try to define degrees of happiness and measure statistically. When the feeling is the object of the discussion, the subjective isn't an illusion. A pill which makes you feel competent when you aren't is producing an illusion, but a pill that makes you feel happy when you want to is a pill that works. (This likewise reduces the point of blinding. If we're trying to figure out whether a chemical makes you happy, it makes sense to run it against a placebo. But if we're trying to figure out whether an activity makes you happy, being aware that you did it is appropriate. Whether it's the physical motion or the awareness that makes you happy is immaterial to whether the activity works.) All of that is to say, if the guy is trying to prove he is well-rested in the sense that he performs better, he reacts more quickly, his judgement is more sound, all the various things we associate with being well-rested, he should be taking tests. But if he is trying to feel well-rested, he should be asking, "How well-rested do I feel?" Now, it's hard to say exactly which he's after -- feeling well-rested but possibly performing like crap, or feeling like crap but possibly performing well. I'd think he probably wants both things. I certainly would. So I think he's only measuring half of what he should. The result could certainly be made more complete by doing something like the coffee guy did (http://quantifiedself.com/2009/10/the-false-god-of-coffee/ http://quantifiedself.com/2009/10/the-false-god-of-coffee/). It is promising that he measured how long he slept before first waking up -- that's pretty objective -- but I can't tell exactly what he's saying he did. (I can't tell what he means by baseline(3) and random(3), so I can't really critique the results). It would be nice if he'd chosen the exercise randomly for the random portion of the experiment. By just picking one, there's a serious possibility he was picking based on how tired he already felt due to events of the day (it is exercise he's signing up for, after all.) I don't know that it reduces those results to nonsense, but it certainly does reduce them to possibly nonsense. I think the best you can get is, "Huh, that's interesting enough to investigate--do it again, and use a die this time." And then we have the obvious criticism: he only tried it on himself. Whether it works only on him for psychological reasons relating to his life history, or whether it only works on some people for genetic reasons, or what . . . remains to be seen. On the whole, I'd give it about a five out of ten for methodology. I do like that he tried the exercise over a period of months, kept notes, and looked for patterns. That's worth a lot of credit right there. I think it could be improved by stating more clearly what he's trying to prove, by incorporating objective measurements where possible and keeping subjective measurements as clear-cut as possible -- though that sort of thing is really more appropriate for the next iteration, anyway. There's investigating to find interesting patterns, and then there's setting out to demonstrate the truth of what you think you've found. An investigative experiment -- which this certainly was -- is inherently not going to be a demonstrative one. I wouldn't mind clearer labeling of the charts, and I definitely wouldn't mind access to the raw data. That would bolster the case a lot. And a repeat would be great -- someone else should try it and see if it works. Ideally several people. Overall, though, it's far from nonsense, and I certainly wouldn't put it in the "trying to get people to stand on one leg to make a point about how gullible we are when we see graphs" category. I'd mark it investigative/interesting.