3 ms·

A bit more regarding "sufficiently strong and consistent" effects: We all do little experiments all the time and generally reach valid conclusions based on them

by glenra 13y ago

A bit more regarding "sufficiently strong and consistent" effects: We all do little experiments all the time and generally reach valid conclusions based on them. Some examples. Imagine your friend says one of these things: "Spicy pepperoni pizza late at night gives me heartburn; I feel better when I stick to milder foods." "My stomach hurts when I eat cheese." "I feel great when I get around 8 hours of sleep, I get grumpy when I only have 4 hours and I feel groggy if I stay in bed for more than 10 hours." "If I drink caffeine after 7pm, I have trouble getting to sleep." "Gin makes me sick, but vodka is fine." Somehow, all of those seem like PERFECTLY REASONABLE statements, even though they are anecdotal and based on a sample population of one. Nobody says this is invalid information because it wasn't double-blind and there was no control group. Nobody ever says, to paraphrase the upstream comment I responded to: > I wish they wouldn't report such anecdota so strongly. Joe is unlikely to have conducted controlled double-blind studies on himself, so he can't know it's the {pizza, cheese, caffeine, sleep...} doing anything or not. Right? But if Joe says: "When I zap myself with tCDS it helps relieve my depression", he DOES get that sort of complaint! Why is that? Why is it okay to report that Joe said "I feel less depressed when I exercise" or "I feel less depressed when I get to bed earlier" but NOT okay to report that Joe said "I feel less depressed when I use the tCDS"? Why is he allowed to KNOW those first two things based just on personal knowledge of how he feels, but not that last one? There is a weird and (I think) inappropriate double standard at work here.

DanBC 13y ago
Many people used to have knee arthroscopy. Doctors felt it was useful. Patients thought it was useful. It had a credible mechanism of action. When we do double blind placebo controlled studies we find that knee arthroscopy for some conditions is no better than placebo. > Why is it okay to report that Joe said "I feel less depressed when I exercise" or "I feel less depressed when I get to bed earlier" but NOT okay to report that Joe said "I feel less depressed when I use the tCDS"? It isn't okay to tell people to do exercise to treat their depression unless you have some kind of evidence to back that up. The evidence is weak, and the effect is small, but more research needs to be done. (http://summaries.cochrane.org/CD004366/exercise-for-depression http://summaries.cochrane.org/CD004366/exercise-for-depressi...) It is not okay to tell people to go to bed earlier to treat their depression unless you have evidence to back that up. I can think of a bunch of pros and cons for going to be earlier. I wouldn't voice my uninformed opinion in a national newspaper, and I wouldn't want that national newspaper to trumpet my uninformed opinion as a facty truthy comment. > Why is he allowed to KNOW those first two things based just on personal knowledge of how he feels, but not that last one? My position is that he doesn't know any of those things, not that he can know two of them but not the last one. Especially things like your gin / vodka example: I'd want evidence. Get a bunch of people in a room. Dip the rim of some drinking glasses in gin, and fill the glasses with cold tonic water. Observe. They're not drinking any alcohol, but they do have loosened inhibitions. So far I've just spoken about the need for a trial. Good quality trials are always needed. Cognitive biases are very strong, and very hard to protect against, and can cause people to undergo lengthy, risky, poor quality "treatment" that just doesn't work. (There are many good books about this. "Bad Science" is a good quick read.) Let's look at some other flags that might cause us to want better data, rather than just accept his word. He sells the device for $600. Right there is a big flag. Either he knows it doesn't work (I don't think this is the case) and he's a scumbag, cynically exploiting people. Or he really does think it works, and he wants to help people. Either way, better data is a good thing. Better data helps us stop the scumbags, and better data helps people like him sell the device and treat ill people.

by glenra 13y ago

DanBC 13y ago

Many people used to have knee arthroscopy. Doctors felt it was useful. Patients thought it was useful. It had a credible mechanism of action. When we do double blind placebo controlled studies we find that knee arthroscopy for some conditions is no better than placebo. > Why is it okay to report that Joe said "I feel less depressed when I exercise" or "I feel less depressed when I get to bed earlier" but NOT okay to report that Joe said "I feel less depressed when I use the tCDS"? It isn't okay to tell people to do exercise to treat their depression unless you have some kind of evidence to back that up. The evidence is weak, and the effect is small, but more research needs to be done. (http://summaries.cochrane.org/CD004366/exercise-for-depression http://summaries.cochrane.org/CD004366/exercise-for-depressi...) It is not okay to tell people to go to bed earlier to treat their depression unless you have evidence to back that up. I can think of a bunch of pros and cons for going to be earlier. I wouldn't voice my uninformed opinion in a national newspaper, and I wouldn't want that national newspaper to trumpet my uninformed opinion as a facty truthy comment. > Why is he allowed to KNOW those first two things based just on personal knowledge of how he feels, but not that last one? My position is that he doesn't know any of those things, not that he can know two of them but not the last one. Especially things like your gin / vodka example: I'd want evidence. Get a bunch of people in a room. Dip the rim of some drinking glasses in gin, and fill the glasses with cold tonic water. Observe. They're not drinking any alcohol, but they do have loosened inhibitions. So far I've just spoken about the need for a trial. Good quality trials are always needed. Cognitive biases are very strong, and very hard to protect against, and can cause people to undergo lengthy, risky, poor quality "treatment" that just doesn't work. (There are many good books about this. "Bad Science" is a good quick read.) Let's look at some other flags that might cause us to want better data, rather than just accept his word. He sells the device for $600. Right there is a big flag. Either he knows it doesn't work (I don't think this is the case) and he's a scumbag, cynically exploiting people. Or he really does think it works, and he wants to help people. Either way, better data is a good thing. Better data helps us stop the scumbags, and better data helps people like him sell the device and treat ill people.