4 ms·

UPDATE: I'm getting downvoted, most likely from people who don't know what I'm referring to. So let me clarify: The platonic ideal of a traditional drug or medi

by glenra 13y ago

UPDATE: I'm getting downvoted, most likely from people who don't know what I'm referring to. So let me clarify: The platonic ideal of a traditional drug or medical treatment is something that works the same way on EVERYBODY. When you do a controlled double-blind study, you specify one specific treatment regimen - use dose X as a function of unit mass or of condition severity - and test whether that (a) has a statistically significant effect which (b) beats the control. Now imagine a drug which has this effect: half the time it makes people better (often quite a bit better). 30% of the time it has no effect, and 20% of the time it has a NEGATIVE effect - it makes people WORSE along the metric being measured. (one example of a treatment that has this sort of outcome pattern: exercise. If you measure blood pressure or cholesterol levels, it helps many but hurts some (about 12%) and has no effect on others(about 20%)) A double-blind study of THAT sort of intervention will find a low AVERAGE effect because the AVERAGE includes the nonresponders and the negative responders. In contrast, a "quantified self" personal science study where somebody carefully tracks their statistic and tries out an intervention while doing some A/B comparisons, is likely to discover whether they ARE or ARE NOT a positive responder. If they ARE, further experimentation can help find the best dose regimen for their particular situation. (of course, if we knew and could measure exactly which factors make some people positive responders and others negative responders, then we could test in the traditional way. But until we DO know that, studying at the individual level is often the best or even the only available way to figure out how to help individuals.) As for my "IF the effect is sufficiently strong and consistent" bit, I'll just note that big controlled studies are sometimes abandoned when an intervention is SO successful it is deemed unethical to deny the treatment to the control group.

gwern 13y ago
> A double-blind study of THAT sort of intervention will find a low AVERAGE effect because the AVERAGE includes the nonresponders and the negative responders. It'd also find a higher standard deviation in the experimental group, which, especially if the data had been graphically plotted, would trigger an f-test to check for increased variance. Finding the increase would then become the main result and one could start looking for predictive variables. (F-tests might not be statistics 101, perhaps, but it's 102.) > As for my "IF the effect is sufficiently strong and consistent" bit, I'll just note that big controlled studies are sometimes abandoned when an intervention is SO successful it is deemed unethical to deny the treatment to the control group. Usually after checking that the interim data is statistically-significant enough, assuming this wasn't built into the study design from the start as an adaptive experimental design...
- glenra 13y ago
  A bit more regarding "sufficiently strong and consistent" effects: We all do little experiments all the time and generally reach valid conclusions based on them. Some examples. Imagine your friend says one of these things: "Spicy pepperoni pizza late at night gives me heartburn; I feel better when I stick to milder foods." "My stomach hurts when I eat cheese." "I feel great when I get around 8 hours of sleep, I get grumpy when I only have 4 hours and I feel groggy if I stay in bed for more than 10 hours." "If I drink caffeine after 7pm, I have trouble getting to sleep." "Gin makes me sick, but vodka is fine." Somehow, all of those seem like PERFECTLY REASONABLE statements, even though they are anecdotal and based on a sample population of one. Nobody says this is invalid information because it wasn't double-blind and there was no control group. Nobody ever says, to paraphrase the upstream comment I responded to: > I wish they wouldn't report such anecdota so strongly. Joe is unlikely to have conducted controlled double-blind studies on himself, so he can't know it's the {pizza, cheese, caffeine, sleep...} doing anything or not. Right? But if Joe says: "When I zap myself with tCDS it helps relieve my depression", he DOES get that sort of complaint! Why is that? Why is it okay to report that Joe said "I feel less depressed when I exercise" or "I feel less depressed when I get to bed earlier" but NOT okay to report that Joe said "I feel less depressed when I use the tCDS"? Why is he allowed to KNOW those first two things based just on personal knowledge of how he feels, but not that last one? There is a weird and (I think) inappropriate double standard at work here.
  - DanBC 13y ago
    Many people used to have knee arthroscopy. Doctors felt it was useful. Patients thought it was useful. It had a credible mechanism of action. When we do double blind placebo controlled studies we find that knee arthroscopy for some conditions is no better than placebo. > Why is it okay to report that Joe said "I feel less depressed when I exercise" or "I feel less depressed when I get to bed earlier" but NOT okay to report that Joe said "I feel less depressed when I use the tCDS"? It isn't okay to tell people to do exercise to treat their depression unless you have some kind of evidence to back that up. The evidence is weak, and the effect is small, but more research needs to be done. (http://summaries.cochrane.org/CD004366/exercise-for-depression http://summaries.cochrane.org/CD004366/exercise-for-depressi...) It is not okay to tell people to go to bed earlier to treat their depression unless you have evidence to back that up. I can think of a bunch of pros and cons for going to be earlier. I wouldn't voice my uninformed opinion in a national newspaper, and I wouldn't want that national newspaper to trumpet my uninformed opinion as a facty truthy comment. > Why is he allowed to KNOW those first two things based just on personal knowledge of how he feels, but not that last one? My position is that he doesn't know any of those things, not that he can know two of them but not the last one. Especially things like your gin / vodka example: I'd want evidence. Get a bunch of people in a room. Dip the rim of some drinking glasses in gin, and fill the glasses with cold tonic water. Observe. They're not drinking any alcohol, but they do have loosened inhibitions. So far I've just spoken about the need for a trial. Good quality trials are always needed. Cognitive biases are very strong, and very hard to protect against, and can cause people to undergo lengthy, risky, poor quality "treatment" that just doesn't work. (There are many good books about this. "Bad Science" is a good quick read.) Let's look at some other flags that might cause us to want better data, rather than just accept his word. He sells the device for $600. Right there is a big flag. Either he knows it doesn't work (I don't think this is the case) and he's a scumbag, cynically exploiting people. Or he really does think it works, and he wants to help people. Either way, better data is a good thing. Better data helps us stop the scumbags, and better data helps people like him sell the device and treat ill people.

by glenra 13y ago

gwern 13y ago

> A double-blind study of THAT sort of intervention will find a low AVERAGE effect because the AVERAGE includes the nonresponders and the negative responders. It'd also find a higher standard deviation in the experimental group, which, especially if the data had been graphically plotted, would trigger an f-test to check for increased variance. Finding the increase would then become the main result and one could start looking for predictive variables. (F-tests might not be statistics 101, perhaps, but it's 102.) > As for my "IF the effect is sufficiently strong and consistent" bit, I'll just note that big controlled studies are sometimes abandoned when an intervention is SO successful it is deemed unethical to deny the treatment to the control group. Usually after checking that the interim data is statistically-significant enough, assuming this wasn't built into the study design from the start as an adaptive experimental design...

glenra 13y ago
A bit more regarding "sufficiently strong and consistent" effects: We all do little experiments all the time and generally reach valid conclusions based on them. Some examples. Imagine your friend says one of these things: "Spicy pepperoni pizza late at night gives me heartburn; I feel better when I stick to milder foods." "My stomach hurts when I eat cheese." "I feel great when I get around 8 hours of sleep, I get grumpy when I only have 4 hours and I feel groggy if I stay in bed for more than 10 hours." "If I drink caffeine after 7pm, I have trouble getting to sleep." "Gin makes me sick, but vodka is fine." Somehow, all of those seem like PERFECTLY REASONABLE statements, even though they are anecdotal and based on a sample population of one. Nobody says this is invalid information because it wasn't double-blind and there was no control group. Nobody ever says, to paraphrase the upstream comment I responded to: > I wish they wouldn't report such anecdota so strongly. Joe is unlikely to have conducted controlled double-blind studies on himself, so he can't know it's the {pizza, cheese, caffeine, sleep...} doing anything or not. Right? But if Joe says: "When I zap myself with tCDS it helps relieve my depression", he DOES get that sort of complaint! Why is that? Why is it okay to report that Joe said "I feel less depressed when I exercise" or "I feel less depressed when I get to bed earlier" but NOT okay to report that Joe said "I feel less depressed when I use the tCDS"? Why is he allowed to KNOW those first two things based just on personal knowledge of how he feels, but not that last one? There is a weird and (I think) inappropriate double standard at work here.
- DanBC 13y ago
  Many people used to have knee arthroscopy. Doctors felt it was useful. Patients thought it was useful. It had a credible mechanism of action. When we do double blind placebo controlled studies we find that knee arthroscopy for some conditions is no better than placebo. > Why is it okay to report that Joe said "I feel less depressed when I exercise" or "I feel less depressed when I get to bed earlier" but NOT okay to report that Joe said "I feel less depressed when I use the tCDS"? It isn't okay to tell people to do exercise to treat their depression unless you have some kind of evidence to back that up. The evidence is weak, and the effect is small, but more research needs to be done. (http://summaries.cochrane.org/CD004366/exercise-for-depression http://summaries.cochrane.org/CD004366/exercise-for-depressi...) It is not okay to tell people to go to bed earlier to treat their depression unless you have evidence to back that up. I can think of a bunch of pros and cons for going to be earlier. I wouldn't voice my uninformed opinion in a national newspaper, and I wouldn't want that national newspaper to trumpet my uninformed opinion as a facty truthy comment. > Why is he allowed to KNOW those first two things based just on personal knowledge of how he feels, but not that last one? My position is that he doesn't know any of those things, not that he can know two of them but not the last one. Especially things like your gin / vodka example: I'd want evidence. Get a bunch of people in a room. Dip the rim of some drinking glasses in gin, and fill the glasses with cold tonic water. Observe. They're not drinking any alcohol, but they do have loosened inhibitions. So far I've just spoken about the need for a trial. Good quality trials are always needed. Cognitive biases are very strong, and very hard to protect against, and can cause people to undergo lengthy, risky, poor quality "treatment" that just doesn't work. (There are many good books about this. "Bad Science" is a good quick read.) Let's look at some other flags that might cause us to want better data, rather than just accept his word. He sells the device for $600. Right there is a big flag. Either he knows it doesn't work (I don't think this is the case) and he's a scumbag, cynically exploiting people. Or he really does think it works, and he wants to help people. Either way, better data is a good thing. Better data helps us stop the scumbags, and better data helps people like him sell the device and treat ill people.