3 ms·

You don't actually need a controlled double-blind study to notice a medical effect, IF the effect is sufficiently strong and consistent. Traditional "controlled

by glenra 13y ago

You don't actually need a controlled double-blind study to notice a medical effect, IF the effect is sufficiently strong and consistent. Traditional "controlled double-blind studies" are nice to have, but are much too likely to accidentally reject good medicines and medical treatments of the sort that might be discovered and confirmed via this sort of personal experimentation. (Yes, false positives are bad, but false negatives are bad too!) The era of personalized medicine is just beginning and these sort of devices seem unusually well-suited for that sort of approach.

glenra 13y ago
UPDATE: I'm getting downvoted, most likely from people who don't know what I'm referring to. So let me clarify: The platonic ideal of a traditional drug or medical treatment is something that works the same way on EVERYBODY. When you do a controlled double-blind study, you specify one specific treatment regimen - use dose X as a function of unit mass or of condition severity - and test whether that (a) has a statistically significant effect which (b) beats the control. Now imagine a drug which has this effect: half the time it makes people better (often quite a bit better). 30% of the time it has no effect, and 20% of the time it has a NEGATIVE effect - it makes people WORSE along the metric being measured. (one example of a treatment that has this sort of outcome pattern: exercise. If you measure blood pressure or cholesterol levels, it helps many but hurts some (about 12%) and has no effect on others(about 20%)) A double-blind study of THAT sort of intervention will find a low AVERAGE effect because the AVERAGE includes the nonresponders and the negative responders. In contrast, a "quantified self" personal science study where somebody carefully tracks their statistic and tries out an intervention while doing some A/B comparisons, is likely to discover whether they ARE or ARE NOT a positive responder. If they ARE, further experimentation can help find the best dose regimen for their particular situation. (of course, if we knew and could measure exactly which factors make some people positive responders and others negative responders, then we could test in the traditional way. But until we DO know that, studying at the individual level is often the best or even the only available way to figure out how to help individuals.) As for my "IF the effect is sufficiently strong and consistent" bit, I'll just note that big controlled studies are sometimes abandoned when an intervention is SO successful it is deemed unethical to deny the treatment to the control group.
- gwern 13y ago
  > A double-blind study of THAT sort of intervention will find a low AVERAGE effect because the AVERAGE includes the nonresponders and the negative responders. It'd also find a higher standard deviation in the experimental group, which, especially if the data had been graphically plotted, would trigger an f-test to check for increased variance. Finding the increase would then become the main result and one could start looking for predictive variables. (F-tests might not be statistics 101, perhaps, but it's 102.) > As for my "IF the effect is sufficiently strong and consistent" bit, I'll just note that big controlled studies are sometimes abandoned when an intervention is SO successful it is deemed unethical to deny the treatment to the control group. Usually after checking that the interim data is statistically-significant enough, assuming this wasn't built into the study design from the start as an adaptive experimental design...
  - glenra 13y ago
    A bit more regarding "sufficiently strong and consistent" effects: We all do little experiments all the time and generally reach valid conclusions based on them. Some examples. Imagine your friend says one of these things: "Spicy pepperoni pizza late at night gives me heartburn; I feel better when I stick to milder foods." "My stomach hurts when I eat cheese." "I feel great when I get around 8 hours of sleep, I get grumpy when I only have 4 hours and I feel groggy if I stay in bed for more than 10 hours." "If I drink caffeine after 7pm, I have trouble getting to sleep." "Gin makes me sick, but vodka is fine." Somehow, all of those seem like PERFECTLY REASONABLE statements, even though they are anecdotal and based on a sample population of one. Nobody says this is invalid information because it wasn't double-blind and there was no control group. Nobody ever says, to paraphrase the upstream comment I responded to: > I wish they wouldn't report such anecdota so strongly. Joe is unlikely to have conducted controlled double-blind studies on himself, so he can't know it's the {pizza, cheese, caffeine, sleep...} doing anything or not. Right? But if Joe says: "When I zap myself with tCDS it helps relieve my depression", he DOES get that sort of complaint! Why is that? Why is it okay to report that Joe said "I feel less depressed when I exercise" or "I feel less depressed when I get to bed earlier" but NOT okay to report that Joe said "I feel less depressed when I use the tCDS"? Why is he allowed to KNOW those first two things based just on personal knowledge of how he feels, but not that last one? There is a weird and (I think) inappropriate double standard at work here.

by glenra 13y ago

glenra 13y ago

UPDATE: I'm getting downvoted, most likely from people who don't know what I'm referring to. So let me clarify: The platonic ideal of a traditional drug or medical treatment is something that works the same way on EVERYBODY. When you do a controlled double-blind study, you specify one specific treatment regimen - use dose X as a function of unit mass or of condition severity - and test whether that (a) has a statistically significant effect which (b) beats the control. Now imagine a drug which has this effect: half the time it makes people better (often quite a bit better). 30% of the time it has no effect, and 20% of the time it has a NEGATIVE effect - it makes people WORSE along the metric being measured. (one example of a treatment that has this sort of outcome pattern: exercise. If you measure blood pressure or cholesterol levels, it helps many but hurts some (about 12%) and has no effect on others(about 20%)) A double-blind study of THAT sort of intervention will find a low AVERAGE effect because the AVERAGE includes the nonresponders and the negative responders. In contrast, a "quantified self" personal science study where somebody carefully tracks their statistic and tries out an intervention while doing some A/B comparisons, is likely to discover whether they ARE or ARE NOT a positive responder. If they ARE, further experimentation can help find the best dose regimen for their particular situation. (of course, if we knew and could measure exactly which factors make some people positive responders and others negative responders, then we could test in the traditional way. But until we DO know that, studying at the individual level is often the best or even the only available way to figure out how to help individuals.) As for my "IF the effect is sufficiently strong and consistent" bit, I'll just note that big controlled studies are sometimes abandoned when an intervention is SO successful it is deemed unethical to deny the treatment to the control group.

gwern 13y ago
> A double-blind study of THAT sort of intervention will find a low AVERAGE effect because the AVERAGE includes the nonresponders and the negative responders. It'd also find a higher standard deviation in the experimental group, which, especially if the data had been graphically plotted, would trigger an f-test to check for increased variance. Finding the increase would then become the main result and one could start looking for predictive variables. (F-tests might not be statistics 101, perhaps, but it's 102.) > As for my "IF the effect is sufficiently strong and consistent" bit, I'll just note that big controlled studies are sometimes abandoned when an intervention is SO successful it is deemed unethical to deny the treatment to the control group. Usually after checking that the interim data is statistically-significant enough, assuming this wasn't built into the study design from the start as an adaptive experimental design...
- glenra 13y ago
  A bit more regarding "sufficiently strong and consistent" effects: We all do little experiments all the time and generally reach valid conclusions based on them. Some examples. Imagine your friend says one of these things: "Spicy pepperoni pizza late at night gives me heartburn; I feel better when I stick to milder foods." "My stomach hurts when I eat cheese." "I feel great when I get around 8 hours of sleep, I get grumpy when I only have 4 hours and I feel groggy if I stay in bed for more than 10 hours." "If I drink caffeine after 7pm, I have trouble getting to sleep." "Gin makes me sick, but vodka is fine." Somehow, all of those seem like PERFECTLY REASONABLE statements, even though they are anecdotal and based on a sample population of one. Nobody says this is invalid information because it wasn't double-blind and there was no control group. Nobody ever says, to paraphrase the upstream comment I responded to: > I wish they wouldn't report such anecdota so strongly. Joe is unlikely to have conducted controlled double-blind studies on himself, so he can't know it's the {pizza, cheese, caffeine, sleep...} doing anything or not. Right? But if Joe says: "When I zap myself with tCDS it helps relieve my depression", he DOES get that sort of complaint! Why is that? Why is it okay to report that Joe said "I feel less depressed when I exercise" or "I feel less depressed when I get to bed earlier" but NOT okay to report that Joe said "I feel less depressed when I use the tCDS"? Why is he allowed to KNOW those first two things based just on personal knowledge of how he feels, but not that last one? There is a weird and (I think) inappropriate double standard at work here.