4 ms·

How does that address the get-better-anyway effect, without a group to measure it?

nonbel 11y ago
Because you are predicting precisely how much better can be accounted for by the placebo effect, no more, no less (of course within error bounds). The get-better-anyway effect is too vague. Say theory A predicts value x is in the range [3.1,3.3] and you observe x=3.36 +/-.1. Also, there is a theory B that predicts x>0. Theory A has been much more severely tested so is supported by the evidence. Theory B, meh.
- ashearer 11y ago
  Say theory A (the placebo effect theory) predicts a 25% improvement, because it's been measured in the past with that result. Then there's also theory B (the get-better-anyway theory, or null hypothesis), which wasn't measured initially but in recent studies shows a similar 25% improvement. I don't understand how more studies testing theory A and getting 25%, while excluding theory B from testing, can show that A should be preferred over B. The rationale that A has been tested more thoroughly and therefore has a smaller confidence interval sounds circular.
  - nonbel 11y ago
    >"Say theory A (the placebo effect theory) predicts a 25% improvement, because it's been measured in the past with that result." That is not a theory. You have to come up with an explanation (a theory) for why it should be that value. Then from that theory you deduce predictions for what it should be under other circumstances. If the predictions are consistently close to the observations it indicates you are onto something.
    - ashearer 11y ago
      The discussion is about how to test the placebo effect (which corresponds to your example's "theory A"), so the proposed alternative to RCT would have to include the process of first deducing the effect size. For this problem, how would you realistically do that without referring to past experimental results? There are so many unknowns, including psychology. The 1955 paper was itself a survey of experimental results. I expect it to be much easier to come up with a statistical prediction for the get-better-anyway effect (regression to the mean), with more specificity than x>0. It would be needed regardless, in order to exclude the null hypothesis.
      - nonbel 11y ago
        My proposal includes collecting experimental data. It is just not in RCT form (which was claimed to be "the only way"). There is no reason to ignore RCT results, but that is not the only way to get the necessary data (which is what I disagreed with). An RCT is also not sufficient on its own, you need the theory that explains the results as well.
        nkurz 11y ago
        As an untrained outside observer (but one who has been thinking a lot recently about the statistics of causality) I'll chime in to say that I'm siding with 'ashearer' here. As I'm reading it, your approach doesn't make sense to me. If you want to distinguish between two treatments (placebo=treated and do-nothing=untreated) you need to measure outcomes for both treated and untreated individuals, and these individuals need to be "exchangeable" to some degree. Perhaps when you say in the first post "give people placebo", you mean "give the placebo to some subset of the individuals leaving the remainder untreated"? I agree that you can often draw useful conclusions from an observational study as long as some comparable individuals receive each treatment, and as long as you can control for the initial differences between the treated and untreated. But if you are saying that you don't need to look at the results for the untreated group at all, then I think you are mistaken.
        nonbel 11y ago
        >"Perhaps when you say in the first post "give people placebo", you mean "give the placebo to some subset of the individuals leaving the remainder untreated"?" I meant just placebo. I am using predictive skill of the theory behind the explanations to distinguish between them, this is common in physics/astronomy/etc. As noted by asherer, placebo effect has been studied for a long time in the manner of seeing how two groups differ. There is nothing wrong with RCTs, but does near sole reliance on them (and their observational analogue) appear to have lead to cumulative knowledge and growth of understanding?
        ashearer 11y ago

Because you are predicting precisely how much better can be accounted for by the placebo effect, no more, no less (of course within error bounds). The get-better-anyway effect is too vague. Say theory A predicts value x is in the range [3.1,3.3] and you observe x=3.36 +/-.1. Also, there is a theory B that predicts x>0. Theory A has been much more severely tested so is supported by the evidence. Theory B, meh.

ashearer 11y ago
Say theory A (the placebo effect theory) predicts a 25% improvement, because it's been measured in the past with that result. Then there's also theory B (the get-better-anyway theory, or null hypothesis), which wasn't measured initially but in recent studies shows a similar 25% improvement. I don't understand how more studies testing theory A and getting 25%, while excluding theory B from testing, can show that A should be preferred over B. The rationale that A has been tested more thoroughly and therefore has a smaller confidence interval sounds circular.
- nonbel 11y ago
  >"Say theory A (the placebo effect theory) predicts a 25% improvement, because it's been measured in the past with that result." That is not a theory. You have to come up with an explanation (a theory) for why it should be that value. Then from that theory you deduce predictions for what it should be under other circumstances. If the predictions are consistently close to the observations it indicates you are onto something.
  - ashearer 11y ago
    The discussion is about how to test the placebo effect (which corresponds to your example's "theory A"), so the proposed alternative to RCT would have to include the process of first deducing the effect size. For this problem, how would you realistically do that without referring to past experimental results? There are so many unknowns, including psychology. The 1955 paper was itself a survey of experimental results. I expect it to be much easier to come up with a statistical prediction for the get-better-anyway effect (regression to the mean), with more specificity than x>0. It would be needed regardless, in order to exclude the null hypothesis.
    - nonbel 11y ago
      My proposal includes collecting experimental data. It is just not in RCT form (which was claimed to be "the only way"). There is no reason to ignore RCT results, but that is not the only way to get the necessary data (which is what I disagreed with). An RCT is also not sufficient on its own, you need the theory that explains the results as well.
      - nkurz 11y ago
        As an untrained outside observer (but one who has been thinking a lot recently about the statistics of causality) I'll chime in to say that I'm siding with 'ashearer' here. As I'm reading it, your approach doesn't make sense to me. If you want to distinguish between two treatments (placebo=treated and do-nothing=untreated) you need to measure outcomes for both treated and untreated individuals, and these individuals need to be "exchangeable" to some degree. Perhaps when you say in the first post "give people placebo", you mean "give the placebo to some subset of the individuals leaving the remainder untreated"? I agree that you can often draw useful conclusions from an observational study as long as some comparable individuals receive each treatment, and as long as you can control for the initial differences between the treated and untreated. But if you are saying that you don't need to look at the results for the untreated group at all, then I think you are mistaken.
        nonbel 11y ago
        >"Perhaps when you say in the first post "give people placebo", you mean "give the placebo to some subset of the individuals leaving the remainder untreated"?" I meant just placebo. I am using predictive skill of the theory behind the explanations to distinguish between them, this is common in physics/astronomy/etc. As noted by asherer, placebo effect has been studied for a long time in the manner of seeing how two groups differ. There is nothing wrong with RCTs, but does near sole reliance on them (and their observational analogue) appear to have lead to cumulative knowledge and growth of understanding?
        ashearer 11y ago