3 ms·

The discussion is about how to test the placebo effect (which corresponds to your example's "theory A"), so the proposed alternative to RCT would have to includ

The discussion is about how to test the placebo effect (which corresponds to your example's "theory A"), so the proposed alternative to RCT would have to include the process of first deducing the effect size. For this problem, how would you realistically do that without referring to past experimental results? There are so many unknowns, including psychology. The 1955 paper was itself a survey of experimental results. I expect it to be much easier to come up with a statistical prediction for the get-better-anyway effect (regression to the mean), with more specificity than x>0. It would be needed regardless, in order to exclude the null hypothesis.

nonbel 11y ago
My proposal includes collecting experimental data. It is just not in RCT form (which was claimed to be "the only way"). There is no reason to ignore RCT results, but that is not the only way to get the necessary data (which is what I disagreed with). An RCT is also not sufficient on its own, you need the theory that explains the results as well.
- nkurz 11y ago
  As an untrained outside observer (but one who has been thinking a lot recently about the statistics of causality) I'll chime in to say that I'm siding with 'ashearer' here. As I'm reading it, your approach doesn't make sense to me. If you want to distinguish between two treatments (placebo=treated and do-nothing=untreated) you need to measure outcomes for both treated and untreated individuals, and these individuals need to be "exchangeable" to some degree. Perhaps when you say in the first post "give people placebo", you mean "give the placebo to some subset of the individuals leaving the remainder untreated"? I agree that you can often draw useful conclusions from an observational study as long as some comparable individuals receive each treatment, and as long as you can control for the initial differences between the treated and untreated. But if you are saying that you don't need to look at the results for the untreated group at all, then I think you are mistaken.
  - nonbel 11y ago
    >"Perhaps when you say in the first post "give people placebo", you mean "give the placebo to some subset of the individuals leaving the remainder untreated"?" I meant just placebo. I am using predictive skill of the theory behind the explanations to distinguish between them, this is common in physics/astronomy/etc. As noted by asherer, placebo effect has been studied for a long time in the manner of seeing how two groups differ. There is nothing wrong with RCTs, but does near sole reliance on them (and their observational analogue) appear to have lead to cumulative knowledge and growth of understanding?
    - ashearer 11y ago
      Both hypotheses in this case are plausible. Beyond that, it would be great to have the tools to numerically calculate a psychological phenomenon such as the placebo effect the way a physicist calculates the energy of a particle. But we're very far from having them, and in the meantime, I don't know of a practical alternative to RCTs that would expand our knowledge at least as much. (I think we can both agree that over-reliance on significance testing is a problem, though not one limited to RCTs.)
  - nonbel 11y ago
    Check out the paper I posted here earlier: https://news.ycombinator.com/item?id=10556521 https://news.ycombinator.com/item?id=10556521

nonbel 11y ago

My proposal includes collecting experimental data. It is just not in RCT form (which was claimed to be "the only way"). There is no reason to ignore RCT results, but that is not the only way to get the necessary data (which is what I disagreed with). An RCT is also not sufficient on its own, you need the theory that explains the results as well.

nkurz 11y ago
As an untrained outside observer (but one who has been thinking a lot recently about the statistics of causality) I'll chime in to say that I'm siding with 'ashearer' here. As I'm reading it, your approach doesn't make sense to me. If you want to distinguish between two treatments (placebo=treated and do-nothing=untreated) you need to measure outcomes for both treated and untreated individuals, and these individuals need to be "exchangeable" to some degree. Perhaps when you say in the first post "give people placebo", you mean "give the placebo to some subset of the individuals leaving the remainder untreated"? I agree that you can often draw useful conclusions from an observational study as long as some comparable individuals receive each treatment, and as long as you can control for the initial differences between the treated and untreated. But if you are saying that you don't need to look at the results for the untreated group at all, then I think you are mistaken.
- nonbel 11y ago
  >"Perhaps when you say in the first post "give people placebo", you mean "give the placebo to some subset of the individuals leaving the remainder untreated"?" I meant just placebo. I am using predictive skill of the theory behind the explanations to distinguish between them, this is common in physics/astronomy/etc. As noted by asherer, placebo effect has been studied for a long time in the manner of seeing how two groups differ. There is nothing wrong with RCTs, but does near sole reliance on them (and their observational analogue) appear to have lead to cumulative knowledge and growth of understanding?
  - ashearer 11y ago
    Both hypotheses in this case are plausible. Beyond that, it would be great to have the tools to numerically calculate a psychological phenomenon such as the placebo effect the way a physicist calculates the energy of a particle. But we're very far from having them, and in the meantime, I don't know of a practical alternative to RCTs that would expand our knowledge at least as much. (I think we can both agree that over-reliance on significance testing is a problem, though not one limited to RCTs.)
- nonbel 11y ago
  Check out the paper I posted here earlier: https://news.ycombinator.com/item?id=10556521 https://news.ycombinator.com/item?id=10556521