3 ms·

That's not how it works, though. First, if it is not possible to blind the patient to the intervention and/or provide a meaningful placebo (which also happens

That's not how it works, though. First, if it is not possible to blind the patient to the intervention and/or provide a meaningful placebo (which also happens with surgery and drugs with prominent side-effects) that's not a big deal, it simply changes the interpretation from "the effect of a particular regime of psychotherapy as such" to "the effect of a particular regime of psychotherapy, including expectation effects". Those expectation effects also exist in real life, if you go to a therapist you really really hope you'll get better and may be motivated to make a few changes in your life independently of the treatment, so if anything the trial becomes more realistic. Note that you'd still use a control arm so that if people get better over time without treatment, you're not exaggerating the effects of the intervention. Second, almost every randomized controlled trial compares an experimental group against a control group, you are assigned to a group and don't move between them, and what that means is that we can only compare whether the treatment works on average, not whether it worked for one person in particular because, as you say, we don't know what would have happened (counterfactually) if that one specific person had been randomly assigned to the other group. But again, this hardly matters, because we can still know how well a treatment works on average. What you refer to, an experiment in which every subject receives every treatment, is known as a crossover design and while that design has its advantages, it's not that common because it is harder to pull off and while it can be more efficient (need fewer patients to attain an estimate with a given precision) it is not "less biased" or inherently more trustworthy.

virtualritz 3y ago
> That's not how it works, though. From the 'Conclusions' of Deaton and Cartwright (2018) [1], which is the one that seemingly comes up in debates among friends most often when this subject arises. I suggest to read that entire part of this paper at least -- it is very sobering. E.g.: "[...] assumptions that RCTs claim to (but do not) avoid, so that if the aim is to use empirical evidence, any credibility advantage that RCTs have in estimation is no longer operative. And because RCTs tell us so little about why results happen, they have a disadvantage over studies that use a wider range of prior information and data to help nail down mechanisms. [...] Without knowing why things happen and why people do things, we run the risk of worthless casual (‘fairy story’) causal theorizing and have given up on one of the central tasks of economics and other social sciences." I do not doubt everything in [1] applies even more so to RCTs in something like psychotherapy where the 'success' of 'failure' rests mostly on the interpretation by the subject and the 'why' is a black box. Until we understand and can observe brains in action much, much better and with vastly more detail than we do today that is. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6019115/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6019115/
- stdbrouw 3y ago
  The paper you link does not mention anything about the need to try every potential treatment (and no treatment) on every person in a study, and it does not mention anything about any "inherent bias" in RCTs -- these were the two claims you made -- so I'm not quite sure why you bring it up. Given that you have brought it up: I like Deaton's work a lot and know the arguments well, but you have to take into account that in this paper they are trying to provide some counterweight to a hype cycle that suddenly led a bunch of social scientists to argue that observational studies are stupid and should just be gotten rid of entirely and replaced by randomized trials, which for many reasons is not a good idea. That hardly implies that we can never know anything about whether psychotherapy works until we can simulate the brain, that's just a totally bizarre conclusion to draw from that paper.

virtualritz 3y ago

> That's not how it works, though. From the 'Conclusions' of Deaton and Cartwright (2018) [1], which is the one that seemingly comes up in debates among friends most often when this subject arises. I suggest to read that entire part of this paper at least -- it is very sobering. E.g.: "[...] assumptions that RCTs claim to (but do not) avoid, so that if the aim is to use empirical evidence, any credibility advantage that RCTs have in estimation is no longer operative. And because RCTs tell us so little about why results happen, they have a disadvantage over studies that use a wider range of prior information and data to help nail down mechanisms. [...] Without knowing why things happen and why people do things, we run the risk of worthless casual (‘fairy story’) causal theorizing and have given up on one of the central tasks of economics and other social sciences." I do not doubt everything in [1] applies even more so to RCTs in something like psychotherapy where the 'success' of 'failure' rests mostly on the interpretation by the subject and the 'why' is a black box. Until we understand and can observe brains in action much, much better and with vastly more detail than we do today that is. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6019115/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6019115/

stdbrouw 3y ago
The paper you link does not mention anything about the need to try every potential treatment (and no treatment) on every person in a study, and it does not mention anything about any "inherent bias" in RCTs -- these were the two claims you made -- so I'm not quite sure why you bring it up. Given that you have brought it up: I like Deaton's work a lot and know the arguments well, but you have to take into account that in this paper they are trying to provide some counterweight to a hype cycle that suddenly led a bunch of social scientists to argue that observational studies are stupid and should just be gotten rid of entirely and replaced by randomized trials, which for many reasons is not a good idea. That hardly implies that we can never know anything about whether psychotherapy works until we can simulate the brain, that's just a totally bizarre conclusion to draw from that paper.