4 ms·

> That's not how it works, though. From the 'Conclusions' of Deaton and Cartwright (2018) [1], which is the one that seemingly comes up in debates among friend

> That's not how it works, though. From the 'Conclusions' of Deaton and Cartwright (2018) [1], which is the one that seemingly comes up in debates among friends most often when this subject arises. I suggest to read that entire part of this paper at least -- it is very sobering. E.g.: "[...] assumptions that RCTs claim to (but do not) avoid, so that if the aim is to use empirical evidence, any credibility advantage that RCTs have in estimation is no longer operative. And because RCTs tell us so little about why results happen, they have a disadvantage over studies that use a wider range of prior information and data to help nail down mechanisms. [...] Without knowing why things happen and why people do things, we run the risk of worthless casual (‘fairy story’) causal theorizing and have given up on one of the central tasks of economics and other social sciences." I do not doubt everything in [1] applies even more so to RCTs in something like psychotherapy where the 'success' of 'failure' rests mostly on the interpretation by the subject and the 'why' is a black box. Until we understand and can observe brains in action much, much better and with vastly more detail than we do today that is. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6019115/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6019115/

stdbrouw 3y ago
The paper you link does not mention anything about the need to try every potential treatment (and no treatment) on every person in a study, and it does not mention anything about any "inherent bias" in RCTs -- these were the two claims you made -- so I'm not quite sure why you bring it up. Given that you have brought it up: I like Deaton's work a lot and know the arguments well, but you have to take into account that in this paper they are trying to provide some counterweight to a hype cycle that suddenly led a bunch of social scientists to argue that observational studies are stupid and should just be gotten rid of entirely and replaced by randomized trials, which for many reasons is not a good idea. That hardly implies that we can never know anything about whether psychotherapy works until we can simulate the brain, that's just a totally bizarre conclusion to draw from that paper.

stdbrouw 3y ago

The paper you link does not mention anything about the need to try every potential treatment (and no treatment) on every person in a study, and it does not mention anything about any "inherent bias" in RCTs -- these were the two claims you made -- so I'm not quite sure why you bring it up. Given that you have brought it up: I like Deaton's work a lot and know the arguments well, but you have to take into account that in this paper they are trying to provide some counterweight to a hype cycle that suddenly led a bunch of social scientists to argue that observational studies are stupid and should just be gotten rid of entirely and replaced by randomized trials, which for many reasons is not a good idea. That hardly implies that we can never know anything about whether psychotherapy works until we can simulate the brain, that's just a totally bizarre conclusion to draw from that paper.