3 ms·

> A subsequent Bayesian reanalysis did show a statistically significant advantage to psilocybin, but this difference was not clinically significant. What does

by p-e-w 3y ago

> A subsequent Bayesian reanalysis did show a statistically significant advantage to psilocybin, but this difference was not clinically significant. What does it mean for something to be statistically significant, but not "clinically significant"?

technocratius 3y ago
It's statistically significant because there is a difference in outcome between the group receiving treatment and the one that did not which could not be attributed to chance alone, but the difference isn't clinically significant because it was a very small difference (in statistics, this is sometimes also referred to as the _effect size_)
- p-e-w 3y ago
  Wouldn't any statistically significant improvement become clinically significant when applied at a large enough scale, e.g. in global clinical practice?
isoprophlex 3y ago
Imagine someone not hitting a nail with a hammer, striking all over the place, all the time. Now a new hammer comes in, that manages, in a very precise manner, to hit much closer to the nail... but still missing it.
jdietrich 3y ago
"Clinical significance" is slightly arbitrary and defined in different ways, but it can be taken to mean "an effect that someone could actually notice in practice". In this trial, the difference between the psilocybin group and the escitalopram group was, on average, two points on the QIDS-SR-16 scale. Those two points might represent someone reporting that their sleep has slightly improved, or that they feel slightly more energetic and slightly more able to concentrate. We can detect that difference in the analysis of the data and we can be somewhat confident that it's a real difference, but it's not the sort of change that most people would be able to distinguish from normal day-to-day variation. Of course those very subtle differences might aggregate to a meaningful benefit across a whole patient population, but we also have to consider the cost-benefit tradeoff. A tiny benefit might be highly significant if the treatments were very similar in cost, but they aren't - escitalopram is a very cheap generic drug that can be safely prescribed by family doctors and nurse prescribers with minimal follow-up, while psilocybin currently requires a full day in a specialised clinic with significant preparation and follow-up. https://alnursing.org/wp-content/uploads/2020/03/Depression-Questionnaire-QIDS-SR-16.pdf https://alnursing.org/wp-content/uploads/2020/03/Depression-...

by p-e-w 3y ago

technocratius 3y ago

It's statistically significant because there is a difference in outcome between the group receiving treatment and the one that did not which could not be attributed to chance alone, but the difference isn't clinically significant because it was a very small difference (in statistics, this is sometimes also referred to as the _effect size_)

p-e-w 3y ago
Wouldn't any statistically significant improvement become clinically significant when applied at a large enough scale, e.g. in global clinical practice?

isoprophlex 3y ago

Imagine someone not hitting a nail with a hammer, striking all over the place, all the time. Now a new hammer comes in, that manages, in a very precise manner, to hit much closer to the nail... but still missing it.

jdietrich 3y ago

"Clinical significance" is slightly arbitrary and defined in different ways, but it can be taken to mean "an effect that someone could actually notice in practice". In this trial, the difference between the psilocybin group and the escitalopram group was, on average, two points on the QIDS-SR-16 scale. Those two points might represent someone reporting that their sleep has slightly improved, or that they feel slightly more energetic and slightly more able to concentrate. We can detect that difference in the analysis of the data and we can be somewhat confident that it's a real difference, but it's not the sort of change that most people would be able to distinguish from normal day-to-day variation. Of course those very subtle differences might aggregate to a meaningful benefit across a whole patient population, but we also have to consider the cost-benefit tradeoff. A tiny benefit might be highly significant if the treatments were very similar in cost, but they aren't - escitalopram is a very cheap generic drug that can be safely prescribed by family doctors and nurse prescribers with minimal follow-up, while psilocybin currently requires a full day in a specialised clinic with significant preparation and follow-up. https://alnursing.org/wp-content/uploads/2020/03/Depression-Questionnaire-QIDS-SR-16.pdf https://alnursing.org/wp-content/uploads/2020/03/Depression-...