14 ms·

Regression to the mean is the main reason ineffective treatments appear to work

137 points by Amorymeltzer 11y ago

ashearer 11y ago
So the research that established the placebo effect—an effect that’s well-known, and widely regarded as an illustration of the importance of control groups in research—itself had no control group? That’s incredible.
- _dark_matter_ 11y ago
  I don't know about that. There seems to be a lot of research investigating the difference between placebo and no-treatment. The publication [0] mentioned in article searches out all the studies that tested placebo treatment, and they all included a no-treatment group. Here's an excerpt: >We included randomised placebo trials with a no-treatment control group investigating any health problem [0] http://www.ncbi.nlm.nih.gov/pubmed/20091554 http://www.ncbi.nlm.nih.gov/pubmed/20091554
  - ashearer 11y ago
    I meant the original 1955 publication of “The Powerful Placebo” that put the idea of the “placebo effect” into the public consciousness. (Use of the term really takes off starting then [1].) That article's more recent (2010) link supports its conclusion, which wouldn't need to be stated unless it contrasted with what it calls “pharmacological folklore” from the original publication, which the author says he “took literally” for many decades after. The point being that the 1955 version of the idea is currently much more widely known than modern no-treatment comparisons, and most people don't realize that its research basis is so flimsy. [1] https://books.google.com/ngrams/graph?content=placebo+effect&year_start=1800&year_end=2000&corpus=15&smoothing=3&share=&direct_url=t1%3B%2Cplacebo%20effect%3B%2Cc0 https://books.google.com/ngrams/graph?content=placebo+effect...
fiatmoney 11y ago
Also known as the Drill Sergeant Paradox. Imagine that shouting has no actual effect on performance, but it is traditional to shout at underlings when they do something particularly poorly. When your trainees screw up, you berate them - and afterwards they actually do tend to do better. Unfortunately, this is because the screwup is more often than not a random variation, and the improvement is due to the mean regression, not the treatment. Conversely, praising them when they do well (again, assuming no underlying effect) actually seems to worsen their performance.
- brewdad 11y ago
  I wonder, has anyone ever tried berating the successes and praising the failures?
  - alphydan 11y ago
    That's a really dumb idea ;) </meta>
    - Jtsummers 11y ago
      As written, I agree. But a more sensible way (well, kinda, still backwards): Praise the individuals even when they screw up (great job cleaning that kitchen, even though there's still dirt under all the tables) and berate them when they do well (nitpick even the tiniest speck of remaining dirt in an otherwise immaculate kitchen). Still seems like a bad idea. Destroyed morale for the ones that do well but get scolded anyways, and encouraged laziness for the ones that do poorly.
      - CPLX 11y ago
        I believe this is called "marriage"
  - nikanj 11y ago
    The VC culture? If you define failure as "didn't make any money" and success as "made some money". Recently there was an article lamenting the "early exit syndrome" that startups have, where they prefer getting some money to potentially getting a lot more or nothing
bigchewy 11y ago
This is, unfortunately, rampant in healthcare. The natural variation amongst people as well as the natural variation of our health throughout out life makes actually analyzing healthcare outcomes incredibly difficult. 90%+ of published outcomes can be invalidated by simply looking at the published data. If you want some chuckles, read blog posts by Al Lewis ripping on research publised by companies touting their own performance. He's acerbic and condescending but also, mostly, correct.
- api 11y ago
  In a bio class a professor summed it up with "the hardest thing about medicine is being able to tell whether anything actually works." I told a friend of mine who's into alt.med once that yes I think a lot of alt.med is bunk but percentage wise it's probably not that much more bunk than "mainstream" medicine. It's easy for emergency medicine. Customer comes in with bullet wound, leaves alive and healthy and without bullet wound. But outside of domains like that it's crushingly hard to tell the difference between treatment and noise. Compounding the fact is that humans aren't a standardized item that can be compared against reference performance metrics. We're all different and our genome and environment are constantly changing.
  - bonoboTP 11y ago
    > alt.med is bunk but percentage wise it's probably not that much more bunk than "mainstream" medicine Oh yes it is! First of all, the actual effort is there in real medicine, while alternative medicine just doesn't care about evidence. Also, most real drugs work as intended. The fringe cases get emphasized a lot, but still, the basic things, like antibiotics, painkillers etc. work very well. Medicine has advanced A LOT in the last 100-150 years. Fake medicine, like homeopathy, is nothing like this. Don't mislead people.
    - api 11y ago
      I agree to some extent, but not completely. Some day population-wide studies fishing for a p value below a certain threshold will be looked at with the same disdain that we look at homeopathy. "They actually believed you could do studies across genetically diverse populations like that?!?! wow! Didn't they understand that everyone's metabolism is different?" Throw in a bit of pharma companies funding their own studies and I don't think the situation is as black and white as you say. Preachy overconfidence in the clear superiority of enlightened scientific medicine actually bolsters public support for alt.med as a reactionary movement.
nonbel 11y ago
>"The only way to measure the size of genuine placebo effects is to compare in an RCT the effect of a dummy treatment with the effect of no treatment at all." This is just wrong, the following procedure would be much more convincing than a RCT: Decide what you are measuring, collect group of people A, measure it, give people placebo, measure it again, record results. Then repeat under different circumstances with a different group of people B. Maybe even go back and do A again. Are you getting consistent measurements? Good. Now come up with an explanation for why the results have that distribution, sources of variation, etc. Use that to quantitatively predict what should be seen a new group of people C. Now go check group C. Did it match the predictions? If so, good, keep at it. Make a prediction for group D. Do the group D results match? If so, you are probably interpreting the results right.
- ashearer 11y ago
  How does that address the get-better-anyway effect, without a group to measure it?
  - nonbel 11y ago
    Because you are predicting precisely how much better can be accounted for by the placebo effect, no more, no less (of course within error bounds). The get-better-anyway effect is too vague. Say theory A predicts value x is in the range [3.1,3.3] and you observe x=3.36 +/-.1. Also, there is a theory B that predicts x>0. Theory A has been much more severely tested so is supported by the evidence. Theory B, meh.
    - ashearer 11y ago
      Say theory A (the placebo effect theory) predicts a 25% improvement, because it's been measured in the past with that result. Then there's also theory B (the get-better-anyway theory, or null hypothesis), which wasn't measured initially but in recent studies shows a similar 25% improvement. I don't understand how more studies testing theory A and getting 25%, while excluding theory B from testing, can show that A should be preferred over B. The rationale that A has been tested more thoroughly and therefore has a smaller confidence interval sounds circular.
deleted 11y ago
[deleted]
tokenadult 11y ago
This factual background deserves to be better known by Hacker News participants, who often avidly discuss statements about medical research without much background in medical research methodology. You, your friends, and all of us deserve to have new proposed treatments and old standard treatments evaluated thoroughly for safety and for effectiveness. A really good source for all of us to follow about the latest research on placebos in human medicine is the group-edited blog Science-Based Medicine,[1] which is edited by several active medical researchers and also includes lawyers, pharmacists, and even a reformed chiropractor among its contributors. The blog includes many informative posts about the placebo effects[2] observed in human medical research that help illuminate issues we all discuss a lot here on Hacker News. Some of my favorite recent articles on placebo effects from Science-Based Medicine include "Are Placebos Getting Stronger?"[3] (21 October 2015) by Steven Novella, a neurologist; "Placebo by Conditioning"[4] (29 July 2015), also by Dr. Novella; "Should placebos be used in randomized controlled trials of surgical interventions?"[5] (25 May 2015) by David Gorski, a surgeon and cancer researcher; and "Placebo, Are You There?"[6] (24 February 2015) a translation by a French-language article, translated by Harriet Hall. All go a long way toward explaining just what has been shown, and just what has not been shown, by previous research on placebo effects in human medicine. "Placebo medicine" has so far only been shown to have any effect at all on self-reported subjective patient symptoms such as pain and nausea that ebb and flow in the natural course of untreated disease. If you broke your arm, you wouldn't seek placebo treatment from your doctor, but actual effective treatment. If you have an injury that causes chronic pain (as I do), you are best off looking for the best available medically verified standard treatment, and not looking for any kind of placebo treatment. [1] https://www.sciencebasedmedicine.org/about-science-based-medicine/ https://www.sciencebasedmedicine.org/about-science-based-med... [2] https://www.sciencebasedmedicine.org/?s=placebo https://www.sciencebasedmedicine.org/?s=placebo [3] https://www.sciencebasedmedicine.org/are-placebos-getting-stronger/ https://www.sciencebasedmedicine.org/are-placebos-getting-st... [4] https://www.sciencebasedmedicine.org/placebo-by-conditioning/ https://www.sciencebasedmedicine.org/placebo-by-conditioning... [5] https://www.sciencebasedmedicine.org/should-placebos-be-used-in-randomized-controlled-trials-of-surgical-interventions/ https://www.sciencebasedmedicine.org/should-placebos-be-used... [6] https://www.sciencebasedmedicine.org/placebo-are-you-there/ https://www.sciencebasedmedicine.org/placebo-are-you-there/
ajarmst 11y ago
At school, I had a friend (studying to be a Physician) who referred to Chiropractic as "applied regression to the mean".

137 points by Amorymeltzer 11y ago

So the research that established the placebo effect—an effect that’s well-known, and widely regarded as an illustration of the importance of control groups in research—itself had no control group? That’s incredible.

_dark_matter_ 11y ago
I don't know about that. There seems to be a lot of research investigating the difference between placebo and no-treatment. The publication [0] mentioned in article searches out all the studies that tested placebo treatment, and they all included a no-treatment group. Here's an excerpt: >We included randomised placebo trials with a no-treatment control group investigating any health problem [0] http://www.ncbi.nlm.nih.gov/pubmed/20091554 http://www.ncbi.nlm.nih.gov/pubmed/20091554
- ashearer 11y ago
  I meant the original 1955 publication of “The Powerful Placebo” that put the idea of the “placebo effect” into the public consciousness. (Use of the term really takes off starting then [1].) That article's more recent (2010) link supports its conclusion, which wouldn't need to be stated unless it contrasted with what it calls “pharmacological folklore” from the original publication, which the author says he “took literally” for many decades after. The point being that the 1955 version of the idea is currently much more widely known than modern no-treatment comparisons, and most people don't realize that its research basis is so flimsy. [1] https://books.google.com/ngrams/graph?content=placebo+effect&year_start=1800&year_end=2000&corpus=15&smoothing=3&share=&direct_url=t1%3B%2Cplacebo%20effect%3B%2Cc0 https://books.google.com/ngrams/graph?content=placebo+effect...

fiatmoney 11y ago

Also known as the Drill Sergeant Paradox. Imagine that shouting has no actual effect on performance, but it is traditional to shout at underlings when they do something particularly poorly. When your trainees screw up, you berate them - and afterwards they actually do tend to do better. Unfortunately, this is because the screwup is more often than not a random variation, and the improvement is due to the mean regression, not the treatment. Conversely, praising them when they do well (again, assuming no underlying effect) actually seems to worsen their performance.

brewdad 11y ago
I wonder, has anyone ever tried berating the successes and praising the failures?
- alphydan 11y ago
  That's a really dumb idea ;) </meta>
  - Jtsummers 11y ago
    As written, I agree. But a more sensible way (well, kinda, still backwards): Praise the individuals even when they screw up (great job cleaning that kitchen, even though there's still dirt under all the tables) and berate them when they do well (nitpick even the tiniest speck of remaining dirt in an otherwise immaculate kitchen). Still seems like a bad idea. Destroyed morale for the ones that do well but get scolded anyways, and encouraged laziness for the ones that do poorly.
    - CPLX 11y ago
      I believe this is called "marriage"
- nikanj 11y ago
  The VC culture? If you define failure as "didn't make any money" and success as "made some money". Recently there was an article lamenting the "early exit syndrome" that startups have, where they prefer getting some money to potentially getting a lot more or nothing

bigchewy 11y ago

This is, unfortunately, rampant in healthcare. The natural variation amongst people as well as the natural variation of our health throughout out life makes actually analyzing healthcare outcomes incredibly difficult. 90%+ of published outcomes can be invalidated by simply looking at the published data. If you want some chuckles, read blog posts by Al Lewis ripping on research publised by companies touting their own performance. He's acerbic and condescending but also, mostly, correct.

api 11y ago
In a bio class a professor summed it up with "the hardest thing about medicine is being able to tell whether anything actually works." I told a friend of mine who's into alt.med once that yes I think a lot of alt.med is bunk but percentage wise it's probably not that much more bunk than "mainstream" medicine. It's easy for emergency medicine. Customer comes in with bullet wound, leaves alive and healthy and without bullet wound. But outside of domains like that it's crushingly hard to tell the difference between treatment and noise. Compounding the fact is that humans aren't a standardized item that can be compared against reference performance metrics. We're all different and our genome and environment are constantly changing.
- bonoboTP 11y ago
  > alt.med is bunk but percentage wise it's probably not that much more bunk than "mainstream" medicine Oh yes it is! First of all, the actual effort is there in real medicine, while alternative medicine just doesn't care about evidence. Also, most real drugs work as intended. The fringe cases get emphasized a lot, but still, the basic things, like antibiotics, painkillers etc. work very well. Medicine has advanced A LOT in the last 100-150 years. Fake medicine, like homeopathy, is nothing like this. Don't mislead people.
  - api 11y ago
    I agree to some extent, but not completely. Some day population-wide studies fishing for a p value below a certain threshold will be looked at with the same disdain that we look at homeopathy. "They actually believed you could do studies across genetically diverse populations like that?!?! wow! Didn't they understand that everyone's metabolism is different?" Throw in a bit of pharma companies funding their own studies and I don't think the situation is as black and white as you say. Preachy overconfidence in the clear superiority of enlightened scientific medicine actually bolsters public support for alt.med as a reactionary movement.

nonbel 11y ago

>"The only way to measure the size of genuine placebo effects is to compare in an RCT the effect of a dummy treatment with the effect of no treatment at all." This is just wrong, the following procedure would be much more convincing than a RCT: Decide what you are measuring, collect group of people A, measure it, give people placebo, measure it again, record results. Then repeat under different circumstances with a different group of people B. Maybe even go back and do A again. Are you getting consistent measurements? Good. Now come up with an explanation for why the results have that distribution, sources of variation, etc. Use that to quantitatively predict what should be seen a new group of people C. Now go check group C. Did it match the predictions? If so, good, keep at it. Make a prediction for group D. Do the group D results match? If so, you are probably interpreting the results right.

ashearer 11y ago
How does that address the get-better-anyway effect, without a group to measure it?
- nonbel 11y ago
  Because you are predicting precisely how much better can be accounted for by the placebo effect, no more, no less (of course within error bounds). The get-better-anyway effect is too vague. Say theory A predicts value x is in the range [3.1,3.3] and you observe x=3.36 +/-.1. Also, there is a theory B that predicts x>0. Theory A has been much more severely tested so is supported by the evidence. Theory B, meh.
  - ashearer 11y ago
    Say theory A (the placebo effect theory) predicts a 25% improvement, because it's been measured in the past with that result. Then there's also theory B (the get-better-anyway theory, or null hypothesis), which wasn't measured initially but in recent studies shows a similar 25% improvement. I don't understand how more studies testing theory A and getting 25%, while excluding theory B from testing, can show that A should be preferred over B. The rationale that A has been tested more thoroughly and therefore has a smaller confidence interval sounds circular.

deleted 11y ago

[deleted]

tokenadult 11y ago

This factual background deserves to be better known by Hacker News participants, who often avidly discuss statements about medical research without much background in medical research methodology. You, your friends, and all of us deserve to have new proposed treatments and old standard treatments evaluated thoroughly for safety and for effectiveness. A really good source for all of us to follow about the latest research on placebos in human medicine is the group-edited blog Science-Based Medicine,[1] which is edited by several active medical researchers and also includes lawyers, pharmacists, and even a reformed chiropractor among its contributors. The blog includes many informative posts about the placebo effects[2] observed in human medical research that help illuminate issues we all discuss a lot here on Hacker News. Some of my favorite recent articles on placebo effects from Science-Based Medicine include "Are Placebos Getting Stronger?"[3] (21 October 2015) by Steven Novella, a neurologist; "Placebo by Conditioning"[4] (29 July 2015), also by Dr. Novella; "Should placebos be used in randomized controlled trials of surgical interventions?"[5] (25 May 2015) by David Gorski, a surgeon and cancer researcher; and "Placebo, Are You There?"[6] (24 February 2015) a translation by a French-language article, translated by Harriet Hall. All go a long way toward explaining just what has been shown, and just what has not been shown, by previous research on placebo effects in human medicine. "Placebo medicine" has so far only been shown to have any effect at all on self-reported subjective patient symptoms such as pain and nausea that ebb and flow in the natural course of untreated disease. If you broke your arm, you wouldn't seek placebo treatment from your doctor, but actual effective treatment. If you have an injury that causes chronic pain (as I do), you are best off looking for the best available medically verified standard treatment, and not looking for any kind of placebo treatment. [1] https://www.sciencebasedmedicine.org/about-science-based-medicine/ https://www.sciencebasedmedicine.org/about-science-based-med... [2] https://www.sciencebasedmedicine.org/?s=placebo https://www.sciencebasedmedicine.org/?s=placebo [3] https://www.sciencebasedmedicine.org/are-placebos-getting-stronger/ https://www.sciencebasedmedicine.org/are-placebos-getting-st... [4] https://www.sciencebasedmedicine.org/placebo-by-conditioning/ https://www.sciencebasedmedicine.org/placebo-by-conditioning... [5] https://www.sciencebasedmedicine.org/should-placebos-be-used-in-randomized-controlled-trials-of-surgical-interventions/ https://www.sciencebasedmedicine.org/should-placebos-be-used... [6] https://www.sciencebasedmedicine.org/placebo-are-you-there/ https://www.sciencebasedmedicine.org/placebo-are-you-there/

ajarmst 11y ago

At school, I had a friend (studying to be a Physician) who referred to Chiropractic as "applied regression to the mean".