7 ms·

Why Almost Everything You Hear About Medicine Is Wrong

76 points by gatsby 16y ago

ffffruit 16y ago
Always relevant, the article from Ioannides - the man who dedicated this entire career on examining the way in which research is done rather than the actual research. http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info:doi/10.1371/journal... #1 in downloads on PLoS for many years.
- ThomPete 16y ago
  Thomas Kuhn is another one worth reading although more philosophical (none the less - relevant) "The Structure of Scientific Revolutions" http://en.wikipedia.org/wiki/The_Structure_of_Scientific_Revolutions http://en.wikipedia.org/wiki/The_Structure_of_Scientific_Rev...
  - stcredzero 16y ago
    And none the less widely misused.
maurits 16y ago
For a bit more detail: The New Yorker http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer?currentPage=all http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_... The actual referenced article (PDF) http://www.edwardtufte.com/files/Study1.pdf http://www.edwardtufte.com/files/Study1.pdf
- GiraffeNecktie 16y ago
  also this one http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/8269/ http://www.theatlantic.com/magazine/archive/2010/11/lies-dam...
alecco 16y ago
And there's the deeper trouble with method: http://en.wikipedia.org/wiki/Paul_Feyerabend#Nature_of_scientific_method http://en.wikipedia.org/wiki/Paul_Feyerabend#Nature_of_scien...
- ThomPete 16y ago
  Kuhn, Feyrabend and Lakatos all worth reading. Even tough they might be a tad to postmodern for some people. None the less they are all valuable thinkers.
  - rbarooah 16y ago
    What do you mean by 'postmodern'? Is this a real criticism of what they are saying?
    - gilesc 16y ago
      Oversimplifying things a bit, these writers are saying that scientific objectivity is impossible. This line of argument tends to reduce the credibility of science in the eyes of the public, with negative consequences for research funding. This is a bad thing because, even if not "objective", science has singlehandedly transformed life for the better in the past century by reducing disease, poverty, manual labor, etc, and should continue to be funded.
      - ThomPete 16y ago
        I don't think any would claim otherwise. But you don't need objective for that all you need is useful.
        derleth 16y ago
        Except if some people think science is not objective, they conclude that religion is just as valid, and that, therefore, Creationism (er, Creation 'Science') needs to be taught in schools alongside the 'fact' that gays and liberals are evil, diseased beings. After all, if it's all subjective, everything is equally valid!
gohat 16y ago
I've been writing about health and medical topics for a long time and this piece does a good job of discussing how the science can get 'icky.' First, let me say that in general, modern medicine has done wonders, increasing duration and quality of life in astonishing ways. That said, the system is set up to promote positive studies/findings, and there are a lot of ways you can play with the data. Very competent statisticians/researchers are capable of helping studies, sometimes, be more positive than they might otherwise be. That, plus the many millions of dollars you can make from a drug, has often resulted in questionable - but predictable - results.
- antidaily 16y ago
  right, less money to be made if people just ate much less and stopped smoking.
alecco 16y ago
Also, Jaynes had a very interesting take on how to address these issues. Probability Theory: The Logic of Science http://bayes.wustl.edu/etj/prob/book.pdf http://bayes.wustl.edu/etj/prob/book.pdf (draft) http://www.amazon.com/Probability-Theory-Logic-Science-Vol/dp/0521592712 http://www.amazon.com/Probability-Theory-Logic-Science-Vol/d...
yummyfajitas 16y ago
Negative results sit in a file drawer, or the trial keeps going in hopes the results turn positive. Therein lies the problem. We know the solution: tweak the publishing process. Currently: 1) Do study 2) Submit paper, including conclusions. 3) Journal accepts, based on correctness of methodology and interestingness of conclusion. To prevent this result, just change the order of things. 1) Submit paper explaining methodology, and "alternate endings" (positive and negative). 2) Journal accepts paper based on methodology only. Methodology section of paper is published. 3) Authors do study and submit results. Journal publishes appropriate conclusion section. 4) If necessary, authors submit additional observational results.
- niels_olson 16y ago
  The International Committee of Medical Journal Editors has done something fairly similar (http://www.icmje.org/urm_main.html http://www.icmje.org/urm_main.html), and, in the US the government backs it up with a mandatory prepublication database of all ongoing trials: http://clinicaltrials.gov/ http://clinicaltrials.gov/ Also see Ionnidis's paper on why most published research is wrong: http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10%2E1371%2Fjournal%2Epmed%2E0020124 http://medicine.plosjournals.org/perlserv/?request=get-docum...
- weego 16y ago
  _Negative results sit in a file drawer, or the trial keeps going in hopes the results turn positive._ There is a more unfortunate scenario that often plays out. That is that they get a different statistician in who has a goal of getting positive looking results out of a negative result set though various means and then often carefully manipulating the wording of the success criteria to match their best attempt at making the data work. This is quite common, my partner (in the girlfriend sense) work within medial journal publishing and sees it quite a lot.
  - yummyfajitas 16y ago
    ...manipulating the wording of the success criteria to match their best attempt at making the data work. While not impossible, it is very tricky to do this before you actually have the data. I'm advocating that the wording of the test criteria, statistical tests to perform, etc, are all decided before the trial. Bringing in a different statistician post-hoc is pointless, all he could do is run the exact predetermined statistical test. In fact, you can do even better: submit code to the journal before the study, and submit data after the study. The journal runs the code on the data and decides which conclusion to publish.
bryanlarsen 16y ago
The vitamin D claim is misleading. Sure, most people have enough vitamin D for bone health. But vitamin D does a lot more than make your bones healthy, so supplementation is still probably useful. Luckily, unlike vitamin C, too much vitamin D is not harmful so all you're wasting is money if the other claims for vitamin D also get disproved.
- michael_dorfman 16y ago
  First of all: it's probably not a good idea to be dispensing (or receiving) medical advice on HN. That being said-- Vitamin D is fat soluble, and too much vitamin D is definitely harmful. Take a look here:http://en.wikipedia.org/wiki/Hypervitaminosis_D http://en.wikipedia.org/wiki/Hypervitaminosis_D Vitamin C, on the other hand, is water soluble, and much harder to overdose on.
  - burgerbrain 16y ago
    Did anyone else find themselves clicking on that link very cautiously? Wikipedia should really institute some sort of official warning system for articles with shocking medical images. (this article does not have images)
  - pjscott 16y ago
    The biggest Vitamin D supplements I've seen have 1000 IU per tablet, and the bottle suggests one tablet per day. According to the article you linked, 10,000 IU/day is safe (we can get that much from sunshine in a day), and long-term overdose has been observed at 77,000 IU/day. You'd have to go pretty hog-wild with the Vitamin D supplements to overdose.
    - gregwebs 16y ago
      That article is just about Hypervitaminosis D, other harmful effects are possible (depending on the individual) when supplementing at high levels like 10,000 IU/day.
    - nkurz 16y ago
Symmetry 16y ago
Which just goes to show that you should learn about settled science before worrying about the latest findings. http://lesswrong.com/lw/ow/the_beauty_of_settled_science/ http://lesswrong.com/lw/ow/the_beauty_of_settled_science/
KirinDave 16y ago
While this article is correct and talks about Ioannides's work—which is good, respectable stuff by the way—it does paint a picture like this should be a shock to you. Folks, it isn't just medicine; Science in general must be comfortable with the proposition of being wrong. The more complex the field, the more often replication will fail (and the tighter standards need to be). There are very few pursuits that our race engages in that are more difficult than understanding life itself. One of the biggest problems is that physicians and the press tend to get very hasty with pushing new research. It's not their fault, mind you, they're trying to inform us about health. If a doctor could save a life with an experimental treatment that has yet to reach thorough replication, shouldn't s/he? And it's the media's job to get us interesting new stories (albeit they could be far better at putting their information into context). The nature of science is such that refinements occur over time. Unfortunately, the nature of most people receiving medical treatment is to believe it works or doesn't work based of extremely subjective data. If we want to attack that problem head on, we should begin with education. All the improvements in methodology imaginable cannot stop science from being what it is, and as such cannot eliminate the sawtoothed slope of medical progress.
tokenadult 16y ago
A place to hear things about medicine that are more often than not correct: http://www.sciencebasedmedicine.org/?p=8198 http://www.sciencebasedmedicine.org/?p=8198 http://www.sciencebasedmedicine.org/?cat=11 http://www.sciencebasedmedicine.org/?cat=11
Mz 16y ago
One of the problems with drugs studies is that the very point is to develop a drug and that intent is generally commercial in nature. At no point does anyone seem to stop and wonder what might be the best solution. In addressing my health issues, I have had to infer a lot of things based on drug studies because there is a dearth of good studies on other topics (at least specific to my medical condition). Yet the general consensus is that diet and lifestyle play a major role in every major deadly medical condition. Encouraging people to live right just doesn't have the splash and dash and isn't generally as a lucrative. It also gets a lot of resistance from the intended audience: An awful lot of folks don't want to do the hard work of changing their lives. In many cases, they just want a new drug, one that won't make them sick...etc (I'm sure someone can quote song lyrics better than I can).

76 points by gatsby 16y ago

Always relevant, the article from Ioannides - the man who dedicated this entire career on examining the way in which research is done rather than the actual research. http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info:doi/10.1371/journal... #1 in downloads on PLoS for many years.

ThomPete 16y ago
Thomas Kuhn is another one worth reading although more philosophical (none the less - relevant) "The Structure of Scientific Revolutions" http://en.wikipedia.org/wiki/The_Structure_of_Scientific_Revolutions http://en.wikipedia.org/wiki/The_Structure_of_Scientific_Rev...
- stcredzero 16y ago
  And none the less widely misused.

maurits 16y ago

For a bit more detail: The New Yorker http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer?currentPage=all http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_... The actual referenced article (PDF) http://www.edwardtufte.com/files/Study1.pdf http://www.edwardtufte.com/files/Study1.pdf

GiraffeNecktie 16y ago
also this one http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/8269/ http://www.theatlantic.com/magazine/archive/2010/11/lies-dam...

alecco 16y ago

And there's the deeper trouble with method: http://en.wikipedia.org/wiki/Paul_Feyerabend#Nature_of_scientific_method http://en.wikipedia.org/wiki/Paul_Feyerabend#Nature_of_scien...

ThomPete 16y ago
Kuhn, Feyrabend and Lakatos all worth reading. Even tough they might be a tad to postmodern for some people. None the less they are all valuable thinkers.
- rbarooah 16y ago
  What do you mean by 'postmodern'? Is this a real criticism of what they are saying?
  - gilesc 16y ago
    Oversimplifying things a bit, these writers are saying that scientific objectivity is impossible. This line of argument tends to reduce the credibility of science in the eyes of the public, with negative consequences for research funding. This is a bad thing because, even if not "objective", science has singlehandedly transformed life for the better in the past century by reducing disease, poverty, manual labor, etc, and should continue to be funded.
    - ThomPete 16y ago
      I don't think any would claim otherwise. But you don't need objective for that all you need is useful.
      - derleth 16y ago
        Except if some people think science is not objective, they conclude that religion is just as valid, and that, therefore, Creationism (er, Creation 'Science') needs to be taught in schools alongside the 'fact' that gays and liberals are evil, diseased beings. After all, if it's all subjective, everything is equally valid!

gohat 16y ago

I've been writing about health and medical topics for a long time and this piece does a good job of discussing how the science can get 'icky.' First, let me say that in general, modern medicine has done wonders, increasing duration and quality of life in astonishing ways. That said, the system is set up to promote positive studies/findings, and there are a lot of ways you can play with the data. Very competent statisticians/researchers are capable of helping studies, sometimes, be more positive than they might otherwise be. That, plus the many millions of dollars you can make from a drug, has often resulted in questionable - but predictable - results.

antidaily 16y ago
right, less money to be made if people just ate much less and stopped smoking.

alecco 16y ago

Also, Jaynes had a very interesting take on how to address these issues. Probability Theory: The Logic of Science http://bayes.wustl.edu/etj/prob/book.pdf http://bayes.wustl.edu/etj/prob/book.pdf (draft) http://www.amazon.com/Probability-Theory-Logic-Science-Vol/dp/0521592712 http://www.amazon.com/Probability-Theory-Logic-Science-Vol/d...

yummyfajitas 16y ago

Negative results sit in a file drawer, or the trial keeps going in hopes the results turn positive. Therein lies the problem. We know the solution: tweak the publishing process. Currently: 1) Do study 2) Submit paper, including conclusions. 3) Journal accepts, based on correctness of methodology and interestingness of conclusion. To prevent this result, just change the order of things. 1) Submit paper explaining methodology, and "alternate endings" (positive and negative). 2) Journal accepts paper based on methodology only. Methodology section of paper is published. 3) Authors do study and submit results. Journal publishes appropriate conclusion section. 4) If necessary, authors submit additional observational results.

niels_olson 16y ago
The International Committee of Medical Journal Editors has done something fairly similar (http://www.icmje.org/urm_main.html http://www.icmje.org/urm_main.html), and, in the US the government backs it up with a mandatory prepublication database of all ongoing trials: http://clinicaltrials.gov/ http://clinicaltrials.gov/ Also see Ionnidis's paper on why most published research is wrong: http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10%2E1371%2Fjournal%2Epmed%2E0020124 http://medicine.plosjournals.org/perlserv/?request=get-docum...
weego 16y ago
_Negative results sit in a file drawer, or the trial keeps going in hopes the results turn positive._ There is a more unfortunate scenario that often plays out. That is that they get a different statistician in who has a goal of getting positive looking results out of a negative result set though various means and then often carefully manipulating the wording of the success criteria to match their best attempt at making the data work. This is quite common, my partner (in the girlfriend sense) work within medial journal publishing and sees it quite a lot.
- yummyfajitas 16y ago
  ...manipulating the wording of the success criteria to match their best attempt at making the data work. While not impossible, it is very tricky to do this before you actually have the data. I'm advocating that the wording of the test criteria, statistical tests to perform, etc, are all decided before the trial. Bringing in a different statistician post-hoc is pointless, all he could do is run the exact predetermined statistical test. In fact, you can do even better: submit code to the journal before the study, and submit data after the study. The journal runs the code on the data and decides which conclusion to publish.

bryanlarsen 16y ago

The vitamin D claim is misleading. Sure, most people have enough vitamin D for bone health. But vitamin D does a lot more than make your bones healthy, so supplementation is still probably useful. Luckily, unlike vitamin C, too much vitamin D is not harmful so all you're wasting is money if the other claims for vitamin D also get disproved.

michael_dorfman 16y ago
First of all: it's probably not a good idea to be dispensing (or receiving) medical advice on HN. That being said-- Vitamin D is fat soluble, and too much vitamin D is definitely harmful. Take a look here:http://en.wikipedia.org/wiki/Hypervitaminosis_D http://en.wikipedia.org/wiki/Hypervitaminosis_D Vitamin C, on the other hand, is water soluble, and much harder to overdose on.
- burgerbrain 16y ago
  Did anyone else find themselves clicking on that link very cautiously? Wikipedia should really institute some sort of official warning system for articles with shocking medical images. (this article does not have images)
- pjscott 16y ago
  The biggest Vitamin D supplements I've seen have 1000 IU per tablet, and the bottle suggests one tablet per day. According to the article you linked, 10,000 IU/day is safe (we can get that much from sunshine in a day), and long-term overdose has been observed at 77,000 IU/day. You'd have to go pretty hog-wild with the Vitamin D supplements to overdose.
  - gregwebs 16y ago
    That article is just about Hypervitaminosis D, other harmful effects are possible (depending on the individual) when supplementing at high levels like 10,000 IU/day.
  - nkurz 16y ago

Symmetry 16y ago

Which just goes to show that you should learn about settled science before worrying about the latest findings. http://lesswrong.com/lw/ow/the_beauty_of_settled_science/ http://lesswrong.com/lw/ow/the_beauty_of_settled_science/

KirinDave 16y ago

While this article is correct and talks about Ioannides's work—which is good, respectable stuff by the way—it does paint a picture like this should be a shock to you. Folks, it isn't just medicine; Science in general must be comfortable with the proposition of being wrong. The more complex the field, the more often replication will fail (and the tighter standards need to be). There are very few pursuits that our race engages in that are more difficult than understanding life itself. One of the biggest problems is that physicians and the press tend to get very hasty with pushing new research. It's not their fault, mind you, they're trying to inform us about health. If a doctor could save a life with an experimental treatment that has yet to reach thorough replication, shouldn't s/he? And it's the media's job to get us interesting new stories (albeit they could be far better at putting their information into context). The nature of science is such that refinements occur over time. Unfortunately, the nature of most people receiving medical treatment is to believe it works or doesn't work based of extremely subjective data. If we want to attack that problem head on, we should begin with education. All the improvements in methodology imaginable cannot stop science from being what it is, and as such cannot eliminate the sawtoothed slope of medical progress.

tokenadult 16y ago

A place to hear things about medicine that are more often than not correct: http://www.sciencebasedmedicine.org/?p=8198 http://www.sciencebasedmedicine.org/?p=8198 http://www.sciencebasedmedicine.org/?cat=11 http://www.sciencebasedmedicine.org/?cat=11

Mz 16y ago

One of the problems with drugs studies is that the very point is to develop a drug and that intent is generally commercial in nature. At no point does anyone seem to stop and wonder what might be the best solution. In addressing my health issues, I have had to infer a lot of things based on drug studies because there is a dearth of good studies on other topics (at least specific to my medical condition). Yet the general consensus is that diet and lifestyle play a major role in every major deadly medical condition. Encouraging people to live right just doesn't have the splash and dash and isn't generally as a lucrative. It also gets a lot of resistance from the intended audience: An awful lot of folks don't want to do the hard work of changing their lives. In many cases, they just want a new drug, one that won't make them sick...etc (I'm sure someone can quote song lyrics better than I can).