6 ms·

When doctors, psychologists, and drug makers can't rely on each other's research

67 points by nkurz 11y ago

Alex3917 11y ago
Why are people so obsessed with publication bias and statistical power? Those are just two of the dozens of reasons why published research is unreliable, and it's not even clear that they're the most important ones. It seems like the skeptic community just randomly glommed onto those issues in the 90s or something and haven't updated their worldview since.
- 467568985476 11y ago
  What are the other, more important reasons that research is unreliable?
  - Alex3917 11y ago
    I have a list of them going here, but it's definitely not comprehensive either: http://www.alexkrupp.com/Citevault.html#pharmaceuticals http://www.alexkrupp.com/Citevault.html#pharmaceuticals Even if you look just at statistical analysis, there are dozens of different issues. E.g. I seem to remember some articles about how basically 100% of research involving brain scans is incorrect, because of a mix of statistical problems and also issues related to the imaging software.
    - tcj_phx 11y ago
      This section is particularly useful to me: http://www.alexkrupp.com/Citevault.html#mentalhealth http://www.alexkrupp.com/Citevault.html#mentalhealth Thanks!
  - aschreyer 11y ago
    A big problem are the perverse incentives researchers are facing. The only thing that matters is publishing in prestigious journals, otherwise your career is pretty much over (no real chance to get tenure). Universities often don't even check the publications, and researchers usually omit the actual title of the publication and only keep the name of the journal. Moreover, if you want to get tenure you need to work on "hot" topics (as in CRISPR at the moment or cancer in general), meaning you don't necessarily spend too much time on one subject. I'm sure researchers are often aware that the research results are not ready for publication yet, but it's either publish or perish, and you usually get away with publishing sub-par results because no one really follows up on them either. Cancer and Alzheimer are two fields were billions have been wasted by research grants and pharma companies because the initially promising drug targets for example could not really be validated. I'm afraid this will stay that way as long as the career dilemma researchers are facing does not change.
- jessaustin 11y ago
  I suppose I haven't read enough about publication bias to get as annoyed by it as you seem. TFA offers a pretty convincing argument that publication bias is bad enough to doom most current scientific efforts to irrelevance. That sounds like something we should talk about? Or we can talk about more important reasons why published research is unreliable, if you care to mention them. What is "the skeptic community"? Is Reason part of it? I'm a bit skeptical that's even a thing... Statistical power is mentioned once in TFA and then forgotten. Amusingly enough, it gets the relevance of statistical power for the subject under discussion backwards. That graf might actually have been added by an "obsessed" editor.
  - orblivion 11y ago
    Reason is generally a libertarian publication and website. For the Skeptic Community go to r/skeptic. They're related to the Atheist Community.
    - akiselev 11y ago
      So by "skeptic community" you actually mean "reddit skeptic community?"
      - deleted 11y ago
        [deleted]
      - Alex3917 11y ago
        There are skeptic conferences, skeptic magazines, skeptic books, skeptic authors, skeptic web forums, skeptic TV shows, etc. I think this easily meets pretty much any definition of community.
        jessaustin 11y ago
        This seems to be focused toward critiques of religion rather than of science? I don't see the connection.
jessaustin 11y ago
"HARKing" is a great coinage I had not seen before.
mhkool 11y ago
In double-blind tests, one deliberately changes only one variable and compares/observes two groups where individual do not know whether they take a new drug or a placebo. If the drug has a positive effect, the double-blind test is an accepted method of proof, but unfortunately this limits our options severely. If cancer can be cured by changing _two_ variables, we will never find a cure. NEVER! Remember that we already do research for 50+ years and have spent trillions, so I tend to think that Einstein was right: it is insane to repeat the same experiments and to expect a different outcome. Let's jump to Alzheimer, a feared disease with also a lot of research and not so good drugs. Dr Bredesen has made a protocol with 35 variables to cure Alzheimer. And in his first test, he reversed Alzheimer in 9 out of 10 patients. The scientific purists say that there is no double-blind study, so no proof. I am convinced that Dr Bredesen in on the right path, not only because of the results but also because of his reasoning. The treatment has no drug, but a health optimization in all possible ways. And then the body heals itself in 4 months. Please do not comment with "that cannot be true" unless you proof that it cannot be true (I do not belief that you can provide proof). So the state of current research methods is weak. Medical research only focuses on one variable, one drug, that will cure a disease. The results of this way of doing medical research are VERY disappointing. Trillions are spent and no drugs that cure cancer, AIDS or Alzheimer have been found. Dr Bredesen has chosen a different path and I support him.
- orblivion 11y ago
  So if you have an X variable cure, that means that an X-1 variable cure is not enough, right? I'm not very familiar with the nuances of the scientific process, but I would think that you could still run tests on one variable at a time. Just turn all but one variable "on", make that your control group. Turn all variables on, that's your experiment group. Now repeat the process isolating each variable, determining whether they're all in fact necessary for the cure.
- mabcat 11y ago
  Unfortunately you have a fundamental misunderstanding of health experimental design. Double-blind means that neither the participants nor the experimenters know which participant is allocated to which group. It doesn't say anything about how many groups there are or what is done to each group. I haven't heard of Dr Bredesen, but they could certainly run a double-blind study to validate their treatment regime. Treatments that haven't been validated in this way have a high risk of being ineffective but appearing effective due to experimenter error, placebo effect, demand characteristics, etc. Requiring convincing evidence in the form of a double-blind trial isn't purism, it's sensible caution based on experience. edit: you're also on shaky ground statistically. There's a rule of thumb that if interactions between several variables are present in a treatment effect, 70% of that effect will show up on the individual variables (as main effects). It's very unlikely that there exists a treatment where all the effect is on a 35-variable interaction and none of the effect is visible when looking at each variable singly.
mwsherman 11y ago
This is largely predicted by Daniel Kahneman and Nassim Taleb, would love to see them cited more often.
golergka 11y ago
What it would take a resarch-funding organization to issue grants for specifically replicating results of existing papers? Why don't they do it? May be they do and I'm just unaware?
- countrybama24 11y ago
  They don't because it's not considered cool.
- Outdoorsman 11y ago
  There are organizations that are attempting to police published research...one such is Retraction Watch... It's a 501(c)3, thus a not-for profit...not sure how much grant funding they receive, but they do actively solicit donations from the public... I think you'd be surprised at the number of public research papers, including some that end up being sources for articles in well-known science journals, that are retracted each year... A list of papers retracted, which is continually updated, can be found at their site: http://retractionwatch.com/ http://retractionwatch.com/
jwmerrill 11y ago
> The model he has in mind is the e-print distribution platform arXive. > Other disciplines are emulating the arXive pre-print model > ...something like a gigantic open-source version of arXive for all scientific research That's not how you spell arXiv.
fizixer 11y ago
Ladies and gentlemen, Sturgeon's law at work.
tokenadult 11y ago
I like this summary of current research on how to do research better. A checklist of suggestions for psychology research journal editors[1] gives a different set of links to current research and current best practice, and is well worth a read too. [1] https://jcoynester.wordpress.com/2016/01/21/ten-suggestions-to-the-new-associate-editors-of-psychological-science/ https://jcoynester.wordpress.com/2016/01/21/ten-suggestions-...
imh 11y ago
>We are living in a time of technological marvels, with advances like CRISPR gene-editing being used to bring back extinct mammoths; lithium-air batteries that store 10 times more energy than conventional lithium-ion batteries; mitochondrial transfers that create healthy babies who have three genetic parents; Ebola vaccines that are nearly 100 percent effective; and cars that drive themselves. This line was sort of ironic, since we don't really have those things. Mammoths are still extinct, lithium-air batteries aren't yet practical, three parent babies aren't legally deemed as safe in most countries, and cars can only drive themselves in limited ideal conditions (and those are only research vehicles!). Was that statement was put in as another example of how you can't believe what you read and it went over my head?
- tim333 11y ago
  I took that line as saying that it was remarkable that those things are happening at all rather than saying they are safe, practical and fully functional. I mean I'm impressed that they have actually inserted mammoth DNA in to elephant cells even if Jurassic Park will have to wait a bit. (http://www.livescience.com/50275-bringing-back-woolly-mammoth-dna.html http://www.livescience.com/50275-bringing-back-woolly-mammot...)

67 points by nkurz 11y ago

Why are people so obsessed with publication bias and statistical power? Those are just two of the dozens of reasons why published research is unreliable, and it's not even clear that they're the most important ones. It seems like the skeptic community just randomly glommed onto those issues in the 90s or something and haven't updated their worldview since.

467568985476 11y ago
What are the other, more important reasons that research is unreliable?
- Alex3917 11y ago
  I have a list of them going here, but it's definitely not comprehensive either: http://www.alexkrupp.com/Citevault.html#pharmaceuticals http://www.alexkrupp.com/Citevault.html#pharmaceuticals Even if you look just at statistical analysis, there are dozens of different issues. E.g. I seem to remember some articles about how basically 100% of research involving brain scans is incorrect, because of a mix of statistical problems and also issues related to the imaging software.
  - tcj_phx 11y ago
    This section is particularly useful to me: http://www.alexkrupp.com/Citevault.html#mentalhealth http://www.alexkrupp.com/Citevault.html#mentalhealth Thanks!
- aschreyer 11y ago
  A big problem are the perverse incentives researchers are facing. The only thing that matters is publishing in prestigious journals, otherwise your career is pretty much over (no real chance to get tenure). Universities often don't even check the publications, and researchers usually omit the actual title of the publication and only keep the name of the journal. Moreover, if you want to get tenure you need to work on "hot" topics (as in CRISPR at the moment or cancer in general), meaning you don't necessarily spend too much time on one subject. I'm sure researchers are often aware that the research results are not ready for publication yet, but it's either publish or perish, and you usually get away with publishing sub-par results because no one really follows up on them either. Cancer and Alzheimer are two fields were billions have been wasted by research grants and pharma companies because the initially promising drug targets for example could not really be validated. I'm afraid this will stay that way as long as the career dilemma researchers are facing does not change.
jessaustin 11y ago
I suppose I haven't read enough about publication bias to get as annoyed by it as you seem. TFA offers a pretty convincing argument that publication bias is bad enough to doom most current scientific efforts to irrelevance. That sounds like something we should talk about? Or we can talk about more important reasons why published research is unreliable, if you care to mention them. What is "the skeptic community"? Is Reason part of it? I'm a bit skeptical that's even a thing... Statistical power is mentioned once in TFA and then forgotten. Amusingly enough, it gets the relevance of statistical power for the subject under discussion backwards. That graf might actually have been added by an "obsessed" editor.
- orblivion 11y ago
  Reason is generally a libertarian publication and website. For the Skeptic Community go to r/skeptic. They're related to the Atheist Community.
  - akiselev 11y ago
    So by "skeptic community" you actually mean "reddit skeptic community?"
    - deleted 11y ago
      [deleted]
    - Alex3917 11y ago
      There are skeptic conferences, skeptic magazines, skeptic books, skeptic authors, skeptic web forums, skeptic TV shows, etc. I think this easily meets pretty much any definition of community.
      - jessaustin 11y ago
        This seems to be focused toward critiques of religion rather than of science? I don't see the connection.

jessaustin 11y ago

"HARKing" is a great coinage I had not seen before.

mhkool 11y ago

In double-blind tests, one deliberately changes only one variable and compares/observes two groups where individual do not know whether they take a new drug or a placebo. If the drug has a positive effect, the double-blind test is an accepted method of proof, but unfortunately this limits our options severely. If cancer can be cured by changing _two_ variables, we will never find a cure. NEVER! Remember that we already do research for 50+ years and have spent trillions, so I tend to think that Einstein was right: it is insane to repeat the same experiments and to expect a different outcome. Let's jump to Alzheimer, a feared disease with also a lot of research and not so good drugs. Dr Bredesen has made a protocol with 35 variables to cure Alzheimer. And in his first test, he reversed Alzheimer in 9 out of 10 patients. The scientific purists say that there is no double-blind study, so no proof. I am convinced that Dr Bredesen in on the right path, not only because of the results but also because of his reasoning. The treatment has no drug, but a health optimization in all possible ways. And then the body heals itself in 4 months. Please do not comment with "that cannot be true" unless you proof that it cannot be true (I do not belief that you can provide proof). So the state of current research methods is weak. Medical research only focuses on one variable, one drug, that will cure a disease. The results of this way of doing medical research are VERY disappointing. Trillions are spent and no drugs that cure cancer, AIDS or Alzheimer have been found. Dr Bredesen has chosen a different path and I support him.

orblivion 11y ago
So if you have an X variable cure, that means that an X-1 variable cure is not enough, right? I'm not very familiar with the nuances of the scientific process, but I would think that you could still run tests on one variable at a time. Just turn all but one variable "on", make that your control group. Turn all variables on, that's your experiment group. Now repeat the process isolating each variable, determining whether they're all in fact necessary for the cure.
mabcat 11y ago
Unfortunately you have a fundamental misunderstanding of health experimental design. Double-blind means that neither the participants nor the experimenters know which participant is allocated to which group. It doesn't say anything about how many groups there are or what is done to each group. I haven't heard of Dr Bredesen, but they could certainly run a double-blind study to validate their treatment regime. Treatments that haven't been validated in this way have a high risk of being ineffective but appearing effective due to experimenter error, placebo effect, demand characteristics, etc. Requiring convincing evidence in the form of a double-blind trial isn't purism, it's sensible caution based on experience. edit: you're also on shaky ground statistically. There's a rule of thumb that if interactions between several variables are present in a treatment effect, 70% of that effect will show up on the individual variables (as main effects). It's very unlikely that there exists a treatment where all the effect is on a 35-variable interaction and none of the effect is visible when looking at each variable singly.

mwsherman 11y ago

This is largely predicted by Daniel Kahneman and Nassim Taleb, would love to see them cited more often.

golergka 11y ago

What it would take a resarch-funding organization to issue grants for specifically replicating results of existing papers? Why don't they do it? May be they do and I'm just unaware?

countrybama24 11y ago
They don't because it's not considered cool.
Outdoorsman 11y ago
There are organizations that are attempting to police published research...one such is Retraction Watch... It's a 501(c)3, thus a not-for profit...not sure how much grant funding they receive, but they do actively solicit donations from the public... I think you'd be surprised at the number of public research papers, including some that end up being sources for articles in well-known science journals, that are retracted each year... A list of papers retracted, which is continually updated, can be found at their site: http://retractionwatch.com/ http://retractionwatch.com/

jwmerrill 11y ago

> The model he has in mind is the e-print distribution platform arXive. > Other disciplines are emulating the arXive pre-print model > ...something like a gigantic open-source version of arXive for all scientific research That's not how you spell arXiv.

fizixer 11y ago

Ladies and gentlemen, Sturgeon's law at work.

tokenadult 11y ago

I like this summary of current research on how to do research better. A checklist of suggestions for psychology research journal editors[1] gives a different set of links to current research and current best practice, and is well worth a read too. [1] https://jcoynester.wordpress.com/2016/01/21/ten-suggestions-to-the-new-associate-editors-of-psychological-science/ https://jcoynester.wordpress.com/2016/01/21/ten-suggestions-...

imh 11y ago

>We are living in a time of technological marvels, with advances like CRISPR gene-editing being used to bring back extinct mammoths; lithium-air batteries that store 10 times more energy than conventional lithium-ion batteries; mitochondrial transfers that create healthy babies who have three genetic parents; Ebola vaccines that are nearly 100 percent effective; and cars that drive themselves. This line was sort of ironic, since we don't really have those things. Mammoths are still extinct, lithium-air batteries aren't yet practical, three parent babies aren't legally deemed as safe in most countries, and cars can only drive themselves in limited ideal conditions (and those are only research vehicles!). Was that statement was put in as another example of how you can't believe what you read and it went over my head?

tim333 11y ago
I took that line as saying that it was remarkable that those things are happening at all rather than saying they are safe, practical and fully functional. I mean I'm impressed that they have actually inserted mammoth DNA in to elephant cells even if Jurassic Park will have to wait a bit. (http://www.livescience.com/50275-bringing-back-woolly-mammoth-dna.html http://www.livescience.com/50275-bringing-back-woolly-mammot...)