4 ms·

Probably false? As in you have a better chance claiming the negation of a scientific paper's conclusion than the actual conclusion? I doubt it.

deleted 13y ago
[deleted]
bigchewy 13y ago
yes, probably false. This is not news, e.g. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ came out in 2005. of course, research "proving" that research tends to be false can get us into a little circular game...
- a_bonobo 13y ago
  There's a follow-up paper to the one you linked that claims that much less published research findings are false: >We estimate that the overall rate of false discoveries among reported results is 14% (s.d. 1%), contrary to previous claims. We also found that there is no a significant increase in the estimated rate of reported false discovery results over time (0.5% more false positives (FP) per year, P=0.18) or with respect to journal submissions (0.5% more FP per 100 submissions, P=0.12). [1] http://biostatistics.oxfordjournals.org/content/early/2013/09/24/biostatistics.kxt007 http://biostatistics.oxfordjournals.org/content/early/2013/0... The circle has begun :)
  - capnrefsmmat 13y ago
    You should also read the responses to that article, such as Ioannidis's, which is scathing. They're all open-access, thankfully, and you can find them here: http://simplystatistics.org/2013/09/25/is-most-science-false-the-titans-weigh-in/ http://simplystatistics.org/2013/09/25/is-most-science-false... Jager and Leek put their code on GitHub, so the commenters were able to review their code and tinker with it to see what would happen.
    - a_bonobo 13y ago
      Thanks for all these links! I skimmed them, and it's fun to see that two of them rip into Ionnadis' work, while Ionnadis rips into the Jager/Leek work... It's also a cool exercise in reproducible research and open peer review, both of which are far from common. In my opinion, the entire exercise of data-mining the published literature is pretty much futile. We already know there are problems in the published literature and that scientists are pretty mediocre at statistics. Pin-pointing the exact value of how mediocre only leads to, as your link shows, a mountain of published works, hurt feelings (on Ionnadis side I guess) and doesn't solve anything.
mjn 13y ago
In certain fields, that's not implausible. Many results are along the lines of "compound X is effective against disease Y", where the negation, "compound X is not effective against disease Y", is a reasonable baseline assumption, because most compounds are not effective against most diseases. Results where the prior odds could plausibly be considered 50/50 are another story. Scientific research that doesn't take the form "rule out the null hypothesis with p values" is also a different matter (lots of CS and physics, among other fields, has a more complex mingling of theory and epistemology than experiment/nullhypothesis/pvalue/repeat).
- jamesaguilar 13y ago
  That's a good point. I didn't consider the fact that possibly many discoveries have a very low prior expectation of truthood.
zzleeper 13y ago
I wouldn't be so sure. Not talking about mathematics or CS, but in many social sciences (and also medicine and so on) the paper is just stating that A->B but i) there can be a lot more things going on that explain whatever correlation you are finding (from reverse causality to bad experimental setup), and *more importantly ii) in order to get published, you need to present a somewhat interesting or controversial statement. If you state something obvious and most likely true, good luck getting to Nature/Science. If you state something unusual, then that will sell (where selling is getting citations) so you will see it published
- XorNot 13y ago
  Asimov wrote an excellent piece on this - The Relativity of Wrong: http://chem.tufts.edu/answersinscience/relativityofwrong.htm http://chem.tufts.edu/answersinscience/relativityofwrong.htm The Earth isn't flat, but it certainly approximates being flat for small measurements of its surface (such as those early civilizations would've been able to make).

deleted 13y ago

[deleted]

bigchewy 13y ago

yes, probably false. This is not news, e.g. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ came out in 2005. of course, research "proving" that research tends to be false can get us into a little circular game...

a_bonobo 13y ago
There's a follow-up paper to the one you linked that claims that much less published research findings are false: >We estimate that the overall rate of false discoveries among reported results is 14% (s.d. 1%), contrary to previous claims. We also found that there is no a significant increase in the estimated rate of reported false discovery results over time (0.5% more false positives (FP) per year, P=0.18) or with respect to journal submissions (0.5% more FP per 100 submissions, P=0.12). [1] http://biostatistics.oxfordjournals.org/content/early/2013/09/24/biostatistics.kxt007 http://biostatistics.oxfordjournals.org/content/early/2013/0... The circle has begun :)
- capnrefsmmat 13y ago
  You should also read the responses to that article, such as Ioannidis's, which is scathing. They're all open-access, thankfully, and you can find them here: http://simplystatistics.org/2013/09/25/is-most-science-false-the-titans-weigh-in/ http://simplystatistics.org/2013/09/25/is-most-science-false... Jager and Leek put their code on GitHub, so the commenters were able to review their code and tinker with it to see what would happen.
  - a_bonobo 13y ago
    Thanks for all these links! I skimmed them, and it's fun to see that two of them rip into Ionnadis' work, while Ionnadis rips into the Jager/Leek work... It's also a cool exercise in reproducible research and open peer review, both of which are far from common. In my opinion, the entire exercise of data-mining the published literature is pretty much futile. We already know there are problems in the published literature and that scientists are pretty mediocre at statistics. Pin-pointing the exact value of how mediocre only leads to, as your link shows, a mountain of published works, hurt feelings (on Ionnadis side I guess) and doesn't solve anything.

mjn 13y ago

In certain fields, that's not implausible. Many results are along the lines of "compound X is effective against disease Y", where the negation, "compound X is not effective against disease Y", is a reasonable baseline assumption, because most compounds are not effective against most diseases. Results where the prior odds could plausibly be considered 50/50 are another story. Scientific research that doesn't take the form "rule out the null hypothesis with p values" is also a different matter (lots of CS and physics, among other fields, has a more complex mingling of theory and epistemology than experiment/nullhypothesis/pvalue/repeat).

jamesaguilar 13y ago
That's a good point. I didn't consider the fact that possibly many discoveries have a very low prior expectation of truthood.

zzleeper 13y ago

I wouldn't be so sure. Not talking about mathematics or CS, but in many social sciences (and also medicine and so on) the paper is just stating that A->B but i) there can be a lot more things going on that explain whatever correlation you are finding (from reverse causality to bad experimental setup), and *more importantly ii) in order to get published, you need to present a somewhat interesting or controversial statement. If you state something obvious and most likely true, good luck getting to Nature/Science. If you state something unusual, then that will sell (where selling is getting citations) so you will see it published

XorNot 13y ago
Asimov wrote an excellent piece on this - The Relativity of Wrong: http://chem.tufts.edu/answersinscience/relativityofwrong.htm http://chem.tufts.edu/answersinscience/relativityofwrong.htm The Earth isn't flat, but it certainly approximates being flat for small measurements of its surface (such as those early civilizations would've been able to make).