7 ms·
Most Published Research Findings Are False (2005)
- lainon 8y ago(2005)
- acqq 8y agoNeeds: [2005]
- chrisseaton 8y agoIncluding this one?
- jonathanstrange 8y agoSince I've seen this paper mentioned a few times now on several different sites and heard varying replies, I'd be very interested in informed opinions about this. I've taken a look at it but don't consider my skill level in statistics high enough to assess the paper on my own.
- vertline3 8y agoCan you elaborate as to why? Snark is not really the goal here.
- chrisseaton 8y ago> Can you elaborate as to why? How can I elaborate on answer that I don't have?
- vertline3 8y agoI think just your first post is a snappy reply to the headline, but doesn't seem to address the content of the paper in any way, so I was interested in your specific thoughts about the content of the paper.
- YorkshireSeason 8y agoIt's a perfectly valid, and indeed important question. Self-application is an important form of scientific methodology. Think e.g. of programming languages development: one of the first things you do is bootstrap a compiler.
- sporkologist 8y agoYou're conflating science publishing philosophy with standard software development?
- YorkshireSeason 8y agoProgramming language development is not standard software development. PL research is a discipline with no really feasible empirical methods for robust and replicable evaluation of programming language ideas. Using self-application is a test of internal coherence -- if your new language is not good at writing a compiler, then maybe it isn't such a good idea. Self-application is a well-known scientific methodology. Would you trust a physical theory that predicted the impossibility of physicists?
- asianthrowaway 8y agoIf this publication is false, its premise is true, which would mean the publication is true?
- lifeformed 8y agoThis sentence is false.
- mensetmanusman 8y agoOnly Gödel can save us now.
- devgutt 8y agoI think it is more like "This is not a sentence."
- carlmr 8y agoIf this publication is false, it's premise may or may not be true, which means we don't know anything.
- tomp 8y agoIsn't this just applying a model (i.e. it's not empirical, just theoretical)?
- buboard 8y agoGenerally, no. The article was published in PLoS Med, and concerns generally medical research studies. It has been more than 10 years and i think most people still don't disagree with it. Here is the authors response to criticisms: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1297555/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1297555/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1896210/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1896210/
- simplystats2 8y agoA couple of other articles of interest: An empirical estimate: https://academic.oup.com/biostatistics/article/15/1/1/244509 https://academic.oup.com/biostatistics/article/15/1/1/244509 And a review with some context: https://www.biorxiv.org/content/10.1101/050575v1 https://www.biorxiv.org/content/10.1101/050575v1
- chmike 8y agoAs stated, this research finding my be false too ! ;)
- samuell 8y agoA nature survey from 2016 among 1500 scientists, revealing a widespread consensus that there exists a reproducibility crisis in science: https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 https://www.nature.com/news/1-500-scientists-lift-the-lid-on...
- jshowa3 8y agoI generally agree with this. The publication game tends to focus on exciting new discoveries way more than reproduced ones. I think this is because universities push you to focus on "original ideas" when publishing theses instead of reproducing other ideas because the former is "harder" than the latter.
- johannesbeil 8y agoDo you think a Github for science would help?
- AlexTWithBeard 8y agoHaving an open, accessible repository of scientific data sounds like a good thing, but I doubt it would help the problem of reproducibility. The biggest problem of such a "github" would be that a scientific repository is only as good as its maintainer. In software development land we already see a lot of egos clashing over the code, but software has a clear "works/doesn't work" criteria. In many scientific areas such a criteria is hard to find and we'll have to rely on the personality of a repository maintainer. The repositories will soon become a brand on its own. Think "nature.github.com", "physrev.github.com" and "lancet.github.com".
- kerkeslager 8y agoIf you don't have a clear mechanism by which a startup could solve this problem, I suspect the result would be the same structures that already exist, but with better marketing and a Web 2.0 interface. If anything, this would make the problem worse, because it creates a company that now has a vested interest in keeping the status quo because it's part of the status quo, and that company will be manipulating politics and public opinion with its marketing. A second problem with this sort of company is that it often doesn't really understand the industry it's "disrupting", so they end up not solving a lot of the problems which older solutions solved. At a more fundamental level, I think that a company is always going to be motivated to make money rather than solve a problem. Even if intentions start off well, the people in the company will believe that the company is good and needed, so they'll make choices that give up a little of the original purpose of the company to make the company survive, because if the company doesn't survive then it can't do any good, right? And after a bunch of one-degree deviations from its original direction, the company is going a completely different direction than its original purpose and isn't even geared toward solving the problem any more. Sure, "We're going to save the world" is great marketing, but at this point I simply don't ever believe that a startup is a solution to a major problem like this.
- 8y ago
- kerkeslager 8y agoI think the solution to this is that science publication needs to be done like science itself. Just as scientists commit to doing experiments without knowing what the results of the experiments will be, publishers need to commit to publishing experiments without knowing what the results of the experiments will be. Publishers need to be involved earlier in the process. Instead of submitting a completed research finding, scientists should apply to publish research before doing it, and if the topic of research is interesting to the publisher, both scientist and publisher should commit to publish in that journal, no matter what the results are. Additionally, publishers should commit to publish only something like 1/3 research into new topics, with the other 2/3 being publishing attempt to reproduce previous interesting results. I don't mean a slavish repetition of existing experiments, but taking a critical eye toward testing the same hypotheses with different (hopefully better) methodology.
- deleted 8y ago[deleted]
- buboard 8y agoScience doesn't seem to have fixed itself since. At this point it is prudent to shift the public's perception from "research is true" to "research is valid". To the extent in which public policy is dictated by research, it should be based, not on journal-published research, but perhaps on meta-analysis or third party research-on-research.
- rthomas6 8y agoBeware the man of one study: https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-study/ https://slatestarcodex.com/2014/12/12/beware-the-man-of-one-...
- aceon48 8y agoI learned this in middle school during the Science Fair. My experiment didn't work. Up against the deadline and not wanting a 0, I simply made up the data. I got a nice big A. Probably exactly what happens in the real world too.
- yesenadam 8y agoInside the Fake Science Factory (2018) is an amazing DEFCON talk, by 3 people who stumbled on and investigated an unbelievably huge worldwide network (400,000 scientists involved) of fake-scientific conferences, journals, websites raking in money and enabling people to make claims that their products are scientifically proven. The presenters got hackers to track down the people behind one of the largest organizations, and analyze the huge numbers of conferences/papers. They present/publish nonsensical talks on graphs and bees+cancer to see what peer review there is... A must watch. Very funny, if depressing. https://www.youtube.com/watch?v=ras_VYgA77Q https://www.youtube.com/watch?v=ras_VYgA77Q
- Certhas 8y agoMost code is buggy despite passing all tests. Publication is a sign of persistence, plausibility and a minimal quality check. Peer review improves articles, it does not guarantee correctness. That's also why one should be extremely careful about things that are not peer reviewed. Not because the bar is so high, but because it's so low.
- kolbe 8y agoJust because the bar is low doesn't mean its height is in any way related to its discernment of quality. Say, for example, the bar was simply based on the institution that the author is employed by. That may be a low bar in that anyone from Harvard can get a paper published, but it also doesn't actually tell us anything about the paper's validity. Whereas I wouldn't be skeptical of a good Rush University paper, simply because its authors weren't prestigious enough for Nature to pay attention to.
- Certhas 8y agoI was talking about getting your paper peer reviewed, not placing it in a particular journal. If you want to get into PLOS One I don't think your university matters very much.