7 ms·

Why Most Published Research Findings Are False (2005)

142 points by NoB4Mouth 8y ago

stochastic_monk 8y ago
Please add a [2005] tag.
- deleted 8y ago
  [deleted]
- sctb 8y ago
  Done. Thanks!
  - joe_the_user 8y ago
    It's a reprint of an article in PlosMed, where the context is obvious in the original but not here, so the title might need changing too.
reasonattlm 8y ago
This is why one has to read papers in context. You can't just read a paper: you have to then go off and read some reviews that cover the surrounding science. Then read some other papers on the topic of interest. Pay attention to dates on papers, as opinions change with time. Note the disagreements. Reading the scientific literature is reading the moment to moment output of a noisy search algorithm. Any given publication is near meaningless on its own. This business of reading the context for a specific item of research isn't that hard. If as a layperson you feel up to having an opinion on a specific paper, then you are certainly equipped to do more reading in the field. Start with review papers, which tend to be a gentler uphill slope, and then fit other papers into what you see there. Take note of the disagreements between reviews, the different emphasis placed on different aspects of the topic. In most scientific fields review papers are usually pretty good at explicitly covering the unknowns and debates of interest. The subtexts and unwritten stuff, such as funding-driven conflicts of research strategy, take longer to figure out. But one has to start somewhere. https://www.fightaging.org/archives/2009/05/how-to-read-the-output-of-the-scientific-method/ https://www.fightaging.org/archives/2009/05/how-to-read-the-... "The scientific community doesn't produce an output of nice, neat tablets of truth, pronouncements come down from the mountain. It produces theories that are then backed by varying weights of evidence: a theory with a lot of support stands until deposed by new results. But it's not that neat in practice either. The array of theories presently in the making is a vastly complex and shifting edifice of debate, contradictory research results, and opinion. You might compare the output of the scientific community in this sense with the output of a financial market: a staggeringly varied torrent of data that is confusing and overwhelming to the layperson, but which - when considered in aggregate - more clearly shows the way to someone who has learned to read the ticker tape."
- TeMPOraL 8y ago
  An interesting perspective, but it doesn't really dissolve the big problems - that the "noise" in the output of this "search algorithm" is absurdly high, and that single papers often drive real-world decisionmaking.
- goatlover 8y ago
  > The array of theories presently in the making is a vastly complex and shifting edifice of debate, contradictory research results, and opinion. You might compare the output of the scientific community in this sense with the output of a financial market: a staggeringly varied torrent of data that is confusing and overwhelming to the layperson, but which - when considered in aggregate - more clearly shows the way to someone who has learned to read the ticker tape. Good point, but the problem is the media and public opinion often don't recognize this. So some research result that gains notoriety is treated as the truth from the mountain top without this context.
mindfulplay 8y ago
This only adds credence to the fact that social survey is not a science. Most of it is glorified door to door salespeople work. It's not even research. We have to draw a clear line soon to prevent good-intentioned people from being lumped into shitty science.
majidazimi 8y ago
The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those absolutely stunning results were just intelligently crafted experiments that are not applicable elsewhere. Basically, my whole thesis is just to show that some paper was wrong and the idea is not applicable in another experiment.
- chrisfinazzo 8y ago
  Not surprised to hear this at all. I can remember being specifically told during both undergrad and grad research seminars 'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'. In social sciences, this is an astoundingly common problem. Then, of course, when I have to defend my comp. exam (for a second time) in front of the faculty, I spent the last 5 minutes lamenting the very phenomenon you described. Research can be deeply engrossing, but it doesn't mean shit when you go out into the world looking to land your next job. Of my 3 reviewers, only 1 had spent significant time in industry before teaching so, think of this as a wake up call for people to get outside the bubble and broaden their horizons once in a while. Tl;dr The program I was (Communications) in changed their requirements right after I left to eliminate this distinction and now students are required to develop and implement an idea (Marketing campaign, Training seminar, Comms. Audit w/ recommendations, etc.). You might get, 'that's nice' or even 'interesting', but this is usually followed up with 'how can you help me solve problems x and y right now on deadline?'
  - nonbel 8y ago
    >"'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'." This is just a tautology. In social science large sample size = statistical significance (there are only false negatives since the null model being tested is always wrong).
tensor 8y ago
It's worth noting that this is talking about research that involves sampling groups of people, or other similar sampling approaches. It is not talking about mathematical results which you can often find in computer science, physics, and elsewhere. Not is it necessarily true all of the physical sciences. It also shouldn't be read as "science is broken and wrong so therefore my opinion should be considered equally." There is definitely a problem with accuracy in many scientific fields that needs to be addressed, but the baby doesn't need to be thrown out with the bath water.
- bad_user 8y ago
  While I agree with the general sentiment that not all science is equal, it’s important to remember that math isn’t science. Science is empirical and involves usage of the _scientific method_ for acquisition of knowledge and math is disqualified. Of course math sits at the foundation of the sciences we are talking about. And truly _proving_ something involves math. That’s not what we are talking about though. And computer science is just like math. It’s not empirical, it’s not characterized by the scientific method. It’s not science either.
  - yesenadam 8y ago
    Beware of bugs in the above code; I have only proved it correct, not tried it. – Donald Knuth How did people learn about the nature of, say, 1D cellular automata, or the Mandelbrot set? By experimenting, classifying, observing, testing hypotheses etc.
- epistasis 8y ago
  It's not even talking about biology research, or even most medically related research. While it was a great point to make at the time, far too much has been made of it. Yes, don't p-hack, but it's also better to publish data than to withold it just because there wasn't a positive result. We need better publication mechanisms for data that doesn't have any significant findings.
  - astazangasta 8y ago
swagatkonchada 8y ago
Maybe not in mathematics
- duckerude 8y ago
  >Anecdotal evidence suggests that as many as a third of all papers published in mathematical journals contain mistakes - not just minor errors, but incorrect theorems and proofs… http://www.gwern.net/The-Existential-Risk-of-Mathematical-Error http://www.gwern.net/The-Existential-Risk-of-Mathematical-Er...
serghiou 8y ago
Super important paper, the implications of which have been corroborated repeatedly within the most prestigious publications in psychology, social sciences/economics and cancer biology. If you'd like to read more about such issues and actually work on doing something about it, you may want to check out the Reddit community (https://www.reddit.com/r/metaresearch/ https://www.reddit.com/r/metaresearch/) and a recent initiative at Stanford (http://reproduciblescience.stanford.edu/ http://reproduciblescience.stanford.edu/).
xte 8y ago
The problem have a name: business. If research is public, well founded by government, and universities are public entity research is accurate and effective, otherwise it's only a matter of making money quickly and moving on.

142 points by NoB4Mouth 8y ago

stochastic_monk 8y ago

Please add a [2005] tag.

deleted 8y ago
[deleted]
sctb 8y ago
Done. Thanks!
- joe_the_user 8y ago
  It's a reprint of an article in PlosMed, where the context is obvious in the original but not here, so the title might need changing too.

reasonattlm 8y ago

This is why one has to read papers in context. You can't just read a paper: you have to then go off and read some reviews that cover the surrounding science. Then read some other papers on the topic of interest. Pay attention to dates on papers, as opinions change with time. Note the disagreements. Reading the scientific literature is reading the moment to moment output of a noisy search algorithm. Any given publication is near meaningless on its own. This business of reading the context for a specific item of research isn't that hard. If as a layperson you feel up to having an opinion on a specific paper, then you are certainly equipped to do more reading in the field. Start with review papers, which tend to be a gentler uphill slope, and then fit other papers into what you see there. Take note of the disagreements between reviews, the different emphasis placed on different aspects of the topic. In most scientific fields review papers are usually pretty good at explicitly covering the unknowns and debates of interest. The subtexts and unwritten stuff, such as funding-driven conflicts of research strategy, take longer to figure out. But one has to start somewhere. https://www.fightaging.org/archives/2009/05/how-to-read-the-output-of-the-scientific-method/ https://www.fightaging.org/archives/2009/05/how-to-read-the-... "The scientific community doesn't produce an output of nice, neat tablets of truth, pronouncements come down from the mountain. It produces theories that are then backed by varying weights of evidence: a theory with a lot of support stands until deposed by new results. But it's not that neat in practice either. The array of theories presently in the making is a vastly complex and shifting edifice of debate, contradictory research results, and opinion. You might compare the output of the scientific community in this sense with the output of a financial market: a staggeringly varied torrent of data that is confusing and overwhelming to the layperson, but which - when considered in aggregate - more clearly shows the way to someone who has learned to read the ticker tape."

TeMPOraL 8y ago
An interesting perspective, but it doesn't really dissolve the big problems - that the "noise" in the output of this "search algorithm" is absurdly high, and that single papers often drive real-world decisionmaking.
goatlover 8y ago
> The array of theories presently in the making is a vastly complex and shifting edifice of debate, contradictory research results, and opinion. You might compare the output of the scientific community in this sense with the output of a financial market: a staggeringly varied torrent of data that is confusing and overwhelming to the layperson, but which - when considered in aggregate - more clearly shows the way to someone who has learned to read the ticker tape. Good point, but the problem is the media and public opinion often don't recognize this. So some research result that gains notoriety is treated as the truth from the mountain top without this context.

mindfulplay 8y ago

This only adds credence to the fact that social survey is not a science. Most of it is glorified door to door salespeople work. It's not even research. We have to draw a clear line soon to prevent good-intentioned people from being lumped into shitty science.

majidazimi 8y ago

The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those absolutely stunning results were just intelligently crafted experiments that are not applicable elsewhere. Basically, my whole thesis is just to show that some paper was wrong and the idea is not applicable in another experiment.

chrisfinazzo 8y ago
Not surprised to hear this at all. I can remember being specifically told during both undergrad and grad research seminars 'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'. In social sciences, this is an astoundingly common problem. Then, of course, when I have to defend my comp. exam (for a second time) in front of the faculty, I spent the last 5 minutes lamenting the very phenomenon you described. Research can be deeply engrossing, but it doesn't mean shit when you go out into the world looking to land your next job. Of my 3 reviewers, only 1 had spent significant time in industry before teaching so, think of this as a wake up call for people to get outside the bubble and broaden their horizons once in a while. Tl;dr The program I was (Communications) in changed their requirements right after I left to eliminate this distinction and now students are required to develop and implement an idea (Marketing campaign, Training seminar, Comms. Audit w/ recommendations, etc.). You might get, 'that's nice' or even 'interesting', but this is usually followed up with 'how can you help me solve problems x and y right now on deadline?'
- nonbel 8y ago
  >"'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'." This is just a tautology. In social science large sample size = statistical significance (there are only false negatives since the null model being tested is always wrong).

tensor 8y ago

It's worth noting that this is talking about research that involves sampling groups of people, or other similar sampling approaches. It is not talking about mathematical results which you can often find in computer science, physics, and elsewhere. Not is it necessarily true all of the physical sciences. It also shouldn't be read as "science is broken and wrong so therefore my opinion should be considered equally." There is definitely a problem with accuracy in many scientific fields that needs to be addressed, but the baby doesn't need to be thrown out with the bath water.

bad_user 8y ago
While I agree with the general sentiment that not all science is equal, it’s important to remember that math isn’t science. Science is empirical and involves usage of the _scientific method_ for acquisition of knowledge and math is disqualified. Of course math sits at the foundation of the sciences we are talking about. And truly _proving_ something involves math. That’s not what we are talking about though. And computer science is just like math. It’s not empirical, it’s not characterized by the scientific method. It’s not science either.
- yesenadam 8y ago
  Beware of bugs in the above code; I have only proved it correct, not tried it. – Donald Knuth How did people learn about the nature of, say, 1D cellular automata, or the Mandelbrot set? By experimenting, classifying, observing, testing hypotheses etc.
epistasis 8y ago
It's not even talking about biology research, or even most medically related research. While it was a great point to make at the time, far too much has been made of it. Yes, don't p-hack, but it's also better to publish data than to withold it just because there wasn't a positive result. We need better publication mechanisms for data that doesn't have any significant findings.
- astazangasta 8y ago

swagatkonchada 8y ago

Maybe not in mathematics

duckerude 8y ago
>Anecdotal evidence suggests that as many as a third of all papers published in mathematical journals contain mistakes - not just minor errors, but incorrect theorems and proofs… http://www.gwern.net/The-Existential-Risk-of-Mathematical-Error http://www.gwern.net/The-Existential-Risk-of-Mathematical-Er...

serghiou 8y ago

Super important paper, the implications of which have been corroborated repeatedly within the most prestigious publications in psychology, social sciences/economics and cancer biology. If you'd like to read more about such issues and actually work on doing something about it, you may want to check out the Reddit community (https://www.reddit.com/r/metaresearch/ https://www.reddit.com/r/metaresearch/) and a recent initiative at Stanford (http://reproduciblescience.stanford.edu/ http://reproduciblescience.stanford.edu/).

xte 8y ago

The problem have a name: business. If research is public, well founded by government, and universities are public entity research is accurate and effective, otherwise it's only a matter of making money quickly and moving on.