5 ms·

One of the most eye-opening studies I saw was that social science papers with the most citations, had the highest likelihood of not being reproducible. Meanin

by q1w2 5y ago

One of the most eye-opening studies I saw was that social science papers with the most citations, had the highest likelihood of not being reproducible. Meaning that Social Sciences are actually pushing out misinformation more than not.

hannob 5y ago
This isn't an issue of social science. This is an issue that has been found in a diverse number of fields, basically everywhere where someone has looked: https://www.frontiersin.org/articles/10.3389/fnhum.2018.00037/full https://www.frontiersin.org/articles/10.3389/fnhum.2018.0003...
- gmadsen 5y ago
  its a little disingenuous to make it seem like all fields are equal. when that is clearly not the case
hulahoof 5y ago
Is this somehow a flow on effect of more commonly cited papers having more reproduction attempts?
- midjji 5y ago
  No, a moderately successful paper has 1-10 k citations, and usually 0-1 reproductions. Extremely successful papers have 10-40 k citations and 1-3 reproductions. They largely dont matter at all. Its mostly because extraordinary claims receive more attention, and thus is found by the group for whom the result supported their prior beliefs. They will use it whenever they need to support such. This is also why such studies tend to keep getting cited as evidence long after a long line of failures to replicate have been published. As the result is non-replicable, there also wont be a newer, improved version of the publication, meaning all cites go to the original.
- q1w2 5y ago
  The theory is that there is a pressure in certain academic fields to reinforce existing popular narratives with new publications, motivating researchers to confirm them, and thus for other researchers to cite them. ...but since such efforts attract low-quality researchers and are based on an implicit initial bias, due to its self-referential nature, that the results are not reproducible by others in the future since teams that try to reproduce results are usually more neutral on the topics.
Nasrudith 5y ago
That doesn't technically mean misinformation although but it does imply major room for improvement in methodology. If I roll a d20 and get a 7 and you roll it and get 13 that doesn't make the prior study inaccurate. The sizes are too low or not properly defined. But the combination of publish-or-perish and grant seeking pushing sensationalism isn't good for honesty let alone accuracy or precision.
nathan_compton 5y ago
"A good scientist, in other words, does not merely ignore conventional wisdom, but makes a special effort to break it. Scientists go looking for trouble. This should be the m.o. of any scholar, but scientists seem much more willing to look under rocks." -- Paul Graham.
- genousti 5y ago
diffeomorphism 5y ago
> Meaning that... No, that does not follow at all. But it seems very appropriate that you quote one statistic and then make a claim about a completely different one. Good exercise/example for the reader.

by q1w2 5y ago

hannob 5y ago

This isn't an issue of social science. This is an issue that has been found in a diverse number of fields, basically everywhere where someone has looked: https://www.frontiersin.org/articles/10.3389/fnhum.2018.00037/full https://www.frontiersin.org/articles/10.3389/fnhum.2018.0003...

gmadsen 5y ago
its a little disingenuous to make it seem like all fields are equal. when that is clearly not the case

hulahoof 5y ago

Is this somehow a flow on effect of more commonly cited papers having more reproduction attempts?

midjji 5y ago
No, a moderately successful paper has 1-10 k citations, and usually 0-1 reproductions. Extremely successful papers have 10-40 k citations and 1-3 reproductions. They largely dont matter at all. Its mostly because extraordinary claims receive more attention, and thus is found by the group for whom the result supported their prior beliefs. They will use it whenever they need to support such. This is also why such studies tend to keep getting cited as evidence long after a long line of failures to replicate have been published. As the result is non-replicable, there also wont be a newer, improved version of the publication, meaning all cites go to the original.
q1w2 5y ago
The theory is that there is a pressure in certain academic fields to reinforce existing popular narratives with new publications, motivating researchers to confirm them, and thus for other researchers to cite them. ...but since such efforts attract low-quality researchers and are based on an implicit initial bias, due to its self-referential nature, that the results are not reproducible by others in the future since teams that try to reproduce results are usually more neutral on the topics.

Nasrudith 5y ago

That doesn't technically mean misinformation although but it does imply major room for improvement in methodology. If I roll a d20 and get a 7 and you roll it and get 13 that doesn't make the prior study inaccurate. The sizes are too low or not properly defined. But the combination of publish-or-perish and grant seeking pushing sensationalism isn't good for honesty let alone accuracy or precision.

nathan_compton 5y ago

"A good scientist, in other words, does not merely ignore conventional wisdom, but makes a special effort to break it. Scientists go looking for trouble. This should be the m.o. of any scholar, but scientists seem much more willing to look under rocks." -- Paul Graham.

genousti 5y ago

diffeomorphism 5y ago

> Meaning that... No, that does not follow at all. But it seems very appropriate that you quote one statistic and then make a claim about a completely different one. Good exercise/example for the reader.