8 ms·

Prestigious Science Journals Struggle to Reach Even Average Reliability

82 points by pjf 9y ago

Vinnl 9y ago
Unfortunately, the main reason these journals get used is not because they are better at selecting reliability, but because they provide authors with the credentials needed to advance their careers... Edit: See also https://medium.com/flockademic/the-ridiculous-number-that-can-make-or-break-academic-careers-704e00ae070a https://medium.com/flockademic/the-ridiculous-number-that-ca...
- taneq 9y ago
  So runaway social proof, then? Sounds about right.
- jack6e 9y ago
  But the reason they are considered qualified to serve as credentialing authorities is, in part, due to their perceived reliability. Of course the significance of published work and its influence on the field is perhaps the primary driver of a journal's reputation for readers, but significant work can only be influential if it is also reliable. Were a journal's readership to realize that its publications could not be trusted, it would lose influence and thus the perception that having work published in it is a meaningful professional achievement.
  - eeeii93 9y ago
    I would like to see these sorts of analyses done over time, although I'm not sure it's possible (in the very least it would require a tremendous amount of work). Anecdotally in speaking to people who have been in academic research for awhile, things started changing in the late 90s, plus or minus several years. I've heard a similar story repeatedly and independently about different aspects of the system, but in aggregate there's some sense that there has been change in academic research culture that started around that time. I wonder if the relative quality signal of different types of journals has changed dramatically over time.
  - Vinnl 9y ago
    Is it, though? Are researchers even aware when previous research has been retracted? Retracted research often continues to amass new citations (which, of course, need not necessarily be positive). Although I would certainly not assert that research in the "top" journals is particularly unreliable, it is not unbelievable that, when they tend to publish more extraordinary result, a smaller percentage of that tends to be true. I have not seen any indication that that leads to such journals losing influence, and it certainly doesn't lead to the Impact Factor being lowered - which often influences the "perception that having work published in it is a meaningful professional achievement".
    - 2020-3030 9y ago
      The Retraction Watch website http://retractionwatch.com/ http://retractionwatch.com/ is one good method of highlighting retracted work. For researchers in any given field, keeping current on literature is hard work given the pace of publication but does include seeing retractions. So, the answer to your question depends on which researchers we mean. Experts in a field have the best chance of noticing relevant retractions in their own field. If you are a researcher in one field who has interests in many other fields, it is probably much less likely that you will be able to follow the retractions in those peripheral fields. That is why Retraction Watch and other mechanisms of highlighting such errors is useful.
- randcraw 9y ago
  I've always perceived the superiority of such journals to be: a) rigorous standards for whether the paper's methodology tests their hypothesis (valid science), b) high impact of the hypothesis on the field (utility). I don't think the OP disputes this, but it does corroborate the old saw, "To make extraordinary claims, extraordinary evidence is required". Thus perhaps the current standards for experimentation not only fail to be extraordinary, but fundamentally are inadequate.
  - Vinnl 9y ago
    I don't think there's much reason to believe the "top" journals to have far more rigorous standards than the bulk of "just good" journals (which I'm sure have rigorous standards as well). It is plausible, though, that the research with potentially most impact gets submitted to the "top" journals first, resulting in the "just good" journals never having the chance to publish them.
jimhefferon 9y ago
This must be mistaken? > While the number of scientists has been growing exponentially over the last decades, the number of journals with a large audience has not kept up, neither has the number of articles published per journal. Consequently, rejection rates at the most prestigious journals has fallen below 10% Do they mean acceptance rates?
- lyk 9y ago
  No, since there are fewer articles being submitted to the journals, they need to accept and publish more of them to meet their quotas.
- Vinnl 9y ago
  Yes, the author says so here: https://twitter.com/brembs/status/966046745854730241 https://twitter.com/brembs/status/966046745854730241
- jonathanstrange 9y ago
  Yes, it's a mistake. For example, in my area the top journals have rejection rates above 95%, acceptance rates below 5%.
tw1010 9y ago
"Struggle" would imply they're actually trying. People do what is incentivized, and as long as reliability is placed towards the bottom of the priority latter, it won't improve.
lokopodium 9y ago
It looks like the modern scientific framework just doesn't scale well. With paper output so great, it's nearly impossible for a researcher to sift through the haystack. Researchers resort to reading the papers from known groups only, which filters out most of the fluff, but perhaps leaves meaningful works from outsiders unnoticed.
- mncharity 9y ago
  > reading the papers from known groups only Known groups. Community gossip provides sometimes-essential context. "You can't trust person X on topic Y, because when he does Z, he sees what he wants to see." Not having access to that context, is another challenge faced by folks less connected.
truculation 9y ago
There are a huge number of bureaucratic hoops to jump through before you can get an article published. The flow chart is huge and gets more elaborate year by year. So if the quality is still poor then I think this simply means that we're doing it wrong. Either science isn't what we think it is or our heart isn't in it (or both). My 'method' of choosing what to read is simple: I follow people whose work I know and enjoy. I trust them. Actually I think that's pretty standard. For example, Leonardo gets published just by having left stuff lying around the place.
- Vinnl 9y ago
  I'm afraid that science indeed isn't what people usually think it is, and that there's a lot that could be improved. Following people whose work you enjoy generally works, but it also leads to potentially valuable work by lesser-known researchers (e.g. Early-career researchers) to go unnoticed.
- Retric 9y ago
  I think the expectations for novelty may be to high. Look at the new drugs list every year and you see a lot of churn with minor variations common, but less novelty than many assume. https://www.centerwatch.com/drug-information/fda-approved-drugs/year/2017 https://www.centerwatch.com/drug-information/fda-approved-dr... And this is with 100+ billion in public + private money world wide per year. Sure, there is plenty of stuff being discovered, but we can't expect a steady or increasing rate of progress across all areas.
  - Vinnl 9y ago
    That's the common explanation for why journals with a high Impact Factor have more retractions. Funders focus on "excellence", which is measured using the Impact Factor, which is higher when its articles are cited more, which happens more often if research is "novel" and spectacular, also known as not replicated and not in line with expectations, also known as more likely to be unreliable.
  - aaavl2821 9y ago
    The relationship between public research funding and drug approvals is actually pretty limited. The bottleneck is actually funding for translational research, i.e. The space after public funding but before the $200B+ in big pharma r&d kicks in. This area is called the "valley of death" and is where most science dies Venture funding is what gets drugs through the valley of death. There's a pretty good correlation between the types of drugs VC funds, the drugs big pharma companies buy from VC, and the drugs that get approved. The kinds of drugs that get approved are less correlated with areas of focus for public research spending Or diseases with most societal impact Wrote something on this last week: http://newbio.tech/blog/vc_basics_1.html http://newbio.tech/blog/vc_basics_1.html
aaavl2821 9y ago
The issue the article doesn't spend too much time on one of the major reasons why reliability is so low: there are too many confounding environmental variables to control for, so work done in one lab often doesn't generalize One seemingly standard, innocuous piece of equipment can make immune cells "aggressive", while a different brand can make them tolerant. No one knows why. Genetically identical mice ordered from one vendor can consistently have different disease progression compared to mice from another vendor. Background noise in animal facilities can mess with experimental outcomes and even cure disease. Two lots of otherwise identical reagents can act almost like different chemicals Many findings are the result of some set of confounding variables rather than the independent variable. It is impossible to know these things in advance or control for them. The best tactic is to do the same experiment in a different lab with a different research team to control for all these "other factors". Which is expensive (often six of seven figures) and can take months or even years. It is best practice when startups license tech from a university to ask for an exclusivity period where they reproduce the key experiments before committing to a full license Imagine that for every open source library you use, all you have is a detailed read me, a set of tests, and some infrequently commented white board code. You reconstruct the code as best you can (which isn't that hard), but for some reason the tests don't check out. The bug isn't with your software. You call the original researchers and find they used a few libraries that you didn't, and these didn't show up in the publication. You have to manually install all these libraries, test them, and integrate them. Still doesn't work. You realize they used a different os and have to account for that. After six months, still nothing. After visiting their office, you realize the computer used to write the software is on the floor on some very fuzzy carpet, and there is some weird electrostatic bug in the hardware that contributed to how their software performed This is perhaps not the best analogy, and I'm not a scientist myself though have managed scientific teams for years, managed a few tech transfers, and evaluated tons of science as potential investments, but I just wanted to drive home the point of how tough science can be. If others can critique this I would welcome it as it would be great for wet lab scientists and engineers to understand each other's worlds better Of course there is also fraud, cherry picking of data, and careful manipulation of study design to make externalities work in your favor, but a lot of the reliability / reproducibility crisis comes down to lots of uncontrollable / unknowable variables and expensive and long development cycles
contact_fusion 9y ago
I know this may be perceived as nitpicking, but I am generally annoyed whenever the reproducibility/reliability crisis is referred to as a problem in science, writ large, rather than a specific problem in specific scientific subfields. This seems to be particularly common when the articles are written by practitioners within these subfields, such as neuroscience, biology, or psychology. There is no reproducibility crisis in my field of science - astrophysics. To back this claim up, I searched at Retraction Watch for any mention of the top tier journals in astronomy & astrophysics - specifically, ApJ [1], MNRAS [2], A&A (none found), Icarus (none found), Nature Astronomy (none found) - and found exactly one correction and one retraction. Now certainly errata are published constantly, and I would be foolish to assume that just because Retraction Watch didn't catch many instances of scientific fraud or abuse within my field, that it practically doesn't exist. But no evidence for a "crisis" seems to exist, at least in my corner of science. Astrophysical research appears to be quite reliable and reproducible. For similar reasons, I haven't heard of a reproducibility crisis in analytical chemistry, or optical physics, or mathematics, to name a few - or, close to the interests of Hacker News, computer science. Feel free to correct me if I'm wrong. I'm not saying this to bash non-astronomers, or non-physicists, or specifically life scientists. Biologists, neuroscientists, and psychologists are crucial participants in the greater scientific enterprise and their efforts lead more directly to alleviating human suffering than my work ever will. But when talking about reproducibility and reliability problems we cannot conflate different scientific disciplines with vastly different cultures, practices, and norms - a crisis in biology does not imply that physics has one too. Physics may have its own problems, but they aren't the same as the ones in biology. Witness the rise of groups such as the Flat Earthers, who are rejecting basic scientific knowledge known for literal millenia.(Eratosthenes, anyone?) Or witness the anti-vaxxers, abandoning modern medicine for charlatanry. Pretending science is a monolithic enterprise and abandoning a fine-grained understanding of the validity and power of different types of scientific evidence just gives these movements strength, feeds their delusions, and weakens the prestige of scientists when we do need to stand together. [1] http://retractionwatch.com/category/by-journal/astrophysical-journal/ http://retractionwatch.com/category/by-journal/astrophysical... [2] http://retractionwatch.com/category/by-journal/monthly-notices-of-roy-astro-soc/ http://retractionwatch.com/category/by-journal/monthly-notic...
SagelyGuru 9y ago
When you read between the lines of these "productivity" academic career policies and measures, you realise that they are aimed squarely at fame rather than quality of science. I guess the argument goes something like this: when you get fame through your high profile journal publications, we will be able to attract more grants and more fee paying bums on seats. Who cares about what you have actually written? Under the current model, nobody really.

82 points by pjf 9y ago

Unfortunately, the main reason these journals get used is not because they are better at selecting reliability, but because they provide authors with the credentials needed to advance their careers... Edit: See also https://medium.com/flockademic/the-ridiculous-number-that-can-make-or-break-academic-careers-704e00ae070a https://medium.com/flockademic/the-ridiculous-number-that-ca...

taneq 9y ago
So runaway social proof, then? Sounds about right.
jack6e 9y ago
But the reason they are considered qualified to serve as credentialing authorities is, in part, due to their perceived reliability. Of course the significance of published work and its influence on the field is perhaps the primary driver of a journal's reputation for readers, but significant work can only be influential if it is also reliable. Were a journal's readership to realize that its publications could not be trusted, it would lose influence and thus the perception that having work published in it is a meaningful professional achievement.
- eeeii93 9y ago
  I would like to see these sorts of analyses done over time, although I'm not sure it's possible (in the very least it would require a tremendous amount of work). Anecdotally in speaking to people who have been in academic research for awhile, things started changing in the late 90s, plus or minus several years. I've heard a similar story repeatedly and independently about different aspects of the system, but in aggregate there's some sense that there has been change in academic research culture that started around that time. I wonder if the relative quality signal of different types of journals has changed dramatically over time.
- Vinnl 9y ago
  Is it, though? Are researchers even aware when previous research has been retracted? Retracted research often continues to amass new citations (which, of course, need not necessarily be positive). Although I would certainly not assert that research in the "top" journals is particularly unreliable, it is not unbelievable that, when they tend to publish more extraordinary result, a smaller percentage of that tends to be true. I have not seen any indication that that leads to such journals losing influence, and it certainly doesn't lead to the Impact Factor being lowered - which often influences the "perception that having work published in it is a meaningful professional achievement".
  - 2020-3030 9y ago
    The Retraction Watch website http://retractionwatch.com/ http://retractionwatch.com/ is one good method of highlighting retracted work. For researchers in any given field, keeping current on literature is hard work given the pace of publication but does include seeing retractions. So, the answer to your question depends on which researchers we mean. Experts in a field have the best chance of noticing relevant retractions in their own field. If you are a researcher in one field who has interests in many other fields, it is probably much less likely that you will be able to follow the retractions in those peripheral fields. That is why Retraction Watch and other mechanisms of highlighting such errors is useful.
randcraw 9y ago
I've always perceived the superiority of such journals to be: a) rigorous standards for whether the paper's methodology tests their hypothesis (valid science), b) high impact of the hypothesis on the field (utility). I don't think the OP disputes this, but it does corroborate the old saw, "To make extraordinary claims, extraordinary evidence is required". Thus perhaps the current standards for experimentation not only fail to be extraordinary, but fundamentally are inadequate.
- Vinnl 9y ago
  I don't think there's much reason to believe the "top" journals to have far more rigorous standards than the bulk of "just good" journals (which I'm sure have rigorous standards as well). It is plausible, though, that the research with potentially most impact gets submitted to the "top" journals first, resulting in the "just good" journals never having the chance to publish them.

jimhefferon 9y ago

This must be mistaken? > While the number of scientists has been growing exponentially over the last decades, the number of journals with a large audience has not kept up, neither has the number of articles published per journal. Consequently, rejection rates at the most prestigious journals has fallen below 10% Do they mean acceptance rates?

lyk 9y ago
No, since there are fewer articles being submitted to the journals, they need to accept and publish more of them to meet their quotas.
Vinnl 9y ago
Yes, the author says so here: https://twitter.com/brembs/status/966046745854730241 https://twitter.com/brembs/status/966046745854730241
jonathanstrange 9y ago
Yes, it's a mistake. For example, in my area the top journals have rejection rates above 95%, acceptance rates below 5%.

tw1010 9y ago

"Struggle" would imply they're actually trying. People do what is incentivized, and as long as reliability is placed towards the bottom of the priority latter, it won't improve.

lokopodium 9y ago

It looks like the modern scientific framework just doesn't scale well. With paper output so great, it's nearly impossible for a researcher to sift through the haystack. Researchers resort to reading the papers from known groups only, which filters out most of the fluff, but perhaps leaves meaningful works from outsiders unnoticed.

mncharity 9y ago
> reading the papers from known groups only Known groups. Community gossip provides sometimes-essential context. "You can't trust person X on topic Y, because when he does Z, he sees what he wants to see." Not having access to that context, is another challenge faced by folks less connected.

truculation 9y ago

There are a huge number of bureaucratic hoops to jump through before you can get an article published. The flow chart is huge and gets more elaborate year by year. So if the quality is still poor then I think this simply means that we're doing it wrong. Either science isn't what we think it is or our heart isn't in it (or both). My 'method' of choosing what to read is simple: I follow people whose work I know and enjoy. I trust them. Actually I think that's pretty standard. For example, Leonardo gets published just by having left stuff lying around the place.

Vinnl 9y ago
I'm afraid that science indeed isn't what people usually think it is, and that there's a lot that could be improved. Following people whose work you enjoy generally works, but it also leads to potentially valuable work by lesser-known researchers (e.g. Early-career researchers) to go unnoticed.
Retric 9y ago
I think the expectations for novelty may be to high. Look at the new drugs list every year and you see a lot of churn with minor variations common, but less novelty than many assume. https://www.centerwatch.com/drug-information/fda-approved-drugs/year/2017 https://www.centerwatch.com/drug-information/fda-approved-dr... And this is with 100+ billion in public + private money world wide per year. Sure, there is plenty of stuff being discovered, but we can't expect a steady or increasing rate of progress across all areas.
- Vinnl 9y ago
  That's the common explanation for why journals with a high Impact Factor have more retractions. Funders focus on "excellence", which is measured using the Impact Factor, which is higher when its articles are cited more, which happens more often if research is "novel" and spectacular, also known as not replicated and not in line with expectations, also known as more likely to be unreliable.
- aaavl2821 9y ago
  The relationship between public research funding and drug approvals is actually pretty limited. The bottleneck is actually funding for translational research, i.e. The space after public funding but before the $200B+ in big pharma r&d kicks in. This area is called the "valley of death" and is where most science dies Venture funding is what gets drugs through the valley of death. There's a pretty good correlation between the types of drugs VC funds, the drugs big pharma companies buy from VC, and the drugs that get approved. The kinds of drugs that get approved are less correlated with areas of focus for public research spending Or diseases with most societal impact Wrote something on this last week: http://newbio.tech/blog/vc_basics_1.html http://newbio.tech/blog/vc_basics_1.html

aaavl2821 9y ago

The issue the article doesn't spend too much time on one of the major reasons why reliability is so low: there are too many confounding environmental variables to control for, so work done in one lab often doesn't generalize One seemingly standard, innocuous piece of equipment can make immune cells "aggressive", while a different brand can make them tolerant. No one knows why. Genetically identical mice ordered from one vendor can consistently have different disease progression compared to mice from another vendor. Background noise in animal facilities can mess with experimental outcomes and even cure disease. Two lots of otherwise identical reagents can act almost like different chemicals Many findings are the result of some set of confounding variables rather than the independent variable. It is impossible to know these things in advance or control for them. The best tactic is to do the same experiment in a different lab with a different research team to control for all these "other factors". Which is expensive (often six of seven figures) and can take months or even years. It is best practice when startups license tech from a university to ask for an exclusivity period where they reproduce the key experiments before committing to a full license Imagine that for every open source library you use, all you have is a detailed read me, a set of tests, and some infrequently commented white board code. You reconstruct the code as best you can (which isn't that hard), but for some reason the tests don't check out. The bug isn't with your software. You call the original researchers and find they used a few libraries that you didn't, and these didn't show up in the publication. You have to manually install all these libraries, test them, and integrate them. Still doesn't work. You realize they used a different os and have to account for that. After six months, still nothing. After visiting their office, you realize the computer used to write the software is on the floor on some very fuzzy carpet, and there is some weird electrostatic bug in the hardware that contributed to how their software performed This is perhaps not the best analogy, and I'm not a scientist myself though have managed scientific teams for years, managed a few tech transfers, and evaluated tons of science as potential investments, but I just wanted to drive home the point of how tough science can be. If others can critique this I would welcome it as it would be great for wet lab scientists and engineers to understand each other's worlds better Of course there is also fraud, cherry picking of data, and careful manipulation of study design to make externalities work in your favor, but a lot of the reliability / reproducibility crisis comes down to lots of uncontrollable / unknowable variables and expensive and long development cycles

contact_fusion 9y ago

I know this may be perceived as nitpicking, but I am generally annoyed whenever the reproducibility/reliability crisis is referred to as a problem in science, writ large, rather than a specific problem in specific scientific subfields. This seems to be particularly common when the articles are written by practitioners within these subfields, such as neuroscience, biology, or psychology. There is no reproducibility crisis in my field of science - astrophysics. To back this claim up, I searched at Retraction Watch for any mention of the top tier journals in astronomy & astrophysics - specifically, ApJ [1], MNRAS [2], A&A (none found), Icarus (none found), Nature Astronomy (none found) - and found exactly one correction and one retraction. Now certainly errata are published constantly, and I would be foolish to assume that just because Retraction Watch didn't catch many instances of scientific fraud or abuse within my field, that it practically doesn't exist. But no evidence for a "crisis" seems to exist, at least in my corner of science. Astrophysical research appears to be quite reliable and reproducible. For similar reasons, I haven't heard of a reproducibility crisis in analytical chemistry, or optical physics, or mathematics, to name a few - or, close to the interests of Hacker News, computer science. Feel free to correct me if I'm wrong. I'm not saying this to bash non-astronomers, or non-physicists, or specifically life scientists. Biologists, neuroscientists, and psychologists are crucial participants in the greater scientific enterprise and their efforts lead more directly to alleviating human suffering than my work ever will. But when talking about reproducibility and reliability problems we cannot conflate different scientific disciplines with vastly different cultures, practices, and norms - a crisis in biology does not imply that physics has one too. Physics may have its own problems, but they aren't the same as the ones in biology. Witness the rise of groups such as the Flat Earthers, who are rejecting basic scientific knowledge known for literal millenia.(Eratosthenes, anyone?) Or witness the anti-vaxxers, abandoning modern medicine for charlatanry. Pretending science is a monolithic enterprise and abandoning a fine-grained understanding of the validity and power of different types of scientific evidence just gives these movements strength, feeds their delusions, and weakens the prestige of scientists when we do need to stand together. [1] http://retractionwatch.com/category/by-journal/astrophysical-journal/ http://retractionwatch.com/category/by-journal/astrophysical... [2] http://retractionwatch.com/category/by-journal/monthly-notices-of-roy-astro-soc/ http://retractionwatch.com/category/by-journal/monthly-notic...

SagelyGuru 9y ago

When you read between the lines of these "productivity" academic career policies and measures, you realise that they are aimed squarely at fame rather than quality of science. I guess the argument goes something like this: when you get fame through your high profile journal publications, we will be able to attract more grants and more fee paying bums on seats. Who cares about what you have actually written? Under the current model, nobody really.