3 ms·

I would like to see the governments that fund research get together and create a rule that your government funding requires that a public version of the paper b

by SiVal 11y ago

I would like to see the governments that fund research get together and create a rule that your government funding requires that a public version of the paper be made available online before that paper could be submitted to a private publisher. This would help to separate the public availability of useful information paid for by the public issue from the academics' career-promotion-via-publication issue. That would, of course, mean that papers would appear prior to acceptance and peer review, which would speed up the pace of research at the cost (if peer review is of any value) of putting more shaky research out there. I doubt this would be a big problem, since researchers in the field know what is going on anyway (they get early copies or at least early discussion at pay-walled conferences already), and others who read the research could see that it had not yet been published in a journal and decide for themselves what to make of it. Papers that got peer reviewed and published would still have the published version used in citations, so you could see how far it had gone in the publishing process at time of citation. I wouldn't be surprised if people in the field knew how much credibility to give to a paper based on the researchers without waiting for the researchers' more established rivals to "review" it to death.

bunderbunder 11y ago
OTOH, I'm not sure if the peer review system isn't ripe for disruption anyway. Peer review is a system that's designed for the old dead-trees world, where space for publishing in print journals was extremely limited, publishing in a print journal was the only way to share knowledge, and you only got one shot at it because there's limited options for fixing errors after the plates are engraved. It's primarily there to ensure a minimum standard for the quality of the paper, not necessarily the research. So what it's really good for ensuring is that the paper explains the research well enough that someone who's reading it (and knows what they're doing) could attempt to replicate the work. What it is not designed to do, though, is ensure that the research is valid. Peer review can't reliably detect fraudulent data, nor can it reliably detect fabricated data. Heck, it can't even detect p-value fishing, which is a really sobering problem when you consider how many scientists really don't seem to understand why p-value fishing is such a problem in the first place. There's supposed to be a different mechanism for detecting that stuff: replication. Buy ironcially, replication is something the current journal system actively discourages. Space in journals is a scarce resource. Furthermore, journals are competing with each other for prestige. They've got very little reason to waste space on replicating old findings (booooooooring) when they could be publishing brand new stuff (exciting!). In turn, researchers have very little incentive to try and replicate each other's findings. It's already hard to justify spending money on replicating existing findings. But there's no way in hell a researcher is going to waste their precious resources on work that they wouldn't even be able to get published. So the long and short of it is, people place far too much trust in peer review. In fact, peer review is a terribly low hurdle. All too often complete junk gets through it about as easily as excellent science. The trick is that it just has to be well written junk. OTOH, a system that removes barriers to doing replication work - or better yet, starts actively encouraging it - could also do much of the work of peer review. Because then we'd have peers reviewing the actual research and not just things people write about research.
- SiVal 11y ago
  Well said, and it explains part of what I meant when I wrote "(if peer review is of any value)" in the post. The other reason I wrote that is because of the political shenanigans inherent in the peer-reviewed journal system. "Anonymous peers" will sabotage the work of their rivals, and the journal editors will often choose them for the purpose of sabotaging research that doesn't support the causes the journal editors are working to promote.

by SiVal 11y ago

bunderbunder 11y ago

OTOH, I'm not sure if the peer review system isn't ripe for disruption anyway. Peer review is a system that's designed for the old dead-trees world, where space for publishing in print journals was extremely limited, publishing in a print journal was the only way to share knowledge, and you only got one shot at it because there's limited options for fixing errors after the plates are engraved. It's primarily there to ensure a minimum standard for the quality of the paper, not necessarily the research. So what it's really good for ensuring is that the paper explains the research well enough that someone who's reading it (and knows what they're doing) could attempt to replicate the work. What it is not designed to do, though, is ensure that the research is valid. Peer review can't reliably detect fraudulent data, nor can it reliably detect fabricated data. Heck, it can't even detect p-value fishing, which is a really sobering problem when you consider how many scientists really don't seem to understand why p-value fishing is such a problem in the first place. There's supposed to be a different mechanism for detecting that stuff: replication. Buy ironcially, replication is something the current journal system actively discourages. Space in journals is a scarce resource. Furthermore, journals are competing with each other for prestige. They've got very little reason to waste space on replicating old findings (booooooooring) when they could be publishing brand new stuff (exciting!). In turn, researchers have very little incentive to try and replicate each other's findings. It's already hard to justify spending money on replicating existing findings. But there's no way in hell a researcher is going to waste their precious resources on work that they wouldn't even be able to get published. So the long and short of it is, people place far too much trust in peer review. In fact, peer review is a terribly low hurdle. All too often complete junk gets through it about as easily as excellent science. The trick is that it just has to be well written junk. OTOH, a system that removes barriers to doing replication work - or better yet, starts actively encouraging it - could also do much of the work of peer review. Because then we'd have peers reviewing the actual research and not just things people write about research.

SiVal 11y ago
Well said, and it explains part of what I meant when I wrote "(if peer review is of any value)" in the post. The other reason I wrote that is because of the political shenanigans inherent in the peer-reviewed journal system. "Anonymous peers" will sabotage the work of their rivals, and the journal editors will often choose them for the purpose of sabotaging research that doesn't support the causes the journal editors are working to promote.