OTOH, I'm not sure if the peer review system isn't ripe for disruption anyway. Peer review is a system that's designed for the old dead-trees world, where spac

OTOH, I'm not sure if the peer review system isn't ripe for disruption anyway. Peer review is a system that's designed for the old dead-trees world, where space for publishing in print journals was extremely limited, publishing in a print journal was the only way to share knowledge, and you only got one shot at it because there's limited options for fixing errors after the plates are engraved. It's primarily there to ensure a minimum standard for the quality of the paper, not necessarily the research. So what it's really good for ensuring is that the paper explains the research well enough that someone who's reading it (and knows what they're doing) could attempt to replicate the work. What it is not designed to do, though, is ensure that the research is valid. Peer review can't reliably detect fraudulent data, nor can it reliably detect fabricated data. Heck, it can't even detect p-value fishing, which is a really sobering problem when you consider how many scientists really don't seem to understand why p-value fishing is such a problem in the first place. There's supposed to be a different mechanism for detecting that stuff: replication. Buy ironcially, replication is something the current journal system actively discourages. Space in journals is a scarce resource. Furthermore, journals are competing with each other for prestige. They've got very little reason to waste space on replicating old findings (booooooooring) when they could be publishing brand new stuff (exciting!). In turn, researchers have very little incentive to try and replicate each other's findings. It's already hard to justify spending money on replicating existing findings. But there's no way in hell a researcher is going to waste their precious resources on work that they wouldn't even be able to get published. So the long and short of it is, people place far too much trust in peer review. In fact, peer review is a terribly low hurdle. All too often complete junk gets through it about as easily as excellent science. The trick is that it just has to be well written junk. OTOH, a system that removes barriers to doing replication work - or better yet, starts actively encouraging it - could also do much of the work of peer review. Because then we'd have peers reviewing the actual research and not just things people write about research.