4 ms·
It's not that cut and dry. Only a small subset of arXiv preprints ever make it into a journal or conference proceedings. On balance it's good that more researc
by throwawaymath 8y ago
It's not that cut and dry. Only a small subset of arXiv preprints ever make it into a journal or conference proceedings.
On balance it's good that more researchers these days will publish preprints before submitting for peer review. But there are still two specific dangers with relying on preprints:
1. Most people reading papers don't try to implement or test them. Unless there is a glaring error, it's hard to tell if a paper is critically incorrect without a lot of effort.
2. Even if most people did try to implement papers, that would still make for a poor heuristic on the paper's merit. In an ideal world every valid paper could be implemented. But even in conference proceedings, it's extremely common for papers to be missing details critical to their implementation. In many cases you can't do the implementation because it requires a vast amount of computation or proprietary data available only to the company whose researchers wrote the paper.
- yorwba 8y agoPeer review doesn't really address those problems, though. Reviewers are just people reading papers, if it's hard to reproduce a paper's results, they can't verify that they are correct. Peer review only helps to decide whether a paper looks worthy of attention, and if you found a paper on arXiv, you probably already have some other reason to think that it might be worth looking at. If the paper comes with code you can simply run to reproduce the results, that's a stronger signal for correctness that whether it went through peer review or not.
- throwawaymath 8y ago> If the paper comes with code you can simply run to reproduce the results, that's a stronger signal for correctness that whether it went through peer review or not. 1. Most peer reviewed papers do not come with code. 2. Many that do come with code don't work out of the box.
- yorwba 8y agoExactly. That's why it's a stronger signal.
- mlevental 8y agoi still don't understand the force of the criticism? what is the danger in reading a "bad" paper in our domain? the person i was responding to >but one should always be careful with non-peer reviewed research. what do i need to be careful about? developing the wrong intuition?