6 ms·

What you lose on arxiv is peer review. That might sound like a feature- you get to let others know of your results faster- but if you make a mistake it will not

What you lose on arxiv is peer review. That might sound like a feature- you get to let others know of your results faster- but if you make a mistake it will not be caught until it's out in the wild (and possibly until everyone has already cited it and based their own work on it). With peer-review you know that someone will read your work that has the background to understand it and to catch errors. So what, you'll say, independent experts can catch your mistake if they read the paper on arxiv and contact you to suggest a correction. Yes, but there's no incentive to do the hard work to correct mistakes if your paper is already on arxiv and people are already citing it (who didn't do their due dilligence). Then errors start cascading. Science is self-correcting, sure, but it's much easier to avoid errors in the first place. That's an important function of modern peer review. I'm speaking from experience, of course. Peer review has certainly helped improve my work. I didn't enjoy having to redo the work, but the end result was something I could be confident about. Also, I'm only speaking about publishing in journals, because peer review in conferences is a very different beast. In my field, of machine learning and artificial intelligence research, I'd go as far as to say that peer review in conferences is broken, getting published or not is a lottery and you're better off putting your stuff on arxiv: less hassle and more people will read it. You might even see your article posted on HN. When was the last time a paper published on a conference server was posted on HN? See?

EGreg 5y ago
Well, this is exactly why we should have more WikiNews and less Twitter. I explore these issues in far more depth here: https://www.youtube.com/watch?v=gv5mI6ClPGc https://www.youtube.com/watch?v=gv5mI6ClPGc (Discussion w Noam Chomsky)
IshKebab 5y ago
> With peer-review you know that someone will read your work that has the background to understand it and to catch errors. That's overly optimistic. What you mean is "with peer-review you know that someone will at least skim your work and maybe catch errors." I agree with the rest of your points though. Zero peer review is probably not a great solution.
- marginalia_nu 5y ago
  Honestly the esteem of peer-review does not hold up to the reality of peer-review. It's elevated to this gold standard for quality science, but in practice, it's just not. Not only is some of the real landmarks in science older than peer review as we know it today (until the middle of 20th century, articles were reviewed by an editor), in several fields, 50-90% of peer-reviewed studies fail to replicate. It's very possible to just keep submitting papers to journals until you get someone who shrugs and lets it through. Reviewers aren't paid, and while reviewing is encouraged it basically does bupkis to further your career (and academia is very cut-throat).
  - hasjekyll 5y ago
    Completely agree - the quality of peer-review is a lottery. Sometimes you get an excellent referee making good points, other times you get a referee solely focussed on making sure you cite their work. And, occasionally, you get a referee who doesn't even appear to read the paper and just waves it through.
  - Karellen 5y ago
    > in several fields, 50-90% of peer-reviewed studies fail to replicate. Is that a failure of peer-review though? Or is it more of a consequence that even if you do everything right you'll still get some false positives some of the time, and negative results tend not to get published at all? (i.e. the problem that pre-registration for medical trials hopes to solve - https://en.wikipedia.org/wiki/Preregistration_(science) https://en.wikipedia.org/wiki/Preregistration_(science) )
    - marginalia_nu 5y ago
      > Is that a failure of peer-review though? It raises questions around using peer-review as a mark of scientific quality. If it is, why does peer reviewed science time and time again turn out to have enormous glaring quality problems?
      - 5y ago
mam2 5y ago
SiempreViernes 5y ago
There's actually some incentive: you get to feel smart by telling someone else is wrong. But it is indeed not a very strong one, and there's plenty of other places to act smart on the internet where you'll actually have an audience too!
hasjekyll 5y ago
Agreed. Peer-review is essential. I do like the idea of a public peer review however, similar to that in the https://joss.theoj.org/ https://joss.theoj.org/ - I had a really pleasant experience reviewing a paper for JOSS. I think sometimes reviewers hide behind their anonymity to give snarky responses, and using their power over you, knowing you will try your very best to implement all of their suggestions in order to publish faster. I just think the whole process should be more transparent.
atomack 5y ago
The way we used arxiv worked well in physics, though this is 15 years ago now so might have changed since. arxiv was about distribution. It didn't replace peer review - articles were still submitted to journals and published there too. If an article was posted to arxiv and not a journal, the odds of a citation went down massively. And the journal it was submitted to was a factor in whether or not we read it. When articles were eventually published, most authors also updated the preprint with the post peer review version. Basically it meant that (1) it was easy to keep up to date with what everyone was working on, and pick up interesting new stuff (2) most citations, post 80s, you saw in whatever paper you were reading, you could look up on arxiv and be reading it in seconds.
- kgwgk 5y ago
  > arxiv was about distribution. It didn't replace peer review - articles were still submitted to journals and published there too. I'm surprised that they no longer use the term "preprint" at all, at least it's nowhere to be found on the homepage or "about" section. The consequences of this amnesia are hilarious: https://twitter.com/gustavnilsonne/status/1389487297314312197 https://twitter.com/gustavnilsonne/status/138948729731431219... > Why do we call it "preprints"? The term seems to imply that work is preliminary or unfinished. As far as I can tell, the term introduced by @arxiv , the first online repository for scientific manuscripts, is "e-print". Is "preprint" a marketing device invented by publishers?
  - monkeyjoe 5y ago
    I think it's because the publisher retains copyright. There is a limit on how "done" the manuscript can be and still be shared for free online. Some universities have started to fight back against this by limiting the scope of copyright restrictions that publishers can impose.
    - krastanov 5y ago
      No, in most of physics there is no such limit in practice. The only difference between my published work and the preprint arxiv versions is the font and whether the layout is two columns or one column. They are word-for-word identical with identical figures.
HerrMonnezza 5y ago
> What you lose on arxiv is peer review. Or, instead, you get a much larger audience reviewing your article, instead of the 2 to 4 reviewers involved in the paper publishing process :-) You can get comments from critical and interested readers, you upload a new version of the paper to arXiv, and repeat the process until the article is ready for submission/publishing. In other words: I think that an article on arXiv (on average and as a whole, individual articles may be exceptional) is subject to much wider and more extensive scrutiny and verification than the average article during the paper publication process.
- YeGoblynQueenne 5y ago
  Well, if all one wants is an "audience" one can simply report the results of their research on twitter. Much bigger audience. But that's not the point, is it? The point is that you want someone who knows their shit, to review your paper, and who will have both the knowledge and the motivation to engage with your work critically and help you see what is behind all those big blind spots we all have for our own work. I'm sure this happens with papers put on arxiv, occasionally, but my expectation is that just because the paper is on arxiv most people will not be that interested in making a serious effort to review it, and that it is much more likely that such a serious effort will be made by a reviewer in a peer-reviewd venue, particularly one of the top journals (conference reviews are a mess). So, yeah, I disagree. The level of scrutiny one gets from putting their work on arxiv doesn't compare with peer review by experts in one's field.
  - bopbeepboop 5y ago
    In mathematics, it’s a continuum: Discussion starts in blog posts, on Twitter, in conversations. Then longer blog posts and code demos. Then pre-prints and more blog posts about that. Then finally journal. You get incredibly less review at the publication stage — and your idea “experts in one’s field” don’t also use the internet is hilariously wrong.
    - YeGoblynQueenne 5y ago
      >> You get incredibly less review at the publication stage — and your idea “experts in one’s field” don’t also use the internet is hilariously wrong. I'm glad to hear you're amused by my comment. Where did I express the idea that '“experts in one’s field” don’t also use the internet'? Can you please show me?
indigochill 5y ago
Borrowing from software engineering, you can publish code to a git repository and someone else can make a pull/merge request with a proposal to make fixes/improvements. There's still work involved in reviewing, revising, and merging those proposals, but it can be less work than needing to make every change yourself. I wonder if academic papers could benefit from a similar process of communal development (though it flies in the face of the dead-tree format of academic papers, but we're talking about Arxiv here, not Elsevier). Even if suggestions don't get merged, having a repository that captures discussion around the central artifact is also useful (such as when a maintainer decides that a feature doesn't belong in a project).
- nuclearnice1 5y ago
  Some of the sentiment of your last sentence is embodied in open reviews. https://openreview.net/group?id=ICLR.cc/2022/Conference&referrer=%5BHomepage%5D(%2F) https://openreview.net/group?id=ICLR.cc/2022/Conference&refe...
dandanua 5y ago
> but there's no incentive to do the hard work to correct mistakes if your paper is already on arxiv and people are already citing it Peer reviewing is essential, of course. But note that in most journals reviewers are not getting paid. They have even less incentives to find mistakes in your work than you are. I have no idea how such an utterly broken system emerged, where journals siphon billions of tax money exploiting the work of scientists.
- YeGoblynQueenne 5y ago
  The incentive to freely review others' work is that others will also freely review your work. This is quite separate than the exploitative aspects of the big scientific publishers. Reviewers are certainly exploited by publishers, because publishers make money from reviewers' free work. But reviewers would do the work for free anyway because that's what they expect others to do.
thereddaikon 5y ago
Peer review is important but lets be sure to not conflate it with the established way it operates. There is nothing that says peer review can only happen through submitting to an established Journal that conducts a traditional peer review process. We already know that process isn't perfect so there is no harm in looking to improve or replace that. Does that mean ArXiv as it is works as a replacement? No, and you make some good points why it doesn't.
time_to_smile 5y ago
> With peer-review you know that someone will read your work that has the background to understand it and to catch errors. This is a remarkably naive (and charmingly optimistic) view of peer-review. For the vast majority of the history of academic research peer review did not play a major part. It wasn't until the 1970s that the modern day peer review system emerged and it did so as a reaction to a funding crisis in academia to produce the illusion of legitimacy. It of course has done no such thing. It has done nothing to limit the reproducibility crisis which has shown it's ugly head in nearly all scientific fields of study. One could even argue that peer review coupled with a publish or perish culture is the cause of such a failure of science. This also means just about any great scientific discovery you can think of prior to 1970 did not go through the peer review process as we know it today. If science seems less exciting today, peer review is certainly one of the reasons for this. I'll leave you with Geoffrey Hinton's words on the subject: > Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad. Laking peering review is a feature, and not just because it speeds up time to results.
- YeGoblynQueenne 5y ago
  >> This is a remarkably naive (and charmingly optimistic) view of peer-review. Great way to join a convesation. Well done. I didn't say anything about "the vast majority of history of academic research". I talked about my experience of peer review of my work. Are you responding to someone else's comment but quoting mine?
simplestats 5y ago
> I'm only speaking about publishing in journals, because peer review in conferences is a very different beast. In my field, of machine learning and artificial intelligence research, I'd go as far as to say that peer review in conferences is broken, getting published or not is a lottery and you're better off putting your stuff on arxiv: less hassle and more people will read it. The situation for conferences in CS is basically how it works in journals in other fields. There may be more low-quality submissions to journals, but at the same time, the extreme competitiveness means most high-quality submissions are rejected also. The "solution" has been the appearance of open-access journals which are less concerned with subjective reasons for rejection like "impact", and scale to accept more papers. But they cost thousands of dollars. Open archives are a more democratic solution.

EGreg 5y ago

Well, this is exactly why we should have more WikiNews and less Twitter. I explore these issues in far more depth here: https://www.youtube.com/watch?v=gv5mI6ClPGc https://www.youtube.com/watch?v=gv5mI6ClPGc (Discussion w Noam Chomsky)

IshKebab 5y ago

> With peer-review you know that someone will read your work that has the background to understand it and to catch errors. That's overly optimistic. What you mean is "with peer-review you know that someone will at least skim your work and maybe catch errors." I agree with the rest of your points though. Zero peer review is probably not a great solution.

marginalia_nu 5y ago
Honestly the esteem of peer-review does not hold up to the reality of peer-review. It's elevated to this gold standard for quality science, but in practice, it's just not. Not only is some of the real landmarks in science older than peer review as we know it today (until the middle of 20th century, articles were reviewed by an editor), in several fields, 50-90% of peer-reviewed studies fail to replicate. It's very possible to just keep submitting papers to journals until you get someone who shrugs and lets it through. Reviewers aren't paid, and while reviewing is encouraged it basically does bupkis to further your career (and academia is very cut-throat).
- hasjekyll 5y ago
  Completely agree - the quality of peer-review is a lottery. Sometimes you get an excellent referee making good points, other times you get a referee solely focussed on making sure you cite their work. And, occasionally, you get a referee who doesn't even appear to read the paper and just waves it through.
- Karellen 5y ago
  > in several fields, 50-90% of peer-reviewed studies fail to replicate. Is that a failure of peer-review though? Or is it more of a consequence that even if you do everything right you'll still get some false positives some of the time, and negative results tend not to get published at all? (i.e. the problem that pre-registration for medical trials hopes to solve - https://en.wikipedia.org/wiki/Preregistration_(science) https://en.wikipedia.org/wiki/Preregistration_(science) )
  - marginalia_nu 5y ago
    > Is that a failure of peer-review though? It raises questions around using peer-review as a mark of scientific quality. If it is, why does peer reviewed science time and time again turn out to have enormous glaring quality problems?
    - 5y ago

mam2 5y ago

SiempreViernes 5y ago

There's actually some incentive: you get to feel smart by telling someone else is wrong. But it is indeed not a very strong one, and there's plenty of other places to act smart on the internet where you'll actually have an audience too!

hasjekyll 5y ago

Agreed. Peer-review is essential. I do like the idea of a public peer review however, similar to that in the https://joss.theoj.org/ https://joss.theoj.org/ - I had a really pleasant experience reviewing a paper for JOSS. I think sometimes reviewers hide behind their anonymity to give snarky responses, and using their power over you, knowing you will try your very best to implement all of their suggestions in order to publish faster. I just think the whole process should be more transparent.

atomack 5y ago

The way we used arxiv worked well in physics, though this is 15 years ago now so might have changed since. arxiv was about distribution. It didn't replace peer review - articles were still submitted to journals and published there too. If an article was posted to arxiv and not a journal, the odds of a citation went down massively. And the journal it was submitted to was a factor in whether or not we read it. When articles were eventually published, most authors also updated the preprint with the post peer review version. Basically it meant that (1) it was easy to keep up to date with what everyone was working on, and pick up interesting new stuff (2) most citations, post 80s, you saw in whatever paper you were reading, you could look up on arxiv and be reading it in seconds.

kgwgk 5y ago
> arxiv was about distribution. It didn't replace peer review - articles were still submitted to journals and published there too. I'm surprised that they no longer use the term "preprint" at all, at least it's nowhere to be found on the homepage or "about" section. The consequences of this amnesia are hilarious: https://twitter.com/gustavnilsonne/status/1389487297314312197 https://twitter.com/gustavnilsonne/status/138948729731431219... > Why do we call it "preprints"? The term seems to imply that work is preliminary or unfinished. As far as I can tell, the term introduced by @arxiv , the first online repository for scientific manuscripts, is "e-print". Is "preprint" a marketing device invented by publishers?
- monkeyjoe 5y ago
  I think it's because the publisher retains copyright. There is a limit on how "done" the manuscript can be and still be shared for free online. Some universities have started to fight back against this by limiting the scope of copyright restrictions that publishers can impose.
  - krastanov 5y ago
    No, in most of physics there is no such limit in practice. The only difference between my published work and the preprint arxiv versions is the font and whether the layout is two columns or one column. They are word-for-word identical with identical figures.

HerrMonnezza 5y ago

> What you lose on arxiv is peer review. Or, instead, you get a much larger audience reviewing your article, instead of the 2 to 4 reviewers involved in the paper publishing process :-) You can get comments from critical and interested readers, you upload a new version of the paper to arXiv, and repeat the process until the article is ready for submission/publishing. In other words: I think that an article on arXiv (on average and as a whole, individual articles may be exceptional) is subject to much wider and more extensive scrutiny and verification than the average article during the paper publication process.

YeGoblynQueenne 5y ago
Well, if all one wants is an "audience" one can simply report the results of their research on twitter. Much bigger audience. But that's not the point, is it? The point is that you want someone who knows their shit, to review your paper, and who will have both the knowledge and the motivation to engage with your work critically and help you see what is behind all those big blind spots we all have for our own work. I'm sure this happens with papers put on arxiv, occasionally, but my expectation is that just because the paper is on arxiv most people will not be that interested in making a serious effort to review it, and that it is much more likely that such a serious effort will be made by a reviewer in a peer-reviewd venue, particularly one of the top journals (conference reviews are a mess). So, yeah, I disagree. The level of scrutiny one gets from putting their work on arxiv doesn't compare with peer review by experts in one's field.
- bopbeepboop 5y ago
  In mathematics, it’s a continuum: Discussion starts in blog posts, on Twitter, in conversations. Then longer blog posts and code demos. Then pre-prints and more blog posts about that. Then finally journal. You get incredibly less review at the publication stage — and your idea “experts in one’s field” don’t also use the internet is hilariously wrong.
  - YeGoblynQueenne 5y ago
    >> You get incredibly less review at the publication stage — and your idea “experts in one’s field” don’t also use the internet is hilariously wrong. I'm glad to hear you're amused by my comment. Where did I express the idea that '“experts in one’s field” don’t also use the internet'? Can you please show me?

indigochill 5y ago

Borrowing from software engineering, you can publish code to a git repository and someone else can make a pull/merge request with a proposal to make fixes/improvements. There's still work involved in reviewing, revising, and merging those proposals, but it can be less work than needing to make every change yourself. I wonder if academic papers could benefit from a similar process of communal development (though it flies in the face of the dead-tree format of academic papers, but we're talking about Arxiv here, not Elsevier). Even if suggestions don't get merged, having a repository that captures discussion around the central artifact is also useful (such as when a maintainer decides that a feature doesn't belong in a project).

nuclearnice1 5y ago
Some of the sentiment of your last sentence is embodied in open reviews. https://openreview.net/group?id=ICLR.cc/2022/Conference&referrer=%5BHomepage%5D(%2F) https://openreview.net/group?id=ICLR.cc/2022/Conference&refe...

dandanua 5y ago

> but there's no incentive to do the hard work to correct mistakes if your paper is already on arxiv and people are already citing it Peer reviewing is essential, of course. But note that in most journals reviewers are not getting paid. They have even less incentives to find mistakes in your work than you are. I have no idea how such an utterly broken system emerged, where journals siphon billions of tax money exploiting the work of scientists.

YeGoblynQueenne 5y ago
The incentive to freely review others' work is that others will also freely review your work. This is quite separate than the exploitative aspects of the big scientific publishers. Reviewers are certainly exploited by publishers, because publishers make money from reviewers' free work. But reviewers would do the work for free anyway because that's what they expect others to do.

thereddaikon 5y ago

Peer review is important but lets be sure to not conflate it with the established way it operates. There is nothing that says peer review can only happen through submitting to an established Journal that conducts a traditional peer review process. We already know that process isn't perfect so there is no harm in looking to improve or replace that. Does that mean ArXiv as it is works as a replacement? No, and you make some good points why it doesn't.

time_to_smile 5y ago

> With peer-review you know that someone will read your work that has the background to understand it and to catch errors. This is a remarkably naive (and charmingly optimistic) view of peer-review. For the vast majority of the history of academic research peer review did not play a major part. It wasn't until the 1970s that the modern day peer review system emerged and it did so as a reaction to a funding crisis in academia to produce the illusion of legitimacy. It of course has done no such thing. It has done nothing to limit the reproducibility crisis which has shown it's ugly head in nearly all scientific fields of study. One could even argue that peer review coupled with a publish or perish culture is the cause of such a failure of science. This also means just about any great scientific discovery you can think of prior to 1970 did not go through the peer review process as we know it today. If science seems less exciting today, peer review is certainly one of the reasons for this. I'll leave you with Geoffrey Hinton's words on the subject: > Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad. Laking peering review is a feature, and not just because it speeds up time to results.

YeGoblynQueenne 5y ago
>> This is a remarkably naive (and charmingly optimistic) view of peer-review. Great way to join a convesation. Well done. I didn't say anything about "the vast majority of history of academic research". I talked about my experience of peer review of my work. Are you responding to someone else's comment but quoting mine?

simplestats 5y ago

> I'm only speaking about publishing in journals, because peer review in conferences is a very different beast. In my field, of machine learning and artificial intelligence research, I'd go as far as to say that peer review in conferences is broken, getting published or not is a lottery and you're better off putting your stuff on arxiv: less hassle and more people will read it. The situation for conferences in CS is basically how it works in journals in other fields. There may be more low-quality submissions to journals, but at the same time, the extreme competitiveness means most high-quality submissions are rejected also. The "solution" has been the appearance of open-access journals which are less concerned with subjective reasons for rejection like "impact", and scale to accept more papers. But they cost thousands of dollars. Open archives are a more democratic solution.