6 ms·

Interesting to see their approach and the reasons why they are not building more features on top of ArXiv. Although comments on papers might be a dangerous area

Interesting to see their approach and the reasons why they are not building more features on top of ArXiv. Although comments on papers might be a dangerous area to venture into there are definitely places on the web where the ability to annotate and comment papers is helping science move forward. A good example is the Polymath project and Terry Tao's blog. Tao's recent solution to the Erdos discrepancy problem,an 80-year-old number theory problem, was actually triggered by a comment on his blog. Another example is www.fermatslibrary.com. Although the papers in the platform are more historical/foundational, they were able to get consistently good/constructive comments that help people understand papers better.

egjerlow 10y ago
Would you elaborate on why you think comments on papers might be dangerous? For me it seems like the most natural step forward, given where we are today. As I see it, it would solve many issues. Ideally, we would move away from a publication-count metric, and more onto a reputation-based metric. It would lower the bar for participation in scientific discussion, make reproducibility more important, and generally be a healthy thing for science as a whole, I think. But I want to hear opposing viewpoints!
- xioxox 10y ago
  I don't think comments would be useful. To make a proper comment on a paper requires a significant amount of work (as does refereeing). Lowering the barrier to entry would produce some first impressions, the usual moans about lack of citation and misunderstandings. If a paper deserves a serious comment, I think the proper format for that is another paper. Although citations are horrible metrics, I think some sort of reputation ratings based on quick likes would be even worse and even easier to game.
  - Aeolun 10y ago
    The only thing you have to do to make comments worthwhile is restrict them to actual academics. Even if they went on to post shit, it would only destroy their own reputation.
    - adrianratnapala 10y ago
      This destroys one of the potential benefits of comments -- which is to open up what might be closed communities. Academic disciplines can at times turn into echo chambers. But there is the rub: if you make comments open, then you will will probably be flooded with. If you close them up, then fail to get the benefits of openness.
    - mjohn 10y ago
      There are plenty of professionals who would be able to contribute, but are not academics affiliated with a university.
  - egjerlow 10y ago
    > If a paper deserves a serious comment, I think the proper format for that is another paper. That's the thing, though - speaking as a former academic, the reaction when you see a paper that needs serious commenting is mostly "I'll just ignore this paper, since it has these issues", and you continue working on your own stuff - which again is because of the need for citations. If comments had larger effects on your career, they would be worthwhile making. I would guess the effect would be less papers and more 'collaborative science' where you start from a paper and either suggest improvements or improve it / replicate it yourself. I don't see the current proliferation of papers to be a good thing. As for the worry that there will be too many 'laymen' doing the commenting, there could (as suggested elsewhere) be a verification process - i.e. there could be a 'Verified Ph.D.' (or some other measure of merit) comment section in addition to the 'layman' section. There could also be a 'replication' section, where replication efforts will be rewarded - possibly by giving the replications some fraction of the paper's total 'reputation'. In short: If comments are made to really count for scientists, I think there will be a more healthy scientific process going on. Viewing a paper as an evolving thing is IMO more in line with how science does work anyway - a result should be replicated before it's accepted, which is not the case now.
  - return0 10y ago
    >To make a proper comment on a paper requires a significant amount of work (as does refereeing). Let's not fool ourselves, not all refereed papers are great, not all of them are significant and not all of them are published in high impact journals. Most of the reviewing is cursory, menial, and often delegated to postdocs/phds. Most scientific topics are highly specialized, and you 're unlikely to see trolls bothering to comment on them. Writing another paper as response is not a solution either: the pace of article publishing is months and years, not seconds. An example: I have often found the discussion of scientific papers here, in HN, to be illuminating, clarifying or countering issues, and offering a wider perspective that is often not mentioned in the article itself. And it's not like everyone in HN is a luminary, just mostly inquisitive people. I also see articles discussed in a useful way in twitter. Why can't we open up this discussion? I don't think comments are to be taken as reputation metrics either. But i do think a simple, helpful open discussion section is missing from every paper that i have read. I believe the main barrier to it is that academics do not want to get off their high horse of untouchability. To be precise, many journals do have a comment section, but nobody uses it.
    - JD557 10y ago
      >To be precise, many journals do have a comment section, but nobody uses it. GitXiv[1] also has a comment section and voting features, and nobody uses it. I know it has a small user base compared to ArXiV, but I think that its format resembles what ArXiv might look like with comments, votes and reproducible code. [1]: http://gitxiv.com/ http://gitxiv.com/
      - jessriedel 10y ago
        > GitXiv[1] also has a comment section and voting features, and nobody uses it. Likewise SciRate is a reasonably popular arXiv overlay, but the comments are never used: https://scirate.com/ https://scirate.com/
  - dalke 10y ago
    We need only to look to journals to see counter-examples to your proposition that "If a paper deserves a serious comment, I think the proper format for that is another paper." For example, a journal might have a "Letters to the Editor" section for people to voice a serious comment, even though such a letter is not "another paper".
- dsqrt 10y ago
  As somebody in the academia, I would see comments as potentially very dangerous. A comment system could be very easily abused by established researchers in some areas to prevent others from entering their niche sub-field. Do not get me wrong: the majority of scientists I know have very good professional integrity. However, I have also witnessed cases where established professors tried in all possible ways to prevent people from competing with them by discrediting or delaying the publication of very solid science. The arXiv as it is now is a great equalizer, because papers are presented without any string attached and this gives a chance to everyone (or at least to experts) to judge the quality of the various works without pre-conceptions. As a fictitious example of what could go wrong. I am a theoretical astrophysicist. I am not established (I am not a tenured professor). Suppose that I come up with a good model to explain some astronomical observations. What I would want to do is to write a paper directed to my astronomers colleagues explaining how my model works and why it is better than competing models. I want to convince them to work with me to analyze their data. As it is now, I would post it on the arXiv, the astronomers would probably find it, read it and evaluate it. However, if the arXiv had comments, a single negative review by a more established theoretical astrophysicist would be enough to discourage any astronomer from even reading my paper. Remember that in the astro-ph section of the arXiv there are of the order of 100 new papers per day and we can realistically read only 1 or 2 papers per day on average. In this situation, the chances of my work being completely ignored because of that one comment would be very significant. I think that the current channels for commenting on scientific work, private email and/or rebuttal papers, are perfectly adequate.
  - return0 10y ago
    "A single negative review by a more established theoretical astrophysicist would be enough" to discredit his opinion completely if your methods and results prove useful to the community. And besides, he can do that anyway if your paper happens to fall in his hands for review, which is a much more vicious system, because all the backstabbing happens in private. I disagree with your premise here.
    - woodson 10y ago
      I think his point is that, because of that negative comment his methods and results will never actually put to the test of whether they will be useful to the community; however, given the volume of new papers coming out it is unlikely for that paper to be picked up by the community as is, with or without comments. The best way to get people interested is in establishing direct relationships (ideally in person: go present at a relevant conference, engage with people in the community).
- Certhas 10y ago
  It changes the nature of the site. I fully think open peer review in various forms, is awesome and should be pursued, but the arxiv serves a more fundamental need as well: To simply make the papers available. Within the fields where arxiv is established this is not an issue, but we need to get adjacent fields, and really everyone to agree to a publicly available database of research first. There is still considerable reluctance in some quarters to upload to the arxiv as I found out after changing fields. The arxiv should be a pure database, then we can think about various ways to build on top. That could be arxiv overlay journals or journals that try out more fancy, new peer reviewing systems that might include curated comments [0]. As long as there is considerable disagreement over what the right format for serious commenting on papers is, arxiv should not be taking sides. [0] http://www.earth-system-dynamics.net/peer_review/interactive_review_process.html http://www.earth-system-dynamics.net/peer_review/interactive...
return0 10y ago
> might be a dangerous area to venture into. The only danger is that the true obnoxious nature of some academics would be aired in public. There is a lot to gain from opening up a discussion in each paper. Asking questions, making clarifications, even suggesting improvements are things that are not possible to do now.
- nl 10y ago
  The only danger is that the true obnoxious nature of some academics would be aired in public. I'm not sure how you can say that. Generally, the comments on the vast majority of internet sites are pretty low quality, and even insightful and useful ones are difficult to find because of all the noise.
  - return0 10y ago
    The vast majority may be useless, but a significant minority are valuable (e.g. maybe ~ 10% in HN). Why throw away that value?
    - nl 10y ago
      Why throw away that value? Because others can extremely negative - to the point where they may put some people off publishing in open forums. The very small amount of value that they may add is easy outweighted by a single person not publishing.
      - return0 10y ago
        I do not believe that. On the other hand not all publishing is valuable: http://www.nature.com/news/the-pressure-to-publish-pushes-down-quality-1.19887 http://www.nature.com/news/the-pressure-to-publish-pushes-do...
    - rubidium 10y ago
      Because right now we have a system that works. Email the author. I've ALWAYS gotten a response.
deleted 10y ago
[deleted]
justifier 10y ago
> Laba says arXiv should focus on its core mission. It’s the job of the journals to moderate, she says, and better for third-party sites to handle the discussion. i actually agree with this though i do think comments and peer review would be great for arxiv papers.. i think it should be done by someone as a webwrapper
westurner 10y ago
Hypothesis (OpenAnnotation) comments (and highlights!) work for any URL. https://hypothes.is/ https://hypothes.is/ https://hypothes.is/embed.js https://hypothes.is/embed.js http://www.openannotation.org/spec/core/ http://www.openannotation.org/spec/core/

egjerlow 10y ago

Would you elaborate on why you think comments on papers might be dangerous? For me it seems like the most natural step forward, given where we are today. As I see it, it would solve many issues. Ideally, we would move away from a publication-count metric, and more onto a reputation-based metric. It would lower the bar for participation in scientific discussion, make reproducibility more important, and generally be a healthy thing for science as a whole, I think. But I want to hear opposing viewpoints!

xioxox 10y ago
I don't think comments would be useful. To make a proper comment on a paper requires a significant amount of work (as does refereeing). Lowering the barrier to entry would produce some first impressions, the usual moans about lack of citation and misunderstandings. If a paper deserves a serious comment, I think the proper format for that is another paper. Although citations are horrible metrics, I think some sort of reputation ratings based on quick likes would be even worse and even easier to game.
- Aeolun 10y ago
  The only thing you have to do to make comments worthwhile is restrict them to actual academics. Even if they went on to post shit, it would only destroy their own reputation.
  - adrianratnapala 10y ago
    This destroys one of the potential benefits of comments -- which is to open up what might be closed communities. Academic disciplines can at times turn into echo chambers. But there is the rub: if you make comments open, then you will will probably be flooded with. If you close them up, then fail to get the benefits of openness.
  - mjohn 10y ago
    There are plenty of professionals who would be able to contribute, but are not academics affiliated with a university.
- egjerlow 10y ago
  > If a paper deserves a serious comment, I think the proper format for that is another paper. That's the thing, though - speaking as a former academic, the reaction when you see a paper that needs serious commenting is mostly "I'll just ignore this paper, since it has these issues", and you continue working on your own stuff - which again is because of the need for citations. If comments had larger effects on your career, they would be worthwhile making. I would guess the effect would be less papers and more 'collaborative science' where you start from a paper and either suggest improvements or improve it / replicate it yourself. I don't see the current proliferation of papers to be a good thing. As for the worry that there will be too many 'laymen' doing the commenting, there could (as suggested elsewhere) be a verification process - i.e. there could be a 'Verified Ph.D.' (or some other measure of merit) comment section in addition to the 'layman' section. There could also be a 'replication' section, where replication efforts will be rewarded - possibly by giving the replications some fraction of the paper's total 'reputation'. In short: If comments are made to really count for scientists, I think there will be a more healthy scientific process going on. Viewing a paper as an evolving thing is IMO more in line with how science does work anyway - a result should be replicated before it's accepted, which is not the case now.
- return0 10y ago
  >To make a proper comment on a paper requires a significant amount of work (as does refereeing). Let's not fool ourselves, not all refereed papers are great, not all of them are significant and not all of them are published in high impact journals. Most of the reviewing is cursory, menial, and often delegated to postdocs/phds. Most scientific topics are highly specialized, and you 're unlikely to see trolls bothering to comment on them. Writing another paper as response is not a solution either: the pace of article publishing is months and years, not seconds. An example: I have often found the discussion of scientific papers here, in HN, to be illuminating, clarifying or countering issues, and offering a wider perspective that is often not mentioned in the article itself. And it's not like everyone in HN is a luminary, just mostly inquisitive people. I also see articles discussed in a useful way in twitter. Why can't we open up this discussion? I don't think comments are to be taken as reputation metrics either. But i do think a simple, helpful open discussion section is missing from every paper that i have read. I believe the main barrier to it is that academics do not want to get off their high horse of untouchability. To be precise, many journals do have a comment section, but nobody uses it.
  - JD557 10y ago
    >To be precise, many journals do have a comment section, but nobody uses it. GitXiv[1] also has a comment section and voting features, and nobody uses it. I know it has a small user base compared to ArXiV, but I think that its format resembles what ArXiv might look like with comments, votes and reproducible code. [1]: http://gitxiv.com/ http://gitxiv.com/
    - jessriedel 10y ago
      > GitXiv[1] also has a comment section and voting features, and nobody uses it. Likewise SciRate is a reasonably popular arXiv overlay, but the comments are never used: https://scirate.com/ https://scirate.com/
- dalke 10y ago
  We need only to look to journals to see counter-examples to your proposition that "If a paper deserves a serious comment, I think the proper format for that is another paper." For example, a journal might have a "Letters to the Editor" section for people to voice a serious comment, even though such a letter is not "another paper".
dsqrt 10y ago
As somebody in the academia, I would see comments as potentially very dangerous. A comment system could be very easily abused by established researchers in some areas to prevent others from entering their niche sub-field. Do not get me wrong: the majority of scientists I know have very good professional integrity. However, I have also witnessed cases where established professors tried in all possible ways to prevent people from competing with them by discrediting or delaying the publication of very solid science. The arXiv as it is now is a great equalizer, because papers are presented without any string attached and this gives a chance to everyone (or at least to experts) to judge the quality of the various works without pre-conceptions. As a fictitious example of what could go wrong. I am a theoretical astrophysicist. I am not established (I am not a tenured professor). Suppose that I come up with a good model to explain some astronomical observations. What I would want to do is to write a paper directed to my astronomers colleagues explaining how my model works and why it is better than competing models. I want to convince them to work with me to analyze their data. As it is now, I would post it on the arXiv, the astronomers would probably find it, read it and evaluate it. However, if the arXiv had comments, a single negative review by a more established theoretical astrophysicist would be enough to discourage any astronomer from even reading my paper. Remember that in the astro-ph section of the arXiv there are of the order of 100 new papers per day and we can realistically read only 1 or 2 papers per day on average. In this situation, the chances of my work being completely ignored because of that one comment would be very significant. I think that the current channels for commenting on scientific work, private email and/or rebuttal papers, are perfectly adequate.
- return0 10y ago
  "A single negative review by a more established theoretical astrophysicist would be enough" to discredit his opinion completely if your methods and results prove useful to the community. And besides, he can do that anyway if your paper happens to fall in his hands for review, which is a much more vicious system, because all the backstabbing happens in private. I disagree with your premise here.
  - woodson 10y ago
    I think his point is that, because of that negative comment his methods and results will never actually put to the test of whether they will be useful to the community; however, given the volume of new papers coming out it is unlikely for that paper to be picked up by the community as is, with or without comments. The best way to get people interested is in establishing direct relationships (ideally in person: go present at a relevant conference, engage with people in the community).
Certhas 10y ago
It changes the nature of the site. I fully think open peer review in various forms, is awesome and should be pursued, but the arxiv serves a more fundamental need as well: To simply make the papers available. Within the fields where arxiv is established this is not an issue, but we need to get adjacent fields, and really everyone to agree to a publicly available database of research first. There is still considerable reluctance in some quarters to upload to the arxiv as I found out after changing fields. The arxiv should be a pure database, then we can think about various ways to build on top. That could be arxiv overlay journals or journals that try out more fancy, new peer reviewing systems that might include curated comments [0]. As long as there is considerable disagreement over what the right format for serious commenting on papers is, arxiv should not be taking sides. [0] http://www.earth-system-dynamics.net/peer_review/interactive_review_process.html http://www.earth-system-dynamics.net/peer_review/interactive...

return0 10y ago

> might be a dangerous area to venture into. The only danger is that the true obnoxious nature of some academics would be aired in public. There is a lot to gain from opening up a discussion in each paper. Asking questions, making clarifications, even suggesting improvements are things that are not possible to do now.

nl 10y ago
The only danger is that the true obnoxious nature of some academics would be aired in public. I'm not sure how you can say that. Generally, the comments on the vast majority of internet sites are pretty low quality, and even insightful and useful ones are difficult to find because of all the noise.
- return0 10y ago
  The vast majority may be useless, but a significant minority are valuable (e.g. maybe ~ 10% in HN). Why throw away that value?
  - nl 10y ago
    Why throw away that value? Because others can extremely negative - to the point where they may put some people off publishing in open forums. The very small amount of value that they may add is easy outweighted by a single person not publishing.
    - return0 10y ago
      I do not believe that. On the other hand not all publishing is valuable: http://www.nature.com/news/the-pressure-to-publish-pushes-down-quality-1.19887 http://www.nature.com/news/the-pressure-to-publish-pushes-do...
  - rubidium 10y ago
    Because right now we have a system that works. Email the author. I've ALWAYS gotten a response.

deleted 10y ago

[deleted]

justifier 10y ago

> Laba says arXiv should focus on its core mission. It’s the job of the journals to moderate, she says, and better for third-party sites to handle the discussion. i actually agree with this though i do think comments and peer review would be great for arxiv papers.. i think it should be done by someone as a webwrapper

westurner 10y ago

Hypothesis (OpenAnnotation) comments (and highlights!) work for any URL. https://hypothes.is/ https://hypothes.is/ https://hypothes.is/embed.js https://hypothes.is/embed.js http://www.openannotation.org/spec/core/ http://www.openannotation.org/spec/core/