4 ms·

As a related aside: Does anyone know why arXiv has no system of public comments or (even better) a public peer review system built in? Whenever this topic comes

As a related aside: Does anyone know why arXiv has no system of public comments or (even better) a public peer review system built in? Whenever this topic comes up at a (CS) conference, everyone is for that, and yet, arXiv is functionally the same for quite a few years now.

xiaq 11y ago
Because it was intended as a repository of preprints instead of a publishing platform. It surely has the potential of becoming one, but that has not happened yet.
- TheOtherHobbes 11y ago
  Besides, if you open up physics papers to public comments you'll be deluged by cranks who think they can prove Einstin woz teh rong!!!! with a bit of algebra and a lot of punctuation. I don't know how you'd do useful public peer review. By definition, "peer" assumes some professional domain competence. The current system is probably too restrictive, but it's hard to imagine a more open system that's at least as useful. Maybe someone should do a PhD on it. :)
  - xiaq 11y ago
    Maybe it can work as follows: Support a concept of "reviewer group" that can be created by anyone but can only be joined on invitation. When you post a review, the groups you belong to are shown; you can also filter articles that are approved by a certain group. Eventually, some groups will gain reputation by producing quality reviews on certain fields and people will treat papers approved by them as high-quality. Heck, you can even create groups like "former reviewers of XXX journal" and as soon as you convince other people, gain reputation very quickly.
  - Coding_Cat 11y ago
    My idea would be to either restrict access (to certain areas, such as comments and reviews) to verified groups. Initially I would say to verify users through universities and/or use a computer automated system to rank each user, determining whose opinion to weight heavily and whose to dismiss. I would also make reviews public and name-and-shame any caught review fixing (if proper evidence is available), which would be the bigger problem imho. But again, I think this is something we should also be able to detect using proper algorithms. If a user Alice always rates the papers by Bob highly, even when others disagree, perhaps her opinion of Bob's work is not to trustworthy.
    - xiaq 11y ago
      Hey, my idea is basically a free-market counterpart to yours: instead of restricting access to verified groups, allow anyone to create groups and let them compete for reputation. This is how some journals come to be more prestigious than others anyway. You don't restrict the creation of journals to verified people; instead everyone gets to run a journal and they compete for reputation.
      - Coding_Cat 11y ago
        I think both ideas have merit. Which works best in practice would depend on the degree-of-shit (that's the scientific term) on your network. I fear special interest groups will, if they want, always outnumber legit researchers. If your network becomes large enough to attract the interest of these groups you might end up with "attacks" on climate research and other 'controversial' issues. Verified users could prevent this. OTOH, I do not like excluding users, but I think public read-access and write-access trough accredited universities and research groups would be a fair balance. (Note, I also think if there is a charge associated with this process we should charge relative to the users country. Subsidizing access for developing countries by charging more in developed countries. To reduce exclusions.)
        xiaq 11y ago
        I don't see how special interest groups will be a problem. Surely, they can flood the reviews with unlegit attacks, but the academia can just ignore them and only stick to the high-reputation review groups. They can make a lot of noise, but I don't think it can be a big problem for the academia.
        Coding_Cat 11y ago
        I'd say that from a UX experience you want the default-settings to work 99.9% of the time. This includes the ratings and reviews of articles. Which you could then only do by effectively censoring these groups when calculating ratings. If you're going to do that, why let them on in the first place?
- NotOscarWilde 11y ago
  I would really like a more comprehensive answer from somebody close to the source; you more or less say "it does not have it because it has not happened yet". Note that arXiv has features that many previous repositories did not have, such as easy access to the original TeX files. Plus, a comment system for a repository of preprints sounds very useful to me -- you can have early birds (such as PhDs or very active scientists) comment on your paper even before you submit to a conference/journal, thus getting good advice for free.
  - xiaq 11y ago
    Wikipedia told me this, but the homepage also advertises itself as an e-print service. Anyway, "it does not have it because it has not happened yet" is exactly my point, because I don't see why it cannot evolve into a publishing platform.
Coding_Cat 11y ago
This is actually something I have been thinking about writing myself. Look at all the major social networks, and all their metrics, analysis, ranking systems and the like. Imaging what you could do with a social-network like publication database (research-gate sort of is, but it doesn't think big enough and the UX is confusing, imho). Every publication became a "post" so to speak. You could easily include discussions (both on the paper, as well as general discussions), ratings, user curated lists (instead of expensive journals), host links to any relevant non-paper data. Funding could be provided by Universities, or by means of a registration fee, or simply with an ad supported model. Targeted ads will probably bring in a relatively high CPM due to the target audience. <rant> Incidentally, while we're at it, we could also get rid of strict formatting guidelines and super-formal writing, some of the best papers I've read are informal. And it's a very modern thing to do anyway, old papers by Kelvin and the like read much better than what we have to write these days. </rant>
- NotOscarWilde 11y ago
  Note that there are some projects that are related to this, for instance a social network for scientists [1], a Github-like site for article creation with public comments [2] and some unknown third party comment systems for arXiv [3]. (I just found this one by googling.) For any success of such a project, the official endorsement from arXiv is crucial. Academia is a very conservative place; anything branded as a social network might hit a snag, especially if "pricing" or closed source recommendation systems pop up. [1]: http://www.researchgate.net/ http://www.researchgate.net/ [2]: https://www.authorea.com/ https://www.authorea.com/ -- It costs quite a lot if you have several papers and you wish to release them only after the preprint is ready. [3]: http://pacs.gutcalc.com/ http://pacs.gutcalc.com/
  - Coding_Cat 11y ago
    Hmm, authorea looks pretty interesting. Researchgate I touched on before. the last one looks more like a quick experiment than a serious attempt. I don't think official endorsement by arXiv is necessary, what is important is to gain a critical mass. I think authorea has the right idea to give users a incentive to use the program. If possible, I would offer more free access to gain that initial traction, but of course this requires an investor with deep enough pockets to take the hit. Networking would also be key, to bring it back to the original article: if you could work with those involved here and offer your services to host all these open-access papers... >Academia is a very conservative place; anything branded as a social network might hit a snag, especially if "pricing" or closed source recommendation systems pop up. I think we need something similar in openness and positioning as Github. AFAIK Github did not market itself as a social platform at first, but it just made it so easy for its users that it ended up being key to its success. I think that is the overall approach one should take, don't force the users into a pattern, that will not work and definitely not in rigid-academia. You have to guide them by a good UX, make them think they want to use your features because that is what's good for them (they're like cats in that regard).
trurl42 11y ago
There is https://scirate.com/ https://scirate.com/ It's only really used by the quantum information community at this point, but you could try promoting it in yours.

xiaq 11y ago

Because it was intended as a repository of preprints instead of a publishing platform. It surely has the potential of becoming one, but that has not happened yet.

TheOtherHobbes 11y ago
Besides, if you open up physics papers to public comments you'll be deluged by cranks who think they can prove Einstin woz teh rong!!!! with a bit of algebra and a lot of punctuation. I don't know how you'd do useful public peer review. By definition, "peer" assumes some professional domain competence. The current system is probably too restrictive, but it's hard to imagine a more open system that's at least as useful. Maybe someone should do a PhD on it. :)
- xiaq 11y ago
  Maybe it can work as follows: Support a concept of "reviewer group" that can be created by anyone but can only be joined on invitation. When you post a review, the groups you belong to are shown; you can also filter articles that are approved by a certain group. Eventually, some groups will gain reputation by producing quality reviews on certain fields and people will treat papers approved by them as high-quality. Heck, you can even create groups like "former reviewers of XXX journal" and as soon as you convince other people, gain reputation very quickly.
- Coding_Cat 11y ago
  My idea would be to either restrict access (to certain areas, such as comments and reviews) to verified groups. Initially I would say to verify users through universities and/or use a computer automated system to rank each user, determining whose opinion to weight heavily and whose to dismiss. I would also make reviews public and name-and-shame any caught review fixing (if proper evidence is available), which would be the bigger problem imho. But again, I think this is something we should also be able to detect using proper algorithms. If a user Alice always rates the papers by Bob highly, even when others disagree, perhaps her opinion of Bob's work is not to trustworthy.
  - xiaq 11y ago
    Hey, my idea is basically a free-market counterpart to yours: instead of restricting access to verified groups, allow anyone to create groups and let them compete for reputation. This is how some journals come to be more prestigious than others anyway. You don't restrict the creation of journals to verified people; instead everyone gets to run a journal and they compete for reputation.
    - Coding_Cat 11y ago
      I think both ideas have merit. Which works best in practice would depend on the degree-of-shit (that's the scientific term) on your network. I fear special interest groups will, if they want, always outnumber legit researchers. If your network becomes large enough to attract the interest of these groups you might end up with "attacks" on climate research and other 'controversial' issues. Verified users could prevent this. OTOH, I do not like excluding users, but I think public read-access and write-access trough accredited universities and research groups would be a fair balance. (Note, I also think if there is a charge associated with this process we should charge relative to the users country. Subsidizing access for developing countries by charging more in developed countries. To reduce exclusions.)
      - xiaq 11y ago
        I don't see how special interest groups will be a problem. Surely, they can flood the reviews with unlegit attacks, but the academia can just ignore them and only stick to the high-reputation review groups. They can make a lot of noise, but I don't think it can be a big problem for the academia.
        Coding_Cat 11y ago
        I'd say that from a UX experience you want the default-settings to work 99.9% of the time. This includes the ratings and reviews of articles. Which you could then only do by effectively censoring these groups when calculating ratings. If you're going to do that, why let them on in the first place?
NotOscarWilde 11y ago
I would really like a more comprehensive answer from somebody close to the source; you more or less say "it does not have it because it has not happened yet". Note that arXiv has features that many previous repositories did not have, such as easy access to the original TeX files. Plus, a comment system for a repository of preprints sounds very useful to me -- you can have early birds (such as PhDs or very active scientists) comment on your paper even before you submit to a conference/journal, thus getting good advice for free.
- xiaq 11y ago
  Wikipedia told me this, but the homepage also advertises itself as an e-print service. Anyway, "it does not have it because it has not happened yet" is exactly my point, because I don't see why it cannot evolve into a publishing platform.

Coding_Cat 11y ago

This is actually something I have been thinking about writing myself. Look at all the major social networks, and all their metrics, analysis, ranking systems and the like. Imaging what you could do with a social-network like publication database (research-gate sort of is, but it doesn't think big enough and the UX is confusing, imho). Every publication became a "post" so to speak. You could easily include discussions (both on the paper, as well as general discussions), ratings, user curated lists (instead of expensive journals), host links to any relevant non-paper data. Funding could be provided by Universities, or by means of a registration fee, or simply with an ad supported model. Targeted ads will probably bring in a relatively high CPM due to the target audience. <rant> Incidentally, while we're at it, we could also get rid of strict formatting guidelines and super-formal writing, some of the best papers I've read are informal. And it's a very modern thing to do anyway, old papers by Kelvin and the like read much better than what we have to write these days. </rant>

NotOscarWilde 11y ago
Note that there are some projects that are related to this, for instance a social network for scientists [1], a Github-like site for article creation with public comments [2] and some unknown third party comment systems for arXiv [3]. (I just found this one by googling.) For any success of such a project, the official endorsement from arXiv is crucial. Academia is a very conservative place; anything branded as a social network might hit a snag, especially if "pricing" or closed source recommendation systems pop up. [1]: http://www.researchgate.net/ http://www.researchgate.net/ [2]: https://www.authorea.com/ https://www.authorea.com/ -- It costs quite a lot if you have several papers and you wish to release them only after the preprint is ready. [3]: http://pacs.gutcalc.com/ http://pacs.gutcalc.com/
- Coding_Cat 11y ago
  Hmm, authorea looks pretty interesting. Researchgate I touched on before. the last one looks more like a quick experiment than a serious attempt. I don't think official endorsement by arXiv is necessary, what is important is to gain a critical mass. I think authorea has the right idea to give users a incentive to use the program. If possible, I would offer more free access to gain that initial traction, but of course this requires an investor with deep enough pockets to take the hit. Networking would also be key, to bring it back to the original article: if you could work with those involved here and offer your services to host all these open-access papers... >Academia is a very conservative place; anything branded as a social network might hit a snag, especially if "pricing" or closed source recommendation systems pop up. I think we need something similar in openness and positioning as Github. AFAIK Github did not market itself as a social platform at first, but it just made it so easy for its users that it ended up being key to its success. I think that is the overall approach one should take, don't force the users into a pattern, that will not work and definitely not in rigid-academia. You have to guide them by a good UX, make them think they want to use your features because that is what's good for them (they're like cats in that regard).

trurl42 11y ago

There is https://scirate.com/ https://scirate.com/ It's only really used by the quantum information community at this point, but you could try promoting it in yours.