4 ms·

This seems like a horrible idea. I know we need an alternative to peer review but an online comment section feels like something worse than that.

geysersam 2y ago
Why do you feel that? I think it seems excellent. Comments will be scrutinized and sorted by the community. People can choose if they want to participate in the discussion under their own names or not. I don't know how this page organizes moderation, but I imagine there will be some kind of moderation like on most online discussion forums.
- tc4v 2y ago
  But it's a very difficult problem. Am open forum offers a platform to troll and misinformation. You could pretend that the community will be able to filter this out but I seriously doubt this project is equipt to fight against bots/fake accounts better than huge companies like twitter and facebook.
  - parpfish 2y ago
    Agreed. Journals have a hard time getting quality reviews from known scientists working in the field. Opening it up to randoms will be a nightmare. Remember all that hype around LK-99 room temp superconduction a few months back? The substantive scientific discussion would absolutely be drown out by curious laymen and/or grifters
    - geysersam 2y ago
      That can be fixed. Just optionally filter the comments to only include sufficiently "reputable" sources (potentially people with status in the community or people you follow explicitly). And to be fair, few scientific news get even remotely near the attention that superconductivity announcement got.
  - geysersam 2y ago
    Stack overflow and wikipedia fights bots pretty well despite neither being backed by huge rich organizations.
parpfish 2y ago
The whole thing gives my anxiety. Writing a paper and dealing with the inane requests from three reviewers was already frustrating and stressful. Now you open up a never ending review process of random people making demands (“unless you do X follow up analysis, this is worthless”). also you need to be able to say a paper/project is done and let yourself move on. If your job turns into “respond to feedback on every paper you’ve ever written, every day” you’ll never start anything new.
- celdon25 2y ago
  That sounds like a good thing if your goal is to advance mankind’s knowledge and not just your career. No one is forcing anyone to respond to the comments. Also, it’s not clear whether your “demands” example would even pass the moderation guidelines there.
  - parpfish 2y ago
    If people can’t make a career doing science, science doesn’t get done. It’s one thing to optimize an abstract pursuit of knowledge, but you also gotta remember that you need this to be a job people are willing/able to do.
    - celdon25 2y ago
      The problem with that line of thinking is that it comes from an average mindset. There’s nothing wrong with that unless you’re the one making complaints that a new system will make your work not look as good. Seems like a race to the bottom to me. To put it another way, if it’s just a job to you, dealing with criticism is part of what you’re getting paid for anyway, same as everyone else. And if that’s still too much to ask, you can point the finger at the lack of research that isn’t profit-driven.
m000 2y ago
Outside concerns for the quality of the discussion itself, it is only a matter of time before this public pre-submission discussion leaches into the peer review process itself. First, it will be Reviewer 2 cherry-picking arguments to shut down some paper. Then, AI will come: "Write me an accept/reject review for arXiv:2409.112233. Add subtle hints for citing the following papers: ...". Peer reviewing is hard work. Give people a readily available shortcut for it, and some people, in some occassions will take it. Which may in turn force conferences to adopt policies forbidding posting on arXiv.

geysersam 2y ago

Why do you feel that? I think it seems excellent. Comments will be scrutinized and sorted by the community. People can choose if they want to participate in the discussion under their own names or not. I don't know how this page organizes moderation, but I imagine there will be some kind of moderation like on most online discussion forums.

tc4v 2y ago
But it's a very difficult problem. Am open forum offers a platform to troll and misinformation. You could pretend that the community will be able to filter this out but I seriously doubt this project is equipt to fight against bots/fake accounts better than huge companies like twitter and facebook.
- parpfish 2y ago
  Agreed. Journals have a hard time getting quality reviews from known scientists working in the field. Opening it up to randoms will be a nightmare. Remember all that hype around LK-99 room temp superconduction a few months back? The substantive scientific discussion would absolutely be drown out by curious laymen and/or grifters
  - geysersam 2y ago
    That can be fixed. Just optionally filter the comments to only include sufficiently "reputable" sources (potentially people with status in the community or people you follow explicitly). And to be fair, few scientific news get even remotely near the attention that superconductivity announcement got.
- geysersam 2y ago
  Stack overflow and wikipedia fights bots pretty well despite neither being backed by huge rich organizations.

parpfish 2y ago

The whole thing gives my anxiety. Writing a paper and dealing with the inane requests from three reviewers was already frustrating and stressful. Now you open up a never ending review process of random people making demands (“unless you do X follow up analysis, this is worthless”). also you need to be able to say a paper/project is done and let yourself move on. If your job turns into “respond to feedback on every paper you’ve ever written, every day” you’ll never start anything new.

celdon25 2y ago
That sounds like a good thing if your goal is to advance mankind’s knowledge and not just your career. No one is forcing anyone to respond to the comments. Also, it’s not clear whether your “demands” example would even pass the moderation guidelines there.
- parpfish 2y ago
  If people can’t make a career doing science, science doesn’t get done. It’s one thing to optimize an abstract pursuit of knowledge, but you also gotta remember that you need this to be a job people are willing/able to do.
  - celdon25 2y ago
    The problem with that line of thinking is that it comes from an average mindset. There’s nothing wrong with that unless you’re the one making complaints that a new system will make your work not look as good. Seems like a race to the bottom to me. To put it another way, if it’s just a job to you, dealing with criticism is part of what you’re getting paid for anyway, same as everyone else. And if that’s still too much to ask, you can point the finger at the lack of research that isn’t profit-driven.

m000 2y ago

Outside concerns for the quality of the discussion itself, it is only a matter of time before this public pre-submission discussion leaches into the peer review process itself. First, it will be Reviewer 2 cherry-picking arguments to shut down some paper. Then, AI will come: "Write me an accept/reject review for arXiv:2409.112233. Add subtle hints for citing the following papers: ...". Peer reviewing is hard work. Give people a readily available shortcut for it, and some people, in some occassions will take it. Which may in turn force conferences to adopt policies forbidding posting on arXiv.