5 ms·

There's a difference with scientific publishing for most disciplines. I work for a non-profit scientific publisher. Much of the research today that is coming ou

There's a difference with scientific publishing for most disciplines. I work for a non-profit scientific publisher. Much of the research today that is coming out is extremely specialized. Thus there aren't many people in the world that have the expertise to referee a paper. So the review is anonymous in order to get a more frank and honest review (we hope). I am also a developer. There are far more people that can give feedback on most code, and also it doesn't take a lot of apparatus or money to get good at coding. Also CS/OSS is relatively young when compared to other disciplines. We've almost always done things with honest and sometimes brutal feedback. Even academic research is usually announced/shared at conferences. Look at what happened when Nature tried to make an AI journal.... [1] I think there is still value in someone getting paid to manage research and referees and ensure a high quality product. Open peer review is just going to be a lot harder for some of these disciplines with limited experts. [1] https://www.sciencemag.org/news/2018/05/why-are-ai-researchers-boycotting-new-nature-journal-and-shunning-others https://www.sciencemag.org/news/2018/05/why-are-ai-researche...

sitkack 8y ago
It sounds like machine checked proofs or other mechanical formalisms need to be employed to ensure scientific validity.
- dharmab 8y ago
  How does a machine proof check a psychology study? Or a drug trial?
  - sitkack 8y ago
    open question! :) We could at least have the data and the statistical analysis (code) accompany the paper.
    - Ericson2314 8y ago
      Yes a fat table, a big proof, and some marginalia which is wonderful pure humanities.
- geoalchimista 8y ago
  Machine checked proofs only work for deductive, non-experimental disciplines. You can't use machine checked proofs to tell if a drug is effective or not. Human referees have to be involved no matter how advanced the AI is.
  - Ericson2314 8y ago
    You can precisely "quarantine" the "human referree part within the framework of a formal proof. That's still a huge step forward from the status quo of reasoning and opinion being smeared together and nothing formal.
    - geoalchimista 8y ago
      > You can precisely "quarantine" the "human referree part within the framework of a formal proof. I think it is unlikely you can separate the two precisely. If it be so, mathematicians can replace doctors who make diagnoses. I don't think all empirical facts can fit nicely into a formal proof system. But still, any effort to reduce the burden on the human referees would be welcome. This said proof system for peer review would be most useful to math and theoretical CS (though not so much to biomedical sciences).
      - Ericson2314 8y ago
        I take it as trivially true that they can be separated shallowly. Then the question is how big the informal "leaps of faith" are. If they are big, the next thing is to look for "strands" of reasoning (lemmas that may not be connected) inside them, as a sort of formal proof reverse-marginalia.
- nl 8y ago
  Sounds like you are trying to solve a social problem with technology.
guscost 8y ago
Anonymous doesn’t have to mean secret, although it isn’t always trivial to achieve.
- kilburn 8y ago
  Examine the facts: - Many research fields have anywhere between 3 and 10 groups working in them. - A review always transpires the background of the reviewer. You just cannot "mask" the shape of your knowledge around a highly-specialized subject. This includes your approach to the problem, the issues you are most interested in (and hence know more about), the references you give, etc. With a closed system, you only get to see: - Reviews of your own paper, without knowing who wrote them. - Reviews of the papers you review (there are usually 3-4 reviewers per paper, and you get to see the other reviewer's reviews and who they are). With these pieces of information, academia is already full of grudges and strong-arming around. Here's an anecdote: I was once in a conference, and met a colleague that made the effort to approach me and comment that he was great friends with my advisor (who didn't attend), and just wouldn't stop praising him. After a while, he switched to ranting about the review process, and how a specific reviewer was a moron that wouldn't understand anything and so on. Of course, that reviewer was my advisor and I did know but he obviously didn't. Given the high egos involved in academia, I am pretty sure that if there was a track record of all reviews, researchers would figure out who reviewed their papers. The backlash would then be ugly, and the entire ecosystem would end up more corrupt that it already is (imho). I've been on all sides of the fence, and I just don't see a better solution than just having secret reviews. Nice words don't matter (much, to most people) when the e-mail starts reading "We are sorry to inform you...".
  - guscost 8y ago
    In the current system, when you get to see the reviews of your paper, there are still the same challenges for anonymity and ugly consequences that you’re pointing out. Perhaps even to a greater degree, because the information is “leaked” through gossip rather than in a formal process. If all reviewer comments were collected and then published at the same time to prevent them influencing each other, how could that make the situation with respect to “frank and honest” reviews any worse than it is already? Are there entirely secret journals where nothing is accessible to non-contributors, for the same reasons? I suspect the answer is no, which is why these feuds often play out in attack-counterattack sequences of published papers.

sitkack 8y ago

It sounds like machine checked proofs or other mechanical formalisms need to be employed to ensure scientific validity.

dharmab 8y ago
How does a machine proof check a psychology study? Or a drug trial?
- sitkack 8y ago
  open question! :) We could at least have the data and the statistical analysis (code) accompany the paper.
  - Ericson2314 8y ago
    Yes a fat table, a big proof, and some marginalia which is wonderful pure humanities.
geoalchimista 8y ago
Machine checked proofs only work for deductive, non-experimental disciplines. You can't use machine checked proofs to tell if a drug is effective or not. Human referees have to be involved no matter how advanced the AI is.
- Ericson2314 8y ago
  You can precisely "quarantine" the "human referree part within the framework of a formal proof. That's still a huge step forward from the status quo of reasoning and opinion being smeared together and nothing formal.
  - geoalchimista 8y ago
    > You can precisely "quarantine" the "human referree part within the framework of a formal proof. I think it is unlikely you can separate the two precisely. If it be so, mathematicians can replace doctors who make diagnoses. I don't think all empirical facts can fit nicely into a formal proof system. But still, any effort to reduce the burden on the human referees would be welcome. This said proof system for peer review would be most useful to math and theoretical CS (though not so much to biomedical sciences).
    - Ericson2314 8y ago
      I take it as trivially true that they can be separated shallowly. Then the question is how big the informal "leaps of faith" are. If they are big, the next thing is to look for "strands" of reasoning (lemmas that may not be connected) inside them, as a sort of formal proof reverse-marginalia.
nl 8y ago
Sounds like you are trying to solve a social problem with technology.

guscost 8y ago

Anonymous doesn’t have to mean secret, although it isn’t always trivial to achieve.

kilburn 8y ago
Examine the facts: - Many research fields have anywhere between 3 and 10 groups working in them. - A review always transpires the background of the reviewer. You just cannot "mask" the shape of your knowledge around a highly-specialized subject. This includes your approach to the problem, the issues you are most interested in (and hence know more about), the references you give, etc. With a closed system, you only get to see: - Reviews of your own paper, without knowing who wrote them. - Reviews of the papers you review (there are usually 3-4 reviewers per paper, and you get to see the other reviewer's reviews and who they are). With these pieces of information, academia is already full of grudges and strong-arming around. Here's an anecdote: I was once in a conference, and met a colleague that made the effort to approach me and comment that he was great friends with my advisor (who didn't attend), and just wouldn't stop praising him. After a while, he switched to ranting about the review process, and how a specific reviewer was a moron that wouldn't understand anything and so on. Of course, that reviewer was my advisor and I did know but he obviously didn't. Given the high egos involved in academia, I am pretty sure that if there was a track record of all reviews, researchers would figure out who reviewed their papers. The backlash would then be ugly, and the entire ecosystem would end up more corrupt that it already is (imho). I've been on all sides of the fence, and I just don't see a better solution than just having secret reviews. Nice words don't matter (much, to most people) when the e-mail starts reading "We are sorry to inform you...".
- guscost 8y ago
  In the current system, when you get to see the reviews of your paper, there are still the same challenges for anonymity and ugly consequences that you’re pointing out. Perhaps even to a greater degree, because the information is “leaked” through gossip rather than in a formal process. If all reviewer comments were collected and then published at the same time to prevent them influencing each other, how could that make the situation with respect to “frank and honest” reviews any worse than it is already? Are there entirely secret journals where nothing is accessible to non-contributors, for the same reasons? I suspect the answer is no, which is why these feuds often play out in attack-counterattack sequences of published papers.