5 ms·

It strikes me that the challenge of peer review is that it is non-transparent, and gate-keeping. It is impossible for one to read the peer review feedback rela

by lumost 8y ago

It strikes me that the challenge of peer review is that it is non-transparent, and gate-keeping. It is impossible for one to read the peer review feedback related to a published paper, and it's similarly difficult to publish controversial pieces that break established norms. Contrast this to the peer review culture of popular open source projects - major pull requests have extensive and transparent dialogue, and disagreements known. Meanwhile there is no barrier to releasing anything new.

avs733 8y ago
gatekeeping in science is a theoretical good...the difficulty in practice is highlighted by the general 'three paths' for citations...articles that never get cited, articles that get cited a lot at first and then are forgotten, and articles that aren't cited until much later and then cited en masse. That last category is really the 'good science' because it does what we want foundational research to do - disrupt current and limiting ways of thinking. Dialogue is an unmitigated good, but dialogue also tends to be dominated by the normative voice. Where science is different than OSPs is that OSPs need to work with what exists now, whereas science needs to engage in tension and informed dialogue in a more foudnational way..
- Nasrudith 8y ago
  Here is a question what is the sort of theoretical good for the gatekeeping? What is the functional purpose of it? I think that answer would prove insightful for determining the future. I can see two noncynical reasons for it one. One is educational purposes. Two would be quality control - if it is already debunkable or widely known there is no point in publishing in Nature "thinking done in the brain". The cynical abuses and perverse incentives however are myraid and academic politicking is already infamous. I don't know what solution will work but understanding the system goals and functions should help decide what to replace and what to keep.
  - avs733 8y ago
    >Here is a question what is the sort of theoretical good for the gatekeeping? What is the functional purpose of it? I think that answer would prove insightful for determining the future. Because I am not at all prepared to assess the quality or accuracy of a quantum physics paper and I doubt a molecular biologist is prepared to assess the quality or accuracy of a sociology paper. In what I have seen there are a lot of cargo cultists who think kthey understand a field, and use big words, and just don't get it. I feel that there should be some filtering function to remove that. As an example: This guy: https://www.mountainproject.com/forum/topic/113602967/near-miss-on-cathedral-peak https://www.mountainproject.com/forum/topic/113602967/near-m... This guy: https://www.reddit.com/r/badphilosophy/comments/7x3t1g/stand_the_test_of_time_where_did_t https://www.reddit.com/r/badphilosophy/comments/7x3t1g/stand... This guy: http://ecclesiastes911.net/ http://ecclesiastes911.net/ and This guy: https://arxiv.org/search/math?searchtype=author&query=Simkin%2C+M https://arxiv.org/search/math?searchtype=author&query=Simkin... are all the same person.
    - x220 8y ago
      That guy has some pretty interesting articles.
- mcbits 8y ago
  I'd favor something like clustering and client-side filtering over gatekeeping. Mainstream academic research can coexist just fine with fringe theories and industry research in the same database as long as one can efficiently distinguish between them. And keeping everything together in one (de)central database would make it possible for people, so inclined, to annotate other people's work with new references to support or debunk the work long after it was published, or to clarify ambiguous language, etc. Those annotations, too, could be subject to filtering as needed. People could build reputations and whole careers around tying up loose ends instead of the "publish or perish" grind.
  - nl 8y ago
    Mainstream academic research can coexist just fine with fringe theories and industry research in the same database as long as one can efficiently distinguish between them How do you propose that is done? For my job I build neural networks for text processing. I spend a lot of time reading papers in the field. And yet if I look at something in an adjacent field (even something as close as something like open information extraction) I have trouble telling which papers are important. How on earth am I supposed to tell if something in a further removed field which attracts more crackpots (say probability theory or something) is a fringe theory or a breakthrough from a new author? I'd note the example of the Gaussian correlation inequality[1] where even people in the field weren't aware it had been proven for 3 years after publication[2]. [1] https://www.quantamagazine.org/statistician-proves-gaussian-correlation-inequality-20170328/ https://www.quantamagazine.org/statistician-proves-gaussian-... [2] https://en.wikipedia.org/wiki/Gaussian_correlation_inequality https://en.wikipedia.org/wiki/Gaussian_correlation_inequalit...
    - mcbits 8y ago
      Arxiv accidentally created a crackpot filter from apparently a simple semantic classifier along with a count of stop words used in the article, which acts as a very coarse "style" metric.[1] Reasonable-looking crackpot work is usually classified as "general physics" instead of being rejected entirely. Unfortunately it also lumps some legitimate but unconventional research in with the crackpots and makes Arxiv itself somewhat of a gatekeeper. What I'm imagining is a sort of layered approach with raw "article" (or some other unit) data at the bottom and indexing, tagging, clustering, filtering, reviewing, commenting, linking, etc. layered on top with possibly many implementations to choose from. The analysis/filtering would also apply to people augmenting the data, so if some users are really good at tagging certain types of junk as junk, you could easily filter out that junk. If there emerges a cluster of users who keep tagging certain interesting material as "woo" then you could filter them out or even use them to discover interesting material. I doubt there's a silver bullet (at least today) that could reliably distinguish between unconventional-bad and unconventional-good work, but keeping the baby and the bathwater together opens the door to such an algorithm in the future. [1] https://arxiv.org/abs/1603.03824 https://arxiv.org/abs/1603.03824
specialp 8y ago
There's a difference with scientific publishing for most disciplines. I work for a non-profit scientific publisher. Much of the research today that is coming out is extremely specialized. Thus there aren't many people in the world that have the expertise to referee a paper. So the review is anonymous in order to get a more frank and honest review (we hope). I am also a developer. There are far more people that can give feedback on most code, and also it doesn't take a lot of apparatus or money to get good at coding. Also CS/OSS is relatively young when compared to other disciplines. We've almost always done things with honest and sometimes brutal feedback. Even academic research is usually announced/shared at conferences. Look at what happened when Nature tried to make an AI journal.... [1] I think there is still value in someone getting paid to manage research and referees and ensure a high quality product. Open peer review is just going to be a lot harder for some of these disciplines with limited experts. [1] https://www.sciencemag.org/news/2018/05/why-are-ai-researchers-boycotting-new-nature-journal-and-shunning-others https://www.sciencemag.org/news/2018/05/why-are-ai-researche...
- sitkack 8y ago
  It sounds like machine checked proofs or other mechanical formalisms need to be employed to ensure scientific validity.
  - dharmab 8y ago
    How does a machine proof check a psychology study? Or a drug trial?
    - sitkack 8y ago
      open question! :) We could at least have the data and the statistical analysis (code) accompany the paper.
      - Ericson2314 8y ago
        Yes a fat table, a big proof, and some marginalia which is wonderful pure humanities.

by lumost 8y ago

avs733 8y ago

gatekeeping in science is a theoretical good...the difficulty in practice is highlighted by the general 'three paths' for citations...articles that never get cited, articles that get cited a lot at first and then are forgotten, and articles that aren't cited until much later and then cited en masse. That last category is really the 'good science' because it does what we want foundational research to do - disrupt current and limiting ways of thinking. Dialogue is an unmitigated good, but dialogue also tends to be dominated by the normative voice. Where science is different than OSPs is that OSPs need to work with what exists now, whereas science needs to engage in tension and informed dialogue in a more foudnational way..

Nasrudith 8y ago
Here is a question what is the sort of theoretical good for the gatekeeping? What is the functional purpose of it? I think that answer would prove insightful for determining the future. I can see two noncynical reasons for it one. One is educational purposes. Two would be quality control - if it is already debunkable or widely known there is no point in publishing in Nature "thinking done in the brain". The cynical abuses and perverse incentives however are myraid and academic politicking is already infamous. I don't know what solution will work but understanding the system goals and functions should help decide what to replace and what to keep.
- avs733 8y ago
  >Here is a question what is the sort of theoretical good for the gatekeeping? What is the functional purpose of it? I think that answer would prove insightful for determining the future. Because I am not at all prepared to assess the quality or accuracy of a quantum physics paper and I doubt a molecular biologist is prepared to assess the quality or accuracy of a sociology paper. In what I have seen there are a lot of cargo cultists who think kthey understand a field, and use big words, and just don't get it. I feel that there should be some filtering function to remove that. As an example: This guy: https://www.mountainproject.com/forum/topic/113602967/near-miss-on-cathedral-peak https://www.mountainproject.com/forum/topic/113602967/near-m... This guy: https://www.reddit.com/r/badphilosophy/comments/7x3t1g/stand_the_test_of_time_where_did_t https://www.reddit.com/r/badphilosophy/comments/7x3t1g/stand... This guy: http://ecclesiastes911.net/ http://ecclesiastes911.net/ and This guy: https://arxiv.org/search/math?searchtype=author&query=Simkin%2C+M https://arxiv.org/search/math?searchtype=author&query=Simkin... are all the same person.
  - x220 8y ago
    That guy has some pretty interesting articles.
mcbits 8y ago
I'd favor something like clustering and client-side filtering over gatekeeping. Mainstream academic research can coexist just fine with fringe theories and industry research in the same database as long as one can efficiently distinguish between them. And keeping everything together in one (de)central database would make it possible for people, so inclined, to annotate other people's work with new references to support or debunk the work long after it was published, or to clarify ambiguous language, etc. Those annotations, too, could be subject to filtering as needed. People could build reputations and whole careers around tying up loose ends instead of the "publish or perish" grind.
- nl 8y ago
  Mainstream academic research can coexist just fine with fringe theories and industry research in the same database as long as one can efficiently distinguish between them How do you propose that is done? For my job I build neural networks for text processing. I spend a lot of time reading papers in the field. And yet if I look at something in an adjacent field (even something as close as something like open information extraction) I have trouble telling which papers are important. How on earth am I supposed to tell if something in a further removed field which attracts more crackpots (say probability theory or something) is a fringe theory or a breakthrough from a new author? I'd note the example of the Gaussian correlation inequality[1] where even people in the field weren't aware it had been proven for 3 years after publication[2]. [1] https://www.quantamagazine.org/statistician-proves-gaussian-correlation-inequality-20170328/ https://www.quantamagazine.org/statistician-proves-gaussian-... [2] https://en.wikipedia.org/wiki/Gaussian_correlation_inequality https://en.wikipedia.org/wiki/Gaussian_correlation_inequalit...
  - mcbits 8y ago
    Arxiv accidentally created a crackpot filter from apparently a simple semantic classifier along with a count of stop words used in the article, which acts as a very coarse "style" metric.[1] Reasonable-looking crackpot work is usually classified as "general physics" instead of being rejected entirely. Unfortunately it also lumps some legitimate but unconventional research in with the crackpots and makes Arxiv itself somewhat of a gatekeeper. What I'm imagining is a sort of layered approach with raw "article" (or some other unit) data at the bottom and indexing, tagging, clustering, filtering, reviewing, commenting, linking, etc. layered on top with possibly many implementations to choose from. The analysis/filtering would also apply to people augmenting the data, so if some users are really good at tagging certain types of junk as junk, you could easily filter out that junk. If there emerges a cluster of users who keep tagging certain interesting material as "woo" then you could filter them out or even use them to discover interesting material. I doubt there's a silver bullet (at least today) that could reliably distinguish between unconventional-bad and unconventional-good work, but keeping the baby and the bathwater together opens the door to such an algorithm in the future. [1] https://arxiv.org/abs/1603.03824 https://arxiv.org/abs/1603.03824

specialp 8y ago

There's a difference with scientific publishing for most disciplines. I work for a non-profit scientific publisher. Much of the research today that is coming out is extremely specialized. Thus there aren't many people in the world that have the expertise to referee a paper. So the review is anonymous in order to get a more frank and honest review (we hope). I am also a developer. There are far more people that can give feedback on most code, and also it doesn't take a lot of apparatus or money to get good at coding. Also CS/OSS is relatively young when compared to other disciplines. We've almost always done things with honest and sometimes brutal feedback. Even academic research is usually announced/shared at conferences. Look at what happened when Nature tried to make an AI journal.... [1] I think there is still value in someone getting paid to manage research and referees and ensure a high quality product. Open peer review is just going to be a lot harder for some of these disciplines with limited experts. [1] https://www.sciencemag.org/news/2018/05/why-are-ai-researchers-boycotting-new-nature-journal-and-shunning-others https://www.sciencemag.org/news/2018/05/why-are-ai-researche...

sitkack 8y ago
It sounds like machine checked proofs or other mechanical formalisms need to be employed to ensure scientific validity.
- dharmab 8y ago
  How does a machine proof check a psychology study? Or a drug trial?
  - sitkack 8y ago
    open question! :) We could at least have the data and the statistical analysis (code) accompany the paper.
    - Ericson2314 8y ago
      Yes a fat table, a big proof, and some marginalia which is wonderful pure humanities.