9 ms·

Add in that the quality of the system is massively massively broken...peer review is about as accurate as the flip of a coin. It does not promote gorund breakin

by avs733 8y ago

Add in that the quality of the system is massively massively broken...peer review is about as accurate as the flip of a coin. It does not promote gorund breaking or novel research, it barely (arguably doesn't) even contribute to quality research. I had a colleague recently be told by a journal editor 'we don't publish critiques from junior scholars.' So much for the nature of peer review being entirely driven by the quality of the work. As one of those academics...I keep getting requests to peer review, I respectfully make clear I don't review for non open source journals anymore. Same with publishing. I'm not tenure-track so am not primarily evaluated based on output. Publishing is broken, but it is really just part of the broader and even more broken nature of academic research.

Angostura 8y ago
What would be a good alternative to peer-review though? Genuinely interested.
- ekianjo 8y ago
  > What would be a good alternative to peer-review though? Genuinely interested. Replicating researches? That's how we prove findings are not wrong or biased, usually.
  - amanaplanacanal 8y ago
    Yes. The fact that we have so much published research that can't be replicated indicates to me that peer-review is doing less work than we think it is.
- lumost 8y ago
  It strikes me that the challenge of peer review is that it is non-transparent, and gate-keeping. It is impossible for one to read the peer review feedback related to a published paper, and it's similarly difficult to publish controversial pieces that break established norms. Contrast this to the peer review culture of popular open source projects - major pull requests have extensive and transparent dialogue, and disagreements known. Meanwhile there is no barrier to releasing anything new.
  - avs733 8y ago
    gatekeeping in science is a theoretical good...the difficulty in practice is highlighted by the general 'three paths' for citations...articles that never get cited, articles that get cited a lot at first and then are forgotten, and articles that aren't cited until much later and then cited en masse. That last category is really the 'good science' because it does what we want foundational research to do - disrupt current and limiting ways of thinking. Dialogue is an unmitigated good, but dialogue also tends to be dominated by the normative voice. Where science is different than OSPs is that OSPs need to work with what exists now, whereas science needs to engage in tension and informed dialogue in a more foudnational way..
    - Nasrudith 8y ago
      Here is a question what is the sort of theoretical good for the gatekeeping? What is the functional purpose of it? I think that answer would prove insightful for determining the future. I can see two noncynical reasons for it one. One is educational purposes. Two would be quality control - if it is already debunkable or widely known there is no point in publishing in Nature "thinking done in the brain". The cynical abuses and perverse incentives however are myraid and academic politicking is already infamous. I don't know what solution will work but understanding the system goals and functions should help decide what to replace and what to keep.
      - avs733 8y ago
        >Here is a question what is the sort of theoretical good for the gatekeeping? What is the functional purpose of it? I think that answer would prove insightful for determining the future. Because I am not at all prepared to assess the quality or accuracy of a quantum physics paper and I doubt a molecular biologist is prepared to assess the quality or accuracy of a sociology paper. In what I have seen there are a lot of cargo cultists who think kthey understand a field, and use big words, and just don't get it. I feel that there should be some filtering function to remove that. As an example: This guy: https://www.mountainproject.com/forum/topic/113602967/near-miss-on-cathedral-peak https://www.mountainproject.com/forum/topic/113602967/near-m... This guy: https://www.reddit.com/r/badphilosophy/comments/7x3t1g/stand_the_test_of_time_where_did_t https://www.reddit.com/r/badphilosophy/comments/7x3t1g/stand... This guy: http://ecclesiastes911.net/ http://ecclesiastes911.net/ and This guy: https://arxiv.org/search/math?searchtype=author&query=Simkin%2C+M https://arxiv.org/search/math?searchtype=author&query=Simkin... are all the same person.
- smaddox 8y ago
  Open publishing and commenting would be a good start---having a dialog, like is done at conferences. Older academic journal articles (pre 1900) read much more like discussions than like the hundred dollar word vomits of modern academic publishing. The broken incentives are at the core of this rotten fruit, though. Just making journals open isn't enough.
  - avip 8y ago
    We have (almost) open publishing and open commenting. Did that improve anything?
    - raziel2701 8y ago
      There's open commenting? I've never seen the back and forth of the review process be published. It should be published.
      - _delirium 8y ago
        Some machine learning conferences use openreview.net for this.
      - westurner 8y ago
        https://hypothes.is https://hypothes.is supports threaded comments on anything with a URI; including PDFs and specific sentences or figures thereof. All you have to do is register an account and install the browser extension or include the JS in the HTML. It's based on open standards and an open platform. W3C Web Annotations: http://w3.org/annotation http://w3.org/annotation About Hypothesis: https://web.hypothes.is/about/ https://web.hypothes.is/about/
    - moultano 8y ago
      I think so. I've followed the back and forth on a few papers on openreview.net and I found the comments nearly as interesting and informative as the paper they were commenting on. There are a lot of things that only exist as suspicions and intuitions inside of a researcher's head, and that sort of information is much more likely to come out in a back and forth reviewing a paper than in actual published literature.
- sgillen 8y ago
  I think the op is not saying that peer review as a concept is broken, more that the current peer review system is broken. I think we could start fixing things by incentivizing good peer review. There’s a lot of ways to do this and I’d be interested in a discussion of different schemes. It will by no means be easy but I think that’s what needs to happen.
  - Fomite 8y ago
    For reference, good peer review is incentivized in some places. A journal I read gives awards every year for the best reviewers, and good reviewers end up as Associate Editors.
    - sgillen 8y ago
      Thanks for pointing that out. I have started to see those recently. Not sure if it is a recent phenomena or if I’ve only noticed it recently.
- DrNuke 8y ago
  For many STEM fields, conference talks, presentations and written contributions are notably faster and better suited to non-tenured applied researchers than the journals route.
  - avs733 8y ago
    unless the people who review their tenure cases don't count them because they are woefully out of date and out of touch. It has been amazing to come to understand just how effective the corporate takeover of universities has been in the thinking of senior faculty. I have had faculty complain about policies that they literally are the only one's empowered to change because it has never occurred to them that they actually have power. Conferences, generally, are a better path to science because of dialogue. Having a conference with 2500 presentations because everyone needs another CV line item don't accomplish that because no one is in the room.
    - DrNuke 8y ago
      It depends on how much waving you do actually need for your career or personal satisfaction, though. All these openness, transparency and calls to dialogue are the exact opposite I can see in my industrial, applied field of research. Most results here are kept private in order to attempt first exploitation or to be ready in case a suitable market arises. I can understand pure sciences and medicine, among STEM curricula, work inherently different, though, and rightly so.
- avs733 8y ago
  So the short answer is I don't know, and I don't think there is a single answer. Different types of scholarship need different standards and processes in my eyes. It isn't so much 'whats the alternative' to what is really a singular step as much as it is a rethink of how scholarship is performed, evaluated, rewarded, and considered. Otherwise the incentives across the playing field don't change enough to make meaningful change meaningful. What would help? My opinions... -FOSS publishing -Abandonment of journal metrics, they really serve no purpose besides trophy hunting at this point -Stronger transparency initiatives -Public review (c.f. Lim) -better journal metadata (it affects citations) -Greater shaming of misbehavior of ALL types -Elimination of stupid policies like 'issue lengths' The article either should be or should not be published...saying there isn't space in a given issue is insane given that >>90% of article access is online. -a hell of a lot of older faculty retiring out of the way of science.
  - avip 8y ago
    The Sphenodon in the room is that the raison d'etre of the entire Academic structure is unclear. So that should be clarified first. Advancing science is one thing. Publishing as many papers as possible, breaking research into MPUs is another. Lecturing to youth completely uninterested to be taught is yet another vertical.
    - avs733 8y ago
      I think you may be confusing vague with unclear... The inclarity, and multiple verticals seems more like an example of Goodhart's law than anything else. Summarized, the goal of academia is and (for most of time in whatever form it took) is to produce and disseminate new knowledge for human society. Creating new knowledge is what we now call research. Publishing papers isn't a raison d'etre...it's a metric for that. I'll note that Ernest Boyer is far more articulate on this than I am...
      - deleted 8y ago
        [deleted]
  - jcranmer 8y ago
- avs733 8y ago
  Actually I came up with one... The top 50 research universities could get together and decide to no longer give consideration to pay-walled articles. It wouldn't affect them AT ALL...the research they do is already treated as pretty much cannon no matter where it appears. Everyone else would follow, and the journals would quickly respond with change. Yeah I know it's unlikely but they have enough social capital to actually have an impact.
  - Something1234 8y ago
    That's not an alternative to peer review. It's a solution to remove the journals from the research, and it has a lot if's.
    - avs733 8y ago
      fair point.
  - deleted 8y ago
    [deleted]
- KevinEldon 8y ago
  ResearchOne(1) was founded to address the issues with peer review and the academic journal system. “A platform for scholarly publishing and peer review that empowers researchers with the Autonomy to pursue their passions, Authority to develop and disseminate their work, and Access to engage with the international community of scholars.” Harry Crane, an Associate Professor of Statistics at Rutgers, is one of the founders and a good follow on Twitter. 1 - https://researchers.one/ https://researchers.one/
- qwerty456127 8y ago
  Why not just build a kind of a social network where people would post research papers, "like" interesting and high-quality papers and reference papers relevant to their research, post their comments and for every participant you would be able to see what papers they have published and reviewed? Sounds fairly simple. Why not?
  - Fomite 8y ago
    People have tried this, or spins on it, and they don't particularly gain traction. Part of the problem is these have to exist alongside the traditional structure, and "Hey academics, do you want to be busier for intangible rewards?" is a hard sell.
    - qwerty456127 8y ago
      What are, if any, the reasons these have to exist alongside the traditional structure other than the fact only specific reputable traditional journals are recognized as valid publication places by universities boards and other relevant institutions?
      - Fomite 8y ago
        So you mean, "other than the fact that reputable journals are the entire basis for the dissemination of new research results, the core of hiring and promotion, and a major aspect of funding"? The simple fact is that like any replacement system, there has to be a reason for it to exist. And for a social network, there have to be network effects in place. Fragmentation is a problem. "Oops, we ran out of VC money..." is a problem. Hell, "We got bought by Spinger" is a problem. In the very long term, they don't have to exist along side each other. In the medium term though, they do. Otherwise, it's like going from "I'm going to quit my job tomorrow with no savings!" to "I live sustainably on my off the grid farm" with no plan for how to eat in the middle.
  - deleted 8y ago
    [deleted]
- barry-cotter 8y ago
  A good alternative to the current system of pre-publication peer review would be post publication peer review. The former, or one pass editorial review, is how scientific publication worked almost universally pre WW2. There are many scientific disciplines where everything important happens in preprints and publication in a journal is for archival purposes more than communication. Economics, Physics, Mathematics, Political Science, Genetics and Genomics, every discipline with substantial coverage on SSRN or the arxiv works on post publication peer review.
- icelancer 8y ago
  Peer-review is a good tool, but it is not the only tool that should be considered. And the problem is not peer-review specifically, but the implementation of the system and rent-seeking middlemen like Elsevier who do nothing but extract payment from taxpayers and researchers. PeerJ and PLOS One are good starts, but until academia is no longer a slave to "impact factor," it will have limited effect.
- scottlocklin 8y ago
  This answer will not please the 'publish or perish' economy which has built up around this: editorial discretion (which is a sort of peer review) and about 10x fewer papers published per capita involved. Its how science worked for most of its history. Really, Elsiever is responsible for the profusion of people publishing fluff to weak journals, subsidized by the government.
dunk010 8y ago
How does the old saw go again? "Academic knowledge moves forward one funeral at a time". People have vested interests and if your research contradicts much of theirs, then they are going to be inclined to reject yours. You'd think that this would be all ivory towers / pursuit of knowledge, but in reality it's dirty and messy and grimy; just like everything else in the world.

by avs733 8y ago

Angostura 8y ago

What would be a good alternative to peer-review though? Genuinely interested.

ekianjo 8y ago
> What would be a good alternative to peer-review though? Genuinely interested. Replicating researches? That's how we prove findings are not wrong or biased, usually.
- amanaplanacanal 8y ago
  Yes. The fact that we have so much published research that can't be replicated indicates to me that peer-review is doing less work than we think it is.
lumost 8y ago
It strikes me that the challenge of peer review is that it is non-transparent, and gate-keeping. It is impossible for one to read the peer review feedback related to a published paper, and it's similarly difficult to publish controversial pieces that break established norms. Contrast this to the peer review culture of popular open source projects - major pull requests have extensive and transparent dialogue, and disagreements known. Meanwhile there is no barrier to releasing anything new.
- avs733 8y ago
  gatekeeping in science is a theoretical good...the difficulty in practice is highlighted by the general 'three paths' for citations...articles that never get cited, articles that get cited a lot at first and then are forgotten, and articles that aren't cited until much later and then cited en masse. That last category is really the 'good science' because it does what we want foundational research to do - disrupt current and limiting ways of thinking. Dialogue is an unmitigated good, but dialogue also tends to be dominated by the normative voice. Where science is different than OSPs is that OSPs need to work with what exists now, whereas science needs to engage in tension and informed dialogue in a more foudnational way..
  - Nasrudith 8y ago
    Here is a question what is the sort of theoretical good for the gatekeeping? What is the functional purpose of it? I think that answer would prove insightful for determining the future. I can see two noncynical reasons for it one. One is educational purposes. Two would be quality control - if it is already debunkable or widely known there is no point in publishing in Nature "thinking done in the brain". The cynical abuses and perverse incentives however are myraid and academic politicking is already infamous. I don't know what solution will work but understanding the system goals and functions should help decide what to replace and what to keep.
    - avs733 8y ago
      >Here is a question what is the sort of theoretical good for the gatekeeping? What is the functional purpose of it? I think that answer would prove insightful for determining the future. Because I am not at all prepared to assess the quality or accuracy of a quantum physics paper and I doubt a molecular biologist is prepared to assess the quality or accuracy of a sociology paper. In what I have seen there are a lot of cargo cultists who think kthey understand a field, and use big words, and just don't get it. I feel that there should be some filtering function to remove that. As an example: This guy: https://www.mountainproject.com/forum/topic/113602967/near-miss-on-cathedral-peak https://www.mountainproject.com/forum/topic/113602967/near-m... This guy: https://www.reddit.com/r/badphilosophy/comments/7x3t1g/stand_the_test_of_time_where_did_t https://www.reddit.com/r/badphilosophy/comments/7x3t1g/stand... This guy: http://ecclesiastes911.net/ http://ecclesiastes911.net/ and This guy: https://arxiv.org/search/math?searchtype=author&query=Simkin%2C+M https://arxiv.org/search/math?searchtype=author&query=Simkin... are all the same person.
smaddox 8y ago
Open publishing and commenting would be a good start---having a dialog, like is done at conferences. Older academic journal articles (pre 1900) read much more like discussions than like the hundred dollar word vomits of modern academic publishing. The broken incentives are at the core of this rotten fruit, though. Just making journals open isn't enough.
- avip 8y ago
  We have (almost) open publishing and open commenting. Did that improve anything?
  - raziel2701 8y ago
    There's open commenting? I've never seen the back and forth of the review process be published. It should be published.
    - _delirium 8y ago
      Some machine learning conferences use openreview.net for this.
    - westurner 8y ago
      https://hypothes.is https://hypothes.is supports threaded comments on anything with a URI; including PDFs and specific sentences or figures thereof. All you have to do is register an account and install the browser extension or include the JS in the HTML. It's based on open standards and an open platform. W3C Web Annotations: http://w3.org/annotation http://w3.org/annotation About Hypothesis: https://web.hypothes.is/about/ https://web.hypothes.is/about/
  - moultano 8y ago
    I think so. I've followed the back and forth on a few papers on openreview.net and I found the comments nearly as interesting and informative as the paper they were commenting on. There are a lot of things that only exist as suspicions and intuitions inside of a researcher's head, and that sort of information is much more likely to come out in a back and forth reviewing a paper than in actual published literature.
sgillen 8y ago
I think the op is not saying that peer review as a concept is broken, more that the current peer review system is broken. I think we could start fixing things by incentivizing good peer review. There’s a lot of ways to do this and I’d be interested in a discussion of different schemes. It will by no means be easy but I think that’s what needs to happen.
- Fomite 8y ago
  For reference, good peer review is incentivized in some places. A journal I read gives awards every year for the best reviewers, and good reviewers end up as Associate Editors.
  - sgillen 8y ago
    Thanks for pointing that out. I have started to see those recently. Not sure if it is a recent phenomena or if I’ve only noticed it recently.
DrNuke 8y ago
For many STEM fields, conference talks, presentations and written contributions are notably faster and better suited to non-tenured applied researchers than the journals route.
- avs733 8y ago
  unless the people who review their tenure cases don't count them because they are woefully out of date and out of touch. It has been amazing to come to understand just how effective the corporate takeover of universities has been in the thinking of senior faculty. I have had faculty complain about policies that they literally are the only one's empowered to change because it has never occurred to them that they actually have power. Conferences, generally, are a better path to science because of dialogue. Having a conference with 2500 presentations because everyone needs another CV line item don't accomplish that because no one is in the room.
  - DrNuke 8y ago
    It depends on how much waving you do actually need for your career or personal satisfaction, though. All these openness, transparency and calls to dialogue are the exact opposite I can see in my industrial, applied field of research. Most results here are kept private in order to attempt first exploitation or to be ready in case a suitable market arises. I can understand pure sciences and medicine, among STEM curricula, work inherently different, though, and rightly so.
avs733 8y ago
So the short answer is I don't know, and I don't think there is a single answer. Different types of scholarship need different standards and processes in my eyes. It isn't so much 'whats the alternative' to what is really a singular step as much as it is a rethink of how scholarship is performed, evaluated, rewarded, and considered. Otherwise the incentives across the playing field don't change enough to make meaningful change meaningful. What would help? My opinions... -FOSS publishing -Abandonment of journal metrics, they really serve no purpose besides trophy hunting at this point -Stronger transparency initiatives -Public review (c.f. Lim) -better journal metadata (it affects citations) -Greater shaming of misbehavior of ALL types -Elimination of stupid policies like 'issue lengths' The article either should be or should not be published...saying there isn't space in a given issue is insane given that >>90% of article access is online. -a hell of a lot of older faculty retiring out of the way of science.
- avip 8y ago
  The Sphenodon in the room is that the raison d'etre of the entire Academic structure is unclear. So that should be clarified first. Advancing science is one thing. Publishing as many papers as possible, breaking research into MPUs is another. Lecturing to youth completely uninterested to be taught is yet another vertical.
  - avs733 8y ago
    I think you may be confusing vague with unclear... The inclarity, and multiple verticals seems more like an example of Goodhart's law than anything else. Summarized, the goal of academia is and (for most of time in whatever form it took) is to produce and disseminate new knowledge for human society. Creating new knowledge is what we now call research. Publishing papers isn't a raison d'etre...it's a metric for that. I'll note that Ernest Boyer is far more articulate on this than I am...
    - deleted 8y ago
      [deleted]
- jcranmer 8y ago
avs733 8y ago
Actually I came up with one... The top 50 research universities could get together and decide to no longer give consideration to pay-walled articles. It wouldn't affect them AT ALL...the research they do is already treated as pretty much cannon no matter where it appears. Everyone else would follow, and the journals would quickly respond with change. Yeah I know it's unlikely but they have enough social capital to actually have an impact.
- Something1234 8y ago
  That's not an alternative to peer review. It's a solution to remove the journals from the research, and it has a lot if's.
  - avs733 8y ago
    fair point.
- deleted 8y ago
  [deleted]
KevinEldon 8y ago
ResearchOne(1) was founded to address the issues with peer review and the academic journal system. “A platform for scholarly publishing and peer review that empowers researchers with the Autonomy to pursue their passions, Authority to develop and disseminate their work, and Access to engage with the international community of scholars.” Harry Crane, an Associate Professor of Statistics at Rutgers, is one of the founders and a good follow on Twitter. 1 - https://researchers.one/ https://researchers.one/
qwerty456127 8y ago
Why not just build a kind of a social network where people would post research papers, "like" interesting and high-quality papers and reference papers relevant to their research, post their comments and for every participant you would be able to see what papers they have published and reviewed? Sounds fairly simple. Why not?
- Fomite 8y ago
  People have tried this, or spins on it, and they don't particularly gain traction. Part of the problem is these have to exist alongside the traditional structure, and "Hey academics, do you want to be busier for intangible rewards?" is a hard sell.
  - qwerty456127 8y ago
    What are, if any, the reasons these have to exist alongside the traditional structure other than the fact only specific reputable traditional journals are recognized as valid publication places by universities boards and other relevant institutions?
    - Fomite 8y ago
      So you mean, "other than the fact that reputable journals are the entire basis for the dissemination of new research results, the core of hiring and promotion, and a major aspect of funding"? The simple fact is that like any replacement system, there has to be a reason for it to exist. And for a social network, there have to be network effects in place. Fragmentation is a problem. "Oops, we ran out of VC money..." is a problem. Hell, "We got bought by Spinger" is a problem. In the very long term, they don't have to exist along side each other. In the medium term though, they do. Otherwise, it's like going from "I'm going to quit my job tomorrow with no savings!" to "I live sustainably on my off the grid farm" with no plan for how to eat in the middle.
- deleted 8y ago
  [deleted]
barry-cotter 8y ago
A good alternative to the current system of pre-publication peer review would be post publication peer review. The former, or one pass editorial review, is how scientific publication worked almost universally pre WW2. There are many scientific disciplines where everything important happens in preprints and publication in a journal is for archival purposes more than communication. Economics, Physics, Mathematics, Political Science, Genetics and Genomics, every discipline with substantial coverage on SSRN or the arxiv works on post publication peer review.
icelancer 8y ago
Peer-review is a good tool, but it is not the only tool that should be considered. And the problem is not peer-review specifically, but the implementation of the system and rent-seeking middlemen like Elsevier who do nothing but extract payment from taxpayers and researchers. PeerJ and PLOS One are good starts, but until academia is no longer a slave to "impact factor," it will have limited effect.
scottlocklin 8y ago
This answer will not please the 'publish or perish' economy which has built up around this: editorial discretion (which is a sort of peer review) and about 10x fewer papers published per capita involved. Its how science worked for most of its history. Really, Elsiever is responsible for the profusion of people publishing fluff to weak journals, subsidized by the government.

dunk010 8y ago

How does the old saw go again? "Academic knowledge moves forward one funeral at a time". People have vested interests and if your research contradicts much of theirs, then they are going to be inclined to reject yours. You'd think that this would be all ivory towers / pursuit of knowledge, but in reality it's dirty and messy and grimy; just like everything else in the world.