10 ms·

I'm gonna get enemies for that .... But let's face it: the Academia peer review process is pretty rotten. - Many organisations could not care less about whate

by adev_ 6y ago

I'm gonna get enemies for that .... But let's face it: the Academia peer review process is pretty rotten. - Many organisations could not care less about whatever crap you publish as long as you publish. And if possible a lot. - Many journal's reviewers are friend of friends and you can often easily guess their names, even in double blinded review. - Many reviewers care more about you quoting their own paper than giving any relevant feedback. - There is plenty "pay2publish" journals that care about quantity not quality that are commonly used by academia to "force" validate crappy researches/PHds - The publishing world is full of oversized-ego individuals that will destroy a paper just because it comes from a competing lab/department/professor and not because it is scientifically irrelevant. And that is only a little subset of the problems...

dash2 6y ago
My experience is different. Peer review is far from perfect but: * Most of my papers have had thoughtful comments. Typically, they improved the paper, sometimes a lot. * I have very rarely been asked to quote a specific paper from a specific author. Of course, pointing out important references that the paper has missed is part of peer review, so this would not necessarily be evidence of corruption. * There are loads of crap pay-to-publish predatory journals, but publishing in those journals would harm my career, not help it. Nobody cares about them. * I've not experienced people rejecting papers from rival labs or departments. I don't say it doesn't happen, but I haven't experienced it. * I myself try hard to give high-quality reviews that explore the paper's value in depth. I am from one particular discipline, country and subfield. Others may have very different experiences. Again, I am not claiming academic peer review and publishing doesn't need a lot of improvement. Removing the parasitic mainstream publishers like Elsevier and Springer would be a great start.
- bjornsing 6y ago
  So how do you explain the information in the OP?
  - dash2 6y ago
    I don't know enough about the specific situation of computer science at the University of Florida to add anything useful. It sounds like a serious scandal.
    - Vhano 6y ago
      yes, this is localized and the system is not corrupt as a whole.
- woofie11 6y ago
  It varies field-by-field. In my field, peer review is a joke. I'm glad it's not a joke in your field yet, but incentive structures say it eventually will be. If incentives don't change, it's a question of when, not if.
  - andi999 6y ago
    What incentive structures? When I get a paper for review, I do my (time limited) best. Which incentive will change me?
    - woofie11 6y ago
      Academic hiring and promotion. High-quality academic curation doesn't effect anything in your career. On the other hand, the impact of your own research does does affect it. I'd like my taxpayer dollars to go a little less towards publish-or-perish, and a little more towards QA to make sure the papers coming out are credible.
- timkam 6y ago
  I largely agree with this comment, but this just means the underlying issues are more subtle. For example: * I typically get thoughtful reviews, and in particular critical reviews are often thoughtful. Still, I know that some people assert influence on the outcome of the reviewing process of their own papers. Also, there's a fine line: for example, many journals allow you to recommend reviewers for your papers. If you're well-connected, you can recommend your best friends and your editor friend follows your advice. Is this misconduct? I'd say it depends. After some decades in a math-heavy community, most people who are experienced and competent in the context of a highly specialized sub-field will be your "friends". * Journals that are outright predatory are, of course, a no-go. Still, there are edge cases like some MDPI journals; sometimes there are special issues from credible, relatively junior people in my community. Should we support them or not? Also, my funding provider wants me to publish in high-profile, "big publisher" journals and pays the open access fee; these journals typically rank much higher than community-owned/not-for-profit venues. On the long run, it's better to fully move to community-owned venues only, but as a junior researcher I feel I cannot afford it at the moment. Personally, for me the problem does not come from outright and obvious ethics violations; sure, I know of some individuals who I consider somewhat problematic, but I don't think these people control the community. For me the problem is to navigate all the grey areas, knowing that the system is largely ethics-agnostic.
- ketzu 6y ago
  I agree with the general point of your comment, but my personal experience with the usefulness of reviews is not so great. Nearly all reviews were superficial, not a single review ever found any technical flaw, even when some were discovered later. The acceptance recommendations seem to be governed by the overall impression the reviewer had from the paper and not by any factual points they make in the review. This manifested in getting 5 reviews at one conference, where all reviewers made nearly the same points in their review: all about the presentation of the paper. The paper received all of the five recommendations: Accept, weak accept, borderline, weak reject, reject. (The most critical and useful with a small correction of one formula was the "Accept" review.) Many reviewers also leave the impression they didn't read the paper in full or at all, or ran out of time while for the review, so looked for a reason to reject the paper and didn't look any further. Note: I am not saying my papers should have been accepted, 2 had serious issues I glad I discovered later! I also had a few very useful reviews to improve the presentation of my paper.
- 8note 6y ago
  I find it intriguing that reproducing the experiment isn't a part of the review, or visiting the lab to look for potential flaws in their set up. These reviews seem far from what you hear about for disproving N rays and the like
  - swiley 6y ago
    Do you checkout your co-worker's branches and run the unit tests for every PR they make? In an ideal world that might be nice but in the real world with normal sized projects that's to time consuming so everyone just reasons through patches.
  - andi999 6y ago
    If reproducing were a requirement for reviewing, publication cost might be exorbitant (like when CERN tries to publish a paper you have to rebuild the particle accelerator?)
CobrastanJorji 6y ago
Oh sure, the system is bad; it has perverse incentives and insufficient checks. But that doesn't also mean that these specific actors aren't more specifically bad in very concrete, condemnable ways.
izacus 6y ago
> - Many reviewers care more about you quoting their own paper than giving any relevant feedback. My supervisor once told me that I should quote a paper from a pair of reviewers (he knew exactly who was reviewing the papers) before he'd allow publishing. It was a CS paper and the reviewers were in the field of geology. A few of those fun cases were the reason why I left the academia - corporate bs doesn't come close to academic bs.
- ntsplnkv2 6y ago
  Academic BS is just as bad, if not worse. You have far less options. It's way easier to just "find another job" than it is to "find another phd program" and colleges/professors abuse the shit out of it.
eptcyka 6y ago
I take it you meant to say 'competing lab' instead of 'concurrent lab' in your 2nd to last comment.
- Tistron 6y ago
  Curious. In Swedish we say "konkurrerande" for "competing", I guess adev speaks a language that does something similar. In Danish they say konkurrence for competition.
  - lou1306 6y ago
    French and Italian also use similar terms (concurrence, concorrenza) for business competition, so it's not a far-fetched hypothesis.
    - rjsw 6y ago
      And for sports competition.
  - gambiting 6y ago
    Konkurencja in Polish, which sounds very very similar to concurrence
  - eptcyka 6y ago
    And in Latvian, probably Slavic labguages too.
- adev_ 6y ago
  Indeed. Fixed.
pca006132 6y ago
But the problem is, apart from peer review process, what measures can be used to ensure the quality of the paper being published? And what should be used as the metric for academics?
jgeada 6y ago
The reason you know about academic fraud is that lots of the process is public and has many unaffiliated people involved that are potential leaks. So, sure, fraud happens and clearly that’s bad. But you do hear about it. By contrast the same events within corporate culture happen behind veils of secrecy and HR departments & heavy hitting lawyers to make life hell if you don’t toe the line. And the cult of personality and oversized egos is very well represented in boards and upper management.
Gatsky 6y ago
Peer review is an ok idea. Academia is the problem. I saw two wasps fighting on the carcass of a dead bird the other day, it was a very apt analogy.
dehrmann 6y ago
This sounds like something like the tragedy of the commons or Gresham's Law for academia. It's in everyone's best interest to do take advantage of the system, at least as long as there's public support for academia. I don't see the situation getting much better. Republicans like free-market solutions, but not facing market constraints is part of the value of academia. Meanwhile, especially with consternation around student debt, Democrats will blindly support academia, failing to put any pressure on it.

by adev_ 6y ago

dash2 6y ago

My experience is different. Peer review is far from perfect but: * Most of my papers have had thoughtful comments. Typically, they improved the paper, sometimes a lot. * I have very rarely been asked to quote a specific paper from a specific author. Of course, pointing out important references that the paper has missed is part of peer review, so this would not necessarily be evidence of corruption. * There are loads of crap pay-to-publish predatory journals, but publishing in those journals would harm my career, not help it. Nobody cares about them. * I've not experienced people rejecting papers from rival labs or departments. I don't say it doesn't happen, but I haven't experienced it. * I myself try hard to give high-quality reviews that explore the paper's value in depth. I am from one particular discipline, country and subfield. Others may have very different experiences. Again, I am not claiming academic peer review and publishing doesn't need a lot of improvement. Removing the parasitic mainstream publishers like Elsevier and Springer would be a great start.

bjornsing 6y ago
So how do you explain the information in the OP?
- dash2 6y ago
  I don't know enough about the specific situation of computer science at the University of Florida to add anything useful. It sounds like a serious scandal.
  - Vhano 6y ago
    yes, this is localized and the system is not corrupt as a whole.
woofie11 6y ago
It varies field-by-field. In my field, peer review is a joke. I'm glad it's not a joke in your field yet, but incentive structures say it eventually will be. If incentives don't change, it's a question of when, not if.
- andi999 6y ago
  What incentive structures? When I get a paper for review, I do my (time limited) best. Which incentive will change me?
  - woofie11 6y ago
    Academic hiring and promotion. High-quality academic curation doesn't effect anything in your career. On the other hand, the impact of your own research does does affect it. I'd like my taxpayer dollars to go a little less towards publish-or-perish, and a little more towards QA to make sure the papers coming out are credible.
timkam 6y ago
I largely agree with this comment, but this just means the underlying issues are more subtle. For example: * I typically get thoughtful reviews, and in particular critical reviews are often thoughtful. Still, I know that some people assert influence on the outcome of the reviewing process of their own papers. Also, there's a fine line: for example, many journals allow you to recommend reviewers for your papers. If you're well-connected, you can recommend your best friends and your editor friend follows your advice. Is this misconduct? I'd say it depends. After some decades in a math-heavy community, most people who are experienced and competent in the context of a highly specialized sub-field will be your "friends". * Journals that are outright predatory are, of course, a no-go. Still, there are edge cases like some MDPI journals; sometimes there are special issues from credible, relatively junior people in my community. Should we support them or not? Also, my funding provider wants me to publish in high-profile, "big publisher" journals and pays the open access fee; these journals typically rank much higher than community-owned/not-for-profit venues. On the long run, it's better to fully move to community-owned venues only, but as a junior researcher I feel I cannot afford it at the moment. Personally, for me the problem does not come from outright and obvious ethics violations; sure, I know of some individuals who I consider somewhat problematic, but I don't think these people control the community. For me the problem is to navigate all the grey areas, knowing that the system is largely ethics-agnostic.
ketzu 6y ago
I agree with the general point of your comment, but my personal experience with the usefulness of reviews is not so great. Nearly all reviews were superficial, not a single review ever found any technical flaw, even when some were discovered later. The acceptance recommendations seem to be governed by the overall impression the reviewer had from the paper and not by any factual points they make in the review. This manifested in getting 5 reviews at one conference, where all reviewers made nearly the same points in their review: all about the presentation of the paper. The paper received all of the five recommendations: Accept, weak accept, borderline, weak reject, reject. (The most critical and useful with a small correction of one formula was the "Accept" review.) Many reviewers also leave the impression they didn't read the paper in full or at all, or ran out of time while for the review, so looked for a reason to reject the paper and didn't look any further. Note: I am not saying my papers should have been accepted, 2 had serious issues I glad I discovered later! I also had a few very useful reviews to improve the presentation of my paper.
8note 6y ago
I find it intriguing that reproducing the experiment isn't a part of the review, or visiting the lab to look for potential flaws in their set up. These reviews seem far from what you hear about for disproving N rays and the like
- swiley 6y ago
  Do you checkout your co-worker's branches and run the unit tests for every PR they make? In an ideal world that might be nice but in the real world with normal sized projects that's to time consuming so everyone just reasons through patches.
- andi999 6y ago
  If reproducing were a requirement for reviewing, publication cost might be exorbitant (like when CERN tries to publish a paper you have to rebuild the particle accelerator?)

CobrastanJorji 6y ago

Oh sure, the system is bad; it has perverse incentives and insufficient checks. But that doesn't also mean that these specific actors aren't more specifically bad in very concrete, condemnable ways.

izacus 6y ago

> - Many reviewers care more about you quoting their own paper than giving any relevant feedback. My supervisor once told me that I should quote a paper from a pair of reviewers (he knew exactly who was reviewing the papers) before he'd allow publishing. It was a CS paper and the reviewers were in the field of geology. A few of those fun cases were the reason why I left the academia - corporate bs doesn't come close to academic bs.

ntsplnkv2 6y ago
Academic BS is just as bad, if not worse. You have far less options. It's way easier to just "find another job" than it is to "find another phd program" and colleges/professors abuse the shit out of it.

eptcyka 6y ago

I take it you meant to say 'competing lab' instead of 'concurrent lab' in your 2nd to last comment.

Tistron 6y ago
Curious. In Swedish we say "konkurrerande" for "competing", I guess adev speaks a language that does something similar. In Danish they say konkurrence for competition.
- lou1306 6y ago
  French and Italian also use similar terms (concurrence, concorrenza) for business competition, so it's not a far-fetched hypothesis.
  - rjsw 6y ago
    And for sports competition.
- gambiting 6y ago
  Konkurencja in Polish, which sounds very very similar to concurrence
- eptcyka 6y ago
  And in Latvian, probably Slavic labguages too.
adev_ 6y ago
Indeed. Fixed.

pca006132 6y ago

But the problem is, apart from peer review process, what measures can be used to ensure the quality of the paper being published? And what should be used as the metric for academics?

jgeada 6y ago

The reason you know about academic fraud is that lots of the process is public and has many unaffiliated people involved that are potential leaks. So, sure, fraud happens and clearly that’s bad. But you do hear about it. By contrast the same events within corporate culture happen behind veils of secrecy and HR departments & heavy hitting lawyers to make life hell if you don’t toe the line. And the cult of personality and oversized egos is very well represented in boards and upper management.

Gatsky 6y ago

Peer review is an ok idea. Academia is the problem. I saw two wasps fighting on the carcass of a dead bird the other day, it was a very apt analogy.

dehrmann 6y ago

This sounds like something like the tragedy of the commons or Gresham's Law for academia. It's in everyone's best interest to do take advantage of the system, at least as long as there's public support for academia. I don't see the situation getting much better. Republicans like free-market solutions, but not facing market constraints is part of the value of academia. Meanwhile, especially with consternation around student debt, Democrats will blindly support academia, failing to put any pressure on it.