6 ms·

The statements made in the original post are so foreign to me. It sounds like the author is digging really deep to try and say something nice about the peer rev

The statements made in the original post are so foreign to me. It sounds like the author is digging really deep to try and say something nice about the peer review process. To be fair, it might be true for CS where they are stuck in the weird trap of publishing in conferences, but in other circles everything goes into journals, and in this case the peer-review processes is definitely not there to "help the authors". The peer review is there to help the journal maintain its reputation by preventing the publication of sub-standard stuff. Period. Sub-standard can mean uninteresting, incomplete, poorly written, or whatever the journal is aiming for. It is not there to safe guard the integrity of the literature against erroneous results...it's purely self-interest on the part of the journal. In reality, a rejected paper will just be submitted elsewhere until it is eventually accepted. The authors cannot afford to spend 1-2 years worth of work on a project then have nothing to show for it, just because a reviewer didn't "get it". So authors will keep submitting it (hopefully with some improvements based on past reviewer comments, but maybe not) until it "gets through" somewhere, and eventually nearly everything gets the seal of "peer review". > There is SO much more I could write on this subject, but I'm trying to stay on point (-:

slimsag 3y ago
Isn't this almost identical to the general admissions process, too? It seems strange to me that we built institutions on the idea of filtering in/out applicants based on relatively arbitrary criteria, and then express shock/surprise when the reward systems inside that institution are.. basically the same? There are parallels everywhere, e.g. scientists feeling they must get positive 'groundbreaking discovery!' news reporting about their publications, not just actually doing impactful work, in the same way good grades aren't enough and you need some other impactful story to tell in order to be accepted to many schools. All of it can be traced back to money, money, money.
- godelski 3y ago
  > It seems strange to me that we built institutions on the idea of filtering in/out applicants based on relatively arbitrary criteria, and then express shock/surprise when the reward systems inside that institution are.. basically the same? I think it is impossible to use anything other than noisy signals in many of these processes. I'm not bothered by the noise. But what I am bothered by is pretending the noise doesn't exist or trying to convince people it is a feature and not a bug. Why not just admit the process is noisy and that we're just doing our best? These processes may still be frustrating, but they will be less so and it leaves the door open to fixing issues if they are actually fixable. With the way we publish science, I find it very odd. We are doing essentially the same thing we have done for centuries with only a few minor tweaks. But unlike the past we don't have many of the same constraints; such as, printing, lack of color, distribution/storage, communication (which is arguably the main purpose of papers), video, and so on. Why are we stuck in the 18th century with a sprinkling of 21st century (plots/graphics and arxiv)? We can track data, reproduction efforts, failures, challenges, and all that stuff. Is not the point of science the pursuit of knowledge? If so, it seems like we're being pretty inefficient. I think there's just something funny about how we publish papers on things like computer graphics and video but constrain our communication methods to a piece of paper.
Blahah 3y ago
True for a particular dominant but antiquated and rapidly aging out model of peer review. Peer review as practiced at PeerJ, eLife, F1000, etc. is collaborative, productive, and maintains integrity in a visible way. Peer review is not inherently terrible. Exploitative rent seeking publishers that commoditize academic careers and outputs, and hold knowledge to ransom, are the problem. Please, everyone, stop publishing in journals that do harm. Think about what the impact of where you publish a paper is and align your choices with your values.
tptacek 3y ago
Right, it's a low bar, and it's meant to be a low bar, and that's fine. It's not no bar. Peer review adds some credibility to a paper. And the venue does as well. It's just less credibility than the popular imagination assumes.
- AlbertCory 3y ago
  This had a refreshing degree of realism. Naturally, any perfection-minded person will be offended by that. A decent journal will stop using reviewers who do a lousy job, and complaints by authors about their paper's reviewers ought to be listened to and not dismissed as sour grapes.
  - godelski 3y ago
    > A decent journal will stop using reviewers who do a lousy job, and complaints by authors about their paper's reviewers ought to be listened to and not dismissed as sour grapes. As a data point in ML conferences, I had a reviewer give me a strong reject with only a few sentences. First was a generic "not novel" with no further explanation. The rest was a complaint that my paper had serious flaws and pointed to an intentionally redacted link and a broken cross-reference. We reported them to the AC and the other two reviewers were borderline and weak accept, but both had low confidence (of course the inane reviewer was high confidence). The result was that the reviewer's rebuttal was much more angry and the weak accept said "authors addressed my concerns, but I've decided to lower my score." I tell this story because at a certain point it is not a problem with bad reviewers, it is a problem with bad management. The most important thing that a system can have is self correcting behavior. A system that gives a bad faith actor a slap on the wrist and allows them to then escalate their bad behavior and influence others is not a well functioning system. Granted, this is just one example, but I've seen many others (such as a theory paper being rejected for lack of experiments). I've started collecting author tweets I find about these kinds of situations, but I think there's a lot more that aren't easily conveyed on social media. I do really want to promote discussion of these things because I think we should always be improving our systems. You're right that nothing is every perfect, but we can still recognize flaws without being consumed with the impossible goal of perfectionism.
whatever1 3y ago
System works as intended. There are filler papers and seminal papers. They both need to pass the sniff test of the peer review process. For the filler papers nobody cares after they get published, they might cite them for the sake of completeness. The seminal papers though will be replicated and any hint of BS will be revealed.
- 2cynykyl 3y ago
  This is a great point about differentiating between filler and seminal papers. And I actually think the word filler is spot on for two reasons. Firstly, this reminds me of a blog post about what it means to get a PhD [1]. The problem with that view is that the circle of human knowledge is drawn as a solid, while in reality it is very holey. So seminal papers push the border outward, while filler papers fill in the holes. Both valid and necessary, but general public only cares about advancing the borders. Also, filler papers are needed not only for the sake of completeness, but scientists need papers to demonstrate productivity. E.g. graduate students needs papers on their CV so they can prove to their next employer that they are productive and capable of serious research. So these filler papers also "fill" the role of "filling" up people's resume. The public seem to think that professors publish papers only to announce to the world about their great discoveries, but 90% of the time its filler papers just reporting on business as usual type progress. [1] https://bretcontreras.com/the-illustrated-guide-to-a-phd/ https://bretcontreras.com/the-illustrated-guide-to-a-phd/
jballer 3y ago
This was what a few academics hypothesized and set out to prove in what is now known as the Grievance Studies Affair [1]. Not only did they get multiple hoax papers published, they got recognized for “excellent scholarship” [2]. [1] https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Grievance_studies_affair [2] https://youtu.be/kVk9a5Jcd1k?t=3m49s https://youtu.be/kVk9a5Jcd1k?t=3m49s
- fenomas 3y ago
  That grievance studies thing was idiotic, and really ought to be dismissed and forgotten. Reasons: 1. The hoax authors didn't submit gibberish or anything, like with Sokal - they submitted coherent papers that they considered absurd, but which they expected the journals to find reasonable. As such, they only thing they can even pretend to have proven is that they and the journals disagree about what's reasonable. 2. IIRC, every one of the initial hoax papers was flat-out rejected! If the authors had any backbone they could have stopped right there, and published a paper called "These grievance journals aren't as bad as we expected." :D But they didn't - they set out to iterate on each paper until a journal accepted it. And if your experimental methodology is "keep trying different things until X happens", you can hardly pretend to have proved something when X happens. I mean, I agree with the premise the hoax authors were trying to prove. But their "experiment" was silly, and I think it's dishonest of them to pretend it proved something. Meh. Meh!
  - foldr 3y ago
    Excellent summary. I’d also mention the lack of a control group. We’re supposed to conclude that it’s easier to (eventually) get fraudulent results published in so-called ‘grievance’ fields than it is in (say) physics. But the hoax authors never made any effort to publish fraudulent results in those fields, so we can’t know if the percentage of papers that were eventually accepted would have been lower or not.
    - jballer 3y ago
      We were typing at the same time - see my other comment (sibling to yours) linking to Thiel arguing things are likely worse in hard sciences, and encouraging people to find ways to expose it.
chmike 3y ago
The peer review system is definitely biased. It's ok for many articles, but there are outliers. For instance really original theories or experimental data. The publication may be stopped to avoid putting the journal's reputation at risk or because the reviewers have personal reasons to not support the article (it would shade their own pet theory, invalidate their research project, they didn't understood it, etc.). The problem with publishing elsewhere is that many people unable to objectively evaluate the validity of the theory juge its value by the reputation of the journal. Also the other journal will most probably have less visibility. There is thus a higher chance that the other theory will not be reported in reviews. There are many assumptions in the above comment that I can't agree with. But my experience is with physics, not computer science.

slimsag 3y ago

Isn't this almost identical to the general admissions process, too? It seems strange to me that we built institutions on the idea of filtering in/out applicants based on relatively arbitrary criteria, and then express shock/surprise when the reward systems inside that institution are.. basically the same? There are parallels everywhere, e.g. scientists feeling they must get positive 'groundbreaking discovery!' news reporting about their publications, not just actually doing impactful work, in the same way good grades aren't enough and you need some other impactful story to tell in order to be accepted to many schools. All of it can be traced back to money, money, money.

godelski 3y ago
> It seems strange to me that we built institutions on the idea of filtering in/out applicants based on relatively arbitrary criteria, and then express shock/surprise when the reward systems inside that institution are.. basically the same? I think it is impossible to use anything other than noisy signals in many of these processes. I'm not bothered by the noise. But what I am bothered by is pretending the noise doesn't exist or trying to convince people it is a feature and not a bug. Why not just admit the process is noisy and that we're just doing our best? These processes may still be frustrating, but they will be less so and it leaves the door open to fixing issues if they are actually fixable. With the way we publish science, I find it very odd. We are doing essentially the same thing we have done for centuries with only a few minor tweaks. But unlike the past we don't have many of the same constraints; such as, printing, lack of color, distribution/storage, communication (which is arguably the main purpose of papers), video, and so on. Why are we stuck in the 18th century with a sprinkling of 21st century (plots/graphics and arxiv)? We can track data, reproduction efforts, failures, challenges, and all that stuff. Is not the point of science the pursuit of knowledge? If so, it seems like we're being pretty inefficient. I think there's just something funny about how we publish papers on things like computer graphics and video but constrain our communication methods to a piece of paper.

Blahah 3y ago

True for a particular dominant but antiquated and rapidly aging out model of peer review. Peer review as practiced at PeerJ, eLife, F1000, etc. is collaborative, productive, and maintains integrity in a visible way. Peer review is not inherently terrible. Exploitative rent seeking publishers that commoditize academic careers and outputs, and hold knowledge to ransom, are the problem. Please, everyone, stop publishing in journals that do harm. Think about what the impact of where you publish a paper is and align your choices with your values.

tptacek 3y ago

Right, it's a low bar, and it's meant to be a low bar, and that's fine. It's not no bar. Peer review adds some credibility to a paper. And the venue does as well. It's just less credibility than the popular imagination assumes.

AlbertCory 3y ago
This had a refreshing degree of realism. Naturally, any perfection-minded person will be offended by that. A decent journal will stop using reviewers who do a lousy job, and complaints by authors about their paper's reviewers ought to be listened to and not dismissed as sour grapes.
- godelski 3y ago
  > A decent journal will stop using reviewers who do a lousy job, and complaints by authors about their paper's reviewers ought to be listened to and not dismissed as sour grapes. As a data point in ML conferences, I had a reviewer give me a strong reject with only a few sentences. First was a generic "not novel" with no further explanation. The rest was a complaint that my paper had serious flaws and pointed to an intentionally redacted link and a broken cross-reference. We reported them to the AC and the other two reviewers were borderline and weak accept, but both had low confidence (of course the inane reviewer was high confidence). The result was that the reviewer's rebuttal was much more angry and the weak accept said "authors addressed my concerns, but I've decided to lower my score." I tell this story because at a certain point it is not a problem with bad reviewers, it is a problem with bad management. The most important thing that a system can have is self correcting behavior. A system that gives a bad faith actor a slap on the wrist and allows them to then escalate their bad behavior and influence others is not a well functioning system. Granted, this is just one example, but I've seen many others (such as a theory paper being rejected for lack of experiments). I've started collecting author tweets I find about these kinds of situations, but I think there's a lot more that aren't easily conveyed on social media. I do really want to promote discussion of these things because I think we should always be improving our systems. You're right that nothing is every perfect, but we can still recognize flaws without being consumed with the impossible goal of perfectionism.

whatever1 3y ago

System works as intended. There are filler papers and seminal papers. They both need to pass the sniff test of the peer review process. For the filler papers nobody cares after they get published, they might cite them for the sake of completeness. The seminal papers though will be replicated and any hint of BS will be revealed.

2cynykyl 3y ago
This is a great point about differentiating between filler and seminal papers. And I actually think the word filler is spot on for two reasons. Firstly, this reminds me of a blog post about what it means to get a PhD [1]. The problem with that view is that the circle of human knowledge is drawn as a solid, while in reality it is very holey. So seminal papers push the border outward, while filler papers fill in the holes. Both valid and necessary, but general public only cares about advancing the borders. Also, filler papers are needed not only for the sake of completeness, but scientists need papers to demonstrate productivity. E.g. graduate students needs papers on their CV so they can prove to their next employer that they are productive and capable of serious research. So these filler papers also "fill" the role of "filling" up people's resume. The public seem to think that professors publish papers only to announce to the world about their great discoveries, but 90% of the time its filler papers just reporting on business as usual type progress. [1] https://bretcontreras.com/the-illustrated-guide-to-a-phd/ https://bretcontreras.com/the-illustrated-guide-to-a-phd/

jballer 3y ago

This was what a few academics hypothesized and set out to prove in what is now known as the Grievance Studies Affair [1]. Not only did they get multiple hoax papers published, they got recognized for “excellent scholarship” [2]. [1] https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Grievance_studies_affair [2] https://youtu.be/kVk9a5Jcd1k?t=3m49s https://youtu.be/kVk9a5Jcd1k?t=3m49s

fenomas 3y ago
That grievance studies thing was idiotic, and really ought to be dismissed and forgotten. Reasons: 1. The hoax authors didn't submit gibberish or anything, like with Sokal - they submitted coherent papers that they considered absurd, but which they expected the journals to find reasonable. As such, they only thing they can even pretend to have proven is that they and the journals disagree about what's reasonable. 2. IIRC, every one of the initial hoax papers was flat-out rejected! If the authors had any backbone they could have stopped right there, and published a paper called "These grievance journals aren't as bad as we expected." :D But they didn't - they set out to iterate on each paper until a journal accepted it. And if your experimental methodology is "keep trying different things until X happens", you can hardly pretend to have proved something when X happens. I mean, I agree with the premise the hoax authors were trying to prove. But their "experiment" was silly, and I think it's dishonest of them to pretend it proved something. Meh. Meh!
- foldr 3y ago
  Excellent summary. I’d also mention the lack of a control group. We’re supposed to conclude that it’s easier to (eventually) get fraudulent results published in so-called ‘grievance’ fields than it is in (say) physics. But the hoax authors never made any effort to publish fraudulent results in those fields, so we can’t know if the percentage of papers that were eventually accepted would have been lower or not.
  - jballer 3y ago
    We were typing at the same time - see my other comment (sibling to yours) linking to Thiel arguing things are likely worse in hard sciences, and encouraging people to find ways to expose it.

chmike 3y ago

The peer review system is definitely biased. It's ok for many articles, but there are outliers. For instance really original theories or experimental data. The publication may be stopped to avoid putting the journal's reputation at risk or because the reviewers have personal reasons to not support the article (it would shade their own pet theory, invalidate their research project, they didn't understood it, etc.). The problem with publishing elsewhere is that many people unable to objectively evaluate the validity of the theory juge its value by the reputation of the journal. Also the other journal will most probably have less visibility. There is thus a higher chance that the other theory will not be reported in reviews. There are many assumptions in the above comment that I can't agree with. But my experience is with physics, not computer science.