3 ms·

Peer review has issues too at the moment. From what I've read, it worked when the world was a smaller place, basically. Now, peer review is another broken cog i

Peer review has issues too at the moment. From what I've read, it worked when the world was a smaller place, basically. Now, peer review is another broken cog in a machine that grew organically out of past practices and which no one really intended and no one knows how to fix.

probably_wrong 4y ago
As someone who does peer review for 2-3 conferences a year, I will say that peer review is still better than any alternative proposed so far. In my circles (Computational Linguistics), getting your paper approved means that you convinced at least three PhD students with a published paper (or higher, all the way up to Professor) that your paper is good PLUS the area chair(s) considered it worth publishing. All of this without knowing who you are nor what your institution is. And once your paper is accepted it is freely available on the internet forever. No human activity is free of issues, but to me double-blind peer review is a fairly effective system. And every year the system is tweaked to try and adapt it to newer developments such as the rise of ArXiv. Most criticism I read uses the term "peer review" to refer to whatever Nature, Science, and/or Elsevier are doing at the moment, but just because they suck it doesn't mean that peer-review as a whole does.
- DoreenMichele 4y ago
  No, no human activity is free of issues. But historically when the world was smaller, people writing papers were more likely to "speak the same language," have experiences in common that fostered clear communication, etc. It's a big, wide world. The sheer variety of experiences these days introduces complications never before seen.
- Schroedingersat 4y ago
  Reproduction by n sources with n-k unaffiliated sources would be much better, but that requires a lot more resources for research that could otherwise go to yachts or the latest ponzi scheme.
- im3w1l 4y ago
  > getting your paper approved means that you convinced at least three PhD students with a published paper This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". If you have a good initial condition maybe it can work for a while, but it seems that time may be coming to an end. > Pournelle's Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people: First, there will be those who are devoted to the goals of the organization. Examples are dedicated classroom teachers in an educational bureaucracy, many of the engineers and launch technicians and scientists at NASA, even some agricultural scientists and advisors in the former Soviet Union collective farming administration. Secondly, there will be those dedicated to the organization itself. Examples are many of the administrators in the education system, many professors of education, many teachers union officials, much of the NASA headquarters staff, etc. The Iron Law states that in every case the second group will gain and keep control of the organization. It will write the rules, and control promotions within the organization.
  - AshamedCaptain 4y ago
    > This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". If you have a good initial condition maybe it can work for a while, but it seems that time may be coming to an end. There is no formal definition for a "scientist" and likely will never be. Definitely a scientist is not "one who has a paper accepted". Here the commenter is just claiming that in order to publish a paper in a specific journal you do need be accepted by a handful of people who have previously been accepted in the same journal. Barring the obvious chicken and egg problem which is easily solved, it does not seem to be like something which "is coming to an end".
  - amelius 4y ago
    > This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". You're simply reformulating what peer means in peer-review. What other way would you suggest? Layman-review?
    - im3w1l 4y ago
- UncleMeat 4y ago
  What I wish is that peer review did not merge so many things into one step. Peer review is used to test for 1. The validity of the results 2. The novelty of the results 3. The importance of the results Papers are routinely rejected in peer review not because their methods or results are suspect but because their results are not seen as impactful enough to justify space in a conference or journal. This sucks and introduces all sorts of nasty biases into the community. I'd love for these to be disconnected so one could demonstrate that their paper had been read by other experts and found to be within reason without having to worry that one person on the PC who just doesn't think that a certain research direction is valuable will sink my career.
  - amelius 4y ago
    Why is space even an issue these days. Everything is digital anyway.
- koalakeeper 4y ago
  “Even when the experts all agree, they may well be mistaken.” - Bertrand Russell

probably_wrong 4y ago

As someone who does peer review for 2-3 conferences a year, I will say that peer review is still better than any alternative proposed so far. In my circles (Computational Linguistics), getting your paper approved means that you convinced at least three PhD students with a published paper (or higher, all the way up to Professor) that your paper is good PLUS the area chair(s) considered it worth publishing. All of this without knowing who you are nor what your institution is. And once your paper is accepted it is freely available on the internet forever. No human activity is free of issues, but to me double-blind peer review is a fairly effective system. And every year the system is tweaked to try and adapt it to newer developments such as the rise of ArXiv. Most criticism I read uses the term "peer review" to refer to whatever Nature, Science, and/or Elsevier are doing at the moment, but just because they suck it doesn't mean that peer-review as a whole does.

DoreenMichele 4y ago
No, no human activity is free of issues. But historically when the world was smaller, people writing papers were more likely to "speak the same language," have experiences in common that fostered clear communication, etc. It's a big, wide world. The sheer variety of experiences these days introduces complications never before seen.
Schroedingersat 4y ago
Reproduction by n sources with n-k unaffiliated sources would be much better, but that requires a lot more resources for research that could otherwise go to yachts or the latest ponzi scheme.
im3w1l 4y ago
> getting your paper approved means that you convinced at least three PhD students with a published paper This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". If you have a good initial condition maybe it can work for a while, but it seems that time may be coming to an end. > Pournelle's Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people: First, there will be those who are devoted to the goals of the organization. Examples are dedicated classroom teachers in an educational bureaucracy, many of the engineers and launch technicians and scientists at NASA, even some agricultural scientists and advisors in the former Soviet Union collective farming administration. Secondly, there will be those dedicated to the organization itself. Examples are many of the administrators in the education system, many professors of education, many teachers union officials, much of the NASA headquarters staff, etc. The Iron Law states that in every case the second group will gain and keep control of the organization. It will write the rules, and control promotions within the organization.
- AshamedCaptain 4y ago
  > This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". If you have a good initial condition maybe it can work for a while, but it seems that time may be coming to an end. There is no formal definition for a "scientist" and likely will never be. Definitely a scientist is not "one who has a paper accepted". Here the commenter is just claiming that in order to publish a paper in a specific journal you do need be accepted by a handful of people who have previously been accepted in the same journal. Barring the obvious chicken and egg problem which is easily solved, it does not seem to be like something which "is coming to an end".
- amelius 4y ago
  > This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". You're simply reformulating what peer means in peer-review. What other way would you suggest? Layman-review?
  - im3w1l 4y ago
UncleMeat 4y ago
What I wish is that peer review did not merge so many things into one step. Peer review is used to test for 1. The validity of the results 2. The novelty of the results 3. The importance of the results Papers are routinely rejected in peer review not because their methods or results are suspect but because their results are not seen as impactful enough to justify space in a conference or journal. This sucks and introduces all sorts of nasty biases into the community. I'd love for these to be disconnected so one could demonstrate that their paper had been read by other experts and found to be within reason without having to worry that one person on the PC who just doesn't think that a certain research direction is valuable will sink my career.
- amelius 4y ago
  Why is space even an issue these days. Everything is digital anyway.
koalakeeper 4y ago
“Even when the experts all agree, they may well be mistaken.” - Bertrand Russell