6 ms·

As someone who does peer review for 2-3 conferences a year, I will say that peer review is still better than any alternative proposed so far. In my circles (Co

As someone who does peer review for 2-3 conferences a year, I will say that peer review is still better than any alternative proposed so far. In my circles (Computational Linguistics), getting your paper approved means that you convinced at least three PhD students with a published paper (or higher, all the way up to Professor) that your paper is good PLUS the area chair(s) considered it worth publishing. All of this without knowing who you are nor what your institution is. And once your paper is accepted it is freely available on the internet forever. No human activity is free of issues, but to me double-blind peer review is a fairly effective system. And every year the system is tweaked to try and adapt it to newer developments such as the rise of ArXiv. Most criticism I read uses the term "peer review" to refer to whatever Nature, Science, and/or Elsevier are doing at the moment, but just because they suck it doesn't mean that peer-review as a whole does.

DoreenMichele 4y ago
No, no human activity is free of issues. But historically when the world was smaller, people writing papers were more likely to "speak the same language," have experiences in common that fostered clear communication, etc. It's a big, wide world. The sheer variety of experiences these days introduces complications never before seen.
Schroedingersat 4y ago
Reproduction by n sources with n-k unaffiliated sources would be much better, but that requires a lot more resources for research that could otherwise go to yachts or the latest ponzi scheme.
im3w1l 4y ago
> getting your paper approved means that you convinced at least three PhD students with a published paper This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". If you have a good initial condition maybe it can work for a while, but it seems that time may be coming to an end. > Pournelle's Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people: First, there will be those who are devoted to the goals of the organization. Examples are dedicated classroom teachers in an educational bureaucracy, many of the engineers and launch technicians and scientists at NASA, even some agricultural scientists and advisors in the former Soviet Union collective farming administration. Secondly, there will be those dedicated to the organization itself. Examples are many of the administrators in the education system, many professors of education, many teachers union officials, much of the NASA headquarters staff, etc. The Iron Law states that in every case the second group will gain and keep control of the organization. It will write the rules, and control promotions within the organization.
- AshamedCaptain 4y ago
  > This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". If you have a good initial condition maybe it can work for a while, but it seems that time may be coming to an end. There is no formal definition for a "scientist" and likely will never be. Definitely a scientist is not "one who has a paper accepted". Here the commenter is just claiming that in order to publish a paper in a specific journal you do need be accepted by a handful of people who have previously been accepted in the same journal. Barring the obvious chicken and egg problem which is easily solved, it does not seem to be like something which "is coming to an end".
- amelius 4y ago
  > This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". You're simply reformulating what peer means in peer-review. What other way would you suggest? Layman-review?
  - im3w1l 4y ago
    I had a couple approaches in mind, but I think I got carried away in the moment when I wrote that comment. A radical overhaul may do more harm then good.
    - still_grokking 4y ago
      But you were right in the first comment! At the moment people do things "for living", in those usual bureaucratic institutions that all institutions become sooner or later, there are initiatives at power that work against the initial idea behind that institution. That's why progress in science occurs at best "one funeral at a time", but usually not before a whole generation bites the dust. This plus bureaucracy seem to almost killed larger parts of science by now. We have collusion rings milking the system. We have broad pure esoteric nonsense "science" circles. We have more and more utter garbage and scam. And we have holy churches that will send the inquisition on you if you dare to disagree. I don't know how to fix that. But I guess the initiatives, and especially the cash flows would need some re-engineering. But how concretely, I have no clue. Throwing away everything you have is of course almost never a good idea. But some things need to change. Change in a radical way, I guess.
UncleMeat 4y ago
What I wish is that peer review did not merge so many things into one step. Peer review is used to test for 1. The validity of the results 2. The novelty of the results 3. The importance of the results Papers are routinely rejected in peer review not because their methods or results are suspect but because their results are not seen as impactful enough to justify space in a conference or journal. This sucks and introduces all sorts of nasty biases into the community. I'd love for these to be disconnected so one could demonstrate that their paper had been read by other experts and found to be within reason without having to worry that one person on the PC who just doesn't think that a certain research direction is valuable will sink my career.
- amelius 4y ago
  Why is space even an issue these days. Everything is digital anyway.
koalakeeper 4y ago
“Even when the experts all agree, they may well be mistaken.” - Bertrand Russell

DoreenMichele 4y ago

No, no human activity is free of issues. But historically when the world was smaller, people writing papers were more likely to "speak the same language," have experiences in common that fostered clear communication, etc. It's a big, wide world. The sheer variety of experiences these days introduces complications never before seen.

Schroedingersat 4y ago

Reproduction by n sources with n-k unaffiliated sources would be much better, but that requires a lot more resources for research that could otherwise go to yachts or the latest ponzi scheme.

im3w1l 4y ago

> getting your paper approved means that you convinced at least three PhD students with a published paper This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". If you have a good initial condition maybe it can work for a while, but it seems that time may be coming to an end. > Pournelle's Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people: First, there will be those who are devoted to the goals of the organization. Examples are dedicated classroom teachers in an educational bureaucracy, many of the engineers and launch technicians and scientists at NASA, even some agricultural scientists and advisors in the former Soviet Union collective farming administration. Secondly, there will be those dedicated to the organization itself. Examples are many of the administrators in the education system, many professors of education, many teachers union officials, much of the NASA headquarters staff, etc. The Iron Law states that in every case the second group will gain and keep control of the organization. It will write the rules, and control promotions within the organization.

AshamedCaptain 4y ago
> This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". If you have a good initial condition maybe it can work for a while, but it seems that time may be coming to an end. There is no formal definition for a "scientist" and likely will never be. Definitely a scientist is not "one who has a paper accepted". Here the commenter is just claiming that in order to publish a paper in a specific journal you do need be accepted by a handful of people who have previously been accepted in the same journal. Barring the obvious chicken and egg problem which is easily solved, it does not seem to be like something which "is coming to an end".
amelius 4y ago
> This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". You're simply reformulating what peer means in peer-review. What other way would you suggest? Layman-review?
- im3w1l 4y ago
  I had a couple approaches in mind, but I think I got carried away in the moment when I wrote that comment. A radical overhaul may do more harm then good.
  - still_grokking 4y ago
    But you were right in the first comment! At the moment people do things "for living", in those usual bureaucratic institutions that all institutions become sooner or later, there are initiatives at power that work against the initial idea behind that institution. That's why progress in science occurs at best "one funeral at a time", but usually not before a whole generation bites the dust. This plus bureaucracy seem to almost killed larger parts of science by now. We have collusion rings milking the system. We have broad pure esoteric nonsense "science" circles. We have more and more utter garbage and scam. And we have holy churches that will send the inquisition on you if you dare to disagree. I don't know how to fix that. But I guess the initiatives, and especially the cash flows would need some re-engineering. But how concretely, I have no clue. Throwing away everything you have is of course almost never a good idea. But some things need to change. Change in a radical way, I guess.

UncleMeat 4y ago

What I wish is that peer review did not merge so many things into one step. Peer review is used to test for 1. The validity of the results 2. The novelty of the results 3. The importance of the results Papers are routinely rejected in peer review not because their methods or results are suspect but because their results are not seen as impactful enough to justify space in a conference or journal. This sucks and introduces all sorts of nasty biases into the community. I'd love for these to be disconnected so one could demonstrate that their paper had been read by other experts and found to be within reason without having to worry that one person on the PC who just doesn't think that a certain research direction is valuable will sink my career.

amelius 4y ago
Why is space even an issue these days. Everything is digital anyway.

koalakeeper 4y ago

“Even when the experts all agree, they may well be mistaken.” - Bertrand Russell