3 ms·

This issue of scientists being OK with totally non-scientific processes is all too common. When I was an academic my Department used to make an enormous fuss ab

This issue of scientists being OK with totally non-scientific processes is all too common. When I was an academic my Department used to make an enormous fuss about the year to year jitter in student teaching evaluations. My colleagues (all scientists) would sit around discussing why they were a heroes because their evaluation went up 10% from last year or what they had to change because it went down 10%. I used to just sit there thinking if this sort of analysis was in a paper they were reviewing they would have ripped the authors to shreds. Peer review fails on all levels. It does not functions as a source of quality control (everything gets published eventually) and even worse it rarely improves the quality of the paper being reviewed. I have published dozens of papers over the years and on only one occasion has the review process improved the paper - in most cases the reviewers demands made the papers worse by forcing me to remove important information or include irrelevant details (citing the reviewers publications mostly).

chunky1994 10y ago
I think that example is a bit unfair, teaching really isn't scientific when you get to the core of it. You cannot expect professors to be able to conduct double blind studies with student grades and so a more overarching non-scientific discussion is probably more amenable to actual improvement in performance.
- danieltillett 10y ago
  You can actually conduct scientific studies on teaching and grading, but that is not the point I was making. What concerned me is how a department full of scientists wasted their time discussing the importance of results which were statistically meaningless. The scatter was just noise yet we wasted hours (days) of our time discussing the supposed importance of this noise.
quantumhobbit 10y ago
Peer review is broken not because the process of review is broken but because the peers are broken. Reviewers, like all academics, have screwed up incentives to get there name on as many papers as possible and those papers cited widely, regardless of the quality of the science involved. They will always find a way game the system to their benefit. Recently heard about reviewers of a friend's paper blocking publication until the reviewer could publish his research on the same topic first. The whole system is just too screwed up to be fixed.
- danieltillett 10y ago
  Yes this sort of bad thing happens, but more in the real high stakes games of grant reviews than publications. You can easily kill the grant of one of your competitors by just giving it a mediocre review (without any basis of course). I would see this happen all the time to my colleagues where they would get a spiteful review on a grant that would kill it and there was nothing they could do.
  - dangerlibrary 10y ago
    At the NIH, at least, I know that enormous effort is put into ensuring that there are not conflicts of interest between the reviewers and those being reviewed. I'm not particularly surprised to hear that some places do not put in that effort - it is a slow, painstaking process - but there are places where this is less of a problem.
    - danieltillett 10y ago
      I am in Australia where the pool of potential grant reviewers is pretty small for most areas. When you have a grant success rate of 15% it is all too easy to nobble your competitors via the review process.
      - jsprogrammer 10y ago
        Why is science viewed as a competition there?
        dflock 10y ago
        For exactly the same reasons it is everywhere in the world: because there isn't enough funding to go around - and the grant application process is explicitly designed to be competitive.
        jsprogrammer 10y ago
xioxox 10y ago
> and even worse it rarely improves the quality of the paper being reviewed. That might be your experience, but I think it has significantly improved the quality of a large fraction of the papers I have been an author or co-author on. I have encountered one bad reviewer in the past, and in retrospect perhaps I should have taken more notice of his report. As a reviewer, I've only had one negative incident, which was when my comments were overridden by the editor. My experience might be specific in my subject, as we only have one referee per paper.
- danieltillett 10y ago
  I am curious as to what improvements the reviewers made to your papers? What areas did they improve? In the case of the one paper of mine that was improved one of the reviewers noticed a connection between our data and a paper published 10 years before in a different field that made our work much more interesting. Of course after incorporating this the editor rejected the paper despite all three reviewers saying it should be published. The editor of the next journal I sent the paper to was happy especially as the paper has since picked up over 1000 citations, but such is the vagaries of peer review.
  - xioxox 10y ago
    The suggestions have included alternative explanations for the results, useful checks on the data analysis procedures, sensible ways to restructure a paper, useful plots which could be included, places where the text was unclear, indicating where our claims were too strong, how to strengthen the paper by comparisons with theory. One referee's questions were useful to reveal that an undergraduate's paper (of which I was a co-author) had some fairly serious flaws which required it was withdrawn. Fortunately this has been the only "rejection" I've suffered so far, other than papers speculatively submitted to Nature or Science which were later published in mainstream journals. In astronomy and astrophysics we are fortunate to have only 3-4 major journals with similar impact factors.
    - danieltillett 10y ago
      I wish that was the way the biological sciences worked. My personal favourite peer review process story was from one of my first papers where I wrote up the whole thing and my supervisor added his name and sent it off. This was in the pre-electronic days where you physically posted off the manuscript. Serveral months when by and we heard nothing. After around six months of silence my supervisor decides to ring the editor and find out what is happening (proactive he was not). The editor tells him "oh didn't you know that paper was published a couple of months ago" - not a single change made or reviewers comment to deal with. I wish all my subsequent papers had gone like that :)

chunky1994 10y ago

I think that example is a bit unfair, teaching really isn't scientific when you get to the core of it. You cannot expect professors to be able to conduct double blind studies with student grades and so a more overarching non-scientific discussion is probably more amenable to actual improvement in performance.

danieltillett 10y ago
You can actually conduct scientific studies on teaching and grading, but that is not the point I was making. What concerned me is how a department full of scientists wasted their time discussing the importance of results which were statistically meaningless. The scatter was just noise yet we wasted hours (days) of our time discussing the supposed importance of this noise.

quantumhobbit 10y ago

Peer review is broken not because the process of review is broken but because the peers are broken. Reviewers, like all academics, have screwed up incentives to get there name on as many papers as possible and those papers cited widely, regardless of the quality of the science involved. They will always find a way game the system to their benefit. Recently heard about reviewers of a friend's paper blocking publication until the reviewer could publish his research on the same topic first. The whole system is just too screwed up to be fixed.

danieltillett 10y ago
Yes this sort of bad thing happens, but more in the real high stakes games of grant reviews than publications. You can easily kill the grant of one of your competitors by just giving it a mediocre review (without any basis of course). I would see this happen all the time to my colleagues where they would get a spiteful review on a grant that would kill it and there was nothing they could do.
- dangerlibrary 10y ago
  At the NIH, at least, I know that enormous effort is put into ensuring that there are not conflicts of interest between the reviewers and those being reviewed. I'm not particularly surprised to hear that some places do not put in that effort - it is a slow, painstaking process - but there are places where this is less of a problem.
  - danieltillett 10y ago
    I am in Australia where the pool of potential grant reviewers is pretty small for most areas. When you have a grant success rate of 15% it is all too easy to nobble your competitors via the review process.
    - jsprogrammer 10y ago
      Why is science viewed as a competition there?
      - dflock 10y ago
        For exactly the same reasons it is everywhere in the world: because there isn't enough funding to go around - and the grant application process is explicitly designed to be competitive.
        jsprogrammer 10y ago

xioxox 10y ago

> and even worse it rarely improves the quality of the paper being reviewed. That might be your experience, but I think it has significantly improved the quality of a large fraction of the papers I have been an author or co-author on. I have encountered one bad reviewer in the past, and in retrospect perhaps I should have taken more notice of his report. As a reviewer, I've only had one negative incident, which was when my comments were overridden by the editor. My experience might be specific in my subject, as we only have one referee per paper.

danieltillett 10y ago
I am curious as to what improvements the reviewers made to your papers? What areas did they improve? In the case of the one paper of mine that was improved one of the reviewers noticed a connection between our data and a paper published 10 years before in a different field that made our work much more interesting. Of course after incorporating this the editor rejected the paper despite all three reviewers saying it should be published. The editor of the next journal I sent the paper to was happy especially as the paper has since picked up over 1000 citations, but such is the vagaries of peer review.
- xioxox 10y ago
  The suggestions have included alternative explanations for the results, useful checks on the data analysis procedures, sensible ways to restructure a paper, useful plots which could be included, places where the text was unclear, indicating where our claims were too strong, how to strengthen the paper by comparisons with theory. One referee's questions were useful to reveal that an undergraduate's paper (of which I was a co-author) had some fairly serious flaws which required it was withdrawn. Fortunately this has been the only "rejection" I've suffered so far, other than papers speculatively submitted to Nature or Science which were later published in mainstream journals. In astronomy and astrophysics we are fortunate to have only 3-4 major journals with similar impact factors.
  - danieltillett 10y ago
    I wish that was the way the biological sciences worked. My personal favourite peer review process story was from one of my first papers where I wrote up the whole thing and my supervisor added his name and sent it off. This was in the pre-electronic days where you physically posted off the manuscript. Serveral months when by and we heard nothing. After around six months of silence my supervisor decides to ring the editor and find out what is happening (proactive he was not). The editor tells him "oh didn't you know that paper was published a couple of months ago" - not a single change made or reviewers comment to deal with. I wish all my subsequent papers had gone like that :)