5 ms·

``we need to get past the antiquated idea that the singular act of publication – or publication in a particular journal – should signal for all eternity that a

``we need to get past the antiquated idea that the singular act of publication – or publication in a particular journal – should signal for all eternity that a paper is valid, let alone important'' Most scientists I know (physics) don't have this idea. They know publication is just a step. Most grad students learn at some point that just because it's published doesn't mean it's correct. Publication is a checkpoint, not a finish line, for any theory or experimental result.

colechristensen 13y ago
You're absolutely correct, scientists don't suffer from any misconception that publishing equals truth. The general public, on the other hand, has little understanding of this.
- talmand 13y ago
  Nor do the reporters and/or editorials that cite such peer-reviewed papers that agree with their opinions. Wait, maybe they do understand but they also understand that the general public doesn't understand.
  - GrumpySimon 13y ago
    I think this is an artefact of how science is taught: In 1953 Watson and Crick published a paper on the structure of DNA. Everyone smiled and nodded. In my experience it's more like: Most scientists ignored it. Others were interested but had too much teaching to pay attention to it. A few people got pissed off and tried to prove it wrong. A few other people got excited and tried to build on it.
- GrumpySimon 13y ago
  Absolutely. I see peer review as a way of - hopefully - filtering out the most egregious crap, and fixing the worst errors in papers while rounding them off. I feel all these debates are kind of pointless, because no-one I know treats a published article as gospel. We make up our own minds about the paper.
jofer 13y ago
Very true, but he does go on to state: > If we had, instead, a system where the review process was transparent and persisted for the useful life of a work That is a very interesting idea. (Reviews are currently confidential and usually anonymous -- only the author and editor sees the reviews, and only the editor knows who wrote them.) Not only would this reduce blatantly petty reviews, it would also give some recognition to the work that goes into reviewing a paper. It's currently an important but very thankless job. It's considered polite to acknowledge reviewers at the end of the paper (even if they remain anonymous), but it's often not done.
- pizza_boy 13y ago
  > it would also give some recognition to the work that goes into reviewing a paper I think you're spot on with this comment. That's exactly what we're building at publons.com. Check it out.
- tigroferoce 13y ago
  Completely agree. I've been into computer science research for few years and not knowing who accepted the papers and which were the reviews always bugged me. I've always advocated that the reviews behind an accepted paper SHOULD all be public and publicly distributed along with the paper.
  - thaumasiotes 13y ago
    This is definitely a benefit to the readers, and potentially (though not definitely) to the community as a whole. However, it's much worse for the reviewers. I hear so much about how researchers provide papers to journals for free, and review papers for journals for free, and then the journals (are presumably evil to) charge high fees to anyone who wants to read the papers. How much can we pile on the reviewers before they stop doing it?
    - tigroferoce 13y ago
      Good point. But it is a key point that making reviews is part of your job as researched and you MUST take it very seriously. I received so many reviews poorly written, just few lines saying "OK for me". I really think that this way of reviewing is wrong. Reviewers should take care in their job and they should be ready to face a public discussion on why they decided to accept a paper. Note that IMHO is OK to be anonymous while refusing a paper because otherwise a reviewer may be put under lot of pressure.
cma 13y ago
Physics is an area where almost all papers have pre-prints available freely on arXive.org... sort of going along with the author's point?
narrator 13y ago
The converse also holds true. Just because it isn't published doesn't mean it isn't science. Lots of trade secret processes and such fall under this.
utopkara 13y ago
So true! The health of scientific literature is not in the articles that claim things, but in the follow-up articles that verify and/or disprove. Skepticism is at the very core of science. Partly to blame are the "non-fiction/science&nature" book category, and the journalist/writer cross-breed who chain-write in newspapers, magazines and churn out books from their articles every year. At first, the really good ones were a welcome addition to Science literature, for bringing the science to our daily lives. But then they became part of the "where it bleeds, it leads" culture, sans proper journalism and sans scientific thought. Scientists may make fun of how journal articles are covered in newspapers and blogs, but the negative effects of such coverage are quite real.
SoftwareMaven 13y ago
Contrast that, though, with the extremely sad state of nutrition, where an absolute difference of 0.25% gets trumpeted immediately to the media at "200% more likely", even though the effects are measured through observational studies with no proper controls. A field where who you are matters much more than the data you're presenting (and it seems reasonably accepted to throw out data that disagrees with your pet hypothesis). I recognize that, in many ways, physics has it easy: there are few sciences that reduce as well as it does and that offer ways to isolate variables like it does (even if it requires multi-billion dollar machines and years to make that happen). But it is still a shining exemplar of gradual evidence building theories to find more evidence, and there are many, many fields that aren't anywhere near that.
haproxy 13y ago
http://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html http://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html
DaveWalk 13y ago
In the biomedical sciences (where the original author is speaking from), there is a dichotomy: yes, most researchers don't believe that a single publication is the final word on a topic and that in theory anything could be retracted or modified, but in practice that's how the system works. Publications--both how many you have and where they are--are the prime metric for promotions, tenure, and overall success. Furthermore, when applying for funding your publication record is taken into account, as well as your promotions, tenure, etc. It is currency in the life of an investigator. Not to mention that research is all about who comes in first; the incentive to validate published work is by and large not there. I recall a Nature article on data reproducibility: only 11% of the TOP, PEER-REVIEWED PAPERS were able to be reproduced[1]. [1] http://www.nature.com/nature/journal/v483/n7391/full/483531a.html http://www.nature.com/nature/journal/v483/n7391/full/483531a...

colechristensen 13y ago

You're absolutely correct, scientists don't suffer from any misconception that publishing equals truth. The general public, on the other hand, has little understanding of this.

talmand 13y ago
Nor do the reporters and/or editorials that cite such peer-reviewed papers that agree with their opinions. Wait, maybe they do understand but they also understand that the general public doesn't understand.
- GrumpySimon 13y ago
  I think this is an artefact of how science is taught: In 1953 Watson and Crick published a paper on the structure of DNA. Everyone smiled and nodded. In my experience it's more like: Most scientists ignored it. Others were interested but had too much teaching to pay attention to it. A few people got pissed off and tried to prove it wrong. A few other people got excited and tried to build on it.
GrumpySimon 13y ago
Absolutely. I see peer review as a way of - hopefully - filtering out the most egregious crap, and fixing the worst errors in papers while rounding them off. I feel all these debates are kind of pointless, because no-one I know treats a published article as gospel. We make up our own minds about the paper.

jofer 13y ago

Very true, but he does go on to state: > If we had, instead, a system where the review process was transparent and persisted for the useful life of a work That is a very interesting idea. (Reviews are currently confidential and usually anonymous -- only the author and editor sees the reviews, and only the editor knows who wrote them.) Not only would this reduce blatantly petty reviews, it would also give some recognition to the work that goes into reviewing a paper. It's currently an important but very thankless job. It's considered polite to acknowledge reviewers at the end of the paper (even if they remain anonymous), but it's often not done.

pizza_boy 13y ago
> it would also give some recognition to the work that goes into reviewing a paper I think you're spot on with this comment. That's exactly what we're building at publons.com. Check it out.
tigroferoce 13y ago
Completely agree. I've been into computer science research for few years and not knowing who accepted the papers and which were the reviews always bugged me. I've always advocated that the reviews behind an accepted paper SHOULD all be public and publicly distributed along with the paper.
- thaumasiotes 13y ago
  This is definitely a benefit to the readers, and potentially (though not definitely) to the community as a whole. However, it's much worse for the reviewers. I hear so much about how researchers provide papers to journals for free, and review papers for journals for free, and then the journals (are presumably evil to) charge high fees to anyone who wants to read the papers. How much can we pile on the reviewers before they stop doing it?
  - tigroferoce 13y ago
    Good point. But it is a key point that making reviews is part of your job as researched and you MUST take it very seriously. I received so many reviews poorly written, just few lines saying "OK for me". I really think that this way of reviewing is wrong. Reviewers should take care in their job and they should be ready to face a public discussion on why they decided to accept a paper. Note that IMHO is OK to be anonymous while refusing a paper because otherwise a reviewer may be put under lot of pressure.

cma 13y ago

Physics is an area where almost all papers have pre-prints available freely on arXive.org... sort of going along with the author's point?

narrator 13y ago

The converse also holds true. Just because it isn't published doesn't mean it isn't science. Lots of trade secret processes and such fall under this.

utopkara 13y ago

So true! The health of scientific literature is not in the articles that claim things, but in the follow-up articles that verify and/or disprove. Skepticism is at the very core of science. Partly to blame are the "non-fiction/science&nature" book category, and the journalist/writer cross-breed who chain-write in newspapers, magazines and churn out books from their articles every year. At first, the really good ones were a welcome addition to Science literature, for bringing the science to our daily lives. But then they became part of the "where it bleeds, it leads" culture, sans proper journalism and sans scientific thought. Scientists may make fun of how journal articles are covered in newspapers and blogs, but the negative effects of such coverage are quite real.

SoftwareMaven 13y ago

Contrast that, though, with the extremely sad state of nutrition, where an absolute difference of 0.25% gets trumpeted immediately to the media at "200% more likely", even though the effects are measured through observational studies with no proper controls. A field where who you are matters much more than the data you're presenting (and it seems reasonably accepted to throw out data that disagrees with your pet hypothesis). I recognize that, in many ways, physics has it easy: there are few sciences that reduce as well as it does and that offer ways to isolate variables like it does (even if it requires multi-billion dollar machines and years to make that happen). But it is still a shining exemplar of gradual evidence building theories to find more evidence, and there are many, many fields that aren't anywhere near that.

haproxy 13y ago

http://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html http://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html

DaveWalk 13y ago

In the biomedical sciences (where the original author is speaking from), there is a dichotomy: yes, most researchers don't believe that a single publication is the final word on a topic and that in theory anything could be retracted or modified, but in practice that's how the system works. Publications--both how many you have and where they are--are the prime metric for promotions, tenure, and overall success. Furthermore, when applying for funding your publication record is taken into account, as well as your promotions, tenure, etc. It is currency in the life of an investigator. Not to mention that research is all about who comes in first; the incentive to validate published work is by and large not there. I recall a Nature article on data reproducibility: only 11% of the TOP, PEER-REVIEWED PAPERS were able to be reproduced[1]. [1] http://www.nature.com/nature/journal/v483/n7391/full/483531a.html http://www.nature.com/nature/journal/v483/n7391/full/483531a...