6 ms·

Well, however, we can argue here that a pre-print is not of the same quality as a published work. Not that published works cannot have errors (and be retracted)

by kensai 10y ago

Well, however, we can argue here that a pre-print is not of the same quality as a published work. Not that published works cannot have errors (and be retracted), but at least they have passed a first-level scrutiny of an editorial board.

dagenleg 10y ago
Yes, but passing through the editorial board takes time. In some fields research moves extremely fast and by the time papers is properly published it's already obsolete.
- kensai 10y ago
  That's the price of scrutiny, man. Time. And these "extremely fast fields" are the same fields that need to backtrack and retract their claims. The loss of time, effort, and resources for other scientists if they use pre-printed claims and data from faulty publications can be really enormous and probably already wildly underestimated.
  - IanCal 10y ago
    The flow is significantly slower than is required for checking work. I'm fairly sure my wifes papers didn't really take nearly two years to review. Machine learning is a field that's moving quickly and can also largely be checked quickly by other people in the field too. The problems that can't be checked easily are also those that won't be checked by the journals anyway (no journal I know of will retrain one of googles deep nets to see if they get the same result before publishing).
    - kensai 10y ago
      If you think peer-reviewing is slow then it's probably because you don't get the PEER part of the reviewing process. The PEERS are simply colleagues who have their own lives, priorities, and research. They don't stop what they are doing to review your wife's papers. And please, I see what's happening here: I have been portrayed as the "traditional journal apologist" although it's definitely what I wanted here. I simply wanted to express my doubts that pre-printing is the answer to it all. Anyway, is there a study that shows how many pre-printed articles in ML have been retracted or refuted so far? Until this is done, and shown as small as in peer-reviewed journals, keep a small basket.
      - IanCal 10y ago
        > If you think peer-reviewing is slow then it's probably because you don't get the PEER part of the reviewing process. I understand the process fully. It doesn't make it fast, nor does it make it a necessary cost to pay. Delaying access to content for several years does not solve a problem. A not-insignificant time was spent bouncing between people to sort out who was paying for the costs, then there is also a delay between acceptance and publication. This now averages just a month in pubmed, but papers can bounce around this point for a lot longer. I am not arguing for pre-prints to replace traditional publishing, but the speed of spreading information is undeniably faster, and that's what started this whole chain of comments. > Anyway, is there a study that shows how many pre-printed articles in ML have been retracted or refuted so far? Until this is done, and shown as small as in peer-reviewed journals, keep a small basket. I don't get the phrase "keep a small basket", but no I've not seen this. The point is simply that PEERS (if we need to shout the word) can in many cases replicate the work and assess the results much more quickly than the traditional review & edit process. I picked the field because I see people re-implement work described in the papers very quickly, or the original authors share the trained models and code. I'd also caution against using "retracted" papers as a measure, some journals charge for a retraction.
        kensai 10y ago
        "I understand the process fully. It doesn't make it fast, nor does it make it a necessary cost to pay. Delaying access to content for several years does not solve a problem. A not-insignificant time was spent bouncing between people to sort out who was paying for the costs, then there is also a delay between acceptance and publication. This now averages just a month in pubmed, but papers can bounce around this point for a lot longer." You don't understand a thing and that was the proof of it, at least for me. The process of peers evaluating a paper takes a lot of time because it cannot be automated and is serious, especially for the better journals. Of course bouncing people (referees) is part of the process, to find the better and/or most available one. Of course you can cut corners and pre-print, especially not having your paper (peer) reviewed, potentially fraught with errors and other inaccuracies. Problem is, not all sciences are like computer science where the proof could be algorithmically verified quickly. Publishing bullshit in biology, for example, may mean others losing two-three years trying to replicate your bullshit pre-printed results. "I am not arguing for pre-prints to replace traditional publishing, but the speed of spreading information is undeniably faster, and that's what started this whole chain of comments." Hey, I was not arguing traditional publishing is inherently better than pre-prints, but still got downvoted. "I don't get the phrase "keep a small basket", but no I've not seen this. The point is simply that PEERS (if we need to shout the word) can in many cases replicate the work and assess the results much more quickly than the traditional review & edit process. I picked the field because I see people re-implement work described in the papers very quickly, or the original authors share the trained models and code." Pre-printing might be the case in fields like CS and its subfields where verification is a very quick thing, but totally unsuitable in fields such as biology or medicine. The PEERS in the latter need also more time, given the complexity of the experimental setups. "I'd also caution against using "retracted" papers as a measure, some journals charge for a retraction." Oh trust me, if you publish publish to a good journal such a Nature or Science they find out (say bogus experiments), they will happily retract your paper. Happened with certain Korean stem cell researchers in the past.
  - leegao 10y ago
    That's not necessarily true across all "fast fields". There's definitely selection pressure against fields sensitive to faulty experimentation to "move fast." Computer Science is one of the bigger subjects within Arxiv. Retraction rates within this discipline is extremely low and the field as a whole moves very rapidly. Having access to preprints makes it much more scalable for researchers to stay at the cutting edge.
  - deleted 10y ago
    [deleted]
  - dagenleg 10y ago
    Look, you are free to abstain from using pre-printed articles for your research if you do have doubts. Why start a crusade though? ArXiv is working very well for a lot of scientists in Machine Learning, since must results can be doublechecked by simply running the code. I won't disregard a proved approach that I've myself seen working just because it hasn't been published yet.
- sgift 10y ago
  > In some fields research moves extremely fast and by the time papers is properly published it's already obsolete. Could you provide some examples for a layman? To me it always seems that science moves far slower than one could hope for (e.g.: Every 'breakthrough' in batteries/Graphene of the last ten years and still no products)
  - whatshisface 10y ago
    Think of it in terms of the speed of the field relative to the size of the paper - you aren't going to see no-graphene to industrial-graphene in a month, but you might see Johnson's 8% graphene be replaced by Smith's 8.01% graphene.
PaulHoule 10y ago
It is pretty controversial. The value of peer review has never really been demonstrated and despite peer review, the median scientific paper is wrong.
- JD557 10y ago
  >the median scientific paper is wrong. Are you being literal about that or is that just a figure of speech? Could you please provide a source for those claims, if true?
  - lkbm 10y ago
    I assume he's alluding to Ioannidis's 2005 paper "Why Most Published Research Findings Are False"[0] and/or the replication crisis[1] in general. [0] http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou... [1] http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 http://www.nature.com/news/1-500-scientists-lift-the-lid-on-...
    - PaulHoule 10y ago
      Partly that and also based on my experience getting a PhD in physics where I found: (i) published physics papers with 30 pages of math in them frequently had errors, (ii) if you have a large number of signs the odds are 50-50 you will get the sign right in a long calculation, and (iii) you can't say high energy physics papers are "right" or "wrong" anyway... I mean, does anybody think we really live in an anti-DeSitter space? Medicine on the other hand has the problem that you can't really afford large enough sample sizes to have sufficient statistical power.
      - nonbel 10y ago
        >"Medicine on the other hand has the problem that you can't really afford large enough sample sizes to have sufficient statistical power." This really isn't the major problem w/ modern medical research. In fact, if they had properly powered studies there would be far too many "discoveries" and the real problem would become obvious. The real issue is that the efforts to come up with and study models capable of precise predictions (eg Armitage-Doll, SIR, Frank-Starling, Hodgekin-Huxly) have been all but choked out in favor of people testing the vague hypothesis "there is a correlation". There is always some effect/correlation in systems like the human body, so it is only a matter of sample size. As explained long ago by Paul Meehl, this is a 180 degree about-face from what was previously called the scientific method: http://www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf http://www.fisme.science.uu.nl/staff/christianb/downloads/me...
Ar-Curunir 10y ago
Hmm actually in my experience the versions of papers uploaded to preprint servers are the final versions, including reviewers comments. This is in cryptography, but I think the same can be said for complexity theory.

by kensai 10y ago

dagenleg 10y ago

Yes, but passing through the editorial board takes time. In some fields research moves extremely fast and by the time papers is properly published it's already obsolete.

kensai 10y ago
That's the price of scrutiny, man. Time. And these "extremely fast fields" are the same fields that need to backtrack and retract their claims. The loss of time, effort, and resources for other scientists if they use pre-printed claims and data from faulty publications can be really enormous and probably already wildly underestimated.
- IanCal 10y ago
  The flow is significantly slower than is required for checking work. I'm fairly sure my wifes papers didn't really take nearly two years to review. Machine learning is a field that's moving quickly and can also largely be checked quickly by other people in the field too. The problems that can't be checked easily are also those that won't be checked by the journals anyway (no journal I know of will retrain one of googles deep nets to see if they get the same result before publishing).
  - kensai 10y ago
    If you think peer-reviewing is slow then it's probably because you don't get the PEER part of the reviewing process. The PEERS are simply colleagues who have their own lives, priorities, and research. They don't stop what they are doing to review your wife's papers. And please, I see what's happening here: I have been portrayed as the "traditional journal apologist" although it's definitely what I wanted here. I simply wanted to express my doubts that pre-printing is the answer to it all. Anyway, is there a study that shows how many pre-printed articles in ML have been retracted or refuted so far? Until this is done, and shown as small as in peer-reviewed journals, keep a small basket.
    - IanCal 10y ago
      > If you think peer-reviewing is slow then it's probably because you don't get the PEER part of the reviewing process. I understand the process fully. It doesn't make it fast, nor does it make it a necessary cost to pay. Delaying access to content for several years does not solve a problem. A not-insignificant time was spent bouncing between people to sort out who was paying for the costs, then there is also a delay between acceptance and publication. This now averages just a month in pubmed, but papers can bounce around this point for a lot longer. I am not arguing for pre-prints to replace traditional publishing, but the speed of spreading information is undeniably faster, and that's what started this whole chain of comments. > Anyway, is there a study that shows how many pre-printed articles in ML have been retracted or refuted so far? Until this is done, and shown as small as in peer-reviewed journals, keep a small basket. I don't get the phrase "keep a small basket", but no I've not seen this. The point is simply that PEERS (if we need to shout the word) can in many cases replicate the work and assess the results much more quickly than the traditional review & edit process. I picked the field because I see people re-implement work described in the papers very quickly, or the original authors share the trained models and code. I'd also caution against using "retracted" papers as a measure, some journals charge for a retraction.
      - kensai 10y ago
        "I understand the process fully. It doesn't make it fast, nor does it make it a necessary cost to pay. Delaying access to content for several years does not solve a problem. A not-insignificant time was spent bouncing between people to sort out who was paying for the costs, then there is also a delay between acceptance and publication. This now averages just a month in pubmed, but papers can bounce around this point for a lot longer." You don't understand a thing and that was the proof of it, at least for me. The process of peers evaluating a paper takes a lot of time because it cannot be automated and is serious, especially for the better journals. Of course bouncing people (referees) is part of the process, to find the better and/or most available one. Of course you can cut corners and pre-print, especially not having your paper (peer) reviewed, potentially fraught with errors and other inaccuracies. Problem is, not all sciences are like computer science where the proof could be algorithmically verified quickly. Publishing bullshit in biology, for example, may mean others losing two-three years trying to replicate your bullshit pre-printed results. "I am not arguing for pre-prints to replace traditional publishing, but the speed of spreading information is undeniably faster, and that's what started this whole chain of comments." Hey, I was not arguing traditional publishing is inherently better than pre-prints, but still got downvoted. "I don't get the phrase "keep a small basket", but no I've not seen this. The point is simply that PEERS (if we need to shout the word) can in many cases replicate the work and assess the results much more quickly than the traditional review & edit process. I picked the field because I see people re-implement work described in the papers very quickly, or the original authors share the trained models and code." Pre-printing might be the case in fields like CS and its subfields where verification is a very quick thing, but totally unsuitable in fields such as biology or medicine. The PEERS in the latter need also more time, given the complexity of the experimental setups. "I'd also caution against using "retracted" papers as a measure, some journals charge for a retraction." Oh trust me, if you publish publish to a good journal such a Nature or Science they find out (say bogus experiments), they will happily retract your paper. Happened with certain Korean stem cell researchers in the past.
- leegao 10y ago
  That's not necessarily true across all "fast fields". There's definitely selection pressure against fields sensitive to faulty experimentation to "move fast." Computer Science is one of the bigger subjects within Arxiv. Retraction rates within this discipline is extremely low and the field as a whole moves very rapidly. Having access to preprints makes it much more scalable for researchers to stay at the cutting edge.
- deleted 10y ago
  [deleted]
- dagenleg 10y ago
  Look, you are free to abstain from using pre-printed articles for your research if you do have doubts. Why start a crusade though? ArXiv is working very well for a lot of scientists in Machine Learning, since must results can be doublechecked by simply running the code. I won't disregard a proved approach that I've myself seen working just because it hasn't been published yet.
sgift 10y ago
> In some fields research moves extremely fast and by the time papers is properly published it's already obsolete. Could you provide some examples for a layman? To me it always seems that science moves far slower than one could hope for (e.g.: Every 'breakthrough' in batteries/Graphene of the last ten years and still no products)
- whatshisface 10y ago
  Think of it in terms of the speed of the field relative to the size of the paper - you aren't going to see no-graphene to industrial-graphene in a month, but you might see Johnson's 8% graphene be replaced by Smith's 8.01% graphene.

PaulHoule 10y ago

It is pretty controversial. The value of peer review has never really been demonstrated and despite peer review, the median scientific paper is wrong.

JD557 10y ago
>the median scientific paper is wrong. Are you being literal about that or is that just a figure of speech? Could you please provide a source for those claims, if true?
- lkbm 10y ago
  I assume he's alluding to Ioannidis's 2005 paper "Why Most Published Research Findings Are False"[0] and/or the replication crisis[1] in general. [0] http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou... [1] http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 http://www.nature.com/news/1-500-scientists-lift-the-lid-on-...
  - PaulHoule 10y ago
    Partly that and also based on my experience getting a PhD in physics where I found: (i) published physics papers with 30 pages of math in them frequently had errors, (ii) if you have a large number of signs the odds are 50-50 you will get the sign right in a long calculation, and (iii) you can't say high energy physics papers are "right" or "wrong" anyway... I mean, does anybody think we really live in an anti-DeSitter space? Medicine on the other hand has the problem that you can't really afford large enough sample sizes to have sufficient statistical power.
    - nonbel 10y ago
      >"Medicine on the other hand has the problem that you can't really afford large enough sample sizes to have sufficient statistical power." This really isn't the major problem w/ modern medical research. In fact, if they had properly powered studies there would be far too many "discoveries" and the real problem would become obvious. The real issue is that the efforts to come up with and study models capable of precise predictions (eg Armitage-Doll, SIR, Frank-Starling, Hodgekin-Huxly) have been all but choked out in favor of people testing the vague hypothesis "there is a correlation". There is always some effect/correlation in systems like the human body, so it is only a matter of sample size. As explained long ago by Paul Meehl, this is a 180 degree about-face from what was previously called the scientific method: http://www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf http://www.fisme.science.uu.nl/staff/christianb/downloads/me...

Ar-Curunir 10y ago

Hmm actually in my experience the versions of papers uploaded to preprint servers are the final versions, including reviewers comments. This is in cryptography, but I think the same can be said for complexity theory.