3 ms·

> This post is also an ideological action w.r.t arxiv publishing Anyone else thought that this was very weird? The author appears to be complaining about the f

by a3_nm 9y ago

> This post is also an ideological action w.r.t arxiv publishing Anyone else thought that this was very weird? The author appears to be complaining about the fact that reputable people/labs can post a PDF on arXiv and be taken seriously. How is this avoidable? Without arXiv, they could just post the PDF on their website or anywhere else. The "risk" associated to publishing crap on arXiv is the same as always: have people notice it's crap and get a bad reputation. I'm not sure what ideology has to do with it.

denzil_correa 9y ago
> Anyone else thought that this was very weird? The author appears to be complaining about the fact that reputable people/labs can post a PDF on arXiv and be taken seriously. How is this avoidable? Without arXiv, they could just post the PDF on their website or anywhere else. The issue is "flag planting" : arXiv is one way to do it; the same arguments hold true for other sources but arXiv is used as a canonical example for "flag planting".
naturalgradient 9y ago
No, this is not what this is about at all. We have reached a point where reviewers of reputable conferences will ask why a paper is not referencing unreviewed work that has been uploaded on arxiv a day before the conference deadline. This is not hyperbole, as anyone who is submitting to ICLR or NIPS can confirm. Work of certain labs is taken as established authority as soon as it hits arxiv.
Al-Khwarizmi 9y ago
Indeed it's possible in theory to do flag planting without arxiv, for example by posting a timestamped technical report to an institutional repository, but it has never been a common thing. Probably because you don't get much of an audience, so people won't cite you unless you bug them in reviews. And if someone publishes the idea some months later in a paper, they will probably get credit as there's the reasonable assumption they just hadn't read your report. On the other hand, arxiv has a huge audience, lately maybe even more than DL/NLP conferences, making the flag planting really effective, especially if you are from a prestigious group. So there is a real problem now with large, prestigious groups posting half-assed preliminary results in arxiv, and deterring more modest groups from working on the problem or bogging them down because they now need to compare themselves to the well-known arxiv approach, which often has serious reproducibility issues because it has not been peer-reviewed. I'm not an arxiv hater, in fact I check arxiv every day and for the last year or so I have posted most of my papers in it. But the problem is real and something must be done. Not about arxiv which is just the messenger (and a good one), but about the flag-planting culture using it that has emerged in the field.
- mostafab 9y ago
  >But the problem is real and something must be done I am working on a social network for papers that addresses this problem. Arxiv is too vulnerable to fake news articles because it is an archaic social network. I also think https://openreview.net https://openreview.net can be improved. You can check my project here : http://www.startcrowd.club http://www.startcrowd.club it does not have an anti fake news feature yet, but it is on the product roadmap.
- a3_nm 9y ago
  Thanks a lot for bothering to explain the problem. I have never seen this in my field (theoretical computer science), so I wasn't really aware of the problem. (More accurately: reviewers may ask why you do not compare yourself to arXiv preprints, often with indulgence if they are recent, but I have not seen this culture emerge of posting half-baked results to arXiv to claim priority.) It appears that we agree that the problem is not with arXiv. Part of the problem is unsolvable: if prestigious groups can announce what they are working on and discourage other groups from working on the same problem, this may just be reasonable self-interest from the smaller groups and can hardly be avoided. As for reviewers asking for comparison to these works, I guess the problem lies with the reviewers: if an arXiv preprint has not been refereed and/or is hard to reproduce, it should be OK to say so in another paper and not be blamed for the lack of comparison. In any case, this is an interesting problem, thanks again for making me aware of it.
jhbadger 9y ago
It's very avoidable. This whole thing of taking non-peer reviewed works seriously is very field dependent, and even then only a decade or two old. In physics and computer science it seems to be common, but for example, in biomedical research it is basically unheard of. People have been trying to get a bioarXiv started or post biology papers on arXiv itself, but they have very little influence because people think rightly or wrongly that a non-peer reviewed work must be flawed in some way.

by a3_nm 9y ago

denzil_correa 9y ago

> Anyone else thought that this was very weird? The author appears to be complaining about the fact that reputable people/labs can post a PDF on arXiv and be taken seriously. How is this avoidable? Without arXiv, they could just post the PDF on their website or anywhere else. The issue is "flag planting" : arXiv is one way to do it; the same arguments hold true for other sources but arXiv is used as a canonical example for "flag planting".

naturalgradient 9y ago

No, this is not what this is about at all. We have reached a point where reviewers of reputable conferences will ask why a paper is not referencing unreviewed work that has been uploaded on arxiv a day before the conference deadline. This is not hyperbole, as anyone who is submitting to ICLR or NIPS can confirm. Work of certain labs is taken as established authority as soon as it hits arxiv.

Al-Khwarizmi 9y ago

Indeed it's possible in theory to do flag planting without arxiv, for example by posting a timestamped technical report to an institutional repository, but it has never been a common thing. Probably because you don't get much of an audience, so people won't cite you unless you bug them in reviews. And if someone publishes the idea some months later in a paper, they will probably get credit as there's the reasonable assumption they just hadn't read your report. On the other hand, arxiv has a huge audience, lately maybe even more than DL/NLP conferences, making the flag planting really effective, especially if you are from a prestigious group. So there is a real problem now with large, prestigious groups posting half-assed preliminary results in arxiv, and deterring more modest groups from working on the problem or bogging them down because they now need to compare themselves to the well-known arxiv approach, which often has serious reproducibility issues because it has not been peer-reviewed. I'm not an arxiv hater, in fact I check arxiv every day and for the last year or so I have posted most of my papers in it. But the problem is real and something must be done. Not about arxiv which is just the messenger (and a good one), but about the flag-planting culture using it that has emerged in the field.

mostafab 9y ago
>But the problem is real and something must be done I am working on a social network for papers that addresses this problem. Arxiv is too vulnerable to fake news articles because it is an archaic social network. I also think https://openreview.net https://openreview.net can be improved. You can check my project here : http://www.startcrowd.club http://www.startcrowd.club it does not have an anti fake news feature yet, but it is on the product roadmap.
a3_nm 9y ago
Thanks a lot for bothering to explain the problem. I have never seen this in my field (theoretical computer science), so I wasn't really aware of the problem. (More accurately: reviewers may ask why you do not compare yourself to arXiv preprints, often with indulgence if they are recent, but I have not seen this culture emerge of posting half-baked results to arXiv to claim priority.) It appears that we agree that the problem is not with arXiv. Part of the problem is unsolvable: if prestigious groups can announce what they are working on and discourage other groups from working on the same problem, this may just be reasonable self-interest from the smaller groups and can hardly be avoided. As for reviewers asking for comparison to these works, I guess the problem lies with the reviewers: if an arXiv preprint has not been refereed and/or is hard to reproduce, it should be OK to say so in another paper and not be blamed for the lack of comparison. In any case, this is an interesting problem, thanks again for making me aware of it.

jhbadger 9y ago

It's very avoidable. This whole thing of taking non-peer reviewed works seriously is very field dependent, and even then only a decade or two old. In physics and computer science it seems to be common, but for example, in biomedical research it is basically unheard of. People have been trying to get a bioarXiv started or post biology papers on arXiv itself, but they have very little influence because people think rightly or wrongly that a non-peer reviewed work must be flawed in some way.