5 ms·

One way forward could be to lower the bar for publications. Once it's no longer about being in the esteemed and scarce "10%", they won't bother because they do

by jll29 5y ago

One way forward could be to lower the bar for publications. Once it's no longer about being in the esteemed and scarce "10%", they won't bother because they don't need to. Imagine a process where the only criteria are technical soundness and novelty, and as long as minimal standards are met, it's a "go". Call it the "ArXiv + quality check" model. Neither formal acceptance to publish nor citation numbers truly mark scientific excellence; perhaps, winning a "test of time award" does, or appearing in a text book 10 years later. I've been reviewing occasionally since ~1995, regularly since ~2004, and I've never heard of collusion rings happening in my sub-area of CS (ML, IR, NLP). I have caught people submitting to multiple conferences without disclosing it. Ignoring past work that is relevant is common, more often our of blissful ignorance, and occasionally likely with full intent. I'm not saying I doubt the report, but I suspect the bigger problem that CS has is a large percentage of poor-quality work that couldn't be replicated. BTW, the most blantant thing I've heard of (from a contact complaining about it on LinkedIn) is someone had their very own core paper from their PhD thesis plagiarised - submitted to another conference (again) but with different author names on it... and they even cited the real author's PhD thesis!

mycologos 5y ago
> Imagine a process where the only criteria are technical soundness and novelty, and as long as minimal standards are met, it's a "go". Call it the "ArXiv + quality check" model. One possible issue is that researchers usually need to justify their research to somebody who's not in their field. Conferences are one way to do this. So are citation counts. Both are highly imperfect, but outsiders typically want some signal that doesn't require being an expert in a person's chosen field. The "Arxiv + quality check" model doesn't seem to provide this. > I suspect the bigger problem that CS has is a large percentage of poor-quality work that couldn't be replicated. As a sort of ML researcher for several years, I agree.
- PartiallyTyped 5y ago
  As a fellow ML researcher, I want to add that the lack of code along with the publication makes the problem worse.$BIGGROUP gets a paper whose core contribution is a library, published, and yet they haven't released the code 6 months after the conference, effectively claiming credits for something unverifiable.
  - Al-Khwarizmi 5y ago
    I guess this can be different depending on your specific field, but in NLP it really changed for the better in the last few years. I don't have data, but from subjective experience, 5-6 years ago most papers in major NLP conferences didn't have an associated code repository. Now, the overwhelming majority do. There are still many other problems, for example a big one is reporting of spurious improvements that can vanish if you get a less lucky random seed. But at least including code is now common practice.
    - Delk 5y ago
      Back when I did a stint at something NLP-ish for my master's, one of the problems seemed to be that, apart from lack of code, the data was also often non-public and specific to the study. That made it impossible to compare different algorithms even as far as the results reported in the publications themselves go because the testing methodology was all over the place and the datasets used for testing various algorithms might have been all different. You couldn't really make much out of the reported results even if you believed the authors reported honestly and had their methodology more or less straight. I suppose the situation regarding common datasets might vary between subfields and NLP tasks, so maybe I just saw a weird corner of it. Of course the code was also nowhere to be seen. Availability of code would of course be even more important, both because of replicability and general verifiability, and also because that would allow you to do a comparison with any number of datasets yourself. Glad to hear that code availability has been improving. > There are still many other problems, for example a big one is reporting of spurious improvements that can vanish if you get a less lucky random seed. Considering that a lot of NLP is at least somewhat based on machine learning, don't people do cross-validation or something?
      - wodenokoto 5y ago
        For NLP, sharing data is a bit of a problem though. You do a paper showing that problem X can be solved slightly better by downloading and training on a billion tweets. But you don’t have the copyright to those tweets, so you can’t share data. > don't people do cross-validation or something A lot of stable problems comes with a dataset already split into train and test.
quantumwannabe 5y ago
What you propose is actually how it used to function until around the mid-20th century. Journals used to be very permissive when accepting papers with the editor only doing a cursory check to make sure the paper wasn't total garbage. More info here: https://michaelnielsen.org/blog/three-myths-about-scientific-peer-review/ https://michaelnielsen.org/blog/three-myths-about-scientific...
jhrmnn 5y ago
That’s how it used to be. The reason behind the shift is the rapid increase in the amount of research output in the past decades. The reason that “arXiv + quality check” won’t work is that the amount of research funding and permanent positions (related) did not increase accordingly. (Think of all the people producing science for low wages as Phds and postdocs and then quitting.) Right now the burden of the (unfair?) selection is mostly on the publishers and their prestigious journals/conferences, which then funding agencies/institutions take into account when funding research/hiring people. If we switch “arXiv + quality check”, that burden will just move to the funding agencies/institutions, but it won’t solve the underlying fundamental problem.
- unishark 5y ago
  In the US, both academic hiring and proposal reviews are already laborious jobs where committee members are supposed to investigate the candidate/proposal deeply on their merits and potential. It shouldn't be a new burden for them to be unable to take shortcuts. I think the recent wave of low-impact submissions and co-authorship rings is the result of developing countries trying to simplify that process and tying hiring/promotion/pay directly to publication count and related easy metrics.
  - jhrmnn 5y ago
    Are supposed to, but doing that would require so much more work than is usually being put into it now. Especially in the first rounds, it would be impossible to review all candidates/proposals thoroughly without having staff dedicated to just that. (IMO institutions should have such highly-skilled staff.) It may be true that the SNR in research output from developing countries is lower, but there is still lots of good science. But essentially no publishers with good reputation. So even with the same SNR, the increased pool of countries producing science would add to the publication pressure.
ylem 5y ago
At it's best, that's what scientific reports is (in a number of subfields)--a place to put research that is reviewed for technical correctness. There may be debate about how well it succeeds, but I think it's useful. I would prefer to have more papers out where people worked on something, found it wasn't necessarily exciting, but it was solid work and it saves other people time.
KirillPanov 5y ago
Well, then you need something else to use as a basis for tenure decisions. Also, a "best 10%" conference/journal is valuable -- I have only so much time in the day. There are a few conferences for which it is always a good use of my time to read all the abstracts, plus one or two of the papers that seem most interesting. I can't do that for every conference, or even most conferences, in my area. So the "best 10%" conference/journal is valuable to the consumers, and the prestiege is valuable to the producers. Therefore I think such a thing would simply re-emerge if you somehow killed it.
utopcell 5y ago
You can't help but be entertained by the amount of plagiarism that exists. A while back I found out that one of my papers [1] was being sold as a final year thesis via ..a video ad on YouTube [2]. [1] https://ieeexplore.ieee.org/abstract/document/6746236 https://ieeexplore.ieee.org/abstract/document/6746236 [2] https://www.youtube.com/watch?v=MFNFScqN47o https://www.youtube.com/watch?v=MFNFScqN47o
- kwhitefoot 5y ago
  Don't students have to defend a thesis any more? I mean I could buy your paper but I would have to know it by heart and understand it in order to defend it. At least that's how it went when I was studying applied physics (1974-1977). If I had just bought a paper I would have had a really hard time in the viva voce. Edit: by -> buy
  - hawk_ 5y ago
    still the effort to come up with something original is order(s) of magnitude more than copying. it's likely that the potential buyers have enough grounding to understand the papers.
  - akjssdk 5y ago
    Depending on the country, the PhD defense is more or less ceremonial. If you have made it to the defense, you can pretty much not fail anymore.
    - physicsguy 5y ago
      That's really not the case here in the UK, I know several people who failed their vivas and had to make substantial changes to their thesis. It's not common, because there are intermediate level mini-theses and vivas, and people who do badly in those often don't finish at all, but it does happen.
      - conjectures 5y ago
        Yes, in UK mine was terrifying. In all likelihood candidate would get destroyed trying that tactic.
cycomanic 5y ago
I mean there are several journals who publish based only on "technical soundness", Nature's Scientific Reports is probably the most well known one. I don't think it helps, in fact IMO it's detrimental because journals would be even more flooded with publications. Also in the article they talk about the conference getting 10,000 submissions, that quickly becomes unmanageable as a conference to accept.
stdbrouw 5y ago
Even novelty is a dubious standard that is often used to discard replications and meta-analyses (not novel), empirical work (no novel theory), reformulations and simplifications of existing research (not novel) and null findings (nothing was found so not novel).
- BlueTemplar 5y ago
  Lolwhat, empirical work is the only thing that can confirm (or rather, refute) the worthiness of a theory !?
stared 5y ago
In deep learning a lot of papers end up only on arXiv, and authors don't bother (or need) to send it anywhere. Or even more, some seminal papers and up only on the author's page (e.g. one introducing GPT-2 https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/). I am not a big fan of the latter, as it gives problems with long-term availability. Moreover, the correlation with acceptance and impact in existing, but not that high: https://medium.com/ai2-blog/what-open-data-tells-us-about-rejected-iclr-submissions-53eca5c2b019 https://medium.com/ai2-blog/what-open-data-tells-us-about-re... Of course, only corporate researchers can rely on not publishing in established journals - as their salary and position does not come from "publish or perish" metrics. Ironically, it means that there is more academic freedom in private companies than in academia.
- dasudasu 5y ago
  Bell Labs in its heyday was still reputed to be publish or perish, or patent or perish, without any tenure system either. At my university, we had a previous Bell Labs researcher that said he had been pushed out. As a corporate researcher, there is still high pressure to output something.

by jll29 5y ago

mycologos 5y ago

> Imagine a process where the only criteria are technical soundness and novelty, and as long as minimal standards are met, it's a "go". Call it the "ArXiv + quality check" model. One possible issue is that researchers usually need to justify their research to somebody who's not in their field. Conferences are one way to do this. So are citation counts. Both are highly imperfect, but outsiders typically want some signal that doesn't require being an expert in a person's chosen field. The "Arxiv + quality check" model doesn't seem to provide this. > I suspect the bigger problem that CS has is a large percentage of poor-quality work that couldn't be replicated. As a sort of ML researcher for several years, I agree.

PartiallyTyped 5y ago
As a fellow ML researcher, I want to add that the lack of code along with the publication makes the problem worse.$BIGGROUP gets a paper whose core contribution is a library, published, and yet they haven't released the code 6 months after the conference, effectively claiming credits for something unverifiable.
- Al-Khwarizmi 5y ago
  I guess this can be different depending on your specific field, but in NLP it really changed for the better in the last few years. I don't have data, but from subjective experience, 5-6 years ago most papers in major NLP conferences didn't have an associated code repository. Now, the overwhelming majority do. There are still many other problems, for example a big one is reporting of spurious improvements that can vanish if you get a less lucky random seed. But at least including code is now common practice.
  - Delk 5y ago
    Back when I did a stint at something NLP-ish for my master's, one of the problems seemed to be that, apart from lack of code, the data was also often non-public and specific to the study. That made it impossible to compare different algorithms even as far as the results reported in the publications themselves go because the testing methodology was all over the place and the datasets used for testing various algorithms might have been all different. You couldn't really make much out of the reported results even if you believed the authors reported honestly and had their methodology more or less straight. I suppose the situation regarding common datasets might vary between subfields and NLP tasks, so maybe I just saw a weird corner of it. Of course the code was also nowhere to be seen. Availability of code would of course be even more important, both because of replicability and general verifiability, and also because that would allow you to do a comparison with any number of datasets yourself. Glad to hear that code availability has been improving. > There are still many other problems, for example a big one is reporting of spurious improvements that can vanish if you get a less lucky random seed. Considering that a lot of NLP is at least somewhat based on machine learning, don't people do cross-validation or something?
    - wodenokoto 5y ago
      For NLP, sharing data is a bit of a problem though. You do a paper showing that problem X can be solved slightly better by downloading and training on a billion tweets. But you don’t have the copyright to those tweets, so you can’t share data. > don't people do cross-validation or something A lot of stable problems comes with a dataset already split into train and test.

quantumwannabe 5y ago

What you propose is actually how it used to function until around the mid-20th century. Journals used to be very permissive when accepting papers with the editor only doing a cursory check to make sure the paper wasn't total garbage. More info here: https://michaelnielsen.org/blog/three-myths-about-scientific-peer-review/ https://michaelnielsen.org/blog/three-myths-about-scientific...

jhrmnn 5y ago

That’s how it used to be. The reason behind the shift is the rapid increase in the amount of research output in the past decades. The reason that “arXiv + quality check” won’t work is that the amount of research funding and permanent positions (related) did not increase accordingly. (Think of all the people producing science for low wages as Phds and postdocs and then quitting.) Right now the burden of the (unfair?) selection is mostly on the publishers and their prestigious journals/conferences, which then funding agencies/institutions take into account when funding research/hiring people. If we switch “arXiv + quality check”, that burden will just move to the funding agencies/institutions, but it won’t solve the underlying fundamental problem.

unishark 5y ago
In the US, both academic hiring and proposal reviews are already laborious jobs where committee members are supposed to investigate the candidate/proposal deeply on their merits and potential. It shouldn't be a new burden for them to be unable to take shortcuts. I think the recent wave of low-impact submissions and co-authorship rings is the result of developing countries trying to simplify that process and tying hiring/promotion/pay directly to publication count and related easy metrics.
- jhrmnn 5y ago
  Are supposed to, but doing that would require so much more work than is usually being put into it now. Especially in the first rounds, it would be impossible to review all candidates/proposals thoroughly without having staff dedicated to just that. (IMO institutions should have such highly-skilled staff.) It may be true that the SNR in research output from developing countries is lower, but there is still lots of good science. But essentially no publishers with good reputation. So even with the same SNR, the increased pool of countries producing science would add to the publication pressure.

ylem 5y ago

At it's best, that's what scientific reports is (in a number of subfields)--a place to put research that is reviewed for technical correctness. There may be debate about how well it succeeds, but I think it's useful. I would prefer to have more papers out where people worked on something, found it wasn't necessarily exciting, but it was solid work and it saves other people time.

KirillPanov 5y ago

Well, then you need something else to use as a basis for tenure decisions. Also, a "best 10%" conference/journal is valuable -- I have only so much time in the day. There are a few conferences for which it is always a good use of my time to read all the abstracts, plus one or two of the papers that seem most interesting. I can't do that for every conference, or even most conferences, in my area. So the "best 10%" conference/journal is valuable to the consumers, and the prestiege is valuable to the producers. Therefore I think such a thing would simply re-emerge if you somehow killed it.

utopcell 5y ago

You can't help but be entertained by the amount of plagiarism that exists. A while back I found out that one of my papers [1] was being sold as a final year thesis via ..a video ad on YouTube [2]. [1] https://ieeexplore.ieee.org/abstract/document/6746236 https://ieeexplore.ieee.org/abstract/document/6746236 [2] https://www.youtube.com/watch?v=MFNFScqN47o https://www.youtube.com/watch?v=MFNFScqN47o

kwhitefoot 5y ago
Don't students have to defend a thesis any more? I mean I could buy your paper but I would have to know it by heart and understand it in order to defend it. At least that's how it went when I was studying applied physics (1974-1977). If I had just bought a paper I would have had a really hard time in the viva voce. Edit: by -> buy
- hawk_ 5y ago
  still the effort to come up with something original is order(s) of magnitude more than copying. it's likely that the potential buyers have enough grounding to understand the papers.
- akjssdk 5y ago
  Depending on the country, the PhD defense is more or less ceremonial. If you have made it to the defense, you can pretty much not fail anymore.
  - physicsguy 5y ago
    That's really not the case here in the UK, I know several people who failed their vivas and had to make substantial changes to their thesis. It's not common, because there are intermediate level mini-theses and vivas, and people who do badly in those often don't finish at all, but it does happen.
    - conjectures 5y ago
      Yes, in UK mine was terrifying. In all likelihood candidate would get destroyed trying that tactic.

cycomanic 5y ago

I mean there are several journals who publish based only on "technical soundness", Nature's Scientific Reports is probably the most well known one. I don't think it helps, in fact IMO it's detrimental because journals would be even more flooded with publications. Also in the article they talk about the conference getting 10,000 submissions, that quickly becomes unmanageable as a conference to accept.

stdbrouw 5y ago

Even novelty is a dubious standard that is often used to discard replications and meta-analyses (not novel), empirical work (no novel theory), reformulations and simplifications of existing research (not novel) and null findings (nothing was found so not novel).

BlueTemplar 5y ago
Lolwhat, empirical work is the only thing that can confirm (or rather, refute) the worthiness of a theory !?

stared 5y ago

In deep learning a lot of papers end up only on arXiv, and authors don't bother (or need) to send it anywhere. Or even more, some seminal papers and up only on the author's page (e.g. one introducing GPT-2 https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/). I am not a big fan of the latter, as it gives problems with long-term availability. Moreover, the correlation with acceptance and impact in existing, but not that high: https://medium.com/ai2-blog/what-open-data-tells-us-about-rejected-iclr-submissions-53eca5c2b019 https://medium.com/ai2-blog/what-open-data-tells-us-about-re... Of course, only corporate researchers can rely on not publishing in established journals - as their salary and position does not come from "publish or perish" metrics. Ironically, it means that there is more academic freedom in private companies than in academia.

dasudasu 5y ago
Bell Labs in its heyday was still reputed to be publish or perish, or patent or perish, without any tenure system either. At my university, we had a previous Bell Labs researcher that said he had been pushed out. As a corporate researcher, there is still high pressure to output something.