5 ms·

A Case of Plagarism in Machine Learning Research

75 points by EvgeniyZh 5y ago

behnamoh 5y ago
I thought journals use some software to detect plagiarism by scanning the submitted files and checking for copied text. Of course, it's not as straightforward because of all the symbols and math involved. But I think whether the files are submitted as Latex or Word docs, Regex must help extract pieces of text and reverse search them in previous papers. The problem is that this necessitates scanning previous papers too, which journals might not be able or willing to do. Anyway, I always thought research in ML is suspiciously fast paced, but most published work is honestly bs. This post says one reason for this might be the fact that most research isn't actually contributing much to the literature, but rather reiterating previous papers.
- rockinghigh 5y ago
  This paper doesn't seem to have been submitted to a journal/conference and probably didn't go through a standard peer review process.
cs702 5y ago
The individuals listed as authors of the paper who blatantly plagiarized the original research happen to work at prominent government organizations, educational institutions, and major corporations: * Beijing Academy of Artificial Intelligence * Tsinghua University * Wechat, Tencent Inc. * Northeastern University * Renmin University of China * Peking University * Huawei TCS Lab * Institute of Computing Technology, Chinese Academy of Sciences * Shanghai Jiao Tong University * JD AI Research * Harbin Institute of Technology * Columbia University * ByteDance AI Lab * Microsoft Research Asia * Mila-Quebec AI Institute & University of Montreal * New York University * BeiHang University * Institute of Software, Chinese Academy of Sciences * Institute of Automation Chinese Academy of Sciences IMHO, the main authors, who copied and pasted text without giving credit, should be fired on the spot.
- WithinReason 5y ago
  Which of the 100 authors did the copying?
  - myth_drannon 5y ago
    I find it hard to believe that all of them collaborated on the paper or even aware that their name is on it.
  - godelski 5y ago
    Going to be hard to say. The paper itself is 200 pages. I also wouldn't be surprised if more papers were copied in a similar manner given the size of the manuscript. If the authors are using some version controlling (insane not to with this large of text) then it should be relatively easy to check who did it and then check the rest of their writings. But that has to come from internal pressure.
  - ninjin 5y ago
    I am a senior researcher in the same area. From the perspective of ethics enforcement on an institutional, state, or field level, it does not matter which author it was that committed the act. As a co-author you “sign off” on the paper as a whole and is expected to do due diligence to ensure that the paper in its entirety is up to the standards we expect. This is research ethics 101. Now, in practice I would expect that you get off easier on a paper with a ridiculous number of authors such as this one as long as you find the perpetrator or a fall guy. Lastly, there is also the issue of papers at time being submitted without the consent of a co-author (possibly without them even knowing of the paper), which could complicate the matter further. Shame on whoever did this, among the authors from prestigious institutions such as these there ought to be at least one to stop this kind of stupidity or at the very least be competent enough to write the very same text without copying.
    - erosenbe0 5y ago
      It is probably a sloppy mistake, ineptness, and poor process controls by a contributing team, not pure malevolence. In a survey paper some people are sloppy and copy/paste source material directly into their file and edit it, or instruct others to do edits with notes. This creates a large risk that the edit never happens or the wrong version is submitted. Edit: also take into account plenty of these researchers write English as second or third language. Copied text won't necessarily sound as obviously out of place and process becomes even more important.
light_hue_1 5y ago
As an adult, with a PhD, doing research in machine learning I have difficulty reading a paper that's all about BMs ...
- Gigachad 5y ago
  BMs?
lol1lol 5y ago
Der_Einzige 5y ago
As bad as this is, someone lying about their results in a paper is FAR worse than them copying and pasting was is essentially boilerplate as far as the authors and reviewers are concerned... And we all know that outright lies are all too common in ML research...
mywaifuismeta 5y ago
Am I missing something or is this nonsense? The "Big Model" paper is a survey paper that summarizes the current state of the field, while the "original" paper is specific technique. The former, as a small part of the survey, describes what was presented in the original and uses the same language the authors use. Why wouldn't they? Why would they need to re-phrase the facts and methods the authors described themselves when all they're doing is giving an overview? They are not claiming to have done something they didn't. Same argument about copying super generic introductions like "Deep Learning has been successful at...." - does each author really need to come up with their own variation of widely accepted facts to avoid "plagiarism" ?
- godelski 5y ago
  You should describe the same concepts, but you should use your own voice. A survey paper is not just copy pasting blocks of text and adding connecting language. A survey itself should have its own story and therefore your own voice. You're allowed to quote, sure, but what is shown is beyond a reasonable quotation and the "quoter" doesn't reference that the section was quoted.
- bijjgi 5y ago
  Yes, you are missing something. No, this is not nonsense. It’s considered academic plagiarism to copy paste text without indicating the passage is a direct quote. They should re-phrase the text. Not only is it academic plagiarism, it’s also poor writing, because the copied text is providing a different framing than the overview article, and because the copied text is in some places incoherent due to formatting (e.g. we show that over 1 [sic] of the data most models emit). You are wrong, the Big Models paper does in fact claim to have done something that they did not, e.g. “We introduce two complementary methods for performing deduplication.” They do not introduce these methods. The text they are lifting did. What you are missing is long standing norms around academic plagiarism and false claims in the Big Models paper (as a result of copy pasting language).
- deleted 5y ago
  [deleted]
- fwilliams 5y ago
charcircuit 5y ago
>A survey paper that just copies directly from the prior paper hasn't contributed anything new to the field that couldn't be obtained from a list of references. IMO the whole point of a survey paper is to avoid having to read through a whole list of references. There is a benefit in extracting information from those papers and assembling them together into one thing. It's expected that the paper is going to be a compilation of information from the papers that it is surveying.
frozenport 5y ago
Also a copyright violation. ArXiv should remove the article.
srean 5y ago
This seems to be a case of a bad and a lazy survey paper rather than plagiarism. Plagiarism implies misrepresentation of some kind -- usually passing someone else's work as one's own. With the original works cited this does not seem to be the case here. There are more egregious dirty laundry that perhaps needs more attention than this particular one. While I was a grad student, a labmate's research paper was copied word for word, at least the abstract, introduction and conclusion. If memory serves, this was in around 2002. Funnily enough this plagiarized paper submitted to a conference happened to get assigned to my same labmate for review. The are chairs were informed and the paper rejected, but more interesting was the discussion that ensued within our own lab with other Chinese labmate's. They mentioned that in Chinese universities, at least then, it would be considered perfectly kosher piece of work by a student to replicate word for word a good paper that the student had expended effort in identifying, such an act would not be considered plagiarism, perhaps even a demonstration of good work ethic. There was no clear consensus on whether submitting it in their own name in a prominent ML / datamining conference would have been perceived as an act of plagiarism. My labmate who's paper got plagiarized got lucky that it came to him for review. I can easily imagine scenarios where it needn't have. Also its only because it was submitted to an international conference that it got caught out. I wouldnt be surprised if more of these were going on in more home brew conferences. The original paper was published in ICML. I dont seem to recall with high confidence the exact conference the plagiarized copy was sent to.

75 points by EvgeniyZh 5y ago

I thought journals use some software to detect plagiarism by scanning the submitted files and checking for copied text. Of course, it's not as straightforward because of all the symbols and math involved. But I think whether the files are submitted as Latex or Word docs, Regex must help extract pieces of text and reverse search them in previous papers. The problem is that this necessitates scanning previous papers too, which journals might not be able or willing to do. Anyway, I always thought research in ML is suspiciously fast paced, but most published work is honestly bs. This post says one reason for this might be the fact that most research isn't actually contributing much to the literature, but rather reiterating previous papers.

rockinghigh 5y ago
This paper doesn't seem to have been submitted to a journal/conference and probably didn't go through a standard peer review process.

cs702 5y ago

The individuals listed as authors of the paper who blatantly plagiarized the original research happen to work at prominent government organizations, educational institutions, and major corporations: * Beijing Academy of Artificial Intelligence * Tsinghua University * Wechat, Tencent Inc. * Northeastern University * Renmin University of China * Peking University * Huawei TCS Lab * Institute of Computing Technology, Chinese Academy of Sciences * Shanghai Jiao Tong University * JD AI Research * Harbin Institute of Technology * Columbia University * ByteDance AI Lab * Microsoft Research Asia * Mila-Quebec AI Institute & University of Montreal * New York University * BeiHang University * Institute of Software, Chinese Academy of Sciences * Institute of Automation Chinese Academy of Sciences IMHO, the main authors, who copied and pasted text without giving credit, should be fired on the spot.

WithinReason 5y ago
Which of the 100 authors did the copying?
- myth_drannon 5y ago
  I find it hard to believe that all of them collaborated on the paper or even aware that their name is on it.
- godelski 5y ago
  Going to be hard to say. The paper itself is 200 pages. I also wouldn't be surprised if more papers were copied in a similar manner given the size of the manuscript. If the authors are using some version controlling (insane not to with this large of text) then it should be relatively easy to check who did it and then check the rest of their writings. But that has to come from internal pressure.
- ninjin 5y ago
  I am a senior researcher in the same area. From the perspective of ethics enforcement on an institutional, state, or field level, it does not matter which author it was that committed the act. As a co-author you “sign off” on the paper as a whole and is expected to do due diligence to ensure that the paper in its entirety is up to the standards we expect. This is research ethics 101. Now, in practice I would expect that you get off easier on a paper with a ridiculous number of authors such as this one as long as you find the perpetrator or a fall guy. Lastly, there is also the issue of papers at time being submitted without the consent of a co-author (possibly without them even knowing of the paper), which could complicate the matter further. Shame on whoever did this, among the authors from prestigious institutions such as these there ought to be at least one to stop this kind of stupidity or at the very least be competent enough to write the very same text without copying.
  - erosenbe0 5y ago
    It is probably a sloppy mistake, ineptness, and poor process controls by a contributing team, not pure malevolence. In a survey paper some people are sloppy and copy/paste source material directly into their file and edit it, or instruct others to do edits with notes. This creates a large risk that the edit never happens or the wrong version is submitted. Edit: also take into account plenty of these researchers write English as second or third language. Copied text won't necessarily sound as obviously out of place and process becomes even more important.

light_hue_1 5y ago

As an adult, with a PhD, doing research in machine learning I have difficulty reading a paper that's all about BMs ...

Gigachad 5y ago
BMs?

lol1lol 5y ago

Der_Einzige 5y ago

As bad as this is, someone lying about their results in a paper is FAR worse than them copying and pasting was is essentially boilerplate as far as the authors and reviewers are concerned... And we all know that outright lies are all too common in ML research...

mywaifuismeta 5y ago

Am I missing something or is this nonsense? The "Big Model" paper is a survey paper that summarizes the current state of the field, while the "original" paper is specific technique. The former, as a small part of the survey, describes what was presented in the original and uses the same language the authors use. Why wouldn't they? Why would they need to re-phrase the facts and methods the authors described themselves when all they're doing is giving an overview? They are not claiming to have done something they didn't. Same argument about copying super generic introductions like "Deep Learning has been successful at...." - does each author really need to come up with their own variation of widely accepted facts to avoid "plagiarism" ?

godelski 5y ago
You should describe the same concepts, but you should use your own voice. A survey paper is not just copy pasting blocks of text and adding connecting language. A survey itself should have its own story and therefore your own voice. You're allowed to quote, sure, but what is shown is beyond a reasonable quotation and the "quoter" doesn't reference that the section was quoted.
bijjgi 5y ago
Yes, you are missing something. No, this is not nonsense. It’s considered academic plagiarism to copy paste text without indicating the passage is a direct quote. They should re-phrase the text. Not only is it academic plagiarism, it’s also poor writing, because the copied text is providing a different framing than the overview article, and because the copied text is in some places incoherent due to formatting (e.g. we show that over 1 [sic] of the data most models emit). You are wrong, the Big Models paper does in fact claim to have done something that they did not, e.g. “We introduce two complementary methods for performing deduplication.” They do not introduce these methods. The text they are lifting did. What you are missing is long standing norms around academic plagiarism and false claims in the Big Models paper (as a result of copy pasting language).
deleted 5y ago
[deleted]
fwilliams 5y ago

charcircuit 5y ago

>A survey paper that just copies directly from the prior paper hasn't contributed anything new to the field that couldn't be obtained from a list of references. IMO the whole point of a survey paper is to avoid having to read through a whole list of references. There is a benefit in extracting information from those papers and assembling them together into one thing. It's expected that the paper is going to be a compilation of information from the papers that it is surveying.

frozenport 5y ago

Also a copyright violation. ArXiv should remove the article.

srean 5y ago

This seems to be a case of a bad and a lazy survey paper rather than plagiarism. Plagiarism implies misrepresentation of some kind -- usually passing someone else's work as one's own. With the original works cited this does not seem to be the case here. There are more egregious dirty laundry that perhaps needs more attention than this particular one. While I was a grad student, a labmate's research paper was copied word for word, at least the abstract, introduction and conclusion. If memory serves, this was in around 2002. Funnily enough this plagiarized paper submitted to a conference happened to get assigned to my same labmate for review. The are chairs were informed and the paper rejected, but more interesting was the discussion that ensued within our own lab with other Chinese labmate's. They mentioned that in Chinese universities, at least then, it would be considered perfectly kosher piece of work by a student to replicate word for word a good paper that the student had expended effort in identifying, such an act would not be considered plagiarism, perhaps even a demonstration of good work ethic. There was no clear consensus on whether submitting it in their own name in a prominent ML / datamining conference would have been perceived as an act of plagiarism. My labmate who's paper got plagiarized got lucky that it came to him for review. I can easily imagine scenarios where it needn't have. Also its only because it was submitted to an international conference that it got caught out. I wouldnt be surprised if more of these were going on in more home brew conferences. The original paper was published in ICML. I dont seem to recall with high confidence the exact conference the plagiarized copy was sent to.