6 ms·

The fight against fake-paper factories that churn out sham science

27 points by behnamoh 4y ago

ehvatum 4y ago
If only this were something that the rapidly growing cadres of university administrators could address. Perhaps they could erect a new Research Integrity Campus at every major university? If the buildings are eye-catching and the research integrity campuses have their own sport ball teams and diversity coordination hierarchies (diversity provosts, diversity deans, etc), Indian grad students will show up to do all the work. Then there can be international seminars with Chinese representatives donating to our research integrity efforts.
enviclash 4y ago
THIS is a chance to build a start-up.
deleted 4y ago
[deleted]
mike_hearn 4y ago
I spent quite a bit of time researching this topic in 2020/2021 and this is a really good article. I am frequently amazed/impressed that Nature can report on this issue so well, given that they're essentially undermining their core business. The news/journal split does seem to work. I kept wondering if the authors would bring up a particular point or person, and they usually did. Unfortunately, the subject matter is really bleak. As pointed out in the final paragraph, AI image generation has come along at exactly the wrong time for the people trying to combat this stuff, and of course the problems are in no way limited to fake images. Fake text is an even bigger problem. An astonishing number of fake papers can be detected as fake just by reading the abstract, yet this doesn't seem to stop them somehow passing through peer review and the editorial process. A good example of this is covered in an article I wrote earlier this year that looked at the case of a journal retracting 24 papers for being (in their own words) "nonsensical". The content had been fully generated by machines but nothing as sophisticated as GPT-3, just some template driven logic most likely: https://dailysceptic.org/archive/publisher-retracts-24-scientific-papers-for-being-nonsensical-what-happened-to-peer-review/ https://dailysceptic.org/archive/publisher-retracts-24-scien... The journal in question is published by Springer and has 45 people associated with it who have "editor" in their title, mostly professors at western universities. None of them noticed they were publishing nonsense text until it was too late. The irony here is that nothing like this problem exists in any other form of publishing. Scientific journals are one of the most trusted sources of information, yet if a tabloid newspaper from a well known company accidentally published a whole edition consisting of nothing but auto-generated nonsense text it would be jaw dropping; that would itself make world news. Yet this is a semi-regular occurrence in the scientific literature. When looking at what they plan to do about this, you get the feeling they're bringing knives to a gun fight. The paper mills are professional and dedicated operations that make money from faking papers, whereas journals make money from publishing papers. The level of interest in the problem most journals display is exactly what you'd expect from this setup. For example a common response to publishing nonsense text is to announce they're incorporating nonsense text detection programs into their publishing pipelines, which rather implies they think their peer reviewing process has gone AWOL or is unfixable. And although the Nature article talks about Wiley hiring and training people to spot fake images, it's just about looking for duplicates or patterns across large numbers of unrelated papers. They have no response to DNN generation which can make an unlimited number of unique but fake images, nor any way to detect faked raw data. Fundamentally, journals face an unhappy choice: they can double down on their current model and constantly damage trust. Or they can evolve to become full blown audit organizations that verify scientific work submitted was actually done, resulting in a much greater level of staffing and expense simply to get back to the level of trustworthiness they claim to already have. In theory this shouldn't be necessary because the actual institutions that employ the scientists should be doing these audits, but in practice there are too many institutions/governments that have confused the means (giving out funding) for the end (scientific progress) and don't really care how their money is spent as long the resulting artifacts looks superficially like science.
- Kim_Bruning 4y ago
  I stand by my position that the wiki model has the potential to be rather more reliable than the journal model. The elephant in the room: Wikipedia has chosen to be an encyclopedia, so of course that is not the correct venue. However, curated wikis for scientific articles do exist and seem to be operating well. (A quick google search can find many) People who criticize wikis wrt unreliability are technically correct, the best kind of correct. Nothing can be 100% reliable. But (to repurpose an old quote) at this time I think wikis are the worst system except for all the others. [edit: I may need to point out that I'm a strong eventualist[1]: A wiki based approach is not immediately superior. Wiki articles get edited and improved and become more reliable over time. This in opposition to a journal article which must be "perfect" at the time of publication, at which point it becomes frozen. Early on a (bona fide) journal article will obviously be of higher quality. Over time the wiki article can catch up and pass the journal article. ] [1] https://meta.wikimedia.org/wiki/Eventualism https://meta.wikimedia.org/wiki/Eventualism for further reading on how eventualism pans out in the wiki[m|p]edia context.
Kim_Bruning 4y ago
This is Goodhart's law in action. People measure academic worth by publications, so people need to publish a lot. In theory, if one were to switch to alternate measures, this form of pollution would cease overnight. I wonder if there are superior measures already in use?

27 points by behnamoh 4y ago

If only this were something that the rapidly growing cadres of university administrators could address. Perhaps they could erect a new Research Integrity Campus at every major university? If the buildings are eye-catching and the research integrity campuses have their own sport ball teams and diversity coordination hierarchies (diversity provosts, diversity deans, etc), Indian grad students will show up to do all the work. Then there can be international seminars with Chinese representatives donating to our research integrity efforts.

enviclash 4y ago

THIS is a chance to build a start-up.

deleted 4y ago

[deleted]

mike_hearn 4y ago

I spent quite a bit of time researching this topic in 2020/2021 and this is a really good article. I am frequently amazed/impressed that Nature can report on this issue so well, given that they're essentially undermining their core business. The news/journal split does seem to work. I kept wondering if the authors would bring up a particular point or person, and they usually did. Unfortunately, the subject matter is really bleak. As pointed out in the final paragraph, AI image generation has come along at exactly the wrong time for the people trying to combat this stuff, and of course the problems are in no way limited to fake images. Fake text is an even bigger problem. An astonishing number of fake papers can be detected as fake just by reading the abstract, yet this doesn't seem to stop them somehow passing through peer review and the editorial process. A good example of this is covered in an article I wrote earlier this year that looked at the case of a journal retracting 24 papers for being (in their own words) "nonsensical". The content had been fully generated by machines but nothing as sophisticated as GPT-3, just some template driven logic most likely: https://dailysceptic.org/archive/publisher-retracts-24-scientific-papers-for-being-nonsensical-what-happened-to-peer-review/ https://dailysceptic.org/archive/publisher-retracts-24-scien... The journal in question is published by Springer and has 45 people associated with it who have "editor" in their title, mostly professors at western universities. None of them noticed they were publishing nonsense text until it was too late. The irony here is that nothing like this problem exists in any other form of publishing. Scientific journals are one of the most trusted sources of information, yet if a tabloid newspaper from a well known company accidentally published a whole edition consisting of nothing but auto-generated nonsense text it would be jaw dropping; that would itself make world news. Yet this is a semi-regular occurrence in the scientific literature. When looking at what they plan to do about this, you get the feeling they're bringing knives to a gun fight. The paper mills are professional and dedicated operations that make money from faking papers, whereas journals make money from publishing papers. The level of interest in the problem most journals display is exactly what you'd expect from this setup. For example a common response to publishing nonsense text is to announce they're incorporating nonsense text detection programs into their publishing pipelines, which rather implies they think their peer reviewing process has gone AWOL or is unfixable. And although the Nature article talks about Wiley hiring and training people to spot fake images, it's just about looking for duplicates or patterns across large numbers of unrelated papers. They have no response to DNN generation which can make an unlimited number of unique but fake images, nor any way to detect faked raw data. Fundamentally, journals face an unhappy choice: they can double down on their current model and constantly damage trust. Or they can evolve to become full blown audit organizations that verify scientific work submitted was actually done, resulting in a much greater level of staffing and expense simply to get back to the level of trustworthiness they claim to already have. In theory this shouldn't be necessary because the actual institutions that employ the scientists should be doing these audits, but in practice there are too many institutions/governments that have confused the means (giving out funding) for the end (scientific progress) and don't really care how their money is spent as long the resulting artifacts looks superficially like science.

Kim_Bruning 4y ago
I stand by my position that the wiki model has the potential to be rather more reliable than the journal model. The elephant in the room: Wikipedia has chosen to be an encyclopedia, so of course that is not the correct venue. However, curated wikis for scientific articles do exist and seem to be operating well. (A quick google search can find many) People who criticize wikis wrt unreliability are technically correct, the best kind of correct. Nothing can be 100% reliable. But (to repurpose an old quote) at this time I think wikis are the worst system except for all the others. [edit: I may need to point out that I'm a strong eventualist[1]: A wiki based approach is not immediately superior. Wiki articles get edited and improved and become more reliable over time. This in opposition to a journal article which must be "perfect" at the time of publication, at which point it becomes frozen. Early on a (bona fide) journal article will obviously be of higher quality. Over time the wiki article can catch up and pass the journal article. ] [1] https://meta.wikimedia.org/wiki/Eventualism https://meta.wikimedia.org/wiki/Eventualism for further reading on how eventualism pans out in the wiki[m|p]edia context.

Kim_Bruning 4y ago

This is Goodhart's law in action. People measure academic worth by publications, so people need to publish a lot. In theory, if one were to switch to alternate measures, this form of pollution would cease overnight. I wonder if there are superior measures already in use?