11 ms·

Relatedly: unethical researchers could run it on their own work before submitting. It could massively raise the plausibility of fraudulent papers. I hope your

by yojo 2y ago

Relatedly: unethical researchers could run it on their own work before submitting. It could massively raise the plausibility of fraudulent papers. I hope your version of the world wins out. I’m still trying to figure out what a post-trust future looks like.

rererereferred 2y ago
Eventually the unethical researchers will have to make actual research to make their papers pass. Mission fucking accomplished https://xkcd.com/810/ https://xkcd.com/810/
brookst 2y ago
Both will happen. But the world has been post-trust for millennia.
- GuestFAUniverse 2y ago
  Maybe raise the "accountability" part? Baffles me that somebody can be professor, director, whatever, meaning: taking the place of somebody _really_ qualified and not get dragged through court after falsifying a publication until nothing is left of that betrayer. It's not only the damage to society due to false, misleading claims. If those publications decide who gets tenure, a research grant, etc. there are careers of others, that were massively damaged.
  - StableAlkyne 2y ago
    A retraction due to fraud already torches your career. It's a black mark that makes it harder to get funding, and it's one of the few reasons a university might revoke tenure. And you will be explaining it to every future employer in an interview. There generally aren't penalties beyond that in the West because - outside of libel - lying is usually protected as free speech
shkkmo 2y ago
> unethical researchers could run it on their own work before submitting. It could massively raise the plausibility of fraudulent papers The real low hanging fruit that this helps with is detecting accidental errors and preventing researchers with legitimate intent from making mistakes. Research fraud and its detection is always going to be an adversarial process between those trying to commit it and those trying to detect it. Where I see tools like this making a difference against fraud is that it may also make fraud harder to plausibly pass off as errors if the fraudster gets caught. Since the tools can improve over time, I think this increases the risk that research fraud will be detected by tools that didn't exist when the fraud was perpetrated and which will ideally lead to consequences for the fraudster. This risk will hopefully dissuade some researchers from committing fraud.
SubiculumCode 2y ago
I already ask AI it to be a harsh reviewer on a manuscript before submitting it. Sometimes blunders are there because of how close you are to the work. It hadn't occurred to me that bad "scientists" could use it to avoid detection
- SubiculumCode 2y ago
  I would add that I've never gotten anything particularly insightful in return...but it has pointed out somethings that could be written more clearly, or where I forgot to cite a particular standardized measure, etc.
7speter 2y ago
Peer review will still involve human experts, though?
rs186 2y ago
Students and researchers send their own paper to plagiarism checker to look for "real" and unintended flags before actually submitting the papers, and make revisions accordingly. This is a known, standard practice that is widely accepted. And let's say someone modifies their faked lab results so that no AI can detect any evidence of photoshopping images. Their results get published. Well, nobody will be able to reproduce their work (unless other people also publish fraudulent work from there), and fellow researchers will raise questions, like, a lot of them. Also, guess what, even today, badly photoshopped results often don't get caught for a few years, and in hindsight that's just some low effort image manipulation -- copying a part of image and paste it elsewhere. I doubt any of this changes anything. There is a lot of competition in academia, and depending on the field, things may move very fast. Getting away with AI detection of fraudulent work likely doesn't give anyone enough advantage to survive in a competitive field.
- dccsillag 2y ago
  I've never seen this done in a research setting. Not sure about how much of a standard practice it is.
  - StableAlkyne 2y ago
    It may be field specific, but I've also never heard of anyone running a manuscript through a plagiarism checker in chemistry.
    - deleted 2y ago
      [deleted]
- abirch 2y ago
  You're right that this won't change the incentives for the dishonest researchers. Unfortunately there's not an equivalent of "short sellers" in research, people who are incentivized for finding fraud. AI is definitely a good thing (TM) for those honest researchers.
- owl_vision 2y ago
pinko 2y ago
Normally I'm an AI skeptic, but in this case there's a good analogy to post-quantum crypto: even if the current state of the art allows fraudulent researchers to evade detection by today's AI by using today's AI, their results, once published, will remain unchanged as the AI improves, and tomorrow's AI will catch them...
- tmpz22 2y ago
  I think it’s not always a world scale problem as scientific niches tend to be small communities. The challenge is to get these small communities to police themselves. For the rarer world scale papers we can dedicate more resources to getting vetting them.
- atrettel 2y ago
  Based on my own experience as a peer reviewer and scientist, the issue is not necessarily in detecting plagiarism or fraud. It is in getting editors to care after a paper is already published. During peer review, this could be great. It could stop a fraudulent paper before it causes any damage. But in my experience, I have never gotten a journal editor to retract an already-published paper that had obvious plagiarism in it (very obvious plagiarism in one case!). They have no incentive to do extra work after the fact with no obvious benefit to themselves. They choose to ignore it instead. I wish it wasn't true, but that has been my experience.
- mike_hearn 2y ago
  Doesn't matter. Lots of bad papers get caught the moment they're published and read by someone, but there's no followup. The institutions don't care if they publish auto-generated spam that can be detected on literally a single read through, they aren't going to deploy advanced AI on their archives of papers to create consequences a decade later: https://www.nature.com/articles/d41586-021-02134-0 https://www.nature.com/articles/d41586-021-02134-0
  - fc417fc802 2y ago
    Are we talking about "bad papers", "fraud", "academic misconduct", or something else? It's a rather important detail. You would ideally expect blatant fraud to have repercussions, even decades later. You probably would not expect low quality publications to have direct repercussions, now or ever. This is similar to unacceptably low performance at work. You aren't getting immediately reprimanded for it, but if it keeps up consistently then you might not be working there for much longer. > The institutions don't care if they publish auto-generated spam The institutions are generally recognized as having no right to interfere with freedom to publish or freedom to associate. This is a very good thing. So good in fact that it is pretty much the entire point of having a tenure system. They do tend to get involved if someone commits actual (by which I mean legally defined) fraud.
kkylin 2y ago
Every tool cuts both ways. This won't remove the need for people to be good, but hopefully reduces the scale of the problems to the point where good people (and better systems) can manage. FWIW while fraud gets headlines, unintentional errors and simply crappy writing are much more common and bigger problems I think. As reviewer and editor I often feel I'm the first one (counting the authors) to ever read the paper beginning to end: inconsistent notation & terminology, unnecessary repetitions, unexplained background material, etc.
t_mann 2y ago
AI is fundamentally much more of a danger to the fraudsters. Because they can only calibrate their obfuscation to today's tools. But the publications are set in stone and can be analyzed by tomorrow's tools. There are already startups going through old papers with modern tools to detect manipulation [0]. [0] https://imagetwin.ai/ https://imagetwin.ai/
- dgfitz 2y ago
  Training a language model on non-verified publications seems… unproductive.
dsabanin 2y ago
Maybe at least in some cases these checkers will help them actually find and fix their mistakes and they will end up publishing something useful.
callc 2y ago
Humans are already capable of “post-truth”. This is enabled by instant global communication and social media (not dismissing the massive benefits these can bring), and led by dictators who want fealty over independent rational thinking. The limitations of slow news cycles and slow information transmission lends to slow careful thinking. Especially compared to social media. No AI needed.
- hunter2_ 2y ago
  The communication enabled by the internet is incredible, but this aspect of it is so frustrating. The cat is out of the bag, and I struggle to identify a solution. The other day I saw a Facebook post of a national park announcing they'd be closed until further notice. Thousands of comments, 99% of which were divisive political banter assuming this was the result of a top-down order. A very easy-to-miss 1% of the comments were people explaining that the closure was due to a burst pipe or something to that effect. It's reminiscent of the "tragedy of the commons" concept. We are overusing our right to spew nonsense to the point that it's masking the truth. How do we fix this? Guiding people away from the writings of random nobodies in favor of mainstream authorities doesn't feel entirely proper.
  - mmooss 2y ago
    > Guiding people away from the writings of random nobodies in favor of mainstream authorities doesn't feel entirely proper. Why not? I think the issue is the word "mainstream". If by mainstream, we mean pre-Internet authorities, such as leading newspapers, then I think that's inappropriate and an odd prejudice. But we could use 'authorities' to improve the quality of social media - that is, create a category of social media that follows high standards. There's nothing about the medium that prevents it. There's not much difference between a blog entry and scientific journal publication: The founders of the scientific method wrote letters and reports about what they found; they could just as well have posted it on their blogs, if they could. At some point, a few decided they would follow certain standards --- You have to see it yourself. You need publicly verifiable evidence. You need a falsifiable claim. You need to prove that the observed phenomena can be generalized. You should start with a review of prior research following this standard. Etc. --- Journalists follow similar standards, as do courts. There's no reason bloggers can't do the same, or some bloggers and social media posters, and then they could join the group of 'authorities'. Why not? For the ones who are serious and want to be taken seriously, why not? How could they settle for less for their own work product?
Salgat 2y ago
My hope is that ML can be used to point out real world things you can't fake or work around, such as why an idea is considered novel or why the methodology isn't just gaming results or why the statistics was done wrong.
blueboo 2y ago
Just as plagiarism checkers harden the output of plagiarists. This goes back to a principle of safety engineering: the safer, reliable, trustworthy you make the system, the more catastrophic the failures when they happen.
jstummbillig 2y ago
We are "upgrading" from making errors to committing fraud. I think that difference will still be important to most people. In addition I don't really see why an unethical, but not idiotic, researcher would assume, that the same tool that they could use to correct errors, would not allow others to check for and spot the fraud they are thinking of committing instead.
77pt77 2y ago
> I’m still trying to figure out what a post-trust future looks like. Same as the past. What do you think religions are?
deleted 2y ago
[deleted]
miki123211 2y ago
They should work like the Polish plagiarism-detection system, legally required for all students' theses. You can't just put stuff into that system and tweak your work until there are no issues. It only runs after your final submission. If there are issues, appropriate people are notified and can manually resolve them I think (I've never actually hit that pathway).
promptdaddy 2y ago
You're looking at it

by yojo 2y ago

rererereferred 2y ago

Eventually the unethical researchers will have to make actual research to make their papers pass. Mission fucking accomplished https://xkcd.com/810/ https://xkcd.com/810/

brookst 2y ago

Both will happen. But the world has been post-trust for millennia.

GuestFAUniverse 2y ago
Maybe raise the "accountability" part? Baffles me that somebody can be professor, director, whatever, meaning: taking the place of somebody _really_ qualified and not get dragged through court after falsifying a publication until nothing is left of that betrayer. It's not only the damage to society due to false, misleading claims. If those publications decide who gets tenure, a research grant, etc. there are careers of others, that were massively damaged.
- StableAlkyne 2y ago
  A retraction due to fraud already torches your career. It's a black mark that makes it harder to get funding, and it's one of the few reasons a university might revoke tenure. And you will be explaining it to every future employer in an interview. There generally aren't penalties beyond that in the West because - outside of libel - lying is usually protected as free speech

shkkmo 2y ago

> unethical researchers could run it on their own work before submitting. It could massively raise the plausibility of fraudulent papers The real low hanging fruit that this helps with is detecting accidental errors and preventing researchers with legitimate intent from making mistakes. Research fraud and its detection is always going to be an adversarial process between those trying to commit it and those trying to detect it. Where I see tools like this making a difference against fraud is that it may also make fraud harder to plausibly pass off as errors if the fraudster gets caught. Since the tools can improve over time, I think this increases the risk that research fraud will be detected by tools that didn't exist when the fraud was perpetrated and which will ideally lead to consequences for the fraudster. This risk will hopefully dissuade some researchers from committing fraud.

SubiculumCode 2y ago

I already ask AI it to be a harsh reviewer on a manuscript before submitting it. Sometimes blunders are there because of how close you are to the work. It hadn't occurred to me that bad "scientists" could use it to avoid detection

SubiculumCode 2y ago
I would add that I've never gotten anything particularly insightful in return...but it has pointed out somethings that could be written more clearly, or where I forgot to cite a particular standardized measure, etc.

7speter 2y ago

Peer review will still involve human experts, though?

rs186 2y ago

Students and researchers send their own paper to plagiarism checker to look for "real" and unintended flags before actually submitting the papers, and make revisions accordingly. This is a known, standard practice that is widely accepted. And let's say someone modifies their faked lab results so that no AI can detect any evidence of photoshopping images. Their results get published. Well, nobody will be able to reproduce their work (unless other people also publish fraudulent work from there), and fellow researchers will raise questions, like, a lot of them. Also, guess what, even today, badly photoshopped results often don't get caught for a few years, and in hindsight that's just some low effort image manipulation -- copying a part of image and paste it elsewhere. I doubt any of this changes anything. There is a lot of competition in academia, and depending on the field, things may move very fast. Getting away with AI detection of fraudulent work likely doesn't give anyone enough advantage to survive in a competitive field.

dccsillag 2y ago
I've never seen this done in a research setting. Not sure about how much of a standard practice it is.
- StableAlkyne 2y ago
  It may be field specific, but I've also never heard of anyone running a manuscript through a plagiarism checker in chemistry.
  - deleted 2y ago
    [deleted]
abirch 2y ago
You're right that this won't change the incentives for the dishonest researchers. Unfortunately there's not an equivalent of "short sellers" in research, people who are incentivized for finding fraud. AI is definitely a good thing (TM) for those honest researchers.
owl_vision 2y ago

pinko 2y ago

Normally I'm an AI skeptic, but in this case there's a good analogy to post-quantum crypto: even if the current state of the art allows fraudulent researchers to evade detection by today's AI by using today's AI, their results, once published, will remain unchanged as the AI improves, and tomorrow's AI will catch them...

tmpz22 2y ago
I think it’s not always a world scale problem as scientific niches tend to be small communities. The challenge is to get these small communities to police themselves. For the rarer world scale papers we can dedicate more resources to getting vetting them.
atrettel 2y ago
Based on my own experience as a peer reviewer and scientist, the issue is not necessarily in detecting plagiarism or fraud. It is in getting editors to care after a paper is already published. During peer review, this could be great. It could stop a fraudulent paper before it causes any damage. But in my experience, I have never gotten a journal editor to retract an already-published paper that had obvious plagiarism in it (very obvious plagiarism in one case!). They have no incentive to do extra work after the fact with no obvious benefit to themselves. They choose to ignore it instead. I wish it wasn't true, but that has been my experience.
mike_hearn 2y ago
Doesn't matter. Lots of bad papers get caught the moment they're published and read by someone, but there's no followup. The institutions don't care if they publish auto-generated spam that can be detected on literally a single read through, they aren't going to deploy advanced AI on their archives of papers to create consequences a decade later: https://www.nature.com/articles/d41586-021-02134-0 https://www.nature.com/articles/d41586-021-02134-0
- fc417fc802 2y ago
  Are we talking about "bad papers", "fraud", "academic misconduct", or something else? It's a rather important detail. You would ideally expect blatant fraud to have repercussions, even decades later. You probably would not expect low quality publications to have direct repercussions, now or ever. This is similar to unacceptably low performance at work. You aren't getting immediately reprimanded for it, but if it keeps up consistently then you might not be working there for much longer. > The institutions don't care if they publish auto-generated spam The institutions are generally recognized as having no right to interfere with freedom to publish or freedom to associate. This is a very good thing. So good in fact that it is pretty much the entire point of having a tenure system. They do tend to get involved if someone commits actual (by which I mean legally defined) fraud.

kkylin 2y ago

Every tool cuts both ways. This won't remove the need for people to be good, but hopefully reduces the scale of the problems to the point where good people (and better systems) can manage. FWIW while fraud gets headlines, unintentional errors and simply crappy writing are much more common and bigger problems I think. As reviewer and editor I often feel I'm the first one (counting the authors) to ever read the paper beginning to end: inconsistent notation & terminology, unnecessary repetitions, unexplained background material, etc.

t_mann 2y ago

AI is fundamentally much more of a danger to the fraudsters. Because they can only calibrate their obfuscation to today's tools. But the publications are set in stone and can be analyzed by tomorrow's tools. There are already startups going through old papers with modern tools to detect manipulation [0]. [0] https://imagetwin.ai/ https://imagetwin.ai/

dgfitz 2y ago
Training a language model on non-verified publications seems… unproductive.

dsabanin 2y ago

Maybe at least in some cases these checkers will help them actually find and fix their mistakes and they will end up publishing something useful.

callc 2y ago

Humans are already capable of “post-truth”. This is enabled by instant global communication and social media (not dismissing the massive benefits these can bring), and led by dictators who want fealty over independent rational thinking. The limitations of slow news cycles and slow information transmission lends to slow careful thinking. Especially compared to social media. No AI needed.

hunter2_ 2y ago
The communication enabled by the internet is incredible, but this aspect of it is so frustrating. The cat is out of the bag, and I struggle to identify a solution. The other day I saw a Facebook post of a national park announcing they'd be closed until further notice. Thousands of comments, 99% of which were divisive political banter assuming this was the result of a top-down order. A very easy-to-miss 1% of the comments were people explaining that the closure was due to a burst pipe or something to that effect. It's reminiscent of the "tragedy of the commons" concept. We are overusing our right to spew nonsense to the point that it's masking the truth. How do we fix this? Guiding people away from the writings of random nobodies in favor of mainstream authorities doesn't feel entirely proper.
- mmooss 2y ago
  > Guiding people away from the writings of random nobodies in favor of mainstream authorities doesn't feel entirely proper. Why not? I think the issue is the word "mainstream". If by mainstream, we mean pre-Internet authorities, such as leading newspapers, then I think that's inappropriate and an odd prejudice. But we could use 'authorities' to improve the quality of social media - that is, create a category of social media that follows high standards. There's nothing about the medium that prevents it. There's not much difference between a blog entry and scientific journal publication: The founders of the scientific method wrote letters and reports about what they found; they could just as well have posted it on their blogs, if they could. At some point, a few decided they would follow certain standards --- You have to see it yourself. You need publicly verifiable evidence. You need a falsifiable claim. You need to prove that the observed phenomena can be generalized. You should start with a review of prior research following this standard. Etc. --- Journalists follow similar standards, as do courts. There's no reason bloggers can't do the same, or some bloggers and social media posters, and then they could join the group of 'authorities'. Why not? For the ones who are serious and want to be taken seriously, why not? How could they settle for less for their own work product?

Salgat 2y ago

My hope is that ML can be used to point out real world things you can't fake or work around, such as why an idea is considered novel or why the methodology isn't just gaming results or why the statistics was done wrong.

blueboo 2y ago

Just as plagiarism checkers harden the output of plagiarists. This goes back to a principle of safety engineering: the safer, reliable, trustworthy you make the system, the more catastrophic the failures when they happen.

jstummbillig 2y ago

We are "upgrading" from making errors to committing fraud. I think that difference will still be important to most people. In addition I don't really see why an unethical, but not idiotic, researcher would assume, that the same tool that they could use to correct errors, would not allow others to check for and spot the fraud they are thinking of committing instead.

77pt77 2y ago

> I’m still trying to figure out what a post-trust future looks like. Same as the past. What do you think religions are?

deleted 2y ago

[deleted]

miki123211 2y ago

They should work like the Polish plagiarism-detection system, legally required for all students' theses. You can't just put stuff into that system and tweak your work until there are no issues. It only runs after your final submission. If there are issues, appropriate people are notified and can manually resolve them I think (I've never actually hit that pathway).

promptdaddy 2y ago

You're looking at it