9 ms·

AI beats human sleuth at finding problematic images in research papers

148 points by webmaven 3y ago

sideshowb 3y ago
The article seems to discuss manipulation (a valid concern) but the research focuses on duplication. I'm not entirely clear what the link between the two is?
- xnx 3y ago
  Duplicated images are often a substitute for real data. https://www.newyorker.com/science/elements/how-a-sharp-eyed-scientist-became-biologys-image-detective https://www.newyorker.com/science/elements/how-a-sharp-eyed-...
  - explaininjs 3y ago
    Yikes. She contacted the journal to report this clear fraud they were perpetuating and received no response for 6 months. Yet people equate “peer reviewed” and “absolute fact”.
    - Waterluvian 3y ago
      If you want to hear back from any organization, you’ve got to write it on lawyer letterhead.
      - staunton 3y ago
        Which indeed doesn't work for journals because publishing wrong research isn't (and shouldn't be) illegal...
        Waterluvian 3y ago
        I just mean, it’s so easy to ignore inquiries and I’ve found that if you ever get ignored and need to be heard, a lawyer’s letterhead attracts attention. I agree with the legal issue you put forth.
        eru 3y ago
        Yes, but the social engineering aspect of that trick might still work.
    - HideousKojima 3y ago
      Peer review means "has no blatantly obvious methodological issues" at best, and not even that a decent amount of the time. A lot of people take it to mean that the paper's results/conclusions are accurate.
      - eru 3y ago
        Peer review standards also differ between fields and journals.
    - courseofaction 3y ago
      I think this demonstrates a hypothesis: The institution is failing, the technology which exploits the failure is close to irrelevant.
      - wolverine876 3y ago
        > I think this demonstrates a hypothesis: The institution is failing This claim needs to be more specific to be meaningful: How is the 'institution' defined? What evidence do we have that it's performing worse than before, or not functioning (producing valuable scientific knowledge)?
    - maxbond 3y ago
      I think when lay people say "peer reviewed research" they mean it as a metanym for "the scientific consensus as established by empirical research and vigorous debate," which isn't absolute fact to be sure, but is about as sturdy a foundation as you can hope for. I think I know more about peer review and retractions than most people - and I know next to nothing.
    - deleted 3y ago
      [deleted]
  - deleted 3y ago
    [deleted]
ishtanbul 3y ago
Frustrating that an article about images has none.
- deleted 3y ago
  [deleted]
ddtaylor 3y ago
I worry about the ongoing ratcheting of the arms races that AI creates. It's nice that we have better tools to detect problems, but those tools are also readily available to bad actors and they will run their attempted fraud through these tools multiple different times until they succeed and bypass them. We see the same thing in malware where every malware producer make sure before they deploy something that it doesn't get detected by any known tools or methods.
- explaininjs 3y ago
  Soon enough we’ll re-learn that you can’t trust anything except that what you can personally reproduce. Personally I’m glad. Too many years of people subscribing to the most bone headed theories and saying “look, it’s peer reviewed!”.
  - foota 3y ago
    While you shouldn't trust everything, trust is certainly still necessary since few people have the experience or resources to do independent scientific studies.
    - brutusborn 3y ago
      Of course you will still need to trust others to reproduce results in areas with which you are unfamiliar, but this is still better than implicitly trusting peer reviewers. Same thing with software security: if software is open source then I still need to trust someone else to determine if the source code is malicious, but it is better than implicitly trusting software because e.g. “a contractor has done a security audit.” Basically I don’t believe you can truly trust closed-anything, especially scientific analysis. You need _less_ trust when the thing is reproducible, which is the best you can ask for.
      - Guvante 3y ago
        Science is very reproducibile by design, half the goal of peer review is to ensure that. The problem is the incentive structure is not aligned with that and so reproduction is never done. I would instead argue that consensus is the best bar we have to judge science. If the majority of author papers align that means either there is a conspiracy or some fraction of things was reproducibile.
23B1 3y ago
Machines are very good at detecting other machines.
supernova87a 3y ago
Why is the article's title phrased with the word "problematic" rather than "duplicated" or outright "fraudulent"? Are we trying not to offend someone's sensibilities, or not leap to conclusions (when in fact, journals have been far too reluctant to do what they should have been doing about this kind of academic dishonesty)?
- whatshisface 3y ago
  AI can't do a full investigation and identify it as fraudulent, it can only find out when it's weird-looking.
- huytersd 3y ago
  Because it’s trying to collapse multiple types of “problems” into one word in the title.
  - jychang 3y ago
    And also because "creating problems" that need to be double checked, is a much lower bar than "fraudulent"; especially if accusing someone of fraud can carry slander/libel charges.
- deleted 3y ago
  [deleted]
- hyperhopper 3y ago
  I am always skeptical when I see the word "problematic". It is often used to handwave over what the actual problem is, and is very often used when the supposed problem is either trivial or a subjective judgement that many people wouldn't even deem to be a problem.
- bagels 3y ago
  Things like this are why you cannot immediately know it's fraud: a bug in xerox copiers changes numbers to other numbers due to some OCR nonsense. http://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning http://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_...
hella_heely 3y ago
How is the software only 2-3x faster than human review, which lasted months? Little in this article is very clear
- geek_at 3y ago
  I guess they had a system that required humans to confirm the validity or so. I had a similar problem when I realized someone put up CSAM (Children sexual abuse material) on my public demo of my CDN called PictShare [1]. I didn't want to look through all of these images so I built a Raspberry Pi with a Neural compute stick [2] that used an AI model trained by yahoo to filter out "nudity" images and I put them in an encrypted ZIP file along with the access logs and sent them to Interpol. This lead to the arrest of a teacher here in Austria so I'm glad I could do my part. This even lead to a BBC article about my system [3] [1] https://github.com/HaschekSolutions/pictshare https://github.com/HaschekSolutions/pictshare [2] https://blog.haschek.at/2018/fight-child-pornography-with-raspi-and-deep-learning.html https://blog.haschek.at/2018/fight-child-pornography-with-ra... [3] https://www.bbc.com/news/technology-44525358 https://www.bbc.com/news/technology-44525358
  - throwaway290 3y ago
    This is awesome, but since you mention you used a nudity detector and haven't reviewed them... I would expect for a popular service you would have gigabytes of random legal nudity for every actual Interpol-worthy image, right? I guess Interpol is OK sifting through all that? And wouldn't that mean that for all of those nude-but-legal pics you were unnecessarily disclosing regular user PII by sending along access logs to law enforcement? Not to say it wasn't worth it in the end
    - geek_at 3y ago
      I asked interpol upfront if it would be okay to send them the data for them to review and they said OK. There would probably not be a reason to upload "legal nudity" on a demo site of a selfhostable service and also I wipe the whole database in irregular intervals. But actually now I'm using the CSAM detection tool from cloudflare which automatically detects, flags and reports CSAM which takes the hassle out for me. Not sure about false positives though but afterall. It's just a demo site and nobody should upload anything other than test things
opportune 3y ago
For one, this definitely seems like a submarine ad for the AI tool (imagetwin, henceforth called AIT for AI tool to not give them too much more free press). Not sure if Nature has an angle (probably also free scans) but they gave the researcher free access to the AIT per the preprint disclaimer [0]. Also, it's not really giving you enough info to asses whether the AIT is better, because it's basically PR, so allow me: For one, the fact-checking against the AIT per the preprint is "Duplications highlighted by ImageTwin.ai were evaluated as appropriate or inappropriate by80 one reviewer (the author)". Not a great way to reduce false-positives given the conflict of interest (more free scans for the author!) and the fact he'd already reviewed most of the offenders. Also, the article says the AIT missed four papers the author flagged. But in the preprint the category with "At least one inappropriate duplication was identified during the manual review, none were highlighted by the ImageTwin.ai software" has 34 members. Per the author's results text, out of 715 papers with images, the author caught 34 inappropriate duplications the AIT missed, the AIT caught 57 inappropriate duplications that the author subsequently agreed were bad, and they both caught 24 together. But these numbers disagree with the venn diagram shown in figure 3 and referenced in the conclusion, which the Nature article also references. So... am I missing something or is that inconsistent? And were there any AIT flagged papers that the author disagreed were problematic or not? AIT charges a pretty non-trivial amount per scan, at high volume it's still >$2 before you get to custom pricing [1]. It was "only" 2-3x faster than the researcher, and at least per the article it still needs to be checked for false positives. Taking some normal researcher pay and reasonable estimates for the rate at which they can review papers, it seems like the AIT is pretty damn expensive and might be priced competitively to "research intern". This technology has existed for a really long time in the form of reverse image search. Google launched it in 2011, later neutered it (probably not profitable enough), but Yandex still has a pretty good one since 2014 [2]. Overall this seems like a pretty sloppy preprint with an obvious conflict of interest, and a tool that doesn't have any apparent innovation beyond commercializing a probably-underserved vertical. [0] https://www.biorxiv.org/content/10.1101/2023.09.03.556099v2 https://www.biorxiv.org/content/10.1101/2023.09.03.556099v2 [1] https://imagetwin.ai/pricing/ https://imagetwin.ai/pricing/ [2] https://www.searchenginewatch.com/2014/06/19/yandexs-sibir-reverse-image-search-technology-is-pure-awesome/ https://www.searchenginewatch.com/2014/06/19/yandexs-sibir-r...
ACV001 3y ago
This would be a good article title for let's say 1980 or 1970s period..
skywhopper 3y ago
This article is very confusing. It switches between talking about image duplication and image manipulation. It sounds like the software detects image duplication, ie, reusing an image from some other source. Then there are other references to image manipulation being a problem, but no indication that this software detects manipulated images (unless they mean it finds images that are mostly duplicates, but have been manipulated to hide the duplication).

148 points by webmaven 3y ago

The article seems to discuss manipulation (a valid concern) but the research focuses on duplication. I'm not entirely clear what the link between the two is?

xnx 3y ago
Duplicated images are often a substitute for real data. https://www.newyorker.com/science/elements/how-a-sharp-eyed-scientist-became-biologys-image-detective https://www.newyorker.com/science/elements/how-a-sharp-eyed-...
- explaininjs 3y ago
  Yikes. She contacted the journal to report this clear fraud they were perpetuating and received no response for 6 months. Yet people equate “peer reviewed” and “absolute fact”.
  - Waterluvian 3y ago
    If you want to hear back from any organization, you’ve got to write it on lawyer letterhead.
    - staunton 3y ago
      Which indeed doesn't work for journals because publishing wrong research isn't (and shouldn't be) illegal...
      - Waterluvian 3y ago
        I just mean, it’s so easy to ignore inquiries and I’ve found that if you ever get ignored and need to be heard, a lawyer’s letterhead attracts attention. I agree with the legal issue you put forth.
      - eru 3y ago
        Yes, but the social engineering aspect of that trick might still work.
  - HideousKojima 3y ago
    Peer review means "has no blatantly obvious methodological issues" at best, and not even that a decent amount of the time. A lot of people take it to mean that the paper's results/conclusions are accurate.
    - eru 3y ago
      Peer review standards also differ between fields and journals.
  - courseofaction 3y ago
    I think this demonstrates a hypothesis: The institution is failing, the technology which exploits the failure is close to irrelevant.
    - wolverine876 3y ago
      > I think this demonstrates a hypothesis: The institution is failing This claim needs to be more specific to be meaningful: How is the 'institution' defined? What evidence do we have that it's performing worse than before, or not functioning (producing valuable scientific knowledge)?
  - maxbond 3y ago
    I think when lay people say "peer reviewed research" they mean it as a metanym for "the scientific consensus as established by empirical research and vigorous debate," which isn't absolute fact to be sure, but is about as sturdy a foundation as you can hope for. I think I know more about peer review and retractions than most people - and I know next to nothing.
  - deleted 3y ago
    [deleted]
- deleted 3y ago
  [deleted]

ishtanbul 3y ago

Frustrating that an article about images has none.

deleted 3y ago
[deleted]

ddtaylor 3y ago

I worry about the ongoing ratcheting of the arms races that AI creates. It's nice that we have better tools to detect problems, but those tools are also readily available to bad actors and they will run their attempted fraud through these tools multiple different times until they succeed and bypass them. We see the same thing in malware where every malware producer make sure before they deploy something that it doesn't get detected by any known tools or methods.

explaininjs 3y ago
Soon enough we’ll re-learn that you can’t trust anything except that what you can personally reproduce. Personally I’m glad. Too many years of people subscribing to the most bone headed theories and saying “look, it’s peer reviewed!”.
- foota 3y ago
  While you shouldn't trust everything, trust is certainly still necessary since few people have the experience or resources to do independent scientific studies.
  - brutusborn 3y ago
    Of course you will still need to trust others to reproduce results in areas with which you are unfamiliar, but this is still better than implicitly trusting peer reviewers. Same thing with software security: if software is open source then I still need to trust someone else to determine if the source code is malicious, but it is better than implicitly trusting software because e.g. “a contractor has done a security audit.” Basically I don’t believe you can truly trust closed-anything, especially scientific analysis. You need _less_ trust when the thing is reproducible, which is the best you can ask for.
    - Guvante 3y ago
      Science is very reproducibile by design, half the goal of peer review is to ensure that. The problem is the incentive structure is not aligned with that and so reproduction is never done. I would instead argue that consensus is the best bar we have to judge science. If the majority of author papers align that means either there is a conspiracy or some fraction of things was reproducibile.

23B1 3y ago

Machines are very good at detecting other machines.

supernova87a 3y ago

Why is the article's title phrased with the word "problematic" rather than "duplicated" or outright "fraudulent"? Are we trying not to offend someone's sensibilities, or not leap to conclusions (when in fact, journals have been far too reluctant to do what they should have been doing about this kind of academic dishonesty)?

whatshisface 3y ago
AI can't do a full investigation and identify it as fraudulent, it can only find out when it's weird-looking.
huytersd 3y ago
Because it’s trying to collapse multiple types of “problems” into one word in the title.
- jychang 3y ago
  And also because "creating problems" that need to be double checked, is a much lower bar than "fraudulent"; especially if accusing someone of fraud can carry slander/libel charges.
deleted 3y ago
[deleted]
hyperhopper 3y ago
I am always skeptical when I see the word "problematic". It is often used to handwave over what the actual problem is, and is very often used when the supposed problem is either trivial or a subjective judgement that many people wouldn't even deem to be a problem.
bagels 3y ago
Things like this are why you cannot immediately know it's fraud: a bug in xerox copiers changes numbers to other numbers due to some OCR nonsense. http://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning http://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_...

hella_heely 3y ago

How is the software only 2-3x faster than human review, which lasted months? Little in this article is very clear

geek_at 3y ago
I guess they had a system that required humans to confirm the validity or so. I had a similar problem when I realized someone put up CSAM (Children sexual abuse material) on my public demo of my CDN called PictShare [1]. I didn't want to look through all of these images so I built a Raspberry Pi with a Neural compute stick [2] that used an AI model trained by yahoo to filter out "nudity" images and I put them in an encrypted ZIP file along with the access logs and sent them to Interpol. This lead to the arrest of a teacher here in Austria so I'm glad I could do my part. This even lead to a BBC article about my system [3] [1] https://github.com/HaschekSolutions/pictshare https://github.com/HaschekSolutions/pictshare [2] https://blog.haschek.at/2018/fight-child-pornography-with-raspi-and-deep-learning.html https://blog.haschek.at/2018/fight-child-pornography-with-ra... [3] https://www.bbc.com/news/technology-44525358 https://www.bbc.com/news/technology-44525358
- throwaway290 3y ago
  This is awesome, but since you mention you used a nudity detector and haven't reviewed them... I would expect for a popular service you would have gigabytes of random legal nudity for every actual Interpol-worthy image, right? I guess Interpol is OK sifting through all that? And wouldn't that mean that for all of those nude-but-legal pics you were unnecessarily disclosing regular user PII by sending along access logs to law enforcement? Not to say it wasn't worth it in the end
  - geek_at 3y ago
    I asked interpol upfront if it would be okay to send them the data for them to review and they said OK. There would probably not be a reason to upload "legal nudity" on a demo site of a selfhostable service and also I wipe the whole database in irregular intervals. But actually now I'm using the CSAM detection tool from cloudflare which automatically detects, flags and reports CSAM which takes the hassle out for me. Not sure about false positives though but afterall. It's just a demo site and nobody should upload anything other than test things

opportune 3y ago

For one, this definitely seems like a submarine ad for the AI tool (imagetwin, henceforth called AIT for AI tool to not give them too much more free press). Not sure if Nature has an angle (probably also free scans) but they gave the researcher free access to the AIT per the preprint disclaimer [0]. Also, it's not really giving you enough info to asses whether the AIT is better, because it's basically PR, so allow me: For one, the fact-checking against the AIT per the preprint is "Duplications highlighted by ImageTwin.ai were evaluated as appropriate or inappropriate by80 one reviewer (the author)". Not a great way to reduce false-positives given the conflict of interest (more free scans for the author!) and the fact he'd already reviewed most of the offenders. Also, the article says the AIT missed four papers the author flagged. But in the preprint the category with "At least one inappropriate duplication was identified during the manual review, none were highlighted by the ImageTwin.ai software" has 34 members. Per the author's results text, out of 715 papers with images, the author caught 34 inappropriate duplications the AIT missed, the AIT caught 57 inappropriate duplications that the author subsequently agreed were bad, and they both caught 24 together. But these numbers disagree with the venn diagram shown in figure 3 and referenced in the conclusion, which the Nature article also references. So... am I missing something or is that inconsistent? And were there any AIT flagged papers that the author disagreed were problematic or not? AIT charges a pretty non-trivial amount per scan, at high volume it's still >$2 before you get to custom pricing [1]. It was "only" 2-3x faster than the researcher, and at least per the article it still needs to be checked for false positives. Taking some normal researcher pay and reasonable estimates for the rate at which they can review papers, it seems like the AIT is pretty damn expensive and might be priced competitively to "research intern". This technology has existed for a really long time in the form of reverse image search. Google launched it in 2011, later neutered it (probably not profitable enough), but Yandex still has a pretty good one since 2014 [2]. Overall this seems like a pretty sloppy preprint with an obvious conflict of interest, and a tool that doesn't have any apparent innovation beyond commercializing a probably-underserved vertical. [0] https://www.biorxiv.org/content/10.1101/2023.09.03.556099v2 https://www.biorxiv.org/content/10.1101/2023.09.03.556099v2 [1] https://imagetwin.ai/pricing/ https://imagetwin.ai/pricing/ [2] https://www.searchenginewatch.com/2014/06/19/yandexs-sibir-reverse-image-search-technology-is-pure-awesome/ https://www.searchenginewatch.com/2014/06/19/yandexs-sibir-r...

ACV001 3y ago

This would be a good article title for let's say 1980 or 1970s period..

skywhopper 3y ago

This article is very confusing. It switches between talking about image duplication and image manipulation. It sounds like the software detects image duplication, ie, reusing an image from some other source. Then there are other references to image manipulation being a problem, but no indication that this software detects manipulated images (unless they mean it finds images that are mostly duplicates, but have been manipulated to hide the duplication).