3 ms·

Based on the scale and impact of fraudulent results, I wonder if some form of LLM based approach with supervised fine-tuning could help highlight the actually u

Based on the scale and impact of fraudulent results, I wonder if some form of LLM based approach with supervised fine-tuning could help highlight the actually useful research. Papers are typically weighted by citations, but in the case of fraud, citations can be misleading. Perhaps there's a way to embed all the known alzheimer's research, then finetune the embeddings using negative labels for known fraudulent studies. The the resulting embedding space (depending on how its constructed; perhaps with a citation graph under the hood?) might be one way to reweight the existing literature?

deleted 2y ago
[deleted]
rossdavidh 2y ago
I honestly cannot tell if you're being serious or sarcastic.
- shermantanktop 2y ago
  Me neither. But it’s very much in keeping with other seriously-intended suggestions I’ve heard. Optimism is fine until it becomes just dreaming and wishing out loud.
- LarsDu88 2y ago
  Can you explain to me what about the idea I presented is flawed or infeasible?
  - rossdavidh 2y ago
    Sorry, I guess you were not being sarcastic. LLM's are good at vocabulary and syntax, but not good at content (because nothing in their architecture is designed to do that). Since the kind of article we're looking for would be fine if written exactly the same, but it was true, an LLM is not a good match for finding this. Now there might be algorithms that could, for example automatically checking for photo doctoring or reuse of previously used images that are not attributed. These sorts of things would also not be an LLM's forte. My apologies again, it's just that LLMs are the subject of so much hype nowadays that I genuinely thought you might be saying this in jest.
    - LarsDu88 2y ago
      I think you misunderstood my proposal LLMs are good a producing embeddings which are latent representations of the content in the text. That content for research papers includes things like authorship, research directions, and citations to other papers. When you fine-tune a model that generates such embeddings with a labeled dataset representing fraud (consisting of say 1000s of samples), the resulting model will produce different embeddings which can be clustered. The clusterings will be different between the model with the fraudulent information and without the fraudulent information. Now using this embedding generation model, you (may) have a way to discern what truly significant research looks like, versus research that has been tainted from excess regurgitation of untrustworthy data.
mike_hearn 2y ago
Highlighting research that's useful is probably too difficult, but highlighting research that definitely isn't should be well within the bounds of the best existing reasoning LLMs. There are a lot of common patterns in papers that you learn to look for after a while, and it's now absolutely within reach of automation. For example, papers whose conclusions section doesn't match the conclusions in the abstract are a common problem, likewise papers that contain fake citations (the cited document doesn't support the claim, or has nothing to do with the claim, or sometimes is even retracted). For that matter you can get a long way with pre-neural NLP. This database tracks suspicious papers using hand-crafted heuristics: https://dbrech.irit.fr/pls/apex/f?p=9999:1 https://dbrech.irit.fr/pls/apex/f?p=9999:1:::::: What you'll quickly realize though is that none of this matters. Detection happens every day, the problem is that the people who run grant-funded science itself just don't care. At all. Not even a little bit. You can literally send them 100% bulletproof evidence of scientific fraud and they'll just ignore it unless it goes viral and the sort of media they like to read begins asking questions. If it merely goes viral on social media, or if it gets picked up by FOX News or whatever, then they'll not only ignore it but double down and defend or even reward the perpetrators.
- LarsDu88 2y ago
  I think you misunderstand my proposal. I am not describing a fraud classifier. I am describing fine-tuning an embedding space based on papers with known fraud. The content of the fraudelent paper which includes information such as authorship and other citations can be made to exist on an embedding space. Supervised fine-tuning on labels will alter the shape of that embedding space. The resulting embeddings that would come from such a fine-tuned model would generate different clusterings of papers than what you'd get than if you did not have the labeled fraudelent data at all.

deleted 2y ago

[deleted]

rossdavidh 2y ago

I honestly cannot tell if you're being serious or sarcastic.

shermantanktop 2y ago
Me neither. But it’s very much in keeping with other seriously-intended suggestions I’ve heard. Optimism is fine until it becomes just dreaming and wishing out loud.
LarsDu88 2y ago
Can you explain to me what about the idea I presented is flawed or infeasible?
- rossdavidh 2y ago
  Sorry, I guess you were not being sarcastic. LLM's are good at vocabulary and syntax, but not good at content (because nothing in their architecture is designed to do that). Since the kind of article we're looking for would be fine if written exactly the same, but it was true, an LLM is not a good match for finding this. Now there might be algorithms that could, for example automatically checking for photo doctoring or reuse of previously used images that are not attributed. These sorts of things would also not be an LLM's forte. My apologies again, it's just that LLMs are the subject of so much hype nowadays that I genuinely thought you might be saying this in jest.
  - LarsDu88 2y ago
    I think you misunderstood my proposal LLMs are good a producing embeddings which are latent representations of the content in the text. That content for research papers includes things like authorship, research directions, and citations to other papers. When you fine-tune a model that generates such embeddings with a labeled dataset representing fraud (consisting of say 1000s of samples), the resulting model will produce different embeddings which can be clustered. The clusterings will be different between the model with the fraudulent information and without the fraudulent information. Now using this embedding generation model, you (may) have a way to discern what truly significant research looks like, versus research that has been tainted from excess regurgitation of untrustworthy data.

mike_hearn 2y ago

Highlighting research that's useful is probably too difficult, but highlighting research that definitely isn't should be well within the bounds of the best existing reasoning LLMs. There are a lot of common patterns in papers that you learn to look for after a while, and it's now absolutely within reach of automation. For example, papers whose conclusions section doesn't match the conclusions in the abstract are a common problem, likewise papers that contain fake citations (the cited document doesn't support the claim, or has nothing to do with the claim, or sometimes is even retracted). For that matter you can get a long way with pre-neural NLP. This database tracks suspicious papers using hand-crafted heuristics: https://dbrech.irit.fr/pls/apex/f?p=9999:1 https://dbrech.irit.fr/pls/apex/f?p=9999:1:::::: What you'll quickly realize though is that none of this matters. Detection happens every day, the problem is that the people who run grant-funded science itself just don't care. At all. Not even a little bit. You can literally send them 100% bulletproof evidence of scientific fraud and they'll just ignore it unless it goes viral and the sort of media they like to read begins asking questions. If it merely goes viral on social media, or if it gets picked up by FOX News or whatever, then they'll not only ignore it but double down and defend or even reward the perpetrators.

LarsDu88 2y ago
I think you misunderstand my proposal. I am not describing a fraud classifier. I am describing fine-tuning an embedding space based on papers with known fraud. The content of the fraudelent paper which includes information such as authorship and other citations can be made to exist on an embedding space. Supervised fine-tuning on labels will alter the shape of that embedding space. The resulting embeddings that would come from such a fine-tuned model would generate different clusterings of papers than what you'd get than if you did not have the labeled fraudelent data at all.