4 ms·

GPT might make fabricating scientific papers easier, but let's not forget how many humans fabricated scientific research in recent years - they did a great job

by nomilk 2y ago

GPT might make fabricating scientific papers easier, but let's not forget how many humans fabricated scientific research in recent years - they did a great job without AI! For any who haven't seen/heard, this makes for some entertaining and eye-opening viewing! https://www.youtube.com/results?search_query=academic+fraud https://www.youtube.com/results?search_query=academic+fraud

bumby 2y ago
Is there good data on how many are fraudulent? I know there’s reasonable data on replicability issues, but that’s potentially different.
benreesman 2y ago
I think it’s important to remember that while the tidal wave of spam just starting to crest courtesy of the less scrupulous LLM vendors is uh, necessary to address, this century’s war on epistemology was well underway already in the grand traditions of periodic wars on the idea that facts are even aspirationally, directionally worthwhile. The phrase “alternative facts” hit the mainstream in 2016 and the idea that resistance is futile on broad-spectrum digital weaponized bytes was muscular then (that was around the time I was starting to feel ill for being a key architect of it). Now technology is a human artifact and always ends up resembling its creators or financiers or both: I’d have nice fonts on my computer in 2024 most likely either way, but it’s directly because of Jobs they were available in 1984 to a household budget. If someone other than Altman had or some other insight than “this thing can lie in a newly scalable way” was the escape velocity moment on LLMs then we’d still have test sets and metrics and just science going on in the Commanding Heights of the S&P 500, but these people are a symptom of our apathy around any noble instinct. If we had stuck firm on our values no effective altruism cult leader type would even make the press.
- __loam 2y ago
  Post-modernism was a mistake.
  - benreesman 2y ago
    Indeed. I used to think that when it hybridized with Objectivism that was the nastiest malware around but god damn if Amodei and co haven’t rootkitted society to a new level.
- greesil 2y ago
  The metric is in fact the stock price.
- eli_gottlieb 2y ago
  >(that was around the time I was starting to feel ill for being a key architect of it). Now this sounds like a story worth hearing!
croes 2y ago
But AI is to papers what the assembly line was to cars.
anileated 2y ago
Difficulty and scale matter where it comes to fabrication. Academia is a lot about barriers, which while sometimes unpleasant and malfunctioning nevertheless serve a purpose (unfortunately, it is impossible to evaluate everything fully on per-case basis, so humans need shortcuts to filter out noise and determine quicker if it is worth spending attention on). One of the barriers is in the form of the paper itself. The fall of this barrier (notably through often unauthorised use of others’ IP) would likely bring about not sudden idyllic meritocracy but increased noise and/or strengthening of other barriers.
EasyMark 2y ago
Sure, but that takes time, AI has the potential to generate “real sounding”papers in under a second. At least the fake papers before were rate limited.

by nomilk 2y ago

bumby 2y ago

Is there good data on how many are fraudulent? I know there’s reasonable data on replicability issues, but that’s potentially different.

benreesman 2y ago

I think it’s important to remember that while the tidal wave of spam just starting to crest courtesy of the less scrupulous LLM vendors is uh, necessary to address, this century’s war on epistemology was well underway already in the grand traditions of periodic wars on the idea that facts are even aspirationally, directionally worthwhile. The phrase “alternative facts” hit the mainstream in 2016 and the idea that resistance is futile on broad-spectrum digital weaponized bytes was muscular then (that was around the time I was starting to feel ill for being a key architect of it). Now technology is a human artifact and always ends up resembling its creators or financiers or both: I’d have nice fonts on my computer in 2024 most likely either way, but it’s directly because of Jobs they were available in 1984 to a household budget. If someone other than Altman had or some other insight than “this thing can lie in a newly scalable way” was the escape velocity moment on LLMs then we’d still have test sets and metrics and just science going on in the Commanding Heights of the S&P 500, but these people are a symptom of our apathy around any noble instinct. If we had stuck firm on our values no effective altruism cult leader type would even make the press.

__loam 2y ago
Post-modernism was a mistake.
- benreesman 2y ago
  Indeed. I used to think that when it hybridized with Objectivism that was the nastiest malware around but god damn if Amodei and co haven’t rootkitted society to a new level.
greesil 2y ago
The metric is in fact the stock price.
eli_gottlieb 2y ago
>(that was around the time I was starting to feel ill for being a key architect of it). Now this sounds like a story worth hearing!

croes 2y ago

But AI is to papers what the assembly line was to cars.

anileated 2y ago

Difficulty and scale matter where it comes to fabrication. Academia is a lot about barriers, which while sometimes unpleasant and malfunctioning nevertheless serve a purpose (unfortunately, it is impossible to evaluate everything fully on per-case basis, so humans need shortcuts to filter out noise and determine quicker if it is worth spending attention on). One of the barriers is in the form of the paper itself. The fall of this barrier (notably through often unauthorised use of others’ IP) would likely bring about not sudden idyllic meritocracy but increased noise and/or strengthening of other barriers.

EasyMark 2y ago

Sure, but that takes time, AI has the potential to generate “real sounding”papers in under a second. At least the fake papers before were rate limited.