3 ms·

Pure speculation ahead- The other day on Hacker News, there was that article about how scientists could not tell GPT-generated paper abstracts from real ones.

Pure speculation ahead- The other day on Hacker News, there was that article about how scientists could not tell GPT-generated paper abstracts from real ones. Which makes me think- abstracts for scientific papers are high-effort. The corpus of scientific abstracts would understandably have a low count of "garbage" compared to, say, Twitter posts or random blogs. That's not to say that all scientific abstracts are amazing, just that their goal is to sound intelligent and convincing, while probably 60% of the junk fed into GPT is simply clickbait and junk content padded to fit some publisher's SEO requirements. In other words, ask GPT to generate an abstract, and I would expect it to be quite good. Ask it to generate a 5-paragraph essay about Huckleberry Finn, and I would expect it to be the same quality as the corpus- that is to say, high-school English students. So now that we know these models can learn many one-shot tasks, perhaps some cleanup of the training data is required to advance. Imagine GPT trained ONLY on the library of congress, without the shitty travel blogs or 4chan rants.

williamcotton 4y ago
The science is in the reproduction of the methodology, not in the abstract… in fact, a lot of garbage publications with catchy abstracts built on a shaky foundation sounds like one of the issues that plagues contemporary science. That people would stop finding abstracts useful seems a good thing!
JPLeRouzic 4y ago
> "The corpus of scientific abstracts would understandably have a low count of "garbage" compared to, say, Twitter posts or random blogs" That's certainly true, but it's not by a so large margin, at least in biology. For example in ALS (a neurodegenerative disease) there is a real breakthrough perhaps every two years, but most papers about ALS (thousands every year) look like they describe something very important. Similarly for ALZforum the most recent "milestone" paper about Alzheimer disease was in 2012, yet in 2022 alone there were more than 16K papers! So the ratio signal on noise is close to zero. https://www.alzforum.org/papers?type%5Bmilestone%5D=milestone&page=1 https://www.alzforum.org/papers?type%5Bmilestone%5D=mileston... https://pubmed.ncbi.nlm.nih.gov/?term=alzheimer%27s+disease&sort=date https://pubmed.ncbi.nlm.nih.gov/?term=alzheimer%27s+disease&...
deleted 4y ago
[deleted]
ncraig 4y ago
Some might say that abstracts are the original clickbait.

williamcotton 4y ago

The science is in the reproduction of the methodology, not in the abstract… in fact, a lot of garbage publications with catchy abstracts built on a shaky foundation sounds like one of the issues that plagues contemporary science. That people would stop finding abstracts useful seems a good thing!

JPLeRouzic 4y ago

> "The corpus of scientific abstracts would understandably have a low count of "garbage" compared to, say, Twitter posts or random blogs" That's certainly true, but it's not by a so large margin, at least in biology. For example in ALS (a neurodegenerative disease) there is a real breakthrough perhaps every two years, but most papers about ALS (thousands every year) look like they describe something very important. Similarly for ALZforum the most recent "milestone" paper about Alzheimer disease was in 2012, yet in 2022 alone there were more than 16K papers! So the ratio signal on noise is close to zero. https://www.alzforum.org/papers?type%5Bmilestone%5D=milestone&page=1 https://www.alzforum.org/papers?type%5Bmilestone%5D=mileston... https://pubmed.ncbi.nlm.nih.gov/?term=alzheimer%27s+disease&sort=date https://pubmed.ncbi.nlm.nih.gov/?term=alzheimer%27s+disease&...

deleted 4y ago

[deleted]

ncraig 4y ago

Some might say that abstracts are the original clickbait.