3 ms·

Scientific production in the era of large language models [pdf]

62 points by nkko 9mo ago

conditionnumber 9mo ago
Very cool appendix describing how they collected the data. I was kind of surprised to learn that they collected arXiv abstracts + metadata from Kaggle, but it definitely makes sense. I was also surprised that 6 years of SSRN papers was only ~1.3m documents. If you assume 20 pages/document and 400 words/page and 1.3 tokens/word, then it would only cost (ballpark) $1000 to pass the full corpus through the 4o-mini completions API. I think it would be really neat to build out a "Dataset Used", "Model Used" etc table for SSRN papers. I imagine more complicated questions would be harder to answer (because you might have to analyze non-text parts of the documents).
- barishnamazov 9mo ago
  arXiv creator[0] is one of the co-authors, which gives me better assurance on data collection aspect of the paper. [0] https://en.wikipedia.org/wiki/Paul_Ginsparg https://en.wikipedia.org/wiki/Paul_Ginsparg
barishnamazov 9mo ago
The key finding here is the reversal of the relationship between writing complexity and paper quality. Traditionally, sophisticated writing correlated with higher-quality research (or at least higher status/effort). This paper argues that post-LLM, we are seeing a flood of manuscripts that use complex, polished language but contain substantively weaker scientific contribution. They claim LLM adoption increases output by up to 89%, which is a massive productivity shock. If the cost of generating looks-like-science prose drops to near zero, the signal-to-noise ratio in peer review is going to crash. We are entering the era of the polished turd, and likely worse case of publish and perish [0]. [0] https://en.wikipedia.org/wiki/Publish_or_perish https://en.wikipedia.org/wiki/Publish_or_perish
- lo_zamoyski 9mo ago
  I don’t like how we speak of science and knowledge in terms of production. This is not a factory. We are not in the business of churning out units. But what I might expect is not a worse case of publish or perish, but the collapse of this pathology. If churning out crap is cheap, it will render publishing worthless. What I might expect is greater decentralization. Researchers might share reports and articles with colleagues directly. Networks of trust vs. broadcasting.
  - yawnxyz 9mo ago
    we wish it wasn't, but it totally is — in microbiology at least if you don't find anything meaningful to publish even students/post docs in the best unis like Stanford will still put out tons of review papers and microbial resource announcements
  - dr_dshiv 9mo ago
    Some parts of science can be automated and scaled immensely. Like, look at the science that went into building modern large language models. Massive numbers of theory driven empirical experiments. Ironically, not many papers being written about much of it. In many domains, human ego is the bottleneck— or maybe organizational incentives. Most scientists work largely alone—a lot more could be done with the right leadership.
  - postexitus 9mo ago
    The discipline of writing for sharing knowledge is immense - not only for the sharing part, but to look at yourself (or your research) in the mirror with a critical eye and revise it multiple times. If you don't have that discipline, sharing knowledge even with colleagues / networks becomes harder. It's not only LLM churning out slop, but it's also without that discipline, the researcher putting out half finished work.
  - R_D_Olivaw 9mo ago
    I don't like how we talk about humans as factories either. That we re all here to be a "productive" as possible. That are life is measured in how much crap we produced. I would love for this same ethos to spread to other domains. Why DO we hoard things so much? Why don't we just share more? Is it fear? Is it game -theory? How many households have thousands of dollars worth of equipment to maintain that very household that each is used maybe ~30 minutes a week. Why do we keep churning out more plastic crap that just sits in our spaces. Unused
felipeerias 9mo ago
On many fields, the link between “writing papers” and “producing science” was already fraying before the arrival of LLMs. We will have to find better ways to share and promote valuable research, before we all drown in the noise.
hirenj 9mo ago
It is a huge worry for me that unless we decouple the publishing “system” from the career pathways (i.e., rewards), we are going to lose access to both the careers (to robot-weilding bullshitters) and even worse, the shared space where scientific communication took place. Does anyone know of any writing on the network effects of the publishing system? What would happen if the actual value of the journals (of the little they provide!) were to go away? The death of scientific twitter, and the failure to establish any replacement makes me worry that we won’t be able to coalesce around a replacement system. Obviously preprints play a role, but we really need our scientific communities to engage with them in a more serious way.
- naasking 9mo ago
  The arxiv should itself embed a review and commenting system, possibly even blogging. Publishers are archaic, scientific social media should be arxiv's future.
  - R_D_Olivaw 9mo ago
    I like this idea. But it should be a separate entity for robustness's sake. Let's make it! We'll even hold weekly live video/audio discussions on each paper for it's lifetime. Let the errant and emerging thoughts of researchers accompany the original research so we can better iterate and grow. In the open.
  - setopt 9mo ago
    The risk is that publishers might then start opposing the publication of manuscripts that have been shared as preprints on arXiv, if they start perceiving it as a competitor and not a supplement to their "service". But I concur, arXiv with social media features would be nice.
- lmc 9mo ago
  > The death of scientific twitter, and the failure to establish any replacement In my field it's mostly in LinkedIn now.
  - 9mo ago
ggm 9mo ago
Really badly named article at source. Scientific PAPER production in the era of...
hazrmard 9mo ago
The paper finds: - For LLM-assisted output, the more complex the LLM-writing is, the less likely the paper is to be published. From eyeballing, at WC=-30, both have similar chances of publication (~46%). At the upper range of WC=25, LLM-assisted papers are ~17% less likely to be published. - LLM-assisted authors produced more preprints (+36%). I wonder: - What is the distribution of writing complexity? * Does the 17% publication deficit at WC=25 correspond to 17% of the 36% excess LLM-assisted papers being WC=25, thus nullifying the effect? Although, it puts extra strain on the review process.

62 points by nkko 9mo ago

Very cool appendix describing how they collected the data. I was kind of surprised to learn that they collected arXiv abstracts + metadata from Kaggle, but it definitely makes sense. I was also surprised that 6 years of SSRN papers was only ~1.3m documents. If you assume 20 pages/document and 400 words/page and 1.3 tokens/word, then it would only cost (ballpark) $1000 to pass the full corpus through the 4o-mini completions API. I think it would be really neat to build out a "Dataset Used", "Model Used" etc table for SSRN papers. I imagine more complicated questions would be harder to answer (because you might have to analyze non-text parts of the documents).

barishnamazov 9mo ago
arXiv creator[0] is one of the co-authors, which gives me better assurance on data collection aspect of the paper. [0] https://en.wikipedia.org/wiki/Paul_Ginsparg https://en.wikipedia.org/wiki/Paul_Ginsparg

barishnamazov 9mo ago

The key finding here is the reversal of the relationship between writing complexity and paper quality. Traditionally, sophisticated writing correlated with higher-quality research (or at least higher status/effort). This paper argues that post-LLM, we are seeing a flood of manuscripts that use complex, polished language but contain substantively weaker scientific contribution. They claim LLM adoption increases output by up to 89%, which is a massive productivity shock. If the cost of generating looks-like-science prose drops to near zero, the signal-to-noise ratio in peer review is going to crash. We are entering the era of the polished turd, and likely worse case of publish and perish [0]. [0] https://en.wikipedia.org/wiki/Publish_or_perish https://en.wikipedia.org/wiki/Publish_or_perish

lo_zamoyski 9mo ago
I don’t like how we speak of science and knowledge in terms of production. This is not a factory. We are not in the business of churning out units. But what I might expect is not a worse case of publish or perish, but the collapse of this pathology. If churning out crap is cheap, it will render publishing worthless. What I might expect is greater decentralization. Researchers might share reports and articles with colleagues directly. Networks of trust vs. broadcasting.
- yawnxyz 9mo ago
  we wish it wasn't, but it totally is — in microbiology at least if you don't find anything meaningful to publish even students/post docs in the best unis like Stanford will still put out tons of review papers and microbial resource announcements
- dr_dshiv 9mo ago
  Some parts of science can be automated and scaled immensely. Like, look at the science that went into building modern large language models. Massive numbers of theory driven empirical experiments. Ironically, not many papers being written about much of it. In many domains, human ego is the bottleneck— or maybe organizational incentives. Most scientists work largely alone—a lot more could be done with the right leadership.
- postexitus 9mo ago
  The discipline of writing for sharing knowledge is immense - not only for the sharing part, but to look at yourself (or your research) in the mirror with a critical eye and revise it multiple times. If you don't have that discipline, sharing knowledge even with colleagues / networks becomes harder. It's not only LLM churning out slop, but it's also without that discipline, the researcher putting out half finished work.
- R_D_Olivaw 9mo ago
  I don't like how we talk about humans as factories either. That we re all here to be a "productive" as possible. That are life is measured in how much crap we produced. I would love for this same ethos to spread to other domains. Why DO we hoard things so much? Why don't we just share more? Is it fear? Is it game -theory? How many households have thousands of dollars worth of equipment to maintain that very household that each is used maybe ~30 minutes a week. Why do we keep churning out more plastic crap that just sits in our spaces. Unused

felipeerias 9mo ago

On many fields, the link between “writing papers” and “producing science” was already fraying before the arrival of LLMs. We will have to find better ways to share and promote valuable research, before we all drown in the noise.

hirenj 9mo ago

It is a huge worry for me that unless we decouple the publishing “system” from the career pathways (i.e., rewards), we are going to lose access to both the careers (to robot-weilding bullshitters) and even worse, the shared space where scientific communication took place. Does anyone know of any writing on the network effects of the publishing system? What would happen if the actual value of the journals (of the little they provide!) were to go away? The death of scientific twitter, and the failure to establish any replacement makes me worry that we won’t be able to coalesce around a replacement system. Obviously preprints play a role, but we really need our scientific communities to engage with them in a more serious way.

naasking 9mo ago
The arxiv should itself embed a review and commenting system, possibly even blogging. Publishers are archaic, scientific social media should be arxiv's future.
- R_D_Olivaw 9mo ago
  I like this idea. But it should be a separate entity for robustness's sake. Let's make it! We'll even hold weekly live video/audio discussions on each paper for it's lifetime. Let the errant and emerging thoughts of researchers accompany the original research so we can better iterate and grow. In the open.
- setopt 9mo ago
  The risk is that publishers might then start opposing the publication of manuscripts that have been shared as preprints on arXiv, if they start perceiving it as a competitor and not a supplement to their "service". But I concur, arXiv with social media features would be nice.
lmc 9mo ago
> The death of scientific twitter, and the failure to establish any replacement In my field it's mostly in LinkedIn now.
- 9mo ago

ggm 9mo ago

Really badly named article at source. Scientific PAPER production in the era of...

hazrmard 9mo ago

The paper finds: - For LLM-assisted output, the more complex the LLM-writing is, the less likely the paper is to be published. From eyeballing, at WC=-30, both have similar chances of publication (~46%). At the upper range of WC=25, LLM-assisted papers are ~17% less likely to be published. - LLM-assisted authors produced more preprints (+36%). I wonder: - What is the distribution of writing complexity? * Does the 17% publication deficit at WC=25 correspond to 17% of the 36% excess LLM-assisted papers being WC=25, thus nullifying the effect? Although, it puts extra strain on the review process.