4 ms·

Model collapse is already happening

18 points by zdw 7mo ago

FeepingCreature 7mo ago
Source: a bad study from 2023.
- slowmovintarget 7mo ago
  Why is the study bad? https://www.nature.com/articles/s41586-024-07566-y https://www.nature.com/articles/s41586-024-07566-y
  - FeepingCreature 7mo ago
    Because they exclusively used a model that was about as big as the original GPT-2. Which, I mean, fair enough within these constraints, but it's cited like it's a universal law. Really all that can be taken away from the study is "we trained a very small model on data generated from it in a particular way, and this was eventually harmful for the model." Also note that models are nowadays trained on massively self-generated data (task RL post-training) and it seems to significantly improve their performance.
levocardia 7mo ago
Evidence: trust me bro. Really, where is the actual evidence that models are "collapsing" from too much AI-generated training material? Evals are up, subjective perception of model usefulness is up (for me, certainly), and if anything the slop levels are down, or at least stable. I find it hard to believe that seven-figure software engineers at top labs aren't being careful about how much post-ChatGPT-era internet content is going into their training data.
- jrmg 7mo ago
  I find it hard to believe that seven-figure software engineers at top labs aren't being careful about how much post-ChatGPT-era internet content is going into their training data. I agree - but as the Internet descends into all-slop-all-the-time (seriously, just do a search for reviews or travel advice or technical questions -or most anything - to see it), where do you expect the high quality training material on future things to come from? I have a hard time imagining it.
  - ctoth 7mo ago
    Your Claude Code sessions. Every interaction. Every time the model is asked to do something and then gets feedback on that something (this didn't work I got this traceback) Textbooks, company wikis, news corpora, structured reports of all kinds from far more sources than what is available on the web.
    - Terretta 7mo ago
      On your first line -- is it clear that's a good thing? Massive "it depends". Sadly, enterprise fizzbuzz style is wildly successful compared to ghostty style. Put another way, a gem of code versus the masses of mess. It's amazing new models aren't worse. And now most of this human interaction is with vibers. LLMs trained by the crowd risk being medianizers, or rather, mediocritizers. One need not look further than "Absolutely!" to see this in play -- user selection matters for corpus matters for model. Suddenly content everywhere is “Little houses, all alike.” On your second line -- I couldn't agree more strongly. ANTHROP\C has been sitting inside high performance white collar industries with top builders, that signal is priceless compared to feedback farms in Kenya. Bet on models that see spikey pointy mastery at play.
chromacity 7mo ago
There's some comedy in this article having all the hallmarks of LLM writing.
- justonceokay 7mo ago
  Yeah a typo in the subtitle does not especially inspire confidence
  - deleted 7mo ago
    [deleted]
  - niccl 7mo ago
    you've got me. What's the typo?
    - justonceokay 7mo ago
      It seems to me there is a word or two missing between “rich” and “slowly”. If I read the whole thing aloud I cannot parse it into a sentence. Or the word “rich” could be removed. That would be clunky but at least grammatically sensible. “Make data get smoothed out” is a very strange way of saying “smooths out data”
      - quantified 7mo ago
        It might be weird if you haven't read a lot of English. It's actually quite normal to say that process X is a way to make effect Y happen. "Makes your mout water" is more effective than "waters your mouth". "Makes your breath fresh and tolerable" is better than "freshens and tolerablerizes your breath". Etc. Actually, what you are describing is what happens when LLM-generated prose cycles and then trains humans to use equally dull thinking.
        justonceokay 7mo ago
        I have read a lot of English. That’s why it’s weird
kimi 7mo ago
I have a pet-peeve with this. As a non-native English speaker, I find it very useful to dictate multiple notes, in different languages, and have the LLM produce clear English prose out of it. The prose may be LLM-generated, but I edit it when needed to make sure that the contents is 100% mine. It's like dictating to a typist like they did in the 60's - he will make sure that your letter looks professional and will fix your grammar, but you will sign the letter. This is totally different from LLM spam, the kind that inflates a sentence into a three-page article full of nothing. So - is it a problem if the language reverts to a mean? that is the point of a shared language, right?
- mvdtnz 7mo ago
  It's not just the language that reverts to a mean, it's the knowledge embedded in the model. If you're interested in discussing niche topics with ChatGPT, the further the model collapses the less likely you are to get meaningful results from the "tail" - the areas of knowledge that fall at the far ends of the model's bell curve.
  - kimi 7mo ago
    Actually, both will, as they are not separate within the LLM. The thing is, one is a style issue, the other content. You can express original ideas and still use a lot of em dashes, or produce slop with a lot of typos in it.
SunshineTheCat 7mo ago
I always find articles like this very odd and nebulous because they act as though AI models are just Google. Type request, get info. But that's such a narrow/one dimensional view of how LLMs are used. They can gather data or write an article, but that's probably a minority of use cases. People have casual conversations with them, code written, brainstorming sessions, dictating a voice-recorded note, and the list goes on. While data its getting trained on is important, the supposition is that this data consists only of what sits out there on the interwebs. That as oppose to user input/interaction which, I'm guessing, has a pretty large role in training models. Maybe even more so in some cases than AI-written blog spam.
slowmovintarget 7mo ago
[dead]
theAurenVale 7mo ago
you can already see this in AI generated images tbh. compare the outputs from early midjourney v5 to whats coming out of newer models trained on synthetic data, theres this wierd homogeneity creeping in. everything looks techincally clean but increasingly samey. the unique visual accidents and imperfections that made early outputs intresting are getting smoothed away
mikewittie 7mo ago
"What the field actually needs is a robust data provenance infrastructure. We need to know where training data came from, how it was generated, and whether it’s been through a model already. That’s a hard systems problem, and it’s not glamorous enough to attract the attention it deserves. Everyone wants to build the next frontier model. Very few people want to build the plumbing that makes frontier models trustworthy." We did, by guaranteeing agreement and immutability at four orders of magnitude lower cost than Solana: https://arxiv.org/abs/2511.21969 https://arxiv.org/abs/2511.21969

18 points by zdw 7mo ago

FeepingCreature 7mo ago

Source: a bad study from 2023.

slowmovintarget 7mo ago
Why is the study bad? https://www.nature.com/articles/s41586-024-07566-y https://www.nature.com/articles/s41586-024-07566-y
- FeepingCreature 7mo ago
  Because they exclusively used a model that was about as big as the original GPT-2. Which, I mean, fair enough within these constraints, but it's cited like it's a universal law. Really all that can be taken away from the study is "we trained a very small model on data generated from it in a particular way, and this was eventually harmful for the model." Also note that models are nowadays trained on massively self-generated data (task RL post-training) and it seems to significantly improve their performance.

levocardia 7mo ago

Evidence: trust me bro. Really, where is the actual evidence that models are "collapsing" from too much AI-generated training material? Evals are up, subjective perception of model usefulness is up (for me, certainly), and if anything the slop levels are down, or at least stable. I find it hard to believe that seven-figure software engineers at top labs aren't being careful about how much post-ChatGPT-era internet content is going into their training data.

jrmg 7mo ago
I find it hard to believe that seven-figure software engineers at top labs aren't being careful about how much post-ChatGPT-era internet content is going into their training data. I agree - but as the Internet descends into all-slop-all-the-time (seriously, just do a search for reviews or travel advice or technical questions -or most anything - to see it), where do you expect the high quality training material on future things to come from? I have a hard time imagining it.
- ctoth 7mo ago
  Your Claude Code sessions. Every interaction. Every time the model is asked to do something and then gets feedback on that something (this didn't work I got this traceback) Textbooks, company wikis, news corpora, structured reports of all kinds from far more sources than what is available on the web.
  - Terretta 7mo ago
    On your first line -- is it clear that's a good thing? Massive "it depends". Sadly, enterprise fizzbuzz style is wildly successful compared to ghostty style. Put another way, a gem of code versus the masses of mess. It's amazing new models aren't worse. And now most of this human interaction is with vibers. LLMs trained by the crowd risk being medianizers, or rather, mediocritizers. One need not look further than "Absolutely!" to see this in play -- user selection matters for corpus matters for model. Suddenly content everywhere is “Little houses, all alike.” On your second line -- I couldn't agree more strongly. ANTHROP\C has been sitting inside high performance white collar industries with top builders, that signal is priceless compared to feedback farms in Kenya. Bet on models that see spikey pointy mastery at play.

chromacity 7mo ago

There's some comedy in this article having all the hallmarks of LLM writing.

justonceokay 7mo ago
Yeah a typo in the subtitle does not especially inspire confidence
- deleted 7mo ago
  [deleted]
- niccl 7mo ago
  you've got me. What's the typo?
  - justonceokay 7mo ago
    It seems to me there is a word or two missing between “rich” and “slowly”. If I read the whole thing aloud I cannot parse it into a sentence. Or the word “rich” could be removed. That would be clunky but at least grammatically sensible. “Make data get smoothed out” is a very strange way of saying “smooths out data”
    - quantified 7mo ago
      It might be weird if you haven't read a lot of English. It's actually quite normal to say that process X is a way to make effect Y happen. "Makes your mout water" is more effective than "waters your mouth". "Makes your breath fresh and tolerable" is better than "freshens and tolerablerizes your breath". Etc. Actually, what you are describing is what happens when LLM-generated prose cycles and then trains humans to use equally dull thinking.
      - justonceokay 7mo ago
        I have read a lot of English. That’s why it’s weird

kimi 7mo ago

I have a pet-peeve with this. As a non-native English speaker, I find it very useful to dictate multiple notes, in different languages, and have the LLM produce clear English prose out of it. The prose may be LLM-generated, but I edit it when needed to make sure that the contents is 100% mine. It's like dictating to a typist like they did in the 60's - he will make sure that your letter looks professional and will fix your grammar, but you will sign the letter. This is totally different from LLM spam, the kind that inflates a sentence into a three-page article full of nothing. So - is it a problem if the language reverts to a mean? that is the point of a shared language, right?

mvdtnz 7mo ago
It's not just the language that reverts to a mean, it's the knowledge embedded in the model. If you're interested in discussing niche topics with ChatGPT, the further the model collapses the less likely you are to get meaningful results from the "tail" - the areas of knowledge that fall at the far ends of the model's bell curve.
- kimi 7mo ago
  Actually, both will, as they are not separate within the LLM. The thing is, one is a style issue, the other content. You can express original ideas and still use a lot of em dashes, or produce slop with a lot of typos in it.

SunshineTheCat 7mo ago

I always find articles like this very odd and nebulous because they act as though AI models are just Google. Type request, get info. But that's such a narrow/one dimensional view of how LLMs are used. They can gather data or write an article, but that's probably a minority of use cases. People have casual conversations with them, code written, brainstorming sessions, dictating a voice-recorded note, and the list goes on. While data its getting trained on is important, the supposition is that this data consists only of what sits out there on the interwebs. That as oppose to user input/interaction which, I'm guessing, has a pretty large role in training models. Maybe even more so in some cases than AI-written blog spam.

slowmovintarget 7mo ago

[dead]

theAurenVale 7mo ago

you can already see this in AI generated images tbh. compare the outputs from early midjourney v5 to whats coming out of newer models trained on synthetic data, theres this wierd homogeneity creeping in. everything looks techincally clean but increasingly samey. the unique visual accidents and imperfections that made early outputs intresting are getting smoothed away

mikewittie 7mo ago

"What the field actually needs is a robust data provenance infrastructure. We need to know where training data came from, how it was generated, and whether it’s been through a model already. That’s a hard systems problem, and it’s not glamorous enough to attract the attention it deserves. Everyone wants to build the next frontier model. Very few people want to build the plumbing that makes frontier models trustworthy." We did, by guaranteeing agreement and immutability at four orders of magnitude lower cost than Solana: https://arxiv.org/abs/2511.21969 https://arxiv.org/abs/2511.21969