15 ms·
AI models collapse when trained on recursively generated data
- tyingq 2y agoWhich is good background to this story about Reddit locking down robots.txt and trying to get money from the AI teams scraping their content. https://news.ycombinator.com/item?id=41057033 https://news.ycombinator.com/item?id=41057033
- roughly 2y agoIf they're considering Reddit content to be free of generated material, I've got bad news for them. It's not quite the Chernobyl-grade hole that Pinterest has become, but it's hardly "low background".
- tyingq 2y agoSure. I think Reddit is aware though, that time is running out to get paid for whatever human generated content is there that isn't already scraped.
- deleted 2y ago[deleted]
- visarga 2y agoI still believe reddit is an amazing source. Any article you read on reddit, chances are the comments are better than the original text. They will debunk the article, present a diversity of reactions, and most importantly, they will be grounded in public opinion unlike the press which caters to money interests. You just copy-paste a conversation into the LLM and ask for an article. For taste, here is one generated from this very conversation. https://pastebin.com/raw/JFH6PGqg https://pastebin.com/raw/JFH6PGqg
- mvdtnz 2y ago> Any article you read on reddit, chances are the comments are better than the original text. We're talking about reddit dot com here? Seriously? I find it difficult to find any comments worth reading at all on that website. 99% of the stuff that isn't buried is just the same recycled jokes again and again and again.
- tyingq 2y agoI think you're both right. Not all the subreddits are the same.
- daft_pink 2y agoIt’s like the ai generated version of index funds.
- roughly 2y agoBack when I was getting my econ degree, we were taught about the Ultimatum game, which goes like this: You get two participants who don't know each other and will (ostensibly) never see each other again. You give one of them $100, and they make an offer of some portion of it to the other. If the other accepts, both parties keep their portion - so, if A offers B $20, and B accepts, A keeps $80 and B keeps $20, if B rejects, both parties get nothing. Standard economic theory suggests A can offer $1 and B will accept, because otherwise B gets nothing. Spoiler for those of you who haven't seen how standard economic theory plays out in real life, that's not how the game went - typically, offers below ~$30 or so got rejected, because B was a real feeling person who felt like they were getting screwed and opted to punish A for doing so. The exception to this - the people who would take the $1 offer - were people who had been taught economic theory. It turns out you _could_ screw them over and they'd pat themselves on the backs for being very wise. The "tragedy of the commons" is another one of those parts of standard economic theory that never actually played out in reality - we've got examples from all over the world of communities implementing practices and often entire belief systems that led them to be responsible stewards of shared resources without requiring unilateral ownership of that resource and singular acquisition of the benefits of that stewardship, and yet first on the lips of every modern capitalist when describing why they're at a disadvantage if they're not the ones polluting the water supply is the tragedy of the commons.
- asah 2y ago...in the real world, A tells B that he "sourced" the deal and therefore deserves a bigger cut and in the real world, B agrees up to a point (the $30 mark). Over time and rounds of playing the game, the A's of the world learn where the line is and optimize to stay on the correct side of it, only testing the other side 1-2% of the time to see if rules/behavior has changed.
- splwjs 2y agoIt's crazy how most political or economic systems would very obviously collapse in the real world almost instantly without some kind of voluntary moral contract (explicit or implied), yet we've got huge clumps of people demonizing one system or another based on the context of what happens when you implement it in a morally dead societal context. Like there are a ton of people who smirk at your last paragraph and go "nuh uh, hashtag late stage capitalism"
- betenoire 2y agoSeems analogous to the effect of echo chambers on humans
- AnimalMuppet 2y agoOr navel-gazing. In fact, that's one of the classically known flaws. (So well known that it has many names: ivory tower, navel gazing, getting stuck in your own head...) If you don't compare your thoughts to the outside world, it's easy for them to diverge more and more from reality.
- hprotagonist 2y agoI Am Sitting In A Room https://en.wikipedia.org/wiki/I_Am_Sitting_in_a_Room https://en.wikipedia.org/wiki/I_Am_Sitting_in_a_Room
- padraicmahoney 2y agoThis seems extremely interesting, but I don't have the time right now to read this in depth (given I would also need to teach myself a bunch of technical concepts too). Anyone willing to weigh in with a theoretical intuition? The one in the paper is just a little inaccessible to me right now.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- jlos 2y agoAs far as I understand Douglas Hofstadter's Godel, Escher, Bach - self-referential recursive structures (strange loops) are the foundation of consciousness (among other interesting things). I've been watching to see if LLM's becoming self-referential actually improves them as opposed to degrades them.
- mensetmanusman 2y agoThe interesting thing about loops is that they can generate fields (think motion if current generating a magnetic field). Consciousness is more like a field than like a particle (which are also fields), but we haven’t determined how conscious fields fit in physics models.
- hiddencost 2y agoA lot of these papers are wrong. They do something wrong in their setup and then claim their conclusion shows show general truth. Publishing in nature in ML can actually be a red flag, because they're really not well equipped to evaluate a lot of claims. The latest llama model got a lot of its data using labels from llama2, and every frontier lab is talking about self training as the future.
- slashdave 2y agoWho are "they"? And do you actually believe the practice of publishing unvetted preprints is a good thing in ML research?
- hiddencost 2y agoNon sequitur? I never said that. Good venues include main track NeurIPS, ICML, ACL, e.g. Nature is notorious for publishing PR pieces that don't reproduce, and their ML theory publishing has been quite poor. They do pretty well on things like AlphaGo, materials science, or weather modeling because it's more in their wheelhouse and the results don't require a deep understanding of info theory or ML practice.
- slashdave 2y agoThose venues have huge issues with referees. It comes down to who is reviewing the work. The irony in your comment is that it is related to the paper we are discussing. There is a big problem with poisoning from group-think and self reinforcement in current ML research.
- anon291 2y agoIs this an artifact of floating point precision or a fundamental mathematical truth.
- slashdave 2y agoFloating point precision is not involved (most LLM models still function after floating-point quantization). I am puzzled that some find this result at all surprising. You simply cannot generate information from nothing.
- anon291 2y agoI'm not surprised you can't use it to make it better, but one might imagine gradients would go to zero as you fed the model its own output.
- slashdave 2y agoNo, not even close. Gradients don't come to zero in the first place. Training is never perfect.
- anon291 2y agoLet's restate. I'd imagine you end up in local minima that are difficult to escape using model generated data. So sure, non-zero gradients, but if you plot the gradients, I would expect them to orbit at that point. But it seems like they diverge.
- slashdave 2y agoMini-batches and dropout mean that you are constantly jumping out of and into other minima during training of any type (highly-redundant solution space is an important feature of deep learning). This is deliberate and necessary to explore the gigantic parameter space of these huge LLM models.
- 2y ago
- Kuinox 2y agoMeanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works. https://openai.com/index/prover-verifier-games-improve-legibility/ https://openai.com/index/prover-verifier-games-improve-legib... https://www.anthropic.com/research/claude-character https://www.anthropic.com/research/claude-character
- echelon 2y agoI'm long on synthetic data. If you think about evolution and hill climbing, of course it works. You have a pool of information and you accumulate new rearrangements of that information. Fitness selects for the best features within the new pool of data (For primates, opposable thumbs. For AI art, hands that aren't deformed.) It will naturally drift to better optima. RLHF, synthetic data, and enrichment are all we need.
- ainoobler 2y agoAre you sure about this? It's well known that cannibalism in animals leads to degenerative disorders.
- autokad 2y agoI think the direct action of a person taking their idea and thoughts and going through it many times (making changes / updates / fixes) fits better than eating something. however, I do think you still some form of validation data to ensure these are good changes. However, I do get the spirit of the article, that as more information generated online is done by LLms, the validity and use of the output decreases
- megaman821 2y agoThere are other ways AI can help train other AI that aren't generating data. AI could remove low quality data from a training set. It could assist humans in structuring video, 3D and physics simulation datasets for the best learning results.
- simonw 2y ago> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately. I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that well under control.
- jagged-chisel 2y agoI find nothing wrong with your statement. I am curious about the paper's use of "indiscriminate." I read this as "just feed the AI more AI output without care" which one can indeed do deliberately. Seems to me that deliberate discriminate use should yield better against expectations.
- __jl__ 2y agoCame here to say the same. "indiscriminate" doesn't really make sense. It's very deliberate. However, there is one scenario: Scraping of web data. In that case, AI labs might know what is model generated.
- mcswell 2y agoThe question (which I raised in a top-level comment before reading your post) is whether there is any such thing as "discriminate" use of web data. Synthetic data created in the same lab as the LLM is discriminate, but what the authors of the paper are saying (if I read it correctly) is that scraping the web is not currently done in a discriminate way. And it's not at all clear to me that there is a discriminate way to use web scraping, because you can't know for sure what's human-generated and what's LLM-generated.
- simonw 2y agoI get the impression that scraping the web isn't nearly as important a source of LLM training data as it used to be. Everyone is trimming down their training data based on quality - there are plenty of hints about that in the Llama 3.1 paper and Mistral Large 2 announcement. OpenAI are licensing data from sources like the Associated Press. Andrej Karpathy said this: https://twitter.com/karpathy/status/1797313173449764933 https://twitter.com/karpathy/status/1797313173449764933 > Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all.
- simonw 2y agoThe source code that accompanies the paper is available in a zip file here: https://zenodo.org/records/10866595 https://zenodo.org/records/10866595 I copied that into a Gist to make it easier to browse here: https://gist.github.com/simonw/b3ab1588a681dda821da9fb57290d506 https://gist.github.com/simonw/b3ab1588a681dda821da9fb57290d...
- CastIrony 2y ago[flagged]
- bjourne 2y agoThe article contains no proof of theorem 3.1 and finding counterexamples seems trivial. Adult male weight can be modeled by N(85, 20). You can recursively "train" the model on data it generates without having it collapse. It will stay stationary as long as the samples are large enough.
- NaiveBayesian 2y agoI believe that counterexample only works in the limit where the sample size goes to infinity. Every finite sample will have μ≠0 almost surely.(Of course μ will still tend to be very close to 0 for large samples, but still slightly off) So this means the sequence of μₙ will perform a kind of random walk that can stray arbitrarily far from 0 and is almost sure to eventually do so.
- bjourne 2y agoFair point about the mean, but I don't see how the random walk causes the standard deviation to shrink towards zero.
- lostmsu 2y agoI agree. The authors generate a dataset of a similar size as the original and then train on that continuously (e.g. for multiple epochs). That's not what you need to do in order to get new model trained on the knowledge of the teacher. You need to ask the teacher to generate new samples every time, otherwise your generated dataset is not very representative of the totality of knowledge of the teacher. Generating samples every time would (in infinite limit) solve the collapse problem.
- NaiveBayesian 2y agoAgreed, that's what I struggle to see as well. It's not really clear why the variance couldn't stay the same or go to infinity instead. Perhaps it does follow from some property of the underlying Gamma/Wishart distributions.
- mcguire 2y ago
- FredPret 2y agoData in --> slop out. Slop in --> yikes
- vzaliva 2y agoI call this "LLM inbreeding." It's a vicious loop where new models are trained on AI-generated content, resulting in the quality degenerating with each generation.
- tiborsaas 2y agoI like this analogy. With the Cambrian explosion of LLMs, we are getting into safe territory, aren't we? Aren't we?
- dang 2y agoRelated ongoing thread: The problem of 'model collapse': how a lack of human data limits AI progress - https://news.ycombinator.com/item?id=41058867 https://news.ycombinator.com/item?id=41058867 - July 2024 (6 comments)
- swayvil 2y agoThere's a complexity missing there. It's like the effects of incest upon dna. Or an echo chamber upon conversation.
- mcswell 2y agoI must be missing something. Training on the output of your system as if it were validated input seems like an obvious no-no. I'm not talking about using synthetic data (however that might be created in this situation), but rather using anything and everything found on the web as if it were "real", i.e. as if it were human-generated texts rather than the output of the LLM. In this case of course there are multiple LLMs that are creating text which finds its way to the web, but to the extent that the output of the different LLMs have commonalities, this still seems problematic. And afaik, there are no metrics or algorithms that reliably distinguish between human-generated and LLM-generated text, at least not for the current generations of LLMs. What am I missing?
- visarga 2y ago> Training on the output of your system as if it were validated input seems like an obvious no-no. Imagine a scientist inventing theories without testing anything, and then continuing to build on top. Crazy. Not even humans can create absent some kind of feedback or validation from outside. That's why we invented the scientific method.
- meroes 2y agoIsn’t that how math works in some respects? In that, there’s only a hierarchy of consistency (no absolute consistency) for most of math. And we just keep building and building. We tried the absolute consistency route and found it too limiting. Maybe that this doesn’t work for LLMs is a sign they aren’t on the path to AGI… Personally I found LLMs horrendous at this kind of stuff. I’m basically a RLHF peon by trade and if I’m ever needing a quick way to fool a model, I go to simple logical problems, where it can’t lean on external structures, only itself. I don’t mean logical syntax but logical reasoning. I can’t share recent stuff but a just a few months ago the models I work with failed to reason removing 12 cards from a regular deck couldn’t remove an entire suit. That kind of stuff. Why would I want to make my prompt longer and more detailed to provide it extra structure (which is logically superfluous) to ensure it gets the right answer. Im sure a wordy prompt could get it to the right answer. I’m interested in its ability to “reason”, not prompt engineering. Given that math is devoid of external structure, I wonder if there something to this (it’s at least interesting to speculate)
- jksmith 2y agoGiven a time snapshot and enough computing power, isn't recursion inevitable? It's like running out of known universe given time x. So then we're back creating data without a prior dataset, which is still a human domain.
- ziofill 2y agoVery interesting. But wouldn't human preferences still find their way into the datasets of the future?
- asadm 2y ago"Breathing in your own exhaust can be fatal"
- throwthrowuknow 2y agoSo they fine tuned an existing model using its own completions to produce the training set for the next run which uses the fine tuned model as the base. They mention catastrophic forgetting so they are aware of it. I suppose they wanted to get results as quickly as possible but this isn’t an accurate model of reality (pun not intended). They’ve only succeeded in demonstrating something that is well known. If they had made the effort to simulate mitigation of bad data and a growing corpus that included proportionally more synthetic data over time it would have been interesting.
- nostrademons 2y agoThis has happened with much simpler models than LLMs, eg. Google Suggest became noticeably worse when everybody started using Google Suggest to input their queries, because it was trained on real query logs and those query logs started to simply reproduce the output of the Suggest model. SEO and Webspam have similar problems within Google Search. More broadly, this is a reflection of Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." The issue is that any model's purpose is to capture novel, useful data about real human behavior. Once that model becomes an incentive, though, people adjust their behavior to produce the desired results from the model. Authentic behavior disappears, which means there's no useful information content for the model to capture, and future generations of the model instead just reproduce behaviors of the previous generation they were trained on, including quirks. Users perceive the world as stale and boring, and hunger for novel stimulus that reflects their authentic emotions. You could look at this as a full-employment theorem for entrepreneurs and artists.
- mcguire 2y agoFrom my reading of the paper, this is a pretty good description of the problem they identify.
- AceyMan 2y agoSemi off-topic, but I'd put Goodhart's Law up there with Occam's Razor as candidate for 'The most clever (while remaining conceptually simple) thing anybody has ever said.' It amazes me how often it gets to the heart of a problem.
- mcguire 2y agoNature published a computer science paper???! "Given that training a single moderately large model produces twice the American lifetime’s worth of CO2 (ref. 15), we opted to not run such an experiment and instead focus on a more realistic setting for a proof of concept."
- whywhywhywhy 2y agoDunno why people posted this around so much when it’s hard to take it seriously when you read this.
- igorkraw 2y agoIt should be noted that 1. this is nothing that should surprise anyone who has an intuition on control theory and the evolution of unconstrained markov chains 2. there appear to be relatively easy mitigations https://news.ycombinator.com/item?id=41061085 https://news.ycombinator.com/item?id=41061085 (made a separate post because it might be of independent interest to discuss) 3. you still won't get beyond the imititation game boundary without exploration & feedback, i.e. the recursive improvement doomers are, as of now, still wrong
- mvdtnz 2y ago> 1. this is nothing that should surprise anyone who has an intuition on control theory and the evolution of unconstrained markov chains You don't even need to know what a markov chain is. It is intuitively obvious to anyone with two brain cells to rub together that AI can't improve by eating its own vomit.
- lemonwaterlime 2y agoI've been telling people this for the past few years. They would like to find out the hard way what control theorists already know.
- nurettin 2y agoI don't see how this hurts training unless you hurl all hallucinations back at the model. Alpha zero used a similar approach where it trained against itself and that only made it better. I don't think collapse is real.
- zby 2y agoIf the model collapse means that the text produced by it is not statistically identical to the garbage that fills the Internet - then I guess a collapse is the goal.
- m3kw9 2y agoOf course it will collapse if you don’t verify it, I remember OpenAI talking about its research into having a different model verify that data somehow
- exabrial 2y agoMaybe this is true test of intelligence instead of "emulating intelligence"? I can learn from Pythagorus' work, extend it, combine it, apply it, and produce works that are more valuable than the original. Perhaps that gets recognized as important, and others then take that, learn, and repeat the process adding their own experience, increasing the general intelligence.
- myhf 2y agoThis is about language models. They include plenty of real-world concepts that are essential to language. But they are not models of intelligence or knowledge or reasoning. Using generated training data is a good way to ensure that the training includes things that are too obvious to appear in normal writing. (Such as "there are zero giraffes in this photo.") This paper describes the limits of using transformer-generated data to train other transformers.
- DaoVeles 2y agoNo say it ain't so /s I wrote this over a year ago about this. Don't build a city on rock and roll. Don't build a business on a fractal. https://theluddite.org/#!post/the-snake-eats-itself https://theluddite.org/#!post/the-snake-eats-itself
- nfca 2y agoI thought this was fairly obvious. Imperfections would only compound over time. Does anyone remember recursively inter-translating between two languages?
- Strumpfli 2y agoIf I'm correct, we generally percieve AI generated data to be indistinguishable from a human sourced data and we don't have a tool to reliably assess whether a text is AI generated. However, could it be that texts generated by AI models posses some kind of statistical property which causes training to collapse? Then, would it allow us to use it to detect AI texts?
- koliber 2y agoConceptually, a LLM is a lossy compression of all of the data it saw during training. If you feed it lossy data, at each iteration you will get poorer and poorer signal and more noise. Prior generations learned this by copying VHS tapes over and over and making photocopies of photocopies. You can see it today by opening and saving a JPG over and over again.
- hksgdk 2y ago[dead]
- TJM2084 2y agoHere's Gretel's response to the Nature paper on model collapse. Covers the methodology, and how it's flawed, in detail and highlights a lot of other great synthetic data research. https://towardsdatascience.com/addressing-concerns-of-model-collapse-from-synthetic-data-in-ai-7cd380208d14 https://towardsdatascience.com/addressing-concerns-of-model-...