5 ms·

Self-Consuming Generative Models Go MAD

49 points by galaxyLogic 3y ago

tysam_and 3y ago
This is, among other things, a very natural consequence of some of the equations surrounding and involved in Shannon's original noisy channel capacity theorem, where the noise is (in many ways) conditioned upon the structure of the model itself. It is not at all necessarily surprising, I think, from a purely high-level perspective, but I do personally think that I find that it is good to have the analysis. From a purely professional standpoint, I do not believe it is unique or distinctive enough as an individual method to need its own separate name for day-to-day use. From a personal perspective, however, I thought the mad cow disease reference was hilarious and applaud whoever came up with the acronym. I find the benefit in the analysis, and the concerns presented about generated data being present in the data makes sense to me (and if in sufficient quantity, would make sense as biasing the models improperly in a rather significant kind of way). I particularly enjoyed the humor of this line, the tongue-in-cheek nature is very funny/nice to me here: "Ascertaining whether an autophagous loop has gone MAD or not (recall Definition 2.1) requires that we measure how far the synthesized data distribution Gt has drifted from the true data distribution Pr over the generations t." I like their use of color in the paper, I saw a similar orange/green color scheme earlier today and enjoyed it very much as an annotation method. "A fixed real dataset only slows generative model degradation" is again also a natural consequence of Shannon's noisy channel capacity theorem, one can say that with almost nearly perfect certainty that a limited neural network will not be able to perfectly fit the distribution of the data that it is training on, thus it will have bias, variance, or some combination of both, limited ultimately by the model's capacity itself. This w.r.t. the original dataset is noise, and we can choose between whether we want collapse, or recursively encoding the noise patterns of the previous model (which might happen to have an additive effect, or maybe not! Who knows! I do not know for sure here, I have not yet figured this one out myself yet). w.r.t. the real data slowing down degradation, if we are sampling I.I.D. of course then proportionately we still should see some degradation as this is the nature of empirical risk minimization over maximum likelihood estimation. It is still good that they have shown this, however, I thinks. The fresh data loop, I believe, would be an example of actually a kind of noise in and of itself, w.r.t. the original input dataset, and as long as this 'noise' (from the perspective of the model) has a higher SNR than the (potentially slow) collapse of the model's output distribution, then it should (in some kind of proportion at leasts) be constantly-playing 'keep-up' with the fresh data. "First, we find that—regardless of the performance of early generations—the performance of later generations converges to a point that depends only on the amounts of real and synthetic data in the training loop. " -- there we are (I saw this after making the SNR point, this makes sense within this framework of interpretation, then. All in all, I found this paper very aware of itself and what it was studying, it was well-laid out and accessible, and while the points are not necessarily earth-shattering (though I still have to read through some of it, I think), having clear empirical evidence about this phenomenon, detailing it, and cutting away through the forest of (at-least-seemingly) untested battlegrounds is one that I appreciated. Curious to hear what others think about this one. <3 :'))))
galaxyLogic 3y ago
"One doomsday scenario is that, if left uncontrolled for many generations, MAD could poison the data quality and diversity of the entire Internet... "
- t_tsonev 3y ago
  It's the most likely scenario.
Almondsetat 3y ago
If feeding AI generated stuff to train an AI is bad then what does that tell us about humans consuming AI generated stuff? If I give a pig something to eat and he throws up I'm surely not touching that stuff myself
- BartjeD 3y ago
  It reads to me that locking yourself / AI up in a room with only yourself to converse with causes it to go MAD / out of touch with reality. If you interact with people / fresh data regularly you'll avoid that. I'm not sure how much is down to evolving requirements and data formats vs degradation of the training data. The idea sounds common-sense but it's also triggering a bullshit warning for me, because I'd expect more consistency per AI model, and differences to occur on new model interations - regardless of the dataset.
- kvdveer 3y ago
  Not sure you can compare humans and AI this way. It is quite possible that humans have coping mechanisms that AI does not (yet?) have. Or to put it in your analogy, If I feed my cat milk, they'll throw up. That doesn't mean milk is unfit for human consumption.
  - KeplerBoy 3y ago
    milk is unfit for some humans though and I believe excessive AI interaction will come with it's own set of issues for parts of our society. I am of the AI companion startups, that shit can't and won't end pretty.
  - Almondsetat 3y ago
    The example with the pig is because they are famous for eating basically anything without a hitch. If AI generated stuff is so bad you cannot train AI with it why would humans use it to train themselves (aka learn stuff)?
    - gryn 3y ago
      well it not really a good analogy because pigs are very robust and have existed a long time and 'AI'(LLMs) are very brittle fragile embryos that can die from a sneeze. a more relevant analogy for llms is a bullshiter that don't know shit about anything but like talk about everything and they're good at that (talking like they know), I'm sure we all at least know or knew in the past someone like that. at a low enough percentage these bullshiters can thrive and get praise from other people but when there a lot of them and not enough of the real deal, well everyone is in trouble including the bullshiters since they no longer have who to mimic.
_nalply 3y ago
MAD = Model Autophagy Disorder I find that catchy and descriptive on more than one level.
fooker 3y ago
It's articles like this that makes me think it's plausible that we are a small N iterations away from stumbling into a transformer(/newfangled thing) architecture that rivals the reasoning capacity of humans.
- sinuhe69 3y ago
  I wonder how did you reason that? Because their madness looks similar to humans’?
  - dspillett 3y ago
    Perhaps some madness in humans (and other animals with sufficiently complex brains) is part of how we grow our reasoning capabilities. Like a fighter jet that is designed to be a little unstable because that makes them more manoeuvrable with computer aided controls (a stable design will tend to resist changes in flight path more), going off-track is part of how we have ideas: random crap fires in our brains and something triages it, keeping just the useful stuff, or if the less useful stuff gets more attention it is recorded as “let's not do that sort of thing” for future reference. Sometimes those random things are intrusive thoughts (“I wonder what would happen if I snapped and pushed him down those stairs”) sometimes they are bright ideas (“hmm, what if the ground was attracted to the Apple as well?”): the distinction isn't how they are made it is whether the end result would be good or bad overall. Madness in people is sometimes when the control structures stop working to differentiate properly between the two sorts of thoughts, and a dullness in reasoning can be due to these controls being too stringent. Perhaps a future step forward in training AI models for better reasoning and/or more general reasoning is find a way to better emulate this sort of triaged randomness, and some AIs having problems is something we end up accepting as much as we accept natural intelligences sometimes having problems that look similar?
    - ChatGTP 3y ago
      Perhaps reasoning comes from the fact that we're all Aliens wearing VR goolges and playing the game of life together.
Bayes7 3y ago
Summarised by https://xkcd.com/2494 https://xkcd.com/2494
deleted 3y ago
[deleted]
Frummy 3y ago
I wonder if eventually, stronger models exposed to these loops can avoid going mad, like a human monk in isolation
chewxy 3y ago
We're rebranding mode collapse as MAD now?
- kromem 3y ago
  Not "now" - this is like 6 months old. It came out right around the same time as the Stanford paper.
kromem 3y ago
This is like 6 months old. Is there a reason it's suddenly being reposted?
- kromem 3y ago
  Previous post 5 months ago: https://news.ycombinator.com/item?id=37021043 https://news.ycombinator.com/item?id=37021043
- sgt101 3y ago
  There's renewed excitement about synthetic data because of the Deepmind Geometry solving model that appeared yesterday - which was trained on synthetic data. I guess this is why this is being picked up again.
K0balt 3y ago
Perhaps this is vaguely adjacent to a slow motion stream of self reflection, where in the absence of additional (sensory) input, a descent into hallucination is all but inevitable. The same thing happens in a sensory deprivation tank.
tanepiper 3y ago
A better analogy would seem to be coefficient of inbreeding (https://en.wikipedia.org/wiki/Coefficient_of_inbreeding https://en.wikipedia.org/wiki/Coefficient_of_inbreeding)
mrjin 3y ago
Wasn't that natural? Errors accumulate without corrections.
- taylorius 3y ago
  Indeed. Such closed loop dynamics turns the AI into a sort of iterated function system, which gravitate to basins of attraction, or exhibit chaotic behaviour.
red_admiral 3y ago
MAD = Model Autophagy Disorder At least AI hasn't reinvented Mutually Assured Destruction!
visarga 3y ago
How about humans in solitary, do they also go MAD, or the "brain in a vat" scenario. It is unreasonable to train a model on its own outputs for many iterations. > with enough fresh real data, the quality and diversity of the generative models do not degrade over generations When a model gets deployed and is prompted by people it can get fresh data in the prompt - the prompt itself, RAG material, outputs of tools, human responses to its outputs - and this allows for a form of exploration that can go outside the original scope of the model. If you use the model logs to retrain it won't collapse.
- weare138 3y ago
  How does that work for image generation AI? And even with LLMs the prompts are going to contain extremely limited information because the users are primarily requesting information, not providing it. AI is only as good as its models. Now that we're flooding the internet with AI generated content with no way to determine if the content was also produced by AI while endlessly shoveling as much content as possible into these AI models to push up those parameter counts to attract investors, it's inevitable the current generation of popular AI systems will just eat themselves.
  - visarga 3y ago
    LLM interactions can be more than a simple question & answer. Long chats allow the human ample opportunity to steer the model in a direction outside its normal scope. If they also use RAG or other tools like code execution there is even more ample chance to steer the model outside its original experience. Then we can use these sessions as training data to re-tune the base model. It's much cheaper than training a new model, can be done on normal computers. LLMs are not beholden to their training set. They can ingest new information any time, or relate two things that were previously separate to derive new insights.
stcredzero 3y ago
So do genres, art movements, and any number of human "scenes." Hell, I've seen this happening to programming language communities!
galaxyLogic 3y ago
I remember reading somewhere that Mad Cow Disease was perhaps caused by feeding cows brains of dead cows. Luckily we don't hear too much about Mad Cow Disease these days. Let's hope it doesn't come back.

49 points by galaxyLogic 3y ago

This is, among other things, a very natural consequence of some of the equations surrounding and involved in Shannon's original noisy channel capacity theorem, where the noise is (in many ways) conditioned upon the structure of the model itself. It is not at all necessarily surprising, I think, from a purely high-level perspective, but I do personally think that I find that it is good to have the analysis. From a purely professional standpoint, I do not believe it is unique or distinctive enough as an individual method to need its own separate name for day-to-day use. From a personal perspective, however, I thought the mad cow disease reference was hilarious and applaud whoever came up with the acronym. I find the benefit in the analysis, and the concerns presented about generated data being present in the data makes sense to me (and if in sufficient quantity, would make sense as biasing the models improperly in a rather significant kind of way). I particularly enjoyed the humor of this line, the tongue-in-cheek nature is very funny/nice to me here: "Ascertaining whether an autophagous loop has gone MAD or not (recall Definition 2.1) requires that we measure how far the synthesized data distribution Gt has drifted from the true data distribution Pr over the generations t." I like their use of color in the paper, I saw a similar orange/green color scheme earlier today and enjoyed it very much as an annotation method. "A fixed real dataset only slows generative model degradation" is again also a natural consequence of Shannon's noisy channel capacity theorem, one can say that with almost nearly perfect certainty that a limited neural network will not be able to perfectly fit the distribution of the data that it is training on, thus it will have bias, variance, or some combination of both, limited ultimately by the model's capacity itself. This w.r.t. the original dataset is noise, and we can choose between whether we want collapse, or recursively encoding the noise patterns of the previous model (which might happen to have an additive effect, or maybe not! Who knows! I do not know for sure here, I have not yet figured this one out myself yet). w.r.t. the real data slowing down degradation, if we are sampling I.I.D. of course then proportionately we still should see some degradation as this is the nature of empirical risk minimization over maximum likelihood estimation. It is still good that they have shown this, however, I thinks. The fresh data loop, I believe, would be an example of actually a kind of noise in and of itself, w.r.t. the original input dataset, and as long as this 'noise' (from the perspective of the model) has a higher SNR than the (potentially slow) collapse of the model's output distribution, then it should (in some kind of proportion at leasts) be constantly-playing 'keep-up' with the fresh data. "First, we find that—regardless of the performance of early generations—the performance of later generations converges to a point that depends only on the amounts of real and synthetic data in the training loop. " -- there we are (I saw this after making the SNR point, this makes sense within this framework of interpretation, then. All in all, I found this paper very aware of itself and what it was studying, it was well-laid out and accessible, and while the points are not necessarily earth-shattering (though I still have to read through some of it, I think), having clear empirical evidence about this phenomenon, detailing it, and cutting away through the forest of (at-least-seemingly) untested battlegrounds is one that I appreciated. Curious to hear what others think about this one. <3 :'))))

galaxyLogic 3y ago

"One doomsday scenario is that, if left uncontrolled for many generations, MAD could poison the data quality and diversity of the entire Internet... "

t_tsonev 3y ago
It's the most likely scenario.

Almondsetat 3y ago

If feeding AI generated stuff to train an AI is bad then what does that tell us about humans consuming AI generated stuff? If I give a pig something to eat and he throws up I'm surely not touching that stuff myself

BartjeD 3y ago
It reads to me that locking yourself / AI up in a room with only yourself to converse with causes it to go MAD / out of touch with reality. If you interact with people / fresh data regularly you'll avoid that. I'm not sure how much is down to evolving requirements and data formats vs degradation of the training data. The idea sounds common-sense but it's also triggering a bullshit warning for me, because I'd expect more consistency per AI model, and differences to occur on new model interations - regardless of the dataset.
kvdveer 3y ago
Not sure you can compare humans and AI this way. It is quite possible that humans have coping mechanisms that AI does not (yet?) have. Or to put it in your analogy, If I feed my cat milk, they'll throw up. That doesn't mean milk is unfit for human consumption.
- KeplerBoy 3y ago
  milk is unfit for some humans though and I believe excessive AI interaction will come with it's own set of issues for parts of our society. I am of the AI companion startups, that shit can't and won't end pretty.
- Almondsetat 3y ago
  The example with the pig is because they are famous for eating basically anything without a hitch. If AI generated stuff is so bad you cannot train AI with it why would humans use it to train themselves (aka learn stuff)?
  - gryn 3y ago
    well it not really a good analogy because pigs are very robust and have existed a long time and 'AI'(LLMs) are very brittle fragile embryos that can die from a sneeze. a more relevant analogy for llms is a bullshiter that don't know shit about anything but like talk about everything and they're good at that (talking like they know), I'm sure we all at least know or knew in the past someone like that. at a low enough percentage these bullshiters can thrive and get praise from other people but when there a lot of them and not enough of the real deal, well everyone is in trouble including the bullshiters since they no longer have who to mimic.

_nalply 3y ago

MAD = Model Autophagy Disorder I find that catchy and descriptive on more than one level.

fooker 3y ago

It's articles like this that makes me think it's plausible that we are a small N iterations away from stumbling into a transformer(/newfangled thing) architecture that rivals the reasoning capacity of humans.

sinuhe69 3y ago
I wonder how did you reason that? Because their madness looks similar to humans’?
- dspillett 3y ago
  Perhaps some madness in humans (and other animals with sufficiently complex brains) is part of how we grow our reasoning capabilities. Like a fighter jet that is designed to be a little unstable because that makes them more manoeuvrable with computer aided controls (a stable design will tend to resist changes in flight path more), going off-track is part of how we have ideas: random crap fires in our brains and something triages it, keeping just the useful stuff, or if the less useful stuff gets more attention it is recorded as “let's not do that sort of thing” for future reference. Sometimes those random things are intrusive thoughts (“I wonder what would happen if I snapped and pushed him down those stairs”) sometimes they are bright ideas (“hmm, what if the ground was attracted to the Apple as well?”): the distinction isn't how they are made it is whether the end result would be good or bad overall. Madness in people is sometimes when the control structures stop working to differentiate properly between the two sorts of thoughts, and a dullness in reasoning can be due to these controls being too stringent. Perhaps a future step forward in training AI models for better reasoning and/or more general reasoning is find a way to better emulate this sort of triaged randomness, and some AIs having problems is something we end up accepting as much as we accept natural intelligences sometimes having problems that look similar?
  - ChatGTP 3y ago
    Perhaps reasoning comes from the fact that we're all Aliens wearing VR goolges and playing the game of life together.

Bayes7 3y ago

Summarised by https://xkcd.com/2494 https://xkcd.com/2494

deleted 3y ago

[deleted]

Frummy 3y ago

I wonder if eventually, stronger models exposed to these loops can avoid going mad, like a human monk in isolation

chewxy 3y ago

We're rebranding mode collapse as MAD now?

kromem 3y ago
Not "now" - this is like 6 months old. It came out right around the same time as the Stanford paper.

kromem 3y ago

This is like 6 months old. Is there a reason it's suddenly being reposted?

kromem 3y ago
Previous post 5 months ago: https://news.ycombinator.com/item?id=37021043 https://news.ycombinator.com/item?id=37021043
sgt101 3y ago
There's renewed excitement about synthetic data because of the Deepmind Geometry solving model that appeared yesterday - which was trained on synthetic data. I guess this is why this is being picked up again.

K0balt 3y ago

Perhaps this is vaguely adjacent to a slow motion stream of self reflection, where in the absence of additional (sensory) input, a descent into hallucination is all but inevitable. The same thing happens in a sensory deprivation tank.

tanepiper 3y ago

A better analogy would seem to be coefficient of inbreeding (https://en.wikipedia.org/wiki/Coefficient_of_inbreeding https://en.wikipedia.org/wiki/Coefficient_of_inbreeding)

mrjin 3y ago

Wasn't that natural? Errors accumulate without corrections.

taylorius 3y ago
Indeed. Such closed loop dynamics turns the AI into a sort of iterated function system, which gravitate to basins of attraction, or exhibit chaotic behaviour.

red_admiral 3y ago

MAD = Model Autophagy Disorder At least AI hasn't reinvented Mutually Assured Destruction!

visarga 3y ago

How about humans in solitary, do they also go MAD, or the "brain in a vat" scenario. It is unreasonable to train a model on its own outputs for many iterations. > with enough fresh real data, the quality and diversity of the generative models do not degrade over generations When a model gets deployed and is prompted by people it can get fresh data in the prompt - the prompt itself, RAG material, outputs of tools, human responses to its outputs - and this allows for a form of exploration that can go outside the original scope of the model. If you use the model logs to retrain it won't collapse.

weare138 3y ago
How does that work for image generation AI? And even with LLMs the prompts are going to contain extremely limited information because the users are primarily requesting information, not providing it. AI is only as good as its models. Now that we're flooding the internet with AI generated content with no way to determine if the content was also produced by AI while endlessly shoveling as much content as possible into these AI models to push up those parameter counts to attract investors, it's inevitable the current generation of popular AI systems will just eat themselves.
- visarga 3y ago
  LLM interactions can be more than a simple question & answer. Long chats allow the human ample opportunity to steer the model in a direction outside its normal scope. If they also use RAG or other tools like code execution there is even more ample chance to steer the model outside its original experience. Then we can use these sessions as training data to re-tune the base model. It's much cheaper than training a new model, can be done on normal computers. LLMs are not beholden to their training set. They can ingest new information any time, or relate two things that were previously separate to derive new insights.

stcredzero 3y ago

So do genres, art movements, and any number of human "scenes." Hell, I've seen this happening to programming language communities!

galaxyLogic 3y ago

I remember reading somewhere that Mad Cow Disease was perhaps caused by feeding cows brains of dead cows. Luckily we don't hear too much about Mad Cow Disease these days. Let's hope it doesn't come back.