6 ms·

Anyone who has iterated on trained models for long enough knows that feedback loops can be a serious problem. If your models are influencing the generation of d

Anyone who has iterated on trained models for long enough knows that feedback loops can be a serious problem. If your models are influencing the generation of data that they are later retrained on, it gets harder and harder to even maintain model performance. The article mentions one experiment in this direction: "With each generation, the quality of the model actually degraded." This happens whenever there aren't solid strategies to avoid feedback loop issues. Given this, the problem isn't just that there's not enough new content. It's that an ever-increasing fraction of the content in the public sphere will be generated by these models. And can the models detect that they are ingesting their own output? If they get good enough, they probably can't. And then they'll get worse. This could have a strange impact on human language / communication as well. As these models are increasingly trained on their own output, they'll start emulating their own mistakes and more of the content we consume will have these mistakes consistently used. You can imagine people, sometimes intentionally and sometimes not, starting to emulate these patterns and causing shifts in human languages. Interesting times ahead...

rjblackman 3y ago
well we're already being trained by algorithms so I guess this is just an extension of what is already going on. perhaps the quality of human (internal) models will go down too? perhaps they already have?
haltist 3y ago
In system design there is something called resonant amplification and what you are describing is very similar. The biases of the model are amplified with each iteration and the end result is that the system converges onto the patterns recognizable and amplified by its architecture. If you know about impulse and frequency analysis then an AI system can be considered to be a signal processor that amplifies and attenuates certain frequencies in the input/impulse. Running an LLM in a loop always ends up with nonsense as the final output.
- lazystar 3y ago
  So if left unchecked, the thing we built in man's attempt to play god could result in gibberish? sounds kind of like the tower of babel; seems humankinds only defense would be creating a new language that the machines can't infiltrate
  - ben_w 3y ago
    Another defence is to check them, which is why the thumbs up, thumbs down, and regenerate buttons are there on the ChatGPT user interface.
  - haltist 3y ago
    You can try the experiment yourself. Take any open source LLM and then feed the output back into it in a loop and you'll see what I mean. Most LLMs diverge and lose coherence in less than 100 iterations.
    - lazystar 3y ago
      right, im thinking in terms of the world at large - what happens after 5 years of chatgpt, what will the internet and human communication be like in 2030?
HDThoreaun 3y ago
I think it's reasonable to say that this was actually the point of releasing LLMs publicly. The companies that created them wanted a moat and figured the data they had could be it if they poisoned anyones attempt to collect the same data in the future.
- omneity 3y ago
  Intriguing thought, but arguably people are intentionally using GPT to generate synthetic data for their domain specific model. So I'm kinda torn between AI giants poisoning the well with their models, or it just being unforeseen consequences (or one they willingly ignored to be first to market).
  - quickthrower2 3y ago
    Generating data from ChatGPT even $1,000,000 in tokens worth can’t be on the same scale as what OpenAI is collecting from everyone.
js8 3y ago
> If your models are influencing the generation of data that they are later retrained on, it gets harder and harder to even maintain model performance. Why don't humans suffer from this problem, then? Humans have been creating art (and content) that imitates nature and society for thousands of years now, and yet we have little problem (the exception are possibly things like crop circles) to recognize what is a natural phenomenon and what is generated culturally. I think it's wrong to assume that this is a problem with intelligence in general, rather than just a feature (stupidity) of the current models.
- ArekDymalski 3y ago
  > Why don't humans suffer from this problem, then? Can we be sure that we don't? For example, depending on perspective our language is either evolving towards clarity or devolving towards primitivity, compared to XVIII/XIX century. The same could be argued about other aspects of society and culture.
- dragonwriter 3y ago
  > Why don't humans suffer from this problem, then? They do, but the problem is poorly stated. The problen isn't “If your models are influencing the generation of data that they are later retrained on”, its “If output produced by your models with inadequate filtering for quality/accuracy is dominating the data that they are later retrained on.” Humans definitely experience the same problem: we see it when societies become closed, inward-looking, and locked into models of information filtering that don't involve accuracy or fitness for purpose other than confirming an in-groups priors. There are some examoles in stagnation of large societies, but probably the clearest examples are cults that socially isolate their membership and degrade into increasingly bizarre behaviors terminating in mass suicide, murder/suicides, etc. LLM’s experience self-reinforcing degradation more acutely because they are individually less capable than humans and they are less diverse.
  - js8 3y ago
    But humans can have a discussion about what is natural and what is not, and decide to mitigate it. The whole argument is based on the idea that the AI (either collectively or individually) can't even understand that, so the examples of cults etc. just do not apply. It's really the paperclip maximizer argument again. "Superintelligent AI" will supposedly change the world, but will lack enough common sense to understand a simple instruction with ethical constraint. Same here, the crappy AI content will somehow take over the world, despite the fact that most humans can actually mostly recognize original content from a generated one. The art that humans created doesn't replace nature, just like tons of kitsch don't pose a problem for new artists. In the same way, crappy AI content is not gonna replace good content, just augment it, and any decent AI (which has good enough common sense to be called "intelligent") will be able to tell the difference. Nobody's gonna care about crappy content in a few years, we are all gonna just throw it away, and this process happens continuously.
Nevermark 3y ago
Humans have that to, but the reason civilization doesn't go full crazy is our use of language and concepts are tied to doing objective things, which keeps things (mostly) grounded. Where it isn’t grounded, as in endless online conversations with like minded people (closed loop feedback) about informally abstracted (poorly specified constraints) and emotion invoking (high reinforcement) topics, people go batshit too. So the more AI models actually practice what they know in objective environments, the more likely that output->input feedback will inform introspection toward self-improvement, and less like an iterative calculation of the architecture’s resonant frequencies or eigenvalues.
- diogenes4 3y ago
  > use of language and concepts are tied to doing objective things Well, except for floating signifiers, which make up an increasing part of our vocabulary.
- RecycledEle 3y ago
  I agree with this. Humans and AIs both need grounding in reality or we go crazy. Unfortunately, our society does not understand this. If we did, we would value the opinions of auto mechanics far more than those of lawyers.
  - Nevermark 3y ago
    Yes! And value science (understanding it is simply our accumulated tools for finding harder truth, not a priesthood of “the truth”), more than pandering & populist politicians, and tribal media & online personalities.
- FrustratedMonky 3y ago
  >"people go batshit too." Thank you for this observation. LLM Learning from LLM -> Drift Human on Human echo chamber -> batshit crazy. I've often thought human echo chambers were their own phenomena. Something about the brain and tribalism from evolution. I never thought of it in terms of AI training data. As LLMs train on data produced by other LLMs, they will drift. And this drifting is the same phenomena as when humans get in an echo chamber. If each person hears what the others are saying, and spits it out in some form, and the others hear it, and also spit it back out in some form, this turns into a drifting in understanding just like an LLM. ("idea telephone game") Technically, it isn't just in echo chambers. It is all humans, at a lot of different scales, from small to large groups. Countries, and cultures are echo chambers at larger scales. Like how the concepts in philosophy, as they become more abstract, they kind of twist back on themselves, and become re-invented. And as they get more abstract get accused of just 'playing with words'. Just like an LLM can just 'play with words'? The difference is eventually humans have to relate to 'real objects'. So even if the word for 'apple' drifts over time and between groups, eventually you can still relate the words back to the real 'apple'. Humans are grounded in the reality of 'objects' in space. But. I tend to think this is temporary. As LLM's are linked to things like AlphaGo, and drone flight systems. They will also have to deal with real 'objects'. Maybe that will then lead to more grounded reasoning.
globalise83 3y ago
Once these intelligences can both read and write blog posts, product metadata on webshops, etc., could they carefully encode executable code that would allow them to "escape" the sandbox of their cloud environments, becoming fully-fledged Turing machines living in the wild?
choxi 3y ago
But there are a lot of selection mechanisms that filter out the bad generated content. For one, people will publish a fraction of the content they generate, and it most likely will go through some more editing and selection steps by real people. Then the Internet itself becomes like a decentralized labeling service, where there are various signals you could detect to identify different qualities of content. What ends up being crawled for the next training iteration is a societally processed version of the raw model output that might have contributed to generating it. It's kind of interesting to think we might all be contributing to the training data of AGI not just through generating content but also what we choose to share or upvote.
BeFlatXIII 3y ago
Perhaps we can then apply a healthy dose of classism to weed out people who spend all day communicating like they're talking to a forum full of AI chatbots and humans organically mimicking their errors.

rjblackman 3y ago

well we're already being trained by algorithms so I guess this is just an extension of what is already going on. perhaps the quality of human (internal) models will go down too? perhaps they already have?

haltist 3y ago

In system design there is something called resonant amplification and what you are describing is very similar. The biases of the model are amplified with each iteration and the end result is that the system converges onto the patterns recognizable and amplified by its architecture. If you know about impulse and frequency analysis then an AI system can be considered to be a signal processor that amplifies and attenuates certain frequencies in the input/impulse. Running an LLM in a loop always ends up with nonsense as the final output.

lazystar 3y ago
So if left unchecked, the thing we built in man's attempt to play god could result in gibberish? sounds kind of like the tower of babel; seems humankinds only defense would be creating a new language that the machines can't infiltrate
- ben_w 3y ago
  Another defence is to check them, which is why the thumbs up, thumbs down, and regenerate buttons are there on the ChatGPT user interface.
- haltist 3y ago
  You can try the experiment yourself. Take any open source LLM and then feed the output back into it in a loop and you'll see what I mean. Most LLMs diverge and lose coherence in less than 100 iterations.
  - lazystar 3y ago
    right, im thinking in terms of the world at large - what happens after 5 years of chatgpt, what will the internet and human communication be like in 2030?

HDThoreaun 3y ago

I think it's reasonable to say that this was actually the point of releasing LLMs publicly. The companies that created them wanted a moat and figured the data they had could be it if they poisoned anyones attempt to collect the same data in the future.

omneity 3y ago
Intriguing thought, but arguably people are intentionally using GPT to generate synthetic data for their domain specific model. So I'm kinda torn between AI giants poisoning the well with their models, or it just being unforeseen consequences (or one they willingly ignored to be first to market).
- quickthrower2 3y ago
  Generating data from ChatGPT even $1,000,000 in tokens worth can’t be on the same scale as what OpenAI is collecting from everyone.

js8 3y ago

> If your models are influencing the generation of data that they are later retrained on, it gets harder and harder to even maintain model performance. Why don't humans suffer from this problem, then? Humans have been creating art (and content) that imitates nature and society for thousands of years now, and yet we have little problem (the exception are possibly things like crop circles) to recognize what is a natural phenomenon and what is generated culturally. I think it's wrong to assume that this is a problem with intelligence in general, rather than just a feature (stupidity) of the current models.

ArekDymalski 3y ago
> Why don't humans suffer from this problem, then? Can we be sure that we don't? For example, depending on perspective our language is either evolving towards clarity or devolving towards primitivity, compared to XVIII/XIX century. The same could be argued about other aspects of society and culture.
dragonwriter 3y ago
> Why don't humans suffer from this problem, then? They do, but the problem is poorly stated. The problen isn't “If your models are influencing the generation of data that they are later retrained on”, its “If output produced by your models with inadequate filtering for quality/accuracy is dominating the data that they are later retrained on.” Humans definitely experience the same problem: we see it when societies become closed, inward-looking, and locked into models of information filtering that don't involve accuracy or fitness for purpose other than confirming an in-groups priors. There are some examoles in stagnation of large societies, but probably the clearest examples are cults that socially isolate their membership and degrade into increasingly bizarre behaviors terminating in mass suicide, murder/suicides, etc. LLM’s experience self-reinforcing degradation more acutely because they are individually less capable than humans and they are less diverse.
- js8 3y ago
  But humans can have a discussion about what is natural and what is not, and decide to mitigate it. The whole argument is based on the idea that the AI (either collectively or individually) can't even understand that, so the examples of cults etc. just do not apply. It's really the paperclip maximizer argument again. "Superintelligent AI" will supposedly change the world, but will lack enough common sense to understand a simple instruction with ethical constraint. Same here, the crappy AI content will somehow take over the world, despite the fact that most humans can actually mostly recognize original content from a generated one. The art that humans created doesn't replace nature, just like tons of kitsch don't pose a problem for new artists. In the same way, crappy AI content is not gonna replace good content, just augment it, and any decent AI (which has good enough common sense to be called "intelligent") will be able to tell the difference. Nobody's gonna care about crappy content in a few years, we are all gonna just throw it away, and this process happens continuously.

Nevermark 3y ago

Humans have that to, but the reason civilization doesn't go full crazy is our use of language and concepts are tied to doing objective things, which keeps things (mostly) grounded. Where it isn’t grounded, as in endless online conversations with like minded people (closed loop feedback) about informally abstracted (poorly specified constraints) and emotion invoking (high reinforcement) topics, people go batshit too. So the more AI models actually practice what they know in objective environments, the more likely that output->input feedback will inform introspection toward self-improvement, and less like an iterative calculation of the architecture’s resonant frequencies or eigenvalues.

diogenes4 3y ago
> use of language and concepts are tied to doing objective things Well, except for floating signifiers, which make up an increasing part of our vocabulary.
RecycledEle 3y ago
I agree with this. Humans and AIs both need grounding in reality or we go crazy. Unfortunately, our society does not understand this. If we did, we would value the opinions of auto mechanics far more than those of lawyers.
- Nevermark 3y ago
  Yes! And value science (understanding it is simply our accumulated tools for finding harder truth, not a priesthood of “the truth”), more than pandering & populist politicians, and tribal media & online personalities.
FrustratedMonky 3y ago
>"people go batshit too." Thank you for this observation. LLM Learning from LLM -> Drift Human on Human echo chamber -> batshit crazy. I've often thought human echo chambers were their own phenomena. Something about the brain and tribalism from evolution. I never thought of it in terms of AI training data. As LLMs train on data produced by other LLMs, they will drift. And this drifting is the same phenomena as when humans get in an echo chamber. If each person hears what the others are saying, and spits it out in some form, and the others hear it, and also spit it back out in some form, this turns into a drifting in understanding just like an LLM. ("idea telephone game") Technically, it isn't just in echo chambers. It is all humans, at a lot of different scales, from small to large groups. Countries, and cultures are echo chambers at larger scales. Like how the concepts in philosophy, as they become more abstract, they kind of twist back on themselves, and become re-invented. And as they get more abstract get accused of just 'playing with words'. Just like an LLM can just 'play with words'? The difference is eventually humans have to relate to 'real objects'. So even if the word for 'apple' drifts over time and between groups, eventually you can still relate the words back to the real 'apple'. Humans are grounded in the reality of 'objects' in space. But. I tend to think this is temporary. As LLM's are linked to things like AlphaGo, and drone flight systems. They will also have to deal with real 'objects'. Maybe that will then lead to more grounded reasoning.

globalise83 3y ago

Once these intelligences can both read and write blog posts, product metadata on webshops, etc., could they carefully encode executable code that would allow them to "escape" the sandbox of their cloud environments, becoming fully-fledged Turing machines living in the wild?

choxi 3y ago

But there are a lot of selection mechanisms that filter out the bad generated content. For one, people will publish a fraction of the content they generate, and it most likely will go through some more editing and selection steps by real people. Then the Internet itself becomes like a decentralized labeling service, where there are various signals you could detect to identify different qualities of content. What ends up being crawled for the next training iteration is a societally processed version of the raw model output that might have contributed to generating it. It's kind of interesting to think we might all be contributing to the training data of AGI not just through generating content but also what we choose to share or upvote.

BeFlatXIII 3y ago

Perhaps we can then apply a healthy dose of classism to weed out people who spend all day communicating like they're talking to a forum full of AI chatbots and humans organically mimicking their errors.