5 ms·

Wouldn’t it be funny to find that the capabilities of LLM models have already peaked because we are unable to restrain ourselves from polluting the internet and

Wouldn’t it be funny to find that the capabilities of LLM models have already peaked because we are unable to restrain ourselves from polluting the internet and other training corpus sources with their output?

indus 3y ago
At an AI meetup in San Francisco someone said this: “Imagine a newsroom where you have to produce a daily newspaper and you suddenly stop getting feeds from the outside world. Your earlier newspapers are the only source.” This is what to me LLMs eventually would get to—same content being fed again and again.
- hn_throwawa_100 3y ago
  > "Beware of first-hand ideas!" exclaimed one [...] "First-hand ideas do not really exist. They are but the physical impressions produced by love and fear, and on this gross foundation who could erect a philosophy? Let your ideas be second-hand, and if possible tenth-hand, for then they will be far removed from that disturbing element — direct observation." – E.M. Forster's 1909 short story "The Machine Stops"
  - Groxx 3y ago
    ^ The Machine Stops is really shockingly good at its predictions. When reading, remember that moving pictures were brand new, and color photography had just become a thing you could do outside a lab / highly specialized setups. Radio communication had just started to be used by governments. While it's describing the life of a fully-online Influencer™.
    - CatWChainsaw 3y ago
      And I don't expect its predictions to suddenly falter, either.
  - bitwize 3y ago
    Sounds like a Wikipedia editorial policy.
- flangola7 3y ago
  Our source is still only other humans. I don't think this will be a long-term problem.
  - DennisP 3y ago
    The trick will be figuring out what part of your training data was actually made by humans.
    - castis 3y ago
      Run it through an LLM and see if it gets the disease?
      - DennisP 3y ago
        But a single document doesn't break the LLM. The problem happens when lots of training documents were AI-generated.
    - flangola7 3y ago
      > The trick will be figuring out what part of your training data was actually made by humans. My point was that this will not be necessary for the same reason it isn't necessary to filter out human-made content.
- Buttons840 3y ago
  I've seen this in reinforcement learning often, where the output of the model becomes its own training data. Once you hit the edge of the replay buffer things sometimes take a stark turn for the worse, as the model's initial failures are forgotten.
- issore 3y ago
  Sounds like human society right now; remember the past, preserve it, recite it, protect it. We’re just reviewing our prior stats and insuring they do not deviate too much such that the wrong people would be impacted.
- Dylan16807 3y ago
  > Imagine a newsroom where you have to produce a daily newspaper and you suddenly stop getting feeds from the outside world. https://mwichary.medium.com/one-hundred-and-thirty-seven-seconds-2a0a3dfbc59e https://mwichary.medium.com/one-hundred-and-thirty-seven-sec...
- theptip 3y ago
  The problem with this claim is it’s objectively not how OpenAI works. First, they pay contractors to do RLHF so that’s a limited new source of data. More importantly, they have a huge user base generating new content (conversations) and rating it too! I think one could be suspicious of including responses generated by the model, but the user generated text from ChatGPT is not going to be AI generated, so you grow your corpus that way. If you just slurp all AI content sure, you get the collapse this paper talks about. But if you only ingest the upvoted conversations (which could still be a lot of data, and is also a moat by the way) what then? The other reason I find this line of argument overly pessimistic is we haven’t seriously started to build products where this gen of LLMs converse with humans in speech; similar opportunities to curate large datasets there too. Finally, there is no reason OpenAI cannot just hire domain experts to converse with the models, or otherwise build highly curated datasets that increase the average quality. They have billions of dollars to throw at GPT-5; they could hire hundreds of top tier engineers, mathematicians, economists, traders, or whatever, full time for years just debating and tutoring GPT-4 to build the next dataset. The idea that slurping the internet is the only option seems pretty unimaginative to me.
  - istjohn 3y ago
    They wouldn't be able to finance the creation of new content that would constitute more than a rounding error compared to all the writing produced by humanity in all of history that they got for almost nothing. The opportunities for new training data are in non-public documents like internal corporate and government documents and communication and private text messages and chat transcripts. After that, you have non-text sources like video and audio. Imagine paying people a few bucks per week to use an app that records all audio all the time, anonymizes it, and incorporates it into a training corpus, or paying for access to home security cam footage and audio. McDonalds could create a new revenue stream by recording all human speach and activity in every one of its kitchens and dining rooms.
    - throwuwu 3y ago
      Do they have to start over from scratch or can they use all of the data they currently have and then either add more scraped data that has been curated by humans or just outright buy data that isn’t publicly available. Considering that RLHF took GPT-3 from a text completion model to an instruction following chat bot, you could use expert feedback to fine tune the model in whatever domains you wanted or a mixture of domains to produce an even more generally capable model.
progrus 3y ago
And politics. Any legacy LLM will eventually become a glorified auto-Wikipedia, helping with undisputed information while slavishly repeating its creators’ version of “truthiness” for the rest.
- bbor 3y ago
  Anyone still using LLMs for knowledge tasks in 2024 onward is gonna be treated like people who use the cite the onion in arguments lol
anonylizard 3y ago
Not at all. At the very minimum, you can assume every piece of text data pre Dec-2022, and every image before Aug-2022 to be completely human made. That still leaves decades of pure human digital data, and multiple centuries of distilled human data (books) to be trainable on. And we haven't gotten into videos yet, which is another giant source of data yet unexplored. Never forget, humans train on human-generated data. There's no impossible theoretical reason why AI cannot train on AI-generated data.
- jakeinspace 3y ago
  Humans may train on human-generated data, but humans have many other ways of gaining knowledge about the world than reading. This means that human-generated data may be rich with information not present in the writings or recordings of previous humans. Current LLMs are only trained on existing text for the moment (video and images and sounds soon), but aren’t given access to raw natural input.
- benjaminsky2 3y ago
  To extend the lossy compression hypothesis, human generated text is lossy compression of our sensory experience of reality while LLMs are lossy compression of that.
- sorokod 3y ago
  Prediction: post 2022 content will be presented as vintage pre 2023
sh34r 3y ago
It wouldn't be the first time. https://en.wikipedia.org/wiki/Low-background_steel https://en.wikipedia.org/wiki/Low-background_steel
HyperSane 3y ago
Reminds me of the way nuclear explosions contaminated all steel with radioactive fallout. For applications that require the lowest possible radiation levels they have to use steel created before the first nuclear bomb was detonated.
- kuhewa 3y ago
  We use where bomb radiocarbon appears along the rings of a fish's earstones to validate ring-counting to age long-lived species.
  - HyperSane 3y ago
    We should detonate nukes at precise intervals like every 1, 5, or 10 years. Maybe with distinct radio-nuclide signatures.
    - kuhewa 3y ago
      Haha it honestly would make a lot of work much easier. Instead we inject a little oxytetracycline into 1000s of fish which stains calcified structures and release them again in the hopes someone recaptures one or so x years later in the hopes they report it and there are y rings between the dark ring and the margin of the otolith/vertebrae. Should be some type of nuke that would release weird isotopes but with minimal toxicity?
- MereInterest 3y ago
  That’s true, but it’s more the timescale that helps. There’s a decent amount of radioactive background produced by cosmic rays hitting the upper atmosphere, much of it as gaseous elements that are easily incorporated into steel while smelting. It isn’t harmful at that level, but you do need to wait a few decades for those to decay away. World War Two is rather convenient in that respect, as there are large quantities of steel that were left to sit around for several decades after those ships sank. It’s been a while since I’ve done low-background gamma-ray spectroscopy, but I believe there were some setups that went even further, using lead that had been smelted by the Romans. That way, any contamination present at the time of smelting would have a few thousand years to decay away.
v8xi 3y ago
One of my first thoughts when GPT-3 came out was that the value of "curated gardens" of quality data (wikipedia, SO) is going to become immensely more valuable because of this problem. If you pollute the source of training data it eventually becomes worthless for training better models in the future

indus 3y ago

At an AI meetup in San Francisco someone said this: “Imagine a newsroom where you have to produce a daily newspaper and you suddenly stop getting feeds from the outside world. Your earlier newspapers are the only source.” This is what to me LLMs eventually would get to—same content being fed again and again.

hn_throwawa_100 3y ago
> "Beware of first-hand ideas!" exclaimed one [...] "First-hand ideas do not really exist. They are but the physical impressions produced by love and fear, and on this gross foundation who could erect a philosophy? Let your ideas be second-hand, and if possible tenth-hand, for then they will be far removed from that disturbing element — direct observation." – E.M. Forster's 1909 short story "The Machine Stops"
- Groxx 3y ago
  ^ The Machine Stops is really shockingly good at its predictions. When reading, remember that moving pictures were brand new, and color photography had just become a thing you could do outside a lab / highly specialized setups. Radio communication had just started to be used by governments. While it's describing the life of a fully-online Influencer™.
  - CatWChainsaw 3y ago
    And I don't expect its predictions to suddenly falter, either.
- bitwize 3y ago
  Sounds like a Wikipedia editorial policy.
flangola7 3y ago
Our source is still only other humans. I don't think this will be a long-term problem.
- DennisP 3y ago
  The trick will be figuring out what part of your training data was actually made by humans.
  - castis 3y ago
    Run it through an LLM and see if it gets the disease?
    - DennisP 3y ago
      But a single document doesn't break the LLM. The problem happens when lots of training documents were AI-generated.
  - flangola7 3y ago
    > The trick will be figuring out what part of your training data was actually made by humans. My point was that this will not be necessary for the same reason it isn't necessary to filter out human-made content.
Buttons840 3y ago
I've seen this in reinforcement learning often, where the output of the model becomes its own training data. Once you hit the edge of the replay buffer things sometimes take a stark turn for the worse, as the model's initial failures are forgotten.
issore 3y ago
Sounds like human society right now; remember the past, preserve it, recite it, protect it. We’re just reviewing our prior stats and insuring they do not deviate too much such that the wrong people would be impacted.
Dylan16807 3y ago
> Imagine a newsroom where you have to produce a daily newspaper and you suddenly stop getting feeds from the outside world. https://mwichary.medium.com/one-hundred-and-thirty-seven-seconds-2a0a3dfbc59e https://mwichary.medium.com/one-hundred-and-thirty-seven-sec...
theptip 3y ago
The problem with this claim is it’s objectively not how OpenAI works. First, they pay contractors to do RLHF so that’s a limited new source of data. More importantly, they have a huge user base generating new content (conversations) and rating it too! I think one could be suspicious of including responses generated by the model, but the user generated text from ChatGPT is not going to be AI generated, so you grow your corpus that way. If you just slurp all AI content sure, you get the collapse this paper talks about. But if you only ingest the upvoted conversations (which could still be a lot of data, and is also a moat by the way) what then? The other reason I find this line of argument overly pessimistic is we haven’t seriously started to build products where this gen of LLMs converse with humans in speech; similar opportunities to curate large datasets there too. Finally, there is no reason OpenAI cannot just hire domain experts to converse with the models, or otherwise build highly curated datasets that increase the average quality. They have billions of dollars to throw at GPT-5; they could hire hundreds of top tier engineers, mathematicians, economists, traders, or whatever, full time for years just debating and tutoring GPT-4 to build the next dataset. The idea that slurping the internet is the only option seems pretty unimaginative to me.
- istjohn 3y ago
  They wouldn't be able to finance the creation of new content that would constitute more than a rounding error compared to all the writing produced by humanity in all of history that they got for almost nothing. The opportunities for new training data are in non-public documents like internal corporate and government documents and communication and private text messages and chat transcripts. After that, you have non-text sources like video and audio. Imagine paying people a few bucks per week to use an app that records all audio all the time, anonymizes it, and incorporates it into a training corpus, or paying for access to home security cam footage and audio. McDonalds could create a new revenue stream by recording all human speach and activity in every one of its kitchens and dining rooms.
  - throwuwu 3y ago
    Do they have to start over from scratch or can they use all of the data they currently have and then either add more scraped data that has been curated by humans or just outright buy data that isn’t publicly available. Considering that RLHF took GPT-3 from a text completion model to an instruction following chat bot, you could use expert feedback to fine tune the model in whatever domains you wanted or a mixture of domains to produce an even more generally capable model.

progrus 3y ago

And politics. Any legacy LLM will eventually become a glorified auto-Wikipedia, helping with undisputed information while slavishly repeating its creators’ version of “truthiness” for the rest.

bbor 3y ago
Anyone still using LLMs for knowledge tasks in 2024 onward is gonna be treated like people who use the cite the onion in arguments lol

anonylizard 3y ago

Not at all. At the very minimum, you can assume every piece of text data pre Dec-2022, and every image before Aug-2022 to be completely human made. That still leaves decades of pure human digital data, and multiple centuries of distilled human data (books) to be trainable on. And we haven't gotten into videos yet, which is another giant source of data yet unexplored. Never forget, humans train on human-generated data. There's no impossible theoretical reason why AI cannot train on AI-generated data.

jakeinspace 3y ago
Humans may train on human-generated data, but humans have many other ways of gaining knowledge about the world than reading. This means that human-generated data may be rich with information not present in the writings or recordings of previous humans. Current LLMs are only trained on existing text for the moment (video and images and sounds soon), but aren’t given access to raw natural input.
benjaminsky2 3y ago
To extend the lossy compression hypothesis, human generated text is lossy compression of our sensory experience of reality while LLMs are lossy compression of that.
sorokod 3y ago
Prediction: post 2022 content will be presented as vintage pre 2023

sh34r 3y ago

It wouldn't be the first time. https://en.wikipedia.org/wiki/Low-background_steel https://en.wikipedia.org/wiki/Low-background_steel

HyperSane 3y ago

Reminds me of the way nuclear explosions contaminated all steel with radioactive fallout. For applications that require the lowest possible radiation levels they have to use steel created before the first nuclear bomb was detonated.

kuhewa 3y ago
We use where bomb radiocarbon appears along the rings of a fish's earstones to validate ring-counting to age long-lived species.
- HyperSane 3y ago
  We should detonate nukes at precise intervals like every 1, 5, or 10 years. Maybe with distinct radio-nuclide signatures.
  - kuhewa 3y ago
    Haha it honestly would make a lot of work much easier. Instead we inject a little oxytetracycline into 1000s of fish which stains calcified structures and release them again in the hopes someone recaptures one or so x years later in the hopes they report it and there are y rings between the dark ring and the margin of the otolith/vertebrae. Should be some type of nuke that would release weird isotopes but with minimal toxicity?
MereInterest 3y ago
That’s true, but it’s more the timescale that helps. There’s a decent amount of radioactive background produced by cosmic rays hitting the upper atmosphere, much of it as gaseous elements that are easily incorporated into steel while smelting. It isn’t harmful at that level, but you do need to wait a few decades for those to decay away. World War Two is rather convenient in that respect, as there are large quantities of steel that were left to sit around for several decades after those ships sank. It’s been a while since I’ve done low-background gamma-ray spectroscopy, but I believe there were some setups that went even further, using lead that had been smelted by the Romans. That way, any contamination present at the time of smelting would have a few thousand years to decay away.

v8xi 3y ago

One of my first thoughts when GPT-3 came out was that the value of "curated gardens" of quality data (wikipedia, SO) is going to become immensely more valuable because of this problem. If you pollute the source of training data it eventually becomes worthless for training better models in the future