3 ms·

> The internet becoming majority bot content basically guarantees this becomes a real problem for the next generation of models. Only if you assume that people

by eru 7mo ago

> The internet becoming majority bot content basically guarantees this becomes a real problem for the next generation of models. Only if you assume that people who train models are stupid.

troupo 7mo ago
> Only if you assume that people who train models are stupid Someone in the chain will be. Even the smartest people buy a lot of their training datasets. What happens when those get contaminated?
- eru 7mo ago
  You filter them, duh. And you negotiate a contract where the seller bears some of that risk (or you pay less, if they are not willing to make any such warranties.)
  - troupo 7mo ago
    > You filter them, duh. Filters are also not 100% infallible > And you negotiate a contract where the seller bears some of that risk So the training data will be polluted anyway, but "the seller will bear some risk"
    - eru 7mo ago
      > Filters are also not 100% infallible Why would they need to be?
      - troupo 7mo ago
        You want your training data to be clean or contaminated? A small number of samples can poison LLMs of any size https://www.anthropic.com/research/small-samples-poison https://www.anthropic.com/research/small-samples-poison --- start quote --- In a joint study with the UK AI Security Institute and the Alan Turing Institute, we found that as few as 250 malicious documents can produce a "backdoor" vulnerability in a large language model—regardless of model size or training data volume. Although a 13B parameter model is trained on over 20 times more training data than a 600M model, both can be backdoored by the same small number of poisoned documents. --- end quote ---
        eru 7mo ago
        I thought we were talking about model collapse? Poisoning is a completely different topic.
        troupo 7mo ago
        We were talking about this: https://news.ycombinator.com/item?id=47571715 https://news.ycombinator.com/item?id=47571715 and literally every single comment under this is talking about that. It's not a different topic. It's literally the topic of this branch of discussion.
        eru 7mo ago
        Huh? I was replying to https://news.ycombinator.com/item?id=47571319 https://news.ycombinator.com/item?id=47571319
        troupo 7mo ago
        --- start quote --- > The internet becoming majority bot content basically guarantees this becomes a real problem for the next generation of models. Only if you assume that people who train models are stupid. --- end quote --- And then literally everyone who commented on this, including me, was talking about issues with training data contamination. And you are the only one dismissing it as nothing important that can be easily fixed.
        eru 7mo ago
        Look at the whole comment, instead of selectively quoting: > The bigger concern is what happens when AI models start training on AI-generated content at scale. We're already seeing model collapse in research papers where output quality degrades when training data is contaminated with synthetic text. The internet becoming majority bot content basically guarantees this becomes a real problem for the next generation of models. Model collapse.
kdheiwns 7mo ago
Stupidity has nothing to do with it. AI articles and comments are now posted everywhere and presented as human. It's becoming harder and harder to determine whether text was written by humans or AI. Where are they supposed to find content to train AI on that isn't polluted with AI content that'll result in a feedback loop? It's like trying to get pure soil and water for growing that's not contaminated with microplastics/nanoplastics and PFAS. There was a time where it was possible. Not anymore. The filth is everywhere and impossible to filter. And it's simply not reasonable for AI companies to have human hands read through individual comments everywhere from beginning to end to build their training data. There isn't enough time in the universe to advance AI while doing that and also being accurate. Something will always slip through.
- Cthulhu_ 7mo ago
  If you can devise a tool that can detect AI generated content, you can use it to filter data. But the harsh truth is that "gold standard" training data is from before 2022 or whenever the cutoff was. And even that needs to be curated because before AI tools there was bot content filling up the internet. ...and even without bots, a lot of human authored content are low value, poorly written, etc. There are (probably) companies out there whose business is to create, curate and improve training sets.
  - kdheiwns 7mo ago
    And if there's a way to detect that content is AI generated, then there's demand to generate content that seems more human. And we're already at a point where most people have been tricked into believing AI content was real at some point and never even realized it. It'll only get worse. Probably the only real way to validate content is real is building a validation system into devices. Confirm when a photo is taken and send an ID to a server, then when photos are shared, its ID is compared to the image on the camera/phone manufacturer's server. For text, validate every little key press. And there are still ways to game these systems, but I would not be surprised if they're introduced to mitigate AI diffusing everywhere.
- eru 7mo ago
Planktonne 7mo ago
Not stupid, but I think it's fair to say "careless about/unaware of the wider impact of their work".
- eru 7mo ago
  What do you mean by wider impact? Model collapse would be the opposite of a wider impact: it's an immediate impact, and I'm fairly sure the people training these models have good incentives to avoid that. Eg by filtering data, by procuring better data, by applying techniques for making do with more limited data (we used to have a lot of those, and they are still known), or you can also adapt your training process to be less vulnerable to model collapse. Just because some researchers have shown that this happened for the models they tested, doesn't mean it has to be a universal thing.
PowerElectronix 7mo ago
They can be smart and also contrained by the need of training a new model while having already run out of human generated data.
Arainach 7mo ago
Plenty of shortsighted people have done things that are stupid in the long term for short term gains. It's the modus operandi of the US economy since 1981. I don't believe that the people who train models have a secret way of identifying and filtering out bot-generated content that no one else (email spam filters, search engines, etc.) have identified. I do believe that they feel their models need to have up-to-date information on a variety of topics that require regularly ingesting new data. So no, I don't think they have a good way to avoid their inputs rotting from their outputs.
- eru 7mo ago
  What's so special about 1981? And what do you mean about short term gains? If you are training a model, and you see model collapse, where's the short term gain? I don't get it. The incentive for the person who trains a model, even in the short run, is for them to avoid model collapse. > I don't believe that the people who train models have a secret way of identifying and filtering out bot-generated content that no one else (email spam filters, search engines, etc.) have identified. Huh, why would they need a secret filter? A filter is only one way thing you can try. You can also look into using different models, different training, making your approach more resistant to model collapse; training multi-modal models, using approaches to economise on training data; and thousands of other ideas I can't think of in thirty seconds. > So no, I don't think they have a good way to avoid their inputs rotting from their outputs. You lack imagination. People can be remarkably clever if its in their (short term!) interest to find solutions.
  - Arainach 7mo ago
    > What's so special about 1981? Two very significant things happened in 1981. After years of claiming the government couldn't help people, Ronald Reagan was elected and Republicans have been working hard to make that statement more true ever since. A big part of that was deregulation of the financial markets. That same year, Jim Welch became Chairman and CEO of General Electric. He juiced the stock prices by selling off the company's prize jewels, real estate, and future, and for a while (before the utter collapse of the company) artificially raised the stock price so high that executives around the country copied him, and and an entire industry of vultures like Mitt Romney started private equity firms to cannibalize healthy companies for their personal profit.

by eru 7mo ago

> The internet becoming majority bot content basically guarantees this becomes a real problem for the next generation of models. Only if you assume that people who train models are stupid.

troupo 7mo ago

> Only if you assume that people who train models are stupid Someone in the chain will be. Even the smartest people buy a lot of their training datasets. What happens when those get contaminated?

eru 7mo ago
You filter them, duh. And you negotiate a contract where the seller bears some of that risk (or you pay less, if they are not willing to make any such warranties.)
- troupo 7mo ago
  > You filter them, duh. Filters are also not 100% infallible > And you negotiate a contract where the seller bears some of that risk So the training data will be polluted anyway, but "the seller will bear some risk"
  - eru 7mo ago
    > Filters are also not 100% infallible Why would they need to be?
    - troupo 7mo ago
      You want your training data to be clean or contaminated? A small number of samples can poison LLMs of any size https://www.anthropic.com/research/small-samples-poison https://www.anthropic.com/research/small-samples-poison --- start quote --- In a joint study with the UK AI Security Institute and the Alan Turing Institute, we found that as few as 250 malicious documents can produce a "backdoor" vulnerability in a large language model—regardless of model size or training data volume. Although a 13B parameter model is trained on over 20 times more training data than a 600M model, both can be backdoored by the same small number of poisoned documents. --- end quote ---
      - eru 7mo ago
        I thought we were talking about model collapse? Poisoning is a completely different topic.
        troupo 7mo ago
        We were talking about this: https://news.ycombinator.com/item?id=47571715 https://news.ycombinator.com/item?id=47571715 and literally every single comment under this is talking about that. It's not a different topic. It's literally the topic of this branch of discussion.
        eru 7mo ago
        Huh? I was replying to https://news.ycombinator.com/item?id=47571319 https://news.ycombinator.com/item?id=47571319
        troupo 7mo ago
        --- start quote --- > The internet becoming majority bot content basically guarantees this becomes a real problem for the next generation of models. Only if you assume that people who train models are stupid. --- end quote --- And then literally everyone who commented on this, including me, was talking about issues with training data contamination. And you are the only one dismissing it as nothing important that can be easily fixed.
        eru 7mo ago
        Look at the whole comment, instead of selectively quoting: > The bigger concern is what happens when AI models start training on AI-generated content at scale. We're already seeing model collapse in research papers where output quality degrades when training data is contaminated with synthetic text. The internet becoming majority bot content basically guarantees this becomes a real problem for the next generation of models. Model collapse.

kdheiwns 7mo ago

Stupidity has nothing to do with it. AI articles and comments are now posted everywhere and presented as human. It's becoming harder and harder to determine whether text was written by humans or AI. Where are they supposed to find content to train AI on that isn't polluted with AI content that'll result in a feedback loop? It's like trying to get pure soil and water for growing that's not contaminated with microplastics/nanoplastics and PFAS. There was a time where it was possible. Not anymore. The filth is everywhere and impossible to filter. And it's simply not reasonable for AI companies to have human hands read through individual comments everywhere from beginning to end to build their training data. There isn't enough time in the universe to advance AI while doing that and also being accurate. Something will always slip through.

Cthulhu_ 7mo ago
If you can devise a tool that can detect AI generated content, you can use it to filter data. But the harsh truth is that "gold standard" training data is from before 2022 or whenever the cutoff was. And even that needs to be curated because before AI tools there was bot content filling up the internet. ...and even without bots, a lot of human authored content are low value, poorly written, etc. There are (probably) companies out there whose business is to create, curate and improve training sets.
- kdheiwns 7mo ago
  And if there's a way to detect that content is AI generated, then there's demand to generate content that seems more human. And we're already at a point where most people have been tricked into believing AI content was real at some point and never even realized it. It'll only get worse. Probably the only real way to validate content is real is building a validation system into devices. Confirm when a photo is taken and send an ID to a server, then when photos are shared, its ID is compared to the image on the camera/phone manufacturer's server. For text, validate every little key press. And there are still ways to game these systems, but I would not be surprised if they're introduced to mitigate AI diffusing everywhere.
eru 7mo ago

Planktonne 7mo ago

Not stupid, but I think it's fair to say "careless about/unaware of the wider impact of their work".

eru 7mo ago
What do you mean by wider impact? Model collapse would be the opposite of a wider impact: it's an immediate impact, and I'm fairly sure the people training these models have good incentives to avoid that. Eg by filtering data, by procuring better data, by applying techniques for making do with more limited data (we used to have a lot of those, and they are still known), or you can also adapt your training process to be less vulnerable to model collapse. Just because some researchers have shown that this happened for the models they tested, doesn't mean it has to be a universal thing.

PowerElectronix 7mo ago

They can be smart and also contrained by the need of training a new model while having already run out of human generated data.

Arainach 7mo ago

Plenty of shortsighted people have done things that are stupid in the long term for short term gains. It's the modus operandi of the US economy since 1981. I don't believe that the people who train models have a secret way of identifying and filtering out bot-generated content that no one else (email spam filters, search engines, etc.) have identified. I do believe that they feel their models need to have up-to-date information on a variety of topics that require regularly ingesting new data. So no, I don't think they have a good way to avoid their inputs rotting from their outputs.

eru 7mo ago
What's so special about 1981? And what do you mean about short term gains? If you are training a model, and you see model collapse, where's the short term gain? I don't get it. The incentive for the person who trains a model, even in the short run, is for them to avoid model collapse. > I don't believe that the people who train models have a secret way of identifying and filtering out bot-generated content that no one else (email spam filters, search engines, etc.) have identified. Huh, why would they need a secret filter? A filter is only one way thing you can try. You can also look into using different models, different training, making your approach more resistant to model collapse; training multi-modal models, using approaches to economise on training data; and thousands of other ideas I can't think of in thirty seconds. > So no, I don't think they have a good way to avoid their inputs rotting from their outputs. You lack imagination. People can be remarkably clever if its in their (short term!) interest to find solutions.
- Arainach 7mo ago
  > What's so special about 1981? Two very significant things happened in 1981. After years of claiming the government couldn't help people, Ronald Reagan was elected and Republicans have been working hard to make that statement more true ever since. A big part of that was deregulation of the financial markets. That same year, Jim Welch became Chairman and CEO of General Electric. He juiced the stock prices by selling off the company's prize jewels, real estate, and future, and for a while (before the utter collapse of the company) artificially raised the stock price so high that executives around the country copied him, and and an entire industry of vultures like Mitt Romney started private equity firms to cannibalize healthy companies for their personal profit.