6 ms·

Model collapse doomers are such an odd bunch. It's as if they think that the humans training the models have no agency over the data they use, or how that data

Model collapse doomers are such an odd bunch. It's as if they think that the humans training the models have no agency over the data they use, or how that data is processed/weighted/etc.

surfingdino 3y ago
They are only humans and will face the same issues that humans trying to control the quality of the search engine results did in the past and those policing social media networks do today. They too have admitted they cannot keep up and started using automated tools a long time ago.
- CuriouslyC 3y ago
  The difference is that there are perverse incentives keeping garbage in search and social. With AI models, the incentive is accuracy.
  - surfingdino 3y ago
    It will change. Sooner or later AI models will be altered to produce output that embeds advertising/paid content/paid messaging.
    - CuriouslyC 3y ago
      There are open source models, so I don't think people are going to be able to get away with that.
      - surfingdino 3y ago
        Money will choose the model. They need lots of expensive compute power so there will have to be a business model and ad-supported is the way to go on the internet.
harborsong 3y ago
Maybe it's the whole processing and weighting thing that gives people cause for concern.
- CuriouslyC 3y ago
  Well then welcome to being a human, because the gatekeepers of knowledge have been doing this since the beginning of time. What do you think the priestly class's original job was?
andy99 3y ago
I feel like the take there is that they're destroying the internet from the perspective of people who want to make money off of add revenue. Cynically, it's like they want the internet to be kind of bad - as in distraction from ads, begging for money and content arranged to support that, but not so bad it puts people off entirely or drives the incremental value of putting out "content" to zero because of automation. For people writing on the internet for motivations other than ad-driven monetization, and those who read their stuff, there is basically no issue.
- grugagag 3y ago
  > For people writing on the internet for motivations other than ad-driven monetization, and those who read their stuff, there is basically no issue. I doubt these users will not experience a drop in quality as most authorship will roll out LLM laden content which will add more noise to the signal.
- beej71 3y ago
  > For people writing on the internet for motivations other than ad-driven monetization, and those who read their stuff, there is basically no issue. This is the key, I think. But it's small--The Small Web. I think we'll be better off, though. Have a day job and produce quality content in your spare time, is my motto. A few billion people producing content in their spare time is still more than we can ever hope to consume.
- sumeno 3y ago
  I feel like it's the exact opposite, LLMs are great for ad revenue because you can create a ton of garbage and monetize it in a fraction of the time it takes actual content creators to make their content. Meanwhile, the actual good content is drowned in a sea of garbage and much harder to find
  - stavros 3y ago
    You can only monetize garbage for a short time before people catch on and the entire model is destroyed (the good content along with the garbage).
Ensorceled 3y ago
Your "hahahaha those doomers" comments is either intentionally cruel or Pollyannish. Your "people have agency" "criticism" doesn't address the key points of the article: 1. The damage in the form of the destruction of news sources and replacement of writers with AI has happened and continues to happen. Writers are already jobless. News sources are already dead. But sure, these jobless people with no hope for their industry are "doomers". The article doesn't mention artists, but they are also getting hammered. 2. The worst players already have agency over their data choices, they are using that to build silos and destroy their competitors who don't have that ability. 3. Some social places have already become unusable. Twitter for instance. Threads may be able to push back on this, but they are already a dangerous silo.
- skybrian 3y ago
  Writing has always been a tough gig and maybe it’s gotten worse, but it doesn’t seem quite that bad? Aren’t some newer writers making money from subscriptions? From a reader’s point of view, it seems like there’s plenty to read.
  - andy99 3y ago
    And there are plenty of successful writers too. The internet has just enabled unsuccessful ones to complain about it at scale.
- mrweasel 3y ago
  News is a really weird place to start with AI, because it obviously doesn't work. The AI cannot go out into the world and do investigating journalism. It still can't access tons of documents that are still hidden away in archives and libraries around the world and it cannot do an sit though endless hours of court room session (even if that might be a fairly good place to start). The only "news" AIs can report on are press releases, that and rehashed articles written by journalists. Due to the volume of half-arsed news the current AIs can generate your average reader might not notice that some is missing, though I questions that. People are starting to notice that their newspapers doesn't actually have news anymore (and that's just do to cost optimization and competition from ad supported online news). I fear that we're entering a world where some of us pay for news written by real journalists, while the masses consume garbage "news" which is more tailored to them clicking ads, rather that learning about the world.
Kamq 3y ago
That seems like it would mean you're restricted to a dataset from before 2022ish or you have to use data that's significantly more expensive to clean. That seems like it might make the training process untenable.
- coffeebeqn 3y ago
  Is it really true though? If they scrape HN today I don’t think their sampling would be that “polluted” compared to 2022. If you were to teach a LLM to play chess today - wouldn’t you use fully AI generated data because that’s by far the best data ?
- johntb86 3y ago
  Isn't all of recorded human history up until 2022 enough to train a pretty smart ai? It'll miss out on future trends, but some targeted training on trusted news sources should be enough. Eventually languages will change enough that the old training data will seem dated, but that's going to take a while to happen.
  - Kamq 3y ago
    Yeah, but basically all of that comes from post 2000. Information production seems to be on an exponential curve, so it won't be that long before you're training on a minority of all data.
nicklecompte 3y ago
Taken literally this comment suggests that OpenAI et al actually want racist 4chan rants in ChatGPT's training data, or illegal child pornography in DALL-E's training data. Obviously that's not the case: the problem is that it's impossible to effectively audit the amount of data required to make generative AI work. If Big Tech can't even catch pornography or explicit uses of the n-word, there's no way they'll be able to filter subtle LLM / art generator hallucinations.
- jsheard 3y ago
  Going one level deeper, even AI-generated training data which doesn't contain hallucinations or other obvious errors could still negatively affect the performance of a new model. GenAI models have a tendency to repeat the same cliches, in sentence structure or the composition of images or whatever else, so even if their output is "fine" in the sense that it's factually correct and everyone has the right number of fingers, a model trained on it will still internalize those cliches and make them even more likely to show up again. That's even harder to audit out because it would only become apparent in aggregate when you realise your new model has learned to interpret "fantasy artwork" as "fantasy artwork in the generic Midjourney RLHF style" since it was trained on so many examples of that.
  - wruza 3y ago
    That's really just a feature of the default system prompt, although ubiquitous. You can character-prompt local LLMs as you like, but no one seems to do that.
    - CuriouslyC 3y ago
      ChatGPT doesn't take character prompts very well, that's why people don't know. Claude on the other hand is a champ with characters.
- simonw 3y ago
  Has it been demonstrated that there is illegal CSAM in the DALL-E training?
mrspuratic 3y ago
The barriers/costs for pushing AI generated noise are so low, I agree there's the potential for it to overwhelm human written content. In a sense it effectively happened to email, again with lots of $doom, but the timescale turned out to be a decade long frog boil. Nobody, or a close approximation to, runs their own email server any more. In an ironic twist the gatekeepers of the semi-walled gardens/islands of email that arose are now harvesting it for training content. I do so despise those weekly machine generated synopses telling me how I am doing.
marginalia_nu 3y ago
Adding to this, the Internet is extremely noisy and polluted with all manner of crap, this has been the status quo for over a decade. The notion that you could just grab data off the internet completely unvetted, and then train a useful model off that data seems very unrealistic. To get anything useful out of real world crawl data, you need a lot of filtering and processing to select the good bits. This is true if you're training an AI or building a search engine.

Model collapse doomers are such an odd bunch. It's as if they think that the humans training the models have no agency over the data they use, or how that data is processed/weighted/etc.

surfingdino 3y ago

They are only humans and will face the same issues that humans trying to control the quality of the search engine results did in the past and those policing social media networks do today. They too have admitted they cannot keep up and started using automated tools a long time ago.

CuriouslyC 3y ago
The difference is that there are perverse incentives keeping garbage in search and social. With AI models, the incentive is accuracy.
- surfingdino 3y ago
  It will change. Sooner or later AI models will be altered to produce output that embeds advertising/paid content/paid messaging.
  - CuriouslyC 3y ago
    There are open source models, so I don't think people are going to be able to get away with that.
    - surfingdino 3y ago
      Money will choose the model. They need lots of expensive compute power so there will have to be a business model and ad-supported is the way to go on the internet.

harborsong 3y ago

Maybe it's the whole processing and weighting thing that gives people cause for concern.

CuriouslyC 3y ago
Well then welcome to being a human, because the gatekeepers of knowledge have been doing this since the beginning of time. What do you think the priestly class's original job was?

andy99 3y ago

I feel like the take there is that they're destroying the internet from the perspective of people who want to make money off of add revenue. Cynically, it's like they want the internet to be kind of bad - as in distraction from ads, begging for money and content arranged to support that, but not so bad it puts people off entirely or drives the incremental value of putting out "content" to zero because of automation. For people writing on the internet for motivations other than ad-driven monetization, and those who read their stuff, there is basically no issue.

grugagag 3y ago
> For people writing on the internet for motivations other than ad-driven monetization, and those who read their stuff, there is basically no issue. I doubt these users will not experience a drop in quality as most authorship will roll out LLM laden content which will add more noise to the signal.
beej71 3y ago
> For people writing on the internet for motivations other than ad-driven monetization, and those who read their stuff, there is basically no issue. This is the key, I think. But it's small--The Small Web. I think we'll be better off, though. Have a day job and produce quality content in your spare time, is my motto. A few billion people producing content in their spare time is still more than we can ever hope to consume.
sumeno 3y ago
I feel like it's the exact opposite, LLMs are great for ad revenue because you can create a ton of garbage and monetize it in a fraction of the time it takes actual content creators to make their content. Meanwhile, the actual good content is drowned in a sea of garbage and much harder to find
- stavros 3y ago
  You can only monetize garbage for a short time before people catch on and the entire model is destroyed (the good content along with the garbage).

Ensorceled 3y ago

Your "hahahaha those doomers" comments is either intentionally cruel or Pollyannish. Your "people have agency" "criticism" doesn't address the key points of the article: 1. The damage in the form of the destruction of news sources and replacement of writers with AI has happened and continues to happen. Writers are already jobless. News sources are already dead. But sure, these jobless people with no hope for their industry are "doomers". The article doesn't mention artists, but they are also getting hammered. 2. The worst players already have agency over their data choices, they are using that to build silos and destroy their competitors who don't have that ability. 3. Some social places have already become unusable. Twitter for instance. Threads may be able to push back on this, but they are already a dangerous silo.

skybrian 3y ago
Writing has always been a tough gig and maybe it’s gotten worse, but it doesn’t seem quite that bad? Aren’t some newer writers making money from subscriptions? From a reader’s point of view, it seems like there’s plenty to read.
- andy99 3y ago
  And there are plenty of successful writers too. The internet has just enabled unsuccessful ones to complain about it at scale.
mrweasel 3y ago
News is a really weird place to start with AI, because it obviously doesn't work. The AI cannot go out into the world and do investigating journalism. It still can't access tons of documents that are still hidden away in archives and libraries around the world and it cannot do an sit though endless hours of court room session (even if that might be a fairly good place to start). The only "news" AIs can report on are press releases, that and rehashed articles written by journalists. Due to the volume of half-arsed news the current AIs can generate your average reader might not notice that some is missing, though I questions that. People are starting to notice that their newspapers doesn't actually have news anymore (and that's just do to cost optimization and competition from ad supported online news). I fear that we're entering a world where some of us pay for news written by real journalists, while the masses consume garbage "news" which is more tailored to them clicking ads, rather that learning about the world.

Kamq 3y ago

That seems like it would mean you're restricted to a dataset from before 2022ish or you have to use data that's significantly more expensive to clean. That seems like it might make the training process untenable.

coffeebeqn 3y ago
Is it really true though? If they scrape HN today I don’t think their sampling would be that “polluted” compared to 2022. If you were to teach a LLM to play chess today - wouldn’t you use fully AI generated data because that’s by far the best data ?
johntb86 3y ago
Isn't all of recorded human history up until 2022 enough to train a pretty smart ai? It'll miss out on future trends, but some targeted training on trusted news sources should be enough. Eventually languages will change enough that the old training data will seem dated, but that's going to take a while to happen.
- Kamq 3y ago
  Yeah, but basically all of that comes from post 2000. Information production seems to be on an exponential curve, so it won't be that long before you're training on a minority of all data.

nicklecompte 3y ago

Taken literally this comment suggests that OpenAI et al actually want racist 4chan rants in ChatGPT's training data, or illegal child pornography in DALL-E's training data. Obviously that's not the case: the problem is that it's impossible to effectively audit the amount of data required to make generative AI work. If Big Tech can't even catch pornography or explicit uses of the n-word, there's no way they'll be able to filter subtle LLM / art generator hallucinations.

jsheard 3y ago
Going one level deeper, even AI-generated training data which doesn't contain hallucinations or other obvious errors could still negatively affect the performance of a new model. GenAI models have a tendency to repeat the same cliches, in sentence structure or the composition of images or whatever else, so even if their output is "fine" in the sense that it's factually correct and everyone has the right number of fingers, a model trained on it will still internalize those cliches and make them even more likely to show up again. That's even harder to audit out because it would only become apparent in aggregate when you realise your new model has learned to interpret "fantasy artwork" as "fantasy artwork in the generic Midjourney RLHF style" since it was trained on so many examples of that.
- wruza 3y ago
  That's really just a feature of the default system prompt, although ubiquitous. You can character-prompt local LLMs as you like, but no one seems to do that.
  - CuriouslyC 3y ago
    ChatGPT doesn't take character prompts very well, that's why people don't know. Claude on the other hand is a champ with characters.
simonw 3y ago
Has it been demonstrated that there is illegal CSAM in the DALL-E training?

mrspuratic 3y ago

The barriers/costs for pushing AI generated noise are so low, I agree there's the potential for it to overwhelm human written content. In a sense it effectively happened to email, again with lots of $doom, but the timescale turned out to be a decade long frog boil. Nobody, or a close approximation to, runs their own email server any more. In an ironic twist the gatekeepers of the semi-walled gardens/islands of email that arose are now harvesting it for training content. I do so despise those weekly machine generated synopses telling me how I am doing.

marginalia_nu 3y ago

Adding to this, the Internet is extremely noisy and polluted with all manner of crap, this has been the status quo for over a decade. The notion that you could just grab data off the internet completely unvetted, and then train a useful model off that data seems very unrealistic. To get anything useful out of real world crawl data, you need a lot of filtering and processing to select the good bits. This is true if you're training an AI or building a search engine.