3 ms·

My biggest worry is the idea of generating data as training data. We're obviously already unwittingly doing this, but once someone decides to augment low-volume

My biggest worry is the idea of generating data as training data. We're obviously already unwittingly doing this, but once someone decides to augment low-volume segments of the dataset with generative input, we're going to start getting some really crappy feedback loops.

nopeNopeNooope 2y ago
[dead]
bilsbie 2y ago
Do you ever have novel ideas while walking or in the shower? Well, you’re learning from data you generate.
- nkozyra 2y ago
  > Well, you’re learning from data you generate. Sure. I'm producing human output from human input in a generally unconstrained, limitless way. This is producing approximated human output from approximated human input. That second level of abstraction will be constrained, ultimately, by the limits of input.
- deleted 2y ago
  [deleted]
- sottol 2y ago
  I don't necessarily think it is the equivalent. Maybe it's more akin to a high-schooler reading a single book and then being asked to write several new books that hold equal weight in teaching the next generation of children. These new text books could be great at simplifying the subject matter and making the material accessible or they may just never have fully understood the materials and are misleading. Now imagine that over and over again, imo it's pretty likely to introduce inaccuracies if just taking a naiive approach.
- breck 2y ago
  Well put. Also, ever read your old journals? You are training on generated data.
  - Filligree 2y ago
    Ever dream? It’s the same thing. Of course we still need real world data, but it seems like generated data should also play a role. Humans don’t weight dreams equally with reality, however, and that’s a distinction I feel is missing here.
- a_wild_dandan 2y ago
  This is correct. To give more synthetic data examples: 1. Generate inverted problems which are easier to produce than solve. For instance, create an integration math exercise by differentiating a hairy function and reversing the steps. 2. Create (simulated) environmental data. 3. Use adversarial model competition, e.g. self-playing chess or training an artificial image generator/detector model pair. There's evidently a commonplace myth that information quality starts pristine and exclusively gets degraded by systems thereafter. That's just easily, demonstrably false in myriad ways. That's why it's absurd to conclude that LLM output being in internet training data will cause model collapse. Information-rich synthetic data can be created without humans, and it works. (Check out Phi, for instance.)
- vineyardlabs 2y ago
  https://arxiv.org/pdf/2305.17493 https://arxiv.org/pdf/2305.17493 There's some cursory indication that in the long tail, training LLMs on LLM-generated data causes model collapse. Kind of like how if you photocopy a photocopy too many times the document becomes unreadable. This isn't really surprising though. Neural networks at large are a form of lossy compression. You can't do lossy compression on artifacts recovered from lossy compression too many times. The losses stack.
- xboxnolifes 2y ago
  Even hear an idea from another person? You just trained one model with another.
MeImCounting 2y ago
We are already doing this. In fact for the next generation of frontier models the primary electricity cost is running inference to generate training data for these models. As Llama 3 has shown us scale of training data is more important than size of model.
IncreasePosts 2y ago
Shouldn't there be enough known-good training content that we can use it to determine if a document is worth including in a training set?
- devmor 2y ago
  There's not enough human workers to validate that at scale. If you want ML to do it... well that's a bit of a catch-22. How would an ML algorithm know if data is good enough to be trained on unless it has already been trained on that data?
  - HarHarVeryFunny 2y ago
    The way humans do it is via curiosity/boredom/surprise. If we can't predict some thing well (i.e. we're "surprised" by it) then that both acts as learning trigger to predict better next time, as well as retains our interest to explore it. Eventually AGI's will need to learn by experimentation as we do, but in the meantime ability to well predict a potential training sample could be used to decide whether to add it to training set or not. At the moment it seems the main emphasis is on a combination of multi-modality (esp. video) and synthetic data where one generation of LLM generates tailored training samples for the next generation. I guess this synthetic data allows a more selective acquisition of knowledge than just adding surprising texts found in the wild.
  - IncreasePosts 2y ago
    You could use data where you know it was not AI generated, like the library of Congress catalog from prior to 2015. Or highly cited research papers, things of that nature.
  - drdeca 2y ago
    Get a collection of data which is small enough to have humans annotate as being either low quality or high quality. Train a model to predict this annotation. Then on a larger disjoint collection of data, use this model to estimate whether the data points would be considered low quality or high quality, and use this to filter it. This seems doable, and, I think something like it is already done?

nopeNopeNooope 2y ago

[dead]

bilsbie 2y ago

Do you ever have novel ideas while walking or in the shower? Well, you’re learning from data you generate.

nkozyra 2y ago
> Well, you’re learning from data you generate. Sure. I'm producing human output from human input in a generally unconstrained, limitless way. This is producing approximated human output from approximated human input. That second level of abstraction will be constrained, ultimately, by the limits of input.
deleted 2y ago
[deleted]
sottol 2y ago
I don't necessarily think it is the equivalent. Maybe it's more akin to a high-schooler reading a single book and then being asked to write several new books that hold equal weight in teaching the next generation of children. These new text books could be great at simplifying the subject matter and making the material accessible or they may just never have fully understood the materials and are misleading. Now imagine that over and over again, imo it's pretty likely to introduce inaccuracies if just taking a naiive approach.
breck 2y ago
Well put. Also, ever read your old journals? You are training on generated data.
- Filligree 2y ago
  Ever dream? It’s the same thing. Of course we still need real world data, but it seems like generated data should also play a role. Humans don’t weight dreams equally with reality, however, and that’s a distinction I feel is missing here.
a_wild_dandan 2y ago
This is correct. To give more synthetic data examples: 1. Generate inverted problems which are easier to produce than solve. For instance, create an integration math exercise by differentiating a hairy function and reversing the steps. 2. Create (simulated) environmental data. 3. Use adversarial model competition, e.g. self-playing chess or training an artificial image generator/detector model pair. There's evidently a commonplace myth that information quality starts pristine and exclusively gets degraded by systems thereafter. That's just easily, demonstrably false in myriad ways. That's why it's absurd to conclude that LLM output being in internet training data will cause model collapse. Information-rich synthetic data can be created without humans, and it works. (Check out Phi, for instance.)
vineyardlabs 2y ago
https://arxiv.org/pdf/2305.17493 https://arxiv.org/pdf/2305.17493 There's some cursory indication that in the long tail, training LLMs on LLM-generated data causes model collapse. Kind of like how if you photocopy a photocopy too many times the document becomes unreadable. This isn't really surprising though. Neural networks at large are a form of lossy compression. You can't do lossy compression on artifacts recovered from lossy compression too many times. The losses stack.
xboxnolifes 2y ago
Even hear an idea from another person? You just trained one model with another.

MeImCounting 2y ago

We are already doing this. In fact for the next generation of frontier models the primary electricity cost is running inference to generate training data for these models. As Llama 3 has shown us scale of training data is more important than size of model.

IncreasePosts 2y ago

Shouldn't there be enough known-good training content that we can use it to determine if a document is worth including in a training set?

devmor 2y ago
There's not enough human workers to validate that at scale. If you want ML to do it... well that's a bit of a catch-22. How would an ML algorithm know if data is good enough to be trained on unless it has already been trained on that data?
- HarHarVeryFunny 2y ago
  The way humans do it is via curiosity/boredom/surprise. If we can't predict some thing well (i.e. we're "surprised" by it) then that both acts as learning trigger to predict better next time, as well as retains our interest to explore it. Eventually AGI's will need to learn by experimentation as we do, but in the meantime ability to well predict a potential training sample could be used to decide whether to add it to training set or not. At the moment it seems the main emphasis is on a combination of multi-modality (esp. video) and synthetic data where one generation of LLM generates tailored training samples for the next generation. I guess this synthetic data allows a more selective acquisition of knowledge than just adding surprising texts found in the wild.
- IncreasePosts 2y ago
  You could use data where you know it was not AI generated, like the library of Congress catalog from prior to 2015. Or highly cited research papers, things of that nature.
- drdeca 2y ago
  Get a collection of data which is small enough to have humans annotate as being either low quality or high quality. Train a model to predict this annotation. Then on a larger disjoint collection of data, use this model to estimate whether the data points would be considered low quality or high quality, and use this to filter it. This seems doable, and, I think something like it is already done?