5 ms·

I broadly agree with that. Repeated training with self-generated data is the technological equivalent of incest and can lead to nothing good.

rzzzt 3y ago
Putting washing machine parts in the laundry, irradiating pre-war steel, etc.
soks86 3y ago
Only let baby see its own scribbles and scat. Future world leader advisor. Ahem, I hope not.
chmod775 3y ago
Humans train on self-generated data all the time, but compared to today's LLMs, humans have superior reasoning ability, which enables them to separate good from bad data. Humans can confirm many things for themselves if need be and our social hierarchies crystallize authority figures which perform curation for others. On top of that humans have a lifetime of experience with the real world, while LLMs rely only on reports of what the real world is like. All in all LLMs being more gullible than your average 6 year old is really not helping. Humans will have to (continue to) curate the data we train LLMs on.
- jmye 3y ago
  This view of people seems… idealistic, given the number of people who uncritically look to Facebook or Reddit for information (and then happily parrot whatever memes they find) and the number of LLMs that do the same. I think people are substantially worse at figuring out what is “good” data, and I think a huge number of people are/are starting to nominate LLMs as those “authority figures”. And, given that, I think there is a low ceiling for the creators of these tools to clear to make them appealing to users.
  - gorjusborg 3y ago
    > a huge number of people are starting to nominate LLMs as those "authority figures" When LLMs started to go mainstream I said I was not afraid of AI, but was afraid of (unwise) people with AI. This phenomenon is what I foresaw. The worst thinkers are the ones most hungry to outsource their decision making. The fact that LLM output requires good judgement by the recipient to filter the wheat from the chaff, and those with poor judgement skills are most likely to lean on decision tools makes for a potentially volatile situation.
hermitcrab 3y ago
Inbreeding certainly didn't go well for the Habsbergs.
- hermitcrab 3y ago
  Didn't think that would be a controversial topic!
lewhoo 3y ago
I think I'll restrain from buying into this until much more testing is conducted. There is information out there suggesting training on synthetic data is at least on par but cheaper.
- jimbokun 3y ago
  On par at doing what?
  - lewhoo 3y ago
    https://huggingface.co/blog/synthetic-data-save-costs https://huggingface.co/blog/synthetic-data-save-costs At doing whatever the model is suppose to do.
sebzim4500 3y ago
Training on self generated data is not necessarily such a problem, see how successful alphazero/muzero is when it is only trained on self play. The key is that you need some kind of external indicator that tells you which generated examples are good and which are bad. In the case of alphazero you get that by simulating games and seeing who wins, in the case of LLMs you will be only taking the generations that are 'successful', e.g. which HN posts get upvoted.
- vanviegen 3y ago
  Let's indeed hope that it will still be humans doing most of the voting on HN.
__loam 3y ago
Are we sampling reality, or are we sampling the photograph?
twobitshifter 3y ago
With a mixture of experts approach it’s not incest, we intentionally split the training data to add diversity. Incest is bad because the errors compound on themselves, but this isn’t a problem when the AI parents don’t share the same DNA (problem space.) I see it this way, maybe there’s no new ‘knowledge’ but the AI can apply our collective knowledge better than we can. Within that set of knowledge there are surely discoveries never yet realized based on by the fusion of ideas. Pre-AI we rely on individuals Einstein, Bohr, and Oppenheimer and their associations and studies of each others’ work. With AI, we can fuse the corpus of scientific discoveries into a single entity that we each can communicate with. Maybe the AI lacks the spark of creativity needed to make discovery, but put today’s Einstein in front of it and what would he ask? How much boost would it give him? Einstein said - “I have no special talent. I am only passionately curious.”

rzzzt 3y ago

Putting washing machine parts in the laundry, irradiating pre-war steel, etc.

soks86 3y ago

Only let baby see its own scribbles and scat. Future world leader advisor. Ahem, I hope not.

chmod775 3y ago

Humans train on self-generated data all the time, but compared to today's LLMs, humans have superior reasoning ability, which enables them to separate good from bad data. Humans can confirm many things for themselves if need be and our social hierarchies crystallize authority figures which perform curation for others. On top of that humans have a lifetime of experience with the real world, while LLMs rely only on reports of what the real world is like. All in all LLMs being more gullible than your average 6 year old is really not helping. Humans will have to (continue to) curate the data we train LLMs on.

jmye 3y ago
This view of people seems… idealistic, given the number of people who uncritically look to Facebook or Reddit for information (and then happily parrot whatever memes they find) and the number of LLMs that do the same. I think people are substantially worse at figuring out what is “good” data, and I think a huge number of people are/are starting to nominate LLMs as those “authority figures”. And, given that, I think there is a low ceiling for the creators of these tools to clear to make them appealing to users.
- gorjusborg 3y ago
  > a huge number of people are starting to nominate LLMs as those "authority figures" When LLMs started to go mainstream I said I was not afraid of AI, but was afraid of (unwise) people with AI. This phenomenon is what I foresaw. The worst thinkers are the ones most hungry to outsource their decision making. The fact that LLM output requires good judgement by the recipient to filter the wheat from the chaff, and those with poor judgement skills are most likely to lean on decision tools makes for a potentially volatile situation.

hermitcrab 3y ago

Inbreeding certainly didn't go well for the Habsbergs.

hermitcrab 3y ago
Didn't think that would be a controversial topic!

lewhoo 3y ago

I think I'll restrain from buying into this until much more testing is conducted. There is information out there suggesting training on synthetic data is at least on par but cheaper.

jimbokun 3y ago
On par at doing what?
- lewhoo 3y ago
  https://huggingface.co/blog/synthetic-data-save-costs https://huggingface.co/blog/synthetic-data-save-costs At doing whatever the model is suppose to do.

sebzim4500 3y ago

Training on self generated data is not necessarily such a problem, see how successful alphazero/muzero is when it is only trained on self play. The key is that you need some kind of external indicator that tells you which generated examples are good and which are bad. In the case of alphazero you get that by simulating games and seeing who wins, in the case of LLMs you will be only taking the generations that are 'successful', e.g. which HN posts get upvoted.

vanviegen 3y ago
Let's indeed hope that it will still be humans doing most of the voting on HN.

__loam 3y ago

Are we sampling reality, or are we sampling the photograph?

twobitshifter 3y ago

With a mixture of experts approach it’s not incest, we intentionally split the training data to add diversity. Incest is bad because the errors compound on themselves, but this isn’t a problem when the AI parents don’t share the same DNA (problem space.) I see it this way, maybe there’s no new ‘knowledge’ but the AI can apply our collective knowledge better than we can. Within that set of knowledge there are surely discoveries never yet realized based on by the fusion of ideas. Pre-AI we rely on individuals Einstein, Bohr, and Oppenheimer and their associations and studies of each others’ work. With AI, we can fuse the corpus of scientific discoveries into a single entity that we each can communicate with. Maybe the AI lacks the spark of creativity needed to make discovery, but put today’s Einstein in front of it and what would he ask? How much boost would it give him? Einstein said - “I have no special talent. I am only passionately curious.”