3 ms·

As long as the synthetic data is good, how can you tell the difference between it and human generated data? This paper has one huge hole in it: it assumes that

As long as the synthetic data is good, how can you tell the difference between it and human generated data? This paper has one huge hole in it: it assumes that content on the internet is not moderated and that the training dataset will never evolve to take rating into consideration. On social media, the form of moderation is # of likes. Once detected, bots that output bad data will be banned and content deleted. The key issue I have with the paper is for good synthetic data it is impossible to tell it apart from human generated data.

aezart 3y ago
The goal of an LLM, before RLHF, is to accurately predict what token comes next. It cannot do better than that. The perfect outcome is text identical to the training set. Let's say your LLM can generate text with the same quality as the input 98% of the time, and the other 1% of the time, it's wrong. Each round of recursive training amplifies that error. 96% accuracy after the next round. 67% after 20 rounds. There's no way for it to get better without more real human input.
- lyu07282 3y ago
  The amount of training data vastly exceeds the size of the model, it does not just regurgitates what it found on the internet. This ignorant trope needs to die already.
- Solvency 3y ago
  "The perfect outcome is text identical to the training set." Huh? If the LLM was only ever spitting back identical content straight from the training set, that would be a symptom of extreme overfitting, which everyone universally agrees is a bad thing — not a perfect thing.
  - aezart 3y ago
    It's not the most useful outcome for the end user, but it's the perfect outcome from the perspective of the learning algorithm. All these things care about is minimizing their loss function, where loss is deviation from the training set.
- sebzim4500 3y ago
  Yeah but those errors will decrease the chance that the output will end up spreading across the internet, and therefore decrease the chance that it ends up in a future training set. You can see the whole process as a very high latency reinforcement learning system.
  - aezart 3y ago
    That's true but I think you're overestimating how careful people will be in their curation. There are tons of LLM-powered bots on twitter and reddit already, and no one is bothering to delete the output. Also, the curation process counts as "real human input" itself, so I don't think it contradicts my point.
- facu17y 3y ago
  "98% of the time, and the other 1% of the time" Is this a strange take on "off by 1" error?
jasfi 3y ago
I suspect that many people will publish substandard generated data. This will affect the integrity of training data in the long term, unless steps are taken.
sebzim4500 3y ago
>it assumes that content on the internet is not moderated and that the training dataset will never evolve to take rating into consideration Worse than this, since the training set already takes rating into consideration. For example the reddit dataset includes only articles with more than 3 upvotes.

aezart 3y ago

The goal of an LLM, before RLHF, is to accurately predict what token comes next. It cannot do better than that. The perfect outcome is text identical to the training set. Let's say your LLM can generate text with the same quality as the input 98% of the time, and the other 1% of the time, it's wrong. Each round of recursive training amplifies that error. 96% accuracy after the next round. 67% after 20 rounds. There's no way for it to get better without more real human input.

lyu07282 3y ago
The amount of training data vastly exceeds the size of the model, it does not just regurgitates what it found on the internet. This ignorant trope needs to die already.
Solvency 3y ago
"The perfect outcome is text identical to the training set." Huh? If the LLM was only ever spitting back identical content straight from the training set, that would be a symptom of extreme overfitting, which everyone universally agrees is a bad thing — not a perfect thing.
- aezart 3y ago
  It's not the most useful outcome for the end user, but it's the perfect outcome from the perspective of the learning algorithm. All these things care about is minimizing their loss function, where loss is deviation from the training set.
sebzim4500 3y ago
Yeah but those errors will decrease the chance that the output will end up spreading across the internet, and therefore decrease the chance that it ends up in a future training set. You can see the whole process as a very high latency reinforcement learning system.
- aezart 3y ago
  That's true but I think you're overestimating how careful people will be in their curation. There are tons of LLM-powered bots on twitter and reddit already, and no one is bothering to delete the output. Also, the curation process counts as "real human input" itself, so I don't think it contradicts my point.
facu17y 3y ago
"98% of the time, and the other 1% of the time" Is this a strange take on "off by 1" error?

jasfi 3y ago

I suspect that many people will publish substandard generated data. This will affect the integrity of training data in the long term, unless steps are taken.

sebzim4500 3y ago

>it assumes that content on the internet is not moderated and that the training dataset will never evolve to take rating into consideration Worse than this, since the training set already takes rating into consideration. For example the reddit dataset includes only articles with more than 3 upvotes.