4 ms·
Yup. This is my gut to where LLMs will really explode. Let them augment data just a bit, train on improved data, augment more, train again - etcetc. If we take
by unshavedyak 3y ago
Yup. This is my gut to where LLMs will really explode. Let them augment data just a bit, train on improved data, augment more, train again - etcetc. If we take things slow i suspect in the long run it'll be really beneficial to multiple paradigms.
I know people say training bots on bot data is bad, but A: it's happening anyway, and B: it can't be worse than the actual garbage they get trained on in a lot of cases anyway.. can it?
- Filligree 3y agoPixart-alpha provides an example of C: The bot labels can be dramatically better than the human labels. Even though they used LLaVA, and LLaVA isn't all that good compared to gpt-4.
- gaogao 3y agoTraining on bot data can be bad when it's ungrounded and basically hallucinations on hallucinations. Having LLMs help curate something grounded is generally reasonable. Functionally, it's somewhat similar to how some training is using generated subtitles of videos for training video/text pairs; it's very feasible to also go and clean those up, even though it is bot data.
- ori_b 3y ago> Yup. This is my gut to where LLMs will really explode Yes, indeed. This is one place where LLMs can make it look like a bomb went off.
- prosqlinjector 3y agoWe can do polynomial regression of data sets that looks equally plausible, but it's not real data.
- foobarchu 3y ago> A: it's happening anyway This is never a valid defense for doing more of something.