2 ms·
> On one hand if you require people to provide data they just won't. People will never provide the data because it's full of smoking guns. Tangential, but I wo
by skissane 2y ago
> On one hand if you require people to provide data they just won't. People will never provide the data because it's full of smoking guns.
Tangential, but I wonder how well an AI performs when trained on genuine human data, versus a synthetic data set of AI-generated texts.
If performance when trained on the synthetic data set is close to that when trained on the original human dataset – this could be a good way to "launder" the original training data and reduce any potential legal issues with it.
- dartos 2y agoI believe there are several papers which show that synthetic data isn’t as good as real data. It makes sense as any bias in the model generated synthetic data will just get magnified as models are continuously trained on that biased data.
- jart 2y agoThat's basically what companies like Mistral do. Many open source models are trained on OpenAI API request output. That's how a couple guys in Europe are able to build something nearly as good as GPT4 almost overnight and license it Apache 2.0. If you want the training data to be both human and open source, then there aren't many good options besides things like https://en.wikipedia.org/wiki/The_Pile_%28dataset%29 https://en.wikipedia.org/wiki/The_Pile_%28dataset%29 which has Hacker News, Wikipedia, the Enron Emails, GitHub, arXiv, etc.