3 ms·
On one hand if you require people to provide data they just won't. People will never provide the data because it's full of smoking guns. On the other hand if t
by wmf 2y ago
On one hand if you require people to provide data they just won't. People will never provide the data because it's full of smoking guns.
On the other hand if the data isn't open you should probably use the term open weights not open source. They're so close.
- samj 2y agoYes, and Open Source started out with a much smaller set of software that has since grown exponentially thanks to the meaningful Open Source Definition. We risk giving AI the same opportunity to grow in an open direction, and by our own hand. Massive own goal.
- bjornsing 2y ago> Yes, and Open Source started out with a much smaller set of software that has since grown exponentially thanks to the meaningful Open Source Definition. I thought it was thanks to a lot of software developers’ uncompensated labor. Silly me.
- mistrial9 2y ago> ... require people to provide data they just won't. People will never provide the data ... the word "people" is so striking here... teams and companies, corporations and governments.. how can the cast of characters be so completely missed. An extreme opposite to a far previous era where one person could only be their group member. Vocabulary has to evolve in deliberations.
- skissane 2y ago> On one hand if you require people to provide data they just won't. People will never provide the data because it's full of smoking guns. Tangential, but I wonder how well an AI performs when trained on genuine human data, versus a synthetic data set of AI-generated texts. If performance when trained on the synthetic data set is close to that when trained on the original human dataset – this could be a good way to "launder" the original training data and reduce any potential legal issues with it.
- dartos 2y agoI believe there are several papers which show that synthetic data isn’t as good as real data. It makes sense as any bias in the model generated synthetic data will just get magnified as models are continuously trained on that biased data.
- jart 2y agoThat's basically what companies like Mistral do. Many open source models are trained on OpenAI API request output. That's how a couple guys in Europe are able to build something nearly as good as GPT4 almost overnight and license it Apache 2.0. If you want the training data to be both human and open source, then there aren't many good options besides things like https://en.wikipedia.org/wiki/The_Pile_%28dataset%29 https://en.wikipedia.org/wiki/The_Pile_%28dataset%29 which has Hacker News, Wikipedia, the Enron Emails, GitHub, arXiv, etc.