4 ms·
> what would the effects be on the competitiveness of the US AI industry To me, this leads to another question: is it possible to get a GPT-4 quality model us
by cj 3y ago
> what would the effects be on the competitiveness of the US AI industry
To me, this leads to another question: is it possible to get a GPT-4 quality model using only training data from major content providers?
In other words, what if OpenAI didn’t crawl the entire internet and only used data from Facebook, Twitter, Wikipedia, GitHub, NYT, Stackoverflow, and [list 100 more well known high quality data sources here]
If it’s possible to get to GPT-4 model quality with training data from a couple hundred centralized sources, then maybe it won’t have any impact at all (other than OpenAI needing to pay other companies to use their data for training data)
Or, if the long tail “every website on the internet” really is the secret to training LLM models, then that would be much more problematic.
- microtonal 3y agoEven then, e.g. citizens in the EU have the right to be forgotten. So if they use user-submitted data (and in some cases e.g. newspaper articles about specific people), users can request the data to be removed, which will be... an interesting challenge?
- petesergeant 3y agoIt’s possible to get a good image model this way: https://www.gettyimages.ae/ai/generation/about https://www.gettyimages.ae/ai/generation/about
- throwaway4aday 3y agoThere are quite a few papers that indicate that the important factor is the quality of the data, regardless of where it is sourced from. You still need a lot of it so throwing out large chunks of your training set does work against you whether you're doing that for legal reasons or quality reasons. The solution appears to be synthetic, high quality data generated by another model but that makes it a bit of a chicken and egg problem where you need a high quality model like GPT-4 to produce high quality data reliably. I think there are methods for getting around this using less capable models by producing a lot more examples and then using another model to judge the quality of each example and only selecting the best ones. I also suspect you could get pretty far by permuting a smaller set of high quality data to produce a lot more examples that have the same meaning but differ in how they are written.