4 ms·
My immediate thoughts: 1. The human brain must be doing something analogous to this, continuously, to learn from a relatively small numbers of samples. 2. Goi
by cs702 4y ago
My immediate thoughts:
1. The human brain must be doing something analogous to this, continuously, to learn from a relatively small numbers of samples.
2. Going forward, generating high-quality synthetic data looks likely to become a standard practice for training models.
3. Whoever has the largest, highest-quality synthetically generated datasets will have the best-performing AI models.
- version_five 4y agoRe (3) I almost think it implies the opposite: if everyone has access to synthetic data from large foundation models, it will be those that have their own proprietary data that can stand out. This is already true imo. There are lots of competing models with similar performance on imagenet, but when you get into niche computer vision, data is the main differentiator.
- cs702 4y agoIf, as you write, "everyone has access to synthetic data from large foundation models," then I'd agree. But I expect we're going to see a transition in the opposite direction, with more corporations keeping newer models and data proprietary to maximize competitive advantage. I suspect this kind of thinking factored into OpenAI's decision not to release any details about GPT-4.
- version_five 4y agoFair enough. I think that many would like to have their own proprietary models, but for now, these "web scale" datasets really only come from scraping the internet. Companies can do their own cleaning and whatnot, but the data distribution still largely reflects what's on the internet because that's all there is. To your point though, three are definitely some proprietary pockets, like the Shutterstock/openai thing. Or the various RL layers ala chatgpt. But a lot of the value there is in the labeling, and I expect we'll see more competing versions that are open (we already are)
- cs702 4y agoI hope you're right :-)
- fnordpiglet 4y agoYou mean like dreaming?
- cubefox 4y agoTraining on synthetic data has the problem that the models may drift away from the original distribution because errors could get amplified.