3 ms·
If, as you write, "everyone has access to synthetic data from large foundation models," then I'd agree. But I expect we're going to see a transition in the oppo
by cs702 4y ago
If, as you write, "everyone has access to synthetic data from large foundation models," then I'd agree. But I expect we're going to see a transition in the opposite direction, with more corporations keeping newer models and data proprietary to maximize competitive advantage. I suspect this kind of thinking factored into OpenAI's decision not to release any details about GPT-4.
- version_five 4y agoFair enough. I think that many would like to have their own proprietary models, but for now, these "web scale" datasets really only come from scraping the internet. Companies can do their own cleaning and whatnot, but the data distribution still largely reflects what's on the internet because that's all there is. To your point though, three are definitely some proprietary pockets, like the Shutterstock/openai thing. Or the various RL layers ala chatgpt. But a lot of the value there is in the labeling, and I expect we'll see more competing versions that are open (we already are)
- cs702 4y agoI hope you're right :-)