3 ms·
Re (3) I almost think it implies the opposite: if everyone has access to synthetic data from large foundation models, it will be those that have their own propr
by version_five 3y ago
Re (3) I almost think it implies the opposite: if everyone has access to synthetic data from large foundation models, it will be those that have their own proprietary data that can stand out. This is already true imo. There are lots of competing models with similar performance on imagenet, but when you get into niche computer vision, data is the main differentiator.
- cs702 3y agoIf, as you write, "everyone has access to synthetic data from large foundation models," then I'd agree. But I expect we're going to see a transition in the opposite direction, with more corporations keeping newer models and data proprietary to maximize competitive advantage. I suspect this kind of thinking factored into OpenAI's decision not to release any details about GPT-4.
- version_five 3y agoFair enough. I think that many would like to have their own proprietary models, but for now, these "web scale" datasets really only come from scraping the internet. Companies can do their own cleaning and whatnot, but the data distribution still largely reflects what's on the internet because that's all there is. To your point though, three are definitely some proprietary pockets, like the Shutterstock/openai thing. Or the various RL layers ala chatgpt. But a lot of the value there is in the labeling, and I expect we'll see more competing versions that are open (we already are)
- cs702 3y agoI hope you're right :-)