3 ms·
What is the relative difficult of each required part in training GPT3/ChatGPT? E.g. where is OpenAI's moat, and how wide is it? Is it in model design (now publ
by albntomat0 4y ago
What is the relative difficult of each required part in training GPT3/ChatGPT? E.g. where is OpenAI's moat, and how wide is it?
Is it in model design (now published), data set, engineering needed to train it, cost of hardware needed to train it, cost of HLRF on top of GPT3 to make ChatGPT?
- mccorrinall 4y agoIt’s easier to train on all github repos if you own github. There is no real alternative to codex. Stuff like GPT3 is trying to be recreated, but even the eleuther AI guys only collected like 800GB training data, which is much less than what OpenAI has (iirc around 45TB). And apparently their data is very high quality. EleutherAI is pretty much one of the few big model open source competitors with GPT-Neox etc. Plus openai has great branding.
- albntomat0 4y agoInteresting, thanks for sharing! I wonder how LaMDA compares performance wise to ChatGPT. I definitely understand why training on Github is an advantage, but I'd expect Google to also be great at getting a good dataset, across the range of things they'd be interested in.
- tootie 4y agoAlso how the hell do they make money? They began life as a nonprofit and AFAIK have no commerical product.
- albntomat0 4y agoPoking around their website, OpenAI sells API access to DALL-E, and a range of language models. Github Copilot also uses an OpenAI ML model. Their valuation comes from expectations of the value of future iterations of those.
- 93po 4y agoThey've received several billions in investor money and I doubt they're at risk of going through all that.
- throwaway290 4y agoWillingness to enter legally grey areas