4 ms·
They are world-class engineers of course, but it's always been clear OpenAI's core-advantage was simply the access to massive amounts capital without much expec
by oersted 1y ago
They are world-class engineers of course, but it's always been clear OpenAI's core-advantage was simply the access to massive amounts capital without much expectation of a return on investment.
The ML methods they use have always been quite standard, they have been open about that. They just had the gall (or vision) to burn way more money than anyone else on scaling them up. The scale itself carries its own serious engineering challenges of course, but frankly they are not doing anything that any top-of-class CS post-grad couldn't replicate with enough budget.
It's certainly hard, but it's really not that special from an engineering standpoint. What is absolutely unprecedented is the social engineering and political acumen that allowed them to take such risks with so much money, walking that tightrope of mad ambition combined with good scientific discipline to make sure the money wouldn't be completely wasted, and the vision for what was required to make LLMs actually commercially useful (instruction tuning, "safety/censoring"...). But frankly, I really think most of the the engineers themselves are fungible, and I say this as an engineer myself.
- daquisu 1y agoThat is a common narrative but Google had LaMDA as an LLM with over 100B parameters before the ChatGPT release. There was even a Xoogler that claimed it was alive. From my POV Google could have released a good B2C LLM before OpenAI, but it would compete with their own Ads business.
- oersted 1y agoTrue, actually people forget that quite good LLMs existed 2-3 years before ChatGPT, from Google, Microsoft, Facebook… OpenAI itself open-sourced GPT-2 all the way back in 2019 and had a GPT-3 API service for years before ChatGPT. The breakthrough that ChatGPT brought was not technical, but the foresight to bet on laborious human-feedback fine-tuning to make LLMs somewhat controllable and practical. All those previous LLMs where mostly as “intelligent” as the GPT-3.5 that ChatGPT was built on, but they hallucinated so much, and it was so easy to manipulate them to be horribly racist and such. They remained niche tech demos until OpenAI trained them, not with new tech really, just the right vision and lots of expensive experimentation.