5 ms·
What is the bottleneck in building the "largest LLM" today for any interested party? AI expertise? Training data? GPUs?
by paxys 3y ago
What is the bottleneck in building the "largest LLM" today for any interested party? AI expertise? Training data? GPUs?
- xeromal 3y agoMoney?
- olalonde 3y agoGPUs and electricity. GPT-4 reportedly cost 100M$ to train.
- pixl97 3y agoI'd say GPU and people trained in knowing what to do while training. There is a lot of money in the world, as we see Microsoft and other companies handing it out. Power even, if you build in the right place can be cheap, see bitcoin mining.
- patapong 3y agoVery much agree... I think people underestimate how finicky these models are to train. So much so that one of the big announcement around GPT-4 was the fact that OpenAI found a way to make it smoother and more predictable. As evidence by this 114 page log of how engineers resolved problems that came up during the training of OPT-175, which lasted for several months. https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B_Logbook.pdf https://github.com/facebookresearch/metaseq/blob/main/projec...
- api 3y agoIt has to be more than just that. Bard is clearly inferior to GPT-4 and Google has no shortage of GPUs or electricity (or data).
- artdigital 3y ago> Bard is clearly inferior to GPT-4 Have you tried it recently? It got a lot better, and I like it more for research tasks. (GPT4 + Bing are alright)
- _lvbh 3y agoIt’s actually insanely good now, especially with images
- behnamoh 3y ago$100m is relatively cheap for something at the scale and sophistication of gpt-4. I think money is not the issue-it’s all the little things that had to be done right as Altman said in an interview.
- quickthrower2 3y agoH’s for the RLHF too? I guess that is easy to solve by throwing money
- Jeff_Brown 3y agoI believe it's it's humans (for RLHF and other things) and data. If it were just power and chips, the current front-runner would be a corporate behemoth or a state, not OpenAI. (They might become a corporate behemoth soon enough but they aren't so far.)
- WanderPanda 3y agoAt this point I'm pretty convinced that no new gigacorp will be emerge out of this generation of AI (LLM and vision models). OpenAI will have to continuously bring their A-Game to stay 6 months ahead of everyone else. There is durable moat that they could hide behind in sight as of now. Of course they would like to construct a regulatory moat but it doesn't seem like there is a clear majority for this. I'm pretty sure and happy that this will probably turn out as a large consumer windfall. Actually comes right in time to counteract the strong inflationary forces that come with de-globalization and declining efficiency due to increasingly basing the economy on gov spending.
- brianjking 3y agoI'd say expertise and compute. With enough compute I can certainly build the largest LLM. Do I have the expertise to build the BEST largest LLM? No.
- EGreg 3y agoEasy. Human feedback and training. The open secret is that just the AI “revolution” is built upon uncompensated or barely-compensated human labor. Not just everyone uploading their stuff to the internet for 30 years but also this: https://www.forbes.com/sites/kenrickcai/2023/04/11/how-alexandr-wang-turned-an-army-of-clickworkers-into-a-73-billion-ai-unicorn/ https://www.forbes.com/sites/kenrickcai/2023/04/11/how-alexa... Most people on HN have no idea about that https://futurism.com/the-byte/ai-magic-spy-on-you-experts-warn https://futurism.com/the-byte/ai-magic-spy-on-you-experts-wa... And especially OpenAI: https://time.com/6247678/openai-chatgpt-kenya-workers/ https://time.com/6247678/openai-chatgpt-kenya-workers/ That’s the dirty secret of why ChatGPT 4 is better. But they’ll tell you it has to do with chaining ChatGPT 3’s together, more fine tuning etc. They go to these poor countries and recruit people to work on training the AI, scan their eyeballs for crypto and whatever else. And then privatize the profits from all this. But it’s nothing new, capitalists have always exploited labor… and people make their choices. At least it isn’t ivory or blood diamonds https://www.reuters.com/technology/kenya-panel-urges-shutdown-worldcoins-crypto-project-within-country-2023-10-02/ https://www.reuters.com/technology/kenya-panel-urges-shutdow...