4 ms·
GPUs and electricity. GPT-4 reportedly cost 100M$ to train.
by olalonde 3y ago
GPUs and electricity. GPT-4 reportedly cost 100M$ to train.
- pixl97 3y agoI'd say GPU and people trained in knowing what to do while training. There is a lot of money in the world, as we see Microsoft and other companies handing it out. Power even, if you build in the right place can be cheap, see bitcoin mining.
- patapong 3y agoVery much agree... I think people underestimate how finicky these models are to train. So much so that one of the big announcement around GPT-4 was the fact that OpenAI found a way to make it smoother and more predictable. As evidence by this 114 page log of how engineers resolved problems that came up during the training of OPT-175, which lasted for several months. https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B_Logbook.pdf https://github.com/facebookresearch/metaseq/blob/main/projec...
- api 3y agoIt has to be more than just that. Bard is clearly inferior to GPT-4 and Google has no shortage of GPUs or electricity (or data).
- artdigital 3y ago> Bard is clearly inferior to GPT-4 Have you tried it recently? It got a lot better, and I like it more for research tasks. (GPT4 + Bing are alright)
- _lvbh 3y agoIt’s actually insanely good now, especially with images
- behnamoh 3y ago$100m is relatively cheap for something at the scale and sophistication of gpt-4. I think money is not the issue-it’s all the little things that had to be done right as Altman said in an interview.