3 ms·
How is it bullshit lol. GPT4 is genuinely very expensive to train and run inference.
by leesec 3y ago
How is it bullshit lol. GPT4 is genuinely very expensive to train and run inference.
- sva_ 3y agoCould you provide a source for that claim? Other than the very long context model.
- espadrine 3y agoWe can infer from publicly available information. BLOOM[0] was trained for four months on 384 A100 80GB GPUs, excluding architecture search. They specifically indicate (in the Huggingface page): > Estimated cost of training: Equivalent of $2-5M in cloud computing (including preliminary experiments) You can see from the training loss[1] that it was still learning at a good rate when it was stopped. The increased capabilities typically correlate well with the decrease in perplexity. That makes many believe that GPT-4 was trained for vastly more GPU-hours, as also suggested by OpenAI’s CEO[2]. Especially so considering it also included training on images, unlike BLOOM. [0]: https://arxiv.org/pdf/2211.05100.pdf https://arxiv.org/pdf/2211.05100.pdf [1]: https://huggingface.co/bigscience/tr11-176B-logs/tensorboard https://huggingface.co/bigscience/tr11-176B-logs/tensorboard [2]: https://twitter.com/sama/status/1620951010318626817 https://twitter.com/sama/status/1620951010318626817