3 ms·

Could you provide a source for that claim? Other than the very long context model.

• espadrine 4y ago
  We can infer from publicly available information. BLOOM[0] was trained for four months on 384 A100 80GB GPUs, excluding architecture search. They specifically indicate (in the Huggingface page): > Estimated cost of training: Equivalent of $2-5M in cloud computing (including preliminary experiments) You can see from the training loss[1] that it was still learning at a good rate when it was stopped. The increased capabilities typically correlate well with the decrease in perplexity. That makes many believe that GPT-4 was trained for vastly more GPU-hours, as also suggested by OpenAI’s CEO[2]. Especially so considering it also included training on images, unlike BLOOM. [0]: https://arxiv.org/pdf/2211.05100.pdf https://arxiv.org/pdf/2211.05100.pdf [1]: https://huggingface.co/bigscience/tr11-176B-logs/tensorboard https://huggingface.co/bigscience/tr11-176B-logs/tensorboard [2]: https://twitter.com/sama/status/1620951010318626817 https://twitter.com/sama/status/1620951010318626817