BLOOM[0] is another 176B multilingual open-source model whose weights were published. However, running it locally requires 8 A6000 GPUs. The GPU cost alone is

BLOOM[0] is another 176B multilingual open-source model whose weights were published. However, running it locally requires 8 A6000 GPUs. The GPU cost alone is roughly $37K. Also, there is the concern of performance. The Chinchilla[1] paper indicates that most model sizes trained so far received too little token training (or were too big) to reach an optimal performance per energy expenditure. Furthermore, the InstructGPT[2] and more recently Flan[3] papers show that various finetuning techniques achieve significant gains, without which the result is a bit meh. The dark secret of GPT-3 is that the model available on the OpenAI website is heavily fine-tuned, even compared to the InstructGPT paper. [0]: https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom [1]: https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556 [2]: https://arxiv.org/abs/2203.02155 https://arxiv.org/abs/2203.02155 [3]: https://arxiv.org/pdf/2210.11416.pdf https://arxiv.org/pdf/2210.11416.pdf