> I get the sense that LLMs are trained on a very large data set all at once, but that it is difficult to incrementally update them with new data. In 2020, Ope

> I get the sense that LLMs are trained on a very large data set all at once, but that it is difficult to incrementally update them with new data. In 2020, OpenAI wrote that training the 175B parameter GPT-3 took 3e23 fl-operations [1]. A $1600 NVIDIA GeForce RTX 4090 [2] can do 80 tflops (80 × 1e12 operations per second). So 1 GPU would need 120 years to train the model. With $2.4 million, you could buy 1500 GPUs, and train GPT-3 from scratch in 1 month. Partial continuous re-training should be way cheaper, if there is a way. But even training again from scratch every month seems feasible for a big company. Disclaimer: Not an AI expert. [1] https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165, table D.1. [2] https://www.techpowerup.com/gpu-specs/geforce-rtx-4090.c3889 https://www.techpowerup.com/gpu-specs/geforce-rtx-4090.c3889