3 ms·
> they totally would have had it been technically feasible TinyLlama[1] has been made by an individual on their own last year, training a 1.1B model on 3T toke
by littlestymaar 2y ago
> they totally would have had it been technically feasible
TinyLlama[1] has been made by an individual on their own last year, training a 1.1B model on 3T tokens with just 16 A100-40G GPUs in 90 days. It was definitely within reach of any funded org in 2019.
In 2022 (IIRC), Google released the Chinchilla paper about the compute-optimal amount of data to train a given model, for a 1B model, the value was determined to be 20B tokens, which again is 3 orders of magnitude below the current state of the art for the same class of model.
Until very recently (the first llama paper IIRC, and people noticing that the 7B model showed no sign of saturation during its already very long training) the ML community vastly underestimated the amount of training data that was needed to make a LLM perform at its potential.
[1]: https://github.com/jzhang38/TinyLlama https://github.com/jzhang38/TinyLlama