Oh yes, according to [1], training GPT-2 1.5B cost $50k in 2019 (reproduced in 2024 for $672!). [1]: https://www.reddit.com/r/mlscaling/comments/1d3a793/andrej

Oh yes, according to [1], training GPT-2 1.5B cost $50k in 2019 (reproduced in 2024 for $672!). [1]: https://www.reddit.com/r/mlscaling/comments/1d3a793/andrej_karpathy_gpt2_124m_in_llmc_in_90_minutes/ https://www.reddit.com/r/mlscaling/comments/1d3a793/andrej_k...