3 ms·
> ~$1 million: Cost to train a 13 billion parameter model on 1.4 trillion tokens Llama paper mentioned 135,168 A100 hours for training 13 billion model on 1 t
by YetAnotherNick 3y ago
> ~$1 million: Cost to train a 13 billion parameter model on 1.4 trillion tokens
Llama paper mentioned 135,168 A100 hours for training 13 billion model on 1 trillion tokens, which means ~$150k for lambdalabs on demand instance.
- waleedk 3y ago[Author] Good luck trying to use clusters of Lambda machines. Lambda labs are cheap for a reason: their API is not very featureful (we looked at them and we saw they didn't even support machine tagging). If you're looking for a box or two, lambda labs is fine. If you're looking for 1,000, not so much. Plus they don't actually have any actually A100s available at the moment (2022-05-17). CoreWeave is a nice middle ground. You can at least get the A100 machines into a k8s cluster.
- pseg134 3y agoOkay well that is just your experience. If you are brand new in this industry that is undergoing absolutely massive shortages of GPUs right now you probably will not be able to easily source GPUs. It might not seem fair but why would Lamda help someone they never heard of who will move for the next fad as quick as possible versus their long term existing customers?