3 ms·
I am not disputing the premise. I am questioning who is saying it, and their motivation for saying it. Training a model from scratch is an insanely expensive
by 2024throwaway 3y ago
I am not disputing the premise.
I am questioning who is saying it, and their motivation for saying it.
Training a model from scratch is an insanely expensive proposition. Something that anyone attempting to do will likely need to purchase and/or lease quite a bit of, you guessed it, Nvidia hardware to accomplish.
- dawatchusay 3y agoI believe in the article Huang claims it’s not that expensive
- 2024throwaway 3y agoThat claim is wrong. Training even a very basic model like TinyLlama takes 90 days on 16 x A100 GPUs [0]. A p4 box containing 8 X A100 GPUs costs $32/hr [1]. So 90 days * 2 p4 boxes * 24 hours * $32/hr = $138,240 Larger models run into the millions of dollars. Though I suppose it all depends on your personal definition of expensive. [0] https://github.com/jzhang38/TinyLlama https://github.com/jzhang38/TinyLlama [1] https://aws.amazon.com/ec2/instance-types/p4/ https://aws.amazon.com/ec2/instance-types/p4/
- ta1243 3y agomillions of dollars is a rounding error in almost any country's budget.