6 ms·
> that still adds up to $2,451,526.58 to run 1,024 A100 GPUs for 34 days Salary costs are probably even higher than compute costs. Automatic Speech Recognitio
by visarga 5y ago
> that still adds up to $2,451,526.58 to run 1,024 A100 GPUs for 34 days
Salary costs are probably even higher than compute costs. Automatic Speech Recognition is an industrial scale application, it costs a lot to train, but so do many other projects in different fields. How expensive is a plane or a ship? How much can a single building cost? A rocket launch?
- dylanbfox 5y ago> Salary costs are probably even higher than compute costs. Yes exactly. Managing that much compute requires many humans!
- dinvlad 5y agoI wouldn't be so sure :-)
- 6gvONxR4sf7o 5y agoIn what way are salary costs higher? This is on the order of 10 of their people’s annual salaries. This is for a single training run (meaning overall compute costs are higher), and it isn’t the only thing those ten or so people would have done that year (also meaning overall compute costs are higher).
- cpill 5y agoyeah, but a cluster of the resulting model can transcribe thousands of hours of speech / second, 24/7 with a fixed accuracy, what can 10 humans do?
- 6gvONxR4sf7o 5y agoHuh. You and I have two very different readings on this. I'm talking about the ML researcher's time (what if they hired more people instead) and you're talking about human text processors (what if people did this work by hand instead). Kinda neat that we had such different readings.
- visarga 5y agoYou have to work for a long time with a whole team on such a model. It adds up. In my experience there is a lot of work - data pipeline, labeling and data quality, the training, evaluations, deployment and making the deployed model efficient. And then there is also bias analysis, collecting failure cases and iterating on the data engine, and trivial things like measuring usage and billing.
- somebodythere 5y agoThis is an entire seed round's worth of money on an operational expenditure.