One of the reasons ML requires so many GPUs, besides the dataset sizes and the advantages of bigger models is the fact that ML is fundamentally unpredictable: i

One of the reasons ML requires so many GPUs, besides the dataset sizes and the advantages of bigger models is the fact that ML is fundamentally unpredictable: it’s hard to debug, and hard to know if something you try will work before you try it. So if you told my company they could run 2x the number of experiments for the same price they would do it because it doubles the number of hyperparameter settings we can try simultaneously. If that means fine tuning BERT in a day vs 2, that’s a huge increase in iteration velocity. It means we can work on more backlog projects simultaneously. It means engineers aren’t elbowing each other for GPU time (ok that is optimistic).