5 ms·
"Training was done on eight NVIDIA A100 GPUs for 1000 epochs" "trained on four NVIDIA V100 GPUs for 300 epochs" "model was trained on a NVIDIA DGX-1 cluster us
by andrewon 5y ago
"Training was done on eight NVIDIA A100 GPUs for 1000 epochs"
"trained on four NVIDIA V100 GPUs for 300 epochs"
"model was trained on a NVIDIA DGX-1 cluster using eight GPUs"
No doubt these are model innovations. But can't help wondering how much ready access to arrays of GPU might give them advantage... You know how many different models need to be tested before getting to the winning ones.
- setr 5y agoShould GPU choice affect model performance? AFAIK it should only impact training time
- amitport 5y agoWhen you can train and evaluate quickly you can iterate through experiments faster and produce better models...
- aix1 5y agoThe choice of hardware (here, type of GPU) could also impact the model architecture, potentially affecting model quality. For example, if GPU RAM is limited, one might have to reduce a model's depth or its receptive field just to be able to train on smaller hardware.
- irjustin 5y agoA non-trivial amount. But, this is true of basically anything. Teams are resource constrained on hardware, man-power, intelligence, etc. Intelligence to create innovative models, hardware to test/iterate models, etc. AlphaFold is a clear example of what's possible with big money.
- boulos 5y agoDisclosure: I used to work on Google Cloud. I dunno, their A100 results took about 20-30 minutes on 8 x A100s [1]. 8xA100s is like $24/hr on GCP at on-demand rates [2]. So like $12/run. The efficiency was okay but not linear, so if you were more cost constrained you might go with 1xA100 for $3/hr and have ~2.5hr training times. Getting that performance out of a GPU is more challenging than getting access to the GPUs. All the major cloud providers offer them. (Nit: GCP deployed the 40 GiB cards rather than the later 80 GiB parts, but let's ignore that). but it often doesn't matter [1] https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Segmentation/nnUNet#training-accuracy-nvidia-dgx-a100-8x-a100-80g https://github.com/NVIDIA/DeepLearningExamples/tree/master/P... [2] https://cloud.google.com/compute/gpus-pricing https://cloud.google.com/compute/gpus-pricing
- teruakohatu 5y agoThat is the final model right? Not all the training while hyper params, or structure, was developed.
- PeterisP 5y agoNo matter if it takes you a single run or ten thousand runs; qualified manpower to code and supervise it costs more than $12/run, so with prices like that the limiting factor is how much people-time you can devote to it.
- kettleballroll 5y agoI disagree. I've participated and even won a few of those "ML in sciences" competitions in the past, and all other things being equal, the one with the most compute wins. 12$/hour sounds doable, but as soon as you need to grid over more than 4 hyperparameters (and that's small), you're looking at 100s to 1000s of runs in parallel. And you'll iterate over that process several times. Eg It's no coincidence that deepmind (with all of alphabets compute power) won CASP with alphafold. Yes they have a lot of very smart people, but so does everyone else. But no-one has the power to experiment with such large models so extensively for so long.
- brilee 5y agoIf you're doing grid search on more than 4 hyperparameters, that probably explains why you think computation wins. Even random search with a constrained budget will do pretty well without breaking the bank.
- kettleballroll 5y agoPretty well? Sure. Win a competition? No. If you're not running standard models (where you only have learning rate and regularization to grid over) but you're trying more specialized stuff/developing new models, 4 hyperparameters are not too many.
- bigbillheck 5y agoOne of the other contenders agrees: https://twitter.com/HassanMohyudDin/status/1446617536561401862 https://twitter.com/HassanMohyudDin/status/14466175365614018... (Saving you a click, they trained on a single 2080Ti)