3 ms·

Disclosure: I used to work on Google Cloud. I dunno, their A100 results took about 20-30 minutes on 8 x A100s [1]. 8xA100s is like $24/hr on GCP at on-demand r

by boulos 5y ago

Disclosure: I used to work on Google Cloud. I dunno, their A100 results took about 20-30 minutes on 8 x A100s [1]. 8xA100s is like $24/hr on GCP at on-demand rates [2]. So like $12/run. The efficiency was okay but not linear, so if you were more cost constrained you might go with 1xA100 for $3/hr and have ~2.5hr training times. Getting that performance out of a GPU is more challenging than getting access to the GPUs. All the major cloud providers offer them. (Nit: GCP deployed the 40 GiB cards rather than the later 80 GiB parts, but let's ignore that). but it often doesn't matter [1] https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Segmentation/nnUNet#training-accuracy-nvidia-dgx-a100-8x-a100-80g https://github.com/NVIDIA/DeepLearningExamples/tree/master/P... [2] https://cloud.google.com/compute/gpus-pricing https://cloud.google.com/compute/gpus-pricing

teruakohatu 5y ago
That is the final model right? Not all the training while hyper params, or structure, was developed.
- PeterisP 5y ago
  No matter if it takes you a single run or ten thousand runs; qualified manpower to code and supervise it costs more than $12/run, so with prices like that the limiting factor is how much people-time you can devote to it.
  - kettleballroll 5y ago
    I disagree. I've participated and even won a few of those "ML in sciences" competitions in the past, and all other things being equal, the one with the most compute wins. 12$/hour sounds doable, but as soon as you need to grid over more than 4 hyperparameters (and that's small), you're looking at 100s to 1000s of runs in parallel. And you'll iterate over that process several times. Eg It's no coincidence that deepmind (with all of alphabets compute power) won CASP with alphafold. Yes they have a lot of very smart people, but so does everyone else. But no-one has the power to experiment with such large models so extensively for so long.
    - brilee 5y ago
      If you're doing grid search on more than 4 hyperparameters, that probably explains why you think computation wins. Even random search with a constrained budget will do pretty well without breaking the bank.
      - kettleballroll 5y ago
        Pretty well? Sure. Win a competition? No. If you're not running standard models (where you only have learning rate and regularization to grid over) but you're trying more specialized stuff/developing new models, 4 hyperparameters are not too many.

by boulos 5y ago

teruakohatu 5y ago

That is the final model right? Not all the training while hyper params, or structure, was developed.

PeterisP 5y ago
No matter if it takes you a single run or ten thousand runs; qualified manpower to code and supervise it costs more than $12/run, so with prices like that the limiting factor is how much people-time you can devote to it.
- kettleballroll 5y ago
  I disagree. I've participated and even won a few of those "ML in sciences" competitions in the past, and all other things being equal, the one with the most compute wins. 12$/hour sounds doable, but as soon as you need to grid over more than 4 hyperparameters (and that's small), you're looking at 100s to 1000s of runs in parallel. And you'll iterate over that process several times. Eg It's no coincidence that deepmind (with all of alphabets compute power) won CASP with alphafold. Yes they have a lot of very smart people, but so does everyone else. But no-one has the power to experiment with such large models so extensively for so long.
  - brilee 5y ago
    If you're doing grid search on more than 4 hyperparameters, that probably explains why you think computation wins. Even random search with a constrained budget will do pretty well without breaking the bank.
    - kettleballroll 5y ago
      Pretty well? Sure. Win a competition? No. If you're not running standard models (where you only have learning rate and regularization to grid over) but you're trying more specialized stuff/developing new models, 4 hyperparameters are not too many.