4 ms·
I disagree. I've participated and even won a few of those "ML in sciences" competitions in the past, and all other things being equal, the one with the most com
by kettleballroll 5y ago
I disagree. I've participated and even won a few of those "ML in sciences" competitions in the past, and all other things being equal, the one with the most compute wins. 12$/hour sounds doable, but as soon as you need to grid over more than 4 hyperparameters (and that's small), you're looking at 100s to 1000s of runs in parallel. And you'll iterate over that process several times. Eg It's no coincidence that deepmind (with all of alphabets compute power) won CASP with alphafold. Yes they have a lot of very smart people, but so does everyone else. But no-one has the power to experiment with such large models so extensively for so long.
- brilee 5y agoIf you're doing grid search on more than 4 hyperparameters, that probably explains why you think computation wins. Even random search with a constrained budget will do pretty well without breaking the bank.
- kettleballroll 5y agoPretty well? Sure. Win a competition? No. If you're not running standard models (where you only have learning rate and regularization to grid over) but you're trying more specialized stuff/developing new models, 4 hyperparameters are not too many.