3 ms·
I'm not an expert, but I've read highly-cited ML papers where the researchers barely bothered with hyperparameter search, much less throwing a few million dolla
by galimaufry 7y ago
I'm not an expert, but I've read highly-cited ML papers where the researchers barely bothered with hyperparameter search, much less throwing a few million dollars at the problem. You can still get an interesting proof of concept without big money.
And low resource computing is more theoretically and practically interesting. I've heard experts complain of some experiments "they didn't really discover anything, they threw compute at the problem until they got some nice PR." This was coming from M'FAANG people too so it's not just resentment.
- opportune 7y agoAre you sure they barely bothered, or did they just not mention it? I have heard stories of lots of “cutting edge” ML research actually just being the result of extremely fine hyperparameter tuning
- Bartweiss 7y agoThis is a worryingly good question. Most ML papers represent real results, we're not going to see a replication crisis in that sense, but I've heard some fears about another AI winter arriving when people realize that the sum of our reported gains vastly exceeds our actual progress. Hyperparameter tuning is one big concern here; we know it provides good results at the cost of lots of work, so there's a temptation to sic grad students on extensive tuning but publish on technique instead. Dataset bias is another, since nets trained on CIFAR or ImageNet keep turning out to embed database features. Ironically, I'm not sure all this increases the threat of FAANG taking over AI advancements. It sort of suggests that lots of our numerical gains are brute-forced or situational, and there's more benefit in new models work than mere error percentages would imply.