4 ms·
I'm old enough to have started doing machine learning before the deep learning revolution. When it happened, I predicted we'd see a resurgence in metaheuristics
by dimatura 7y ago
I'm old enough to have started doing machine learning before the deep learning revolution. When it happened, I predicted we'd see a resurgence in metaheuristics (of which ACO, used here, is an example), which just like neural networks in the pre-deep learning era, have a poor reputation among researchers. And it pretty much happened, though at first disguised as "bayesian optimization" [1].
I took an undergrad course in evolutionary optimization, and became briefly excited about it, so I'm fairly familiar with the ideas in that area. I think that similarly to neural networks, they don't really have a great mathematical basis to say what will work and what won't -- it's a lot of empirical experimentation. Do they work? Yeah, kinda. There's really few other options when you have to deal with large, combinatorial spaces such as neural network architectures. I do think a lot of research in the metaheuristics area, at least a few years ago (I haven't really kept up with it) is pretty bogus -- I lampooned it in a couple of "papers" (http://oneweirdkerneltrick.com/spectral.pdf http://oneweirdkerneltrick.com/spectral.pdf and http://oneweirdkerneltrick.com/catbasis.pdf http://oneweirdkerneltrick.com/catbasis.pdf). Yes, all the citations are real.
[1] Bayesian optimization is great, though I find it amusing that people who wouldn't touch a genetic or swarm algorithm are totally fine with BO when it's really not that different.
- p1esk 7y agoWhat do you mean they “kinda” work? NAS is all the rage these days. SOA on Imagenet [1], SOA for mobile [2]. Still needs a ton of gpus, but search algorithms getting smarter every month. PS I have to admit, your papers made me laugh :) [1] https://arxiv.org/abs/1811.06965 https://arxiv.org/abs/1811.06965 [2] http://www.arxiv-sanity.com/1807.11626v2 http://www.arxiv-sanity.com/1807.11626v2
- hooloovoo_zoo 7y agoI want to know what 1's CIFAR transfer results are w/o cutout.
- p1esk 7y agoFYI, they compare their cifar results to [1], which is more effective than plain cutout. [1] https://arxiv.org/abs/1805.09501 https://arxiv.org/abs/1805.09501
- hooloovoo_zoo 7y agoHeh, I'm familiar with this one too. It implies that, for instance, the Shake-Shake and Shake-Drop papers employ cutout, which they don't report. It's hard to make apples to apples comparisons when they're changing lots of things at the same time.
- dimatura 7y agoOh yeah, can't argue with results. Similar to deep learning. I've used both deep learning and metaheuristics a fair amount -- I don't care too much about mathematical rigor ;). I just mean, it's the sort of thing that usually needs experimentation, domain knowledge and maybe a bit of luck.
- mv4 7y ago"The normal (left) and paranormal (right) distributions." Your papers are fantastic.
- dimatura 7y ago:) I forgot to say "our papers" -- credit also goes to the coauthor.
- _raoulcousins 7y agoThere are some people fighting the good fight for more disciplined metaheuristics research. Metaheurstics - the Metaphor Exposed is a nice read (https://www.cs.ubc.ca/~hutter/EARG.shtml/stack/2013_Sorensen_MetaheuristicsTheMetaphorExposed.pdf https://www.cs.ubc.ca/~hutter/EARG.shtml/stack/2013_Sorensen...). The author is spot on that a lot of the nature-inspired algorithms don't even make sense as metaphors and they're just obfuscating the descriptions of algorithms unnecessarily.