4 ms·
I agree. I am utterly unsurprised that gradient descent was unable to optimize a “minimal” network to a complete solution of a binary state problem with a Boole
by chaboud 6y ago
I agree. I am utterly unsurprised that gradient descent was unable to optimize a “minimal” network to a complete solution of a binary state problem with a Boolean rule set.
I’d have been shocked if it went any other way.
I’m also reminded of an ‘06 paper by Kreinovich and Shpak, Aggregability is NP-Hard.
http://www.cs.utep.edu/vladik/2006/tr06-29.pdf http://www.cs.utep.edu/vladik/2006/tr06-29.pdf
Neural networks aren’t magic, and optimization isn’t finding the one needle in a haystack with a blindfold on. This is an intuitive result for anyone mildly conversant in the space.
I don’t mind that they went out and did it, though.
- dnautics 6y agoMaybe I'm just not conversant enough in the field (I do ML "ops", not ML) but I'm surprised there aren't clever iterative ways of "reducing" the size of a model, for example, rank order the average activation of each parameter in a reducible group across the dataset, find the weakest, set it to zero, identify new misses, retrain with enrichment in the dead samples, reduce the hyperparameter by one.
- TrainedMonkey 6y agoThere are: https://medium.com/syncedreview/mit-google-propose-automl-for-model-compression-810be5b007d4 https://medium.com/syncedreview/mit-google-propose-automl-fo... I went to a more recent talk by Google and results they handpicked to show were impressive.
- 317070 6y agoThere are, it's called network pruning. You generally can reduce network size 90-95% without loss of accuracy with these methods.
- ChrisRackauckas 6y agoYou "just" have to train a huge network first, which is the downside.