4 ms·
> It is arguable whether Hessian free methods, contrastive divergence or auto-encoder based training methods qualify as 'breakthroughs' but they have definitely
by kmavm 12y ago
> It is arguable whether Hessian free methods, contrastive divergence or auto-encoder based training methods qualify as 'breakthroughs' but they have definitely equipped invigorated researchers in this broad area with their capabilities.
The scientific revolution in computer vision right now is due to deep convnets, trained in a supervised way using backprop and SGD. All of the systems we're talking about that have started blowing away records are members of this family, and were trained this way. If Alex Krizhevsky had not entered ImageNet 2012, we would not be having this conversation (in part because I probably would never have gotten curious enough about it to leave my home territories of systems and programming languages). Second order methods, unsupervised pre-training, RBMs, graphical models, etc. etc. etc. were exciting for those inside the field, definitely provided encouragement to those optimistic about deep models, and still might prove important, but they have had little impact and visibility to skeptical people outside, in the way that entering a computer vision competition and murdering all the computer vision systems did.
- kastnerkyle 12y agoWell, HF training was a pretty big deal IMO. Definitely saved my bacon in training some recurrent nets, much easier to get working and/or recovering from bad optimization but pretty slow. The SGD we use today actually has some strong ties back to that second order optimization work - see some papers by Ilya Sutskever relating a special form of momentum back to second order methods like HF. http://www.cs.utoronto.ca/~ilya/pubs/2013/1051_2.pdf http://www.cs.utoronto.ca/~ilya/pubs/2013/1051_2.pdf His dissertation covers this at some length as well. Using Adagrad, Adadelta, etc. isn't really SGD as it was back in 2012, and this years entrant "GoogLeNet" basically halved the error again using these and other tricks (we think) - which is even more impressive considering 11% - 6.7% is a HUGE increase in difficulty, just my 2 cents. However, there is a good reason the colloquial name for these things is "Alexnets"... that work was truly incredible and has not stopped behind the doors of Google I don't think.
- kmavm 12y agoBy sheer coincidence, Michael Jordan did an AMA yesterday, and made this case better than I could. http://www.reddit.com/r/MachineLearning/comments/2fxi6v/ama_michael_i_jordan/ckep3z6 http://www.reddit.com/r/MachineLearning/comments/2fxi6v/ama_...