4 ms·
What about XOR? From Wikipedia: In 1969 in a famous monograph entitled Perceptrons, Marvin Minsky and Seymour Papert showed that it was impossible for a single
by akuma73 12y ago
What about XOR?
From Wikipedia:
In 1969 in a famous monograph entitled Perceptrons, Marvin Minsky and Seymour Papert showed that it was impossible for a single-layer perceptron network to learn an XOR function.
- pesenti 12y agohttp://en.wikipedia.org/wiki/Feedforward_neural_network#mediaviewer/File:XOR_perceptron_net.png http://en.wikipedia.org/wiki/Feedforward_neural_network#medi...
- Dn_Ab 12y agoIf you join a bunch of perceptrons together that limitation goes away. Another path is to make the problem effectively linear again by transforming into higher dimensions, kernels do this with one clever trick that allows them to avoid the computational cost of doing so explicitly.
- JadeNB 12y ago> kernels do this with one clever trick that allows them to avoid the computational cost of doing so explicitly Minsky and Papert hate him! Use this one weird trick …. (Sorry; I couldn't resist!)
- gwern 12y agoSorry, you were beaten to the joke a long time ago: http://www.oneweirdkerneltrick.com/ http://www.oneweirdkerneltrick.com/
- jbarrow 12y agoThe simple answer is that we no longer use the standard linear perceptrons that McCulloch and Pitts put forth in 1943. The limitations of perceptrons and their training algorithms are well-known and well-studied, and in the last 60 years we've come a long way. A book came out a little bit ago which covers basic neural networks, and culminates in showing a network that can estimate XOR [1]. I strongly recommend reading not only the chapter, but the whole book. [1] http://natureofcode.com/book/chapter-10-neural-networks/ http://natureofcode.com/book/chapter-10-neural-networks/
- dragonwriter 12y ago"Single-layer" is an important restriction there, however.
- JoeAltmaier 12y agoNeural nets now use non-linear functions so they can do things like xor.
- jo_ 12y ago'Single-layer' is an easily overlooked and absolutely essential modifier. Very few networks these days are single layers. (Three is the minimum for infinite dimensional functional approximation, given nonlinear activations.) Deep Networks you see in new research papers these days have, at minimum, perhaps three layers. Some by LeCun et al. will go as high as nine or twelve layers, with four or sixteen layers in breadth.
- szabba 12y agoI'm pretty sure linearity was also part of the picture -- no matter how many layers of neurons you have, as long as they perform linear combinations, they could all be replaced with a single one (ignoring numeric errors).
- jo_ 12y agoI mentioned this in my post ("given nonlinear activations"), but you're correct to point out its importance. The combination of linear functions is always linear. Full stop. You need at least one non-linear layer to get reasonable results.