10 ms·
Frank Rosenblatt's perceptron paved the way for AI 60 years too soon (2019)
- westurner 5y agoPerceptron: https://en.wikipedia.org/wiki/Perceptron https://en.wikipedia.org/wiki/Perceptron #History, #Variants An MLP: Multilayer Perceptron can learn XOR and is considered a feed-forward ANN: Artificial Neural Network: https://en.wikipedia.org/wiki/Multilayer_perceptron https://en.wikipedia.org/wiki/Multilayer_perceptron Backpropagation: 1960-, 1986, https://en.wikipedia.org/wiki/Backpropagation https://en.wikipedia.org/wiki/Backpropagation
- xvilka 5y agoWell, NN aren't even remotely similar to real biological brain. Too simplistic theory for complex things like even a single neuron cell is.
- sydthrowaway 5y agoHow about Numenta's work?
- mach1ne 5y agoNumenta probably has got some things "right", some things wrong and some things missing. The problem with finding the neural code for training a network is that there are so many functional ways to encode data within a network.
- p1esk 5y ago“Digital logic design isn’t even remotely similar to real electrical circuits. Too simplistic theory for complex things like even a single CMOS transistor is.” The problem with your argument is we know too little about higher level brain operation to make any confident comparisons like that.
- lupin_sansei 5y agoFound a short video of the hardware Mark 1 Perceptron machine in action classifying photos. Fascinating https://youtu.be/cNxadbrN_aI https://youtu.be/cNxadbrN_aI
- 29athrowaway 5y agoReverse engineering biology is like studying the Terminator brain chip from Terminator 2. Regular expressions, perceptron, neocognitron, genetic algorithms, deep learning...
- rwallace 5y agoAre you saying regular expressions were inspired by biology? How?
- 29athrowaway 5y agohttps://www.rand.org/content/dam/rand/pubs/research_memoranda/2008/RM704.pdf https://www.rand.org/content/dam/rand/pubs/research_memorand...
- pugworthy 5y agoIt's worth noting that the Perceptron (at least early) was a system of circuits, not a program as we'd think of it now. Here's a 1963 paper, "System and Circuit Designs for The Tobermory Perceptron", which includes circuit diagrams. It would be intriguing to build one, a bit like building a working Babbage machine. https://blogs.umass.edu/brain-wars/files/2016/03/nagy-1963-tobermory-perceptron.pdf https://blogs.umass.edu/brain-wars/files/2016/03/nagy-1963-t...
- sitkack 5y agoGreat find! All the pdfs (use filter) this blog has posted, http://web.archive.org/web/*/blogs.umass.edu/brain-wars/ http://web.archive.org/web/*/blogs.umass.edu/brain-wars/*
- nicholast 5y agoBased on wikipedia it appears the original definition of perceptron was associated with the specific case of a neuron with a heavy-side step function activation output (which maps aggregated weights to either 0 or 1). I've generally adhered to referring to neural networks nodes in my writings as "neurons", which I believe is a catchall for any type of activation output, although have seen others use the term "perceptron" in the same usage in modern practice.
- lqr 5y agoThe Perceptron is also interesting because its classic mistake bound analysis is one of the earliest instances of modern learning theory. The mistake bound states that if the data is linearly separable (i.e. a perfect linear classifier exists), then Perceptron makes no more than (r / m)² misclassifications, where r is the radius of a ball containing the data set and m is the margin between the two classes. The analysis is modern because 1) unlike traditional statistics, it does not make any assumptions about the distributions of each class, and 2) is a non-asymptotic guarantee. It is also an online analysis, in the sense that it only compares performance against the optimal fixed classifier for the sequence that was actually observed, rather than any notion of an underlying distribution. More details: http://www.argmin.net/2021/11/04/perceptron/ http://www.argmin.net/2021/11/04/perceptron/
- canjobear 5y agoThe story of Perceptrons---both the idea and the book of that title---is instructive about how science proceeds in practice. The folklore is that this book killed neural net research, but if you read it you'll find it's not as damning as you might expect. Apparently it circulated widely as a manuscript before publication, and the circulating manuscript was much more negative in tone, and this is what shaped people's perceptions.
- kd5bjo 5y agoMultilayer neural networks weren’t really a viable tool until the backpropagation algorithm for determining internal parameters was developed in 1985 (cf https://apps.dtic.mil/sti/pdfs/ADA164453.pdf https://apps.dtic.mil/sti/pdfs/ADA164453.pdf )
- _0ffh 5y agoI think you'll find that backpropagation was essentially developed (multiple times) during the 60s in the field of control theory and first implemented in the early 70s. Ed: To be clear, the idea to use them to adapt the weights of NNs was also from the 70s but only rediscovered and applied to MLPs by at least two independent groups/individuals in the 80s.
- mindcrime 5y agoI think you'll find that backpropagation was essentially developed (multiple times) during the 60s in the field of control theory and first implemented in the early 70s. Indeed. The book Talking Nets: An Oral History of Neural Networks[1] covers a lot of this ground. Read it and you'll see many people who were involved in the early history NN's mentioning how backprop was discovered and re-discovered over and over again. [1]: https://www.amazon.com/Talking-Nets-History-Neural-Networks/dp/0262511118 https://www.amazon.com/Talking-Nets-History-Neural-Networks/...
- enchiridion 5y agoWhat really made the difference was non-linear activations. Without a non-linearity depth doesn’t buy you anything.
- Jun8 5y agoThe answers here may be of interest: https://ai.stackexchange.com/questions/1288/did-minsky-and-papert-know-that-multi-layer-perceptrons-could-solve-xor https://ai.stackexchange.com/questions/1288/did-minsky-and-p... The unfortunate thing was not the Perceptrons book but the fact that Rosenblatt died prematurely soon after. He was very well-equipped to defend and carry on work on NNs, I think.
- savant_penguin 5y agoThe programmers at stackoverflow must be amazing if they fix it with stackoverflow offline
- gwern 5y agoI was reading through the references in https://blogs.umass.edu/comphon/2017/06/15/did-frank-rosenblatt-invent-deep-learning-in-1962/ https://blogs.umass.edu/comphon/2017/06/15/did-frank-rosenbl... linked in another comment, specifically https://www.gwern.net/docs/ai/1993-olazaran.pdf https://www.gwern.net/docs/ai/1993-olazaran.pdf https://www.gwern.net/docs/ai/1996-olazaran.pdf https://www.gwern.net/docs/ai/1996-olazaran.pdf by a Mikel Olazaran who spent a while interviewing what looks to be almost all the surviving connectionists & Minsky etc. Olazaran argues that all the connectionists were perfectly aware of the _Perceptrons_ headline conclusion about single-layer perceptrons being hopelessly linear, which drafts had been circulating for like 4 years beforehand as well, and most regarded it as unimportant (pointing out that humans can't solve the parity of a grid of dots either without painfully counting them out one by one) and having an obvious solution (multiple layers) that they all, Rosenblatt especially, had put a lot of work into trying. The problem was, none of the multi-layer things worked, and people had run out of ideas. So most of the connectionist researchers got sucked away by things that were working at the time (eg the Stanford group was having huge success with adaptive antennas & telephone filters which accidentally come out of their NN work), and funding dried up (for both exogenous political reasons related to military R&D being cut, and just the lack of results compared to alternative research programs like the symbolics approaches which were enjoying their initial flush of success in theorem proving and checkers etc). So when, years later, _Perceptrons_ came out with all of its i's dotted and ts-crossed, it didn't "kill connectionism" because that had already died. What _Perceptrons_ really did was it served as a kind of excuse or Schelling point to make the death 'official' and cement the dominance of the symbolic approaches. Rosenblatt never gave up, but he had already been left high and dry with no more funding and no research community. Olazaran directly asks several of them whether more funding or work would have helped, and it seems everyone agrees that it would've been useless. The computers just weren't there in the '60s. (One notes that it might have worked in the '70s if anyone had paid attention to the invention of backpropagation, pointing out that Rumelhart et al doing the PDP studies on backprop were using the equivalent of PCs for those studies in the late '80s, so if you were patient you could've done them on minicomputers/mainframes in the '70s. But not the '60s.)
- tomcat27 5y ago[flagged]
- pfisherman 5y agoI think it often goes understated just how much the emergence of massive datasets led to the successes of neural networks. I once heard Daphne Koller say that before big data, neural networks were always the second best way to do anything.
- bmitc 5y agoWhat was the first best way?
- visarga 5y agoSVMs and Random Forests.
- otabdeveloper4 5y ago> successes of neural networks Too early to say that with a straght face yet. "Deep learning" produced some amazing generative art, and some very flashy research papers, but it failed to drive business decisions. (And not for the lack of trying, that's for sure.)
- joefourier 5y agoWhat about machine translation, speech recognition and transcription, speech synthesis, image classification, segmentation and other computer vision tasks, facial recognition, image and audio denoising, beating world champions at Go along with chess engines, language models for text generation, image editing (e.g. FaceApp)... You don’t have to fall for the hype and think deep learning will lead us to AGI in the next 5 years, but dismissing it as generative art and flashy papers isn’t any more accurate.
- otabdeveloper4 5y agoDid you read my post? None of that drives business decisions. As for Alexa, et al - these things fall into the "generative art" bucket. The search results they give aren't any better than the Eliza-tier expert systems of yore. They just feel much better and more human when you use them. (Which is also important, but doesn't drive business decisions, except as part of a marketing strategy.)
- 6gvONxR4sf7o 5y agoHow can an article like this talk about the significance of a single layer perceptron and not talk about statistics’s contributions, like regression models? Binary classification with a single layer perceptron paved the way, but logistic regression isn’t worth mentioning?
- naomisperfume 5y agoThat's also weird to me. Both Probit (1934) and Logistic regression (1943) models, which are also linear classifiers, precede the Perceptron model (1958) by many years.
- inetsee 5y agoI remember taking a class at UCSD in 1971 or 1972 that touched on Perceptrons (among a lot of other things, it was basically a survey course). I wonder sometimes what the world would be like today if they had realized the importance of hidden layers back then.
- rococode 5y agoData and compute power in the 70s probably would've limited the usefulness and led people to try other things (perhaps not even "probably" - that could be what really happened), though maybe it could've been revisited with success by the late 90s. Makes you wonder if there are other abandoned techniques that might be worth circling back to nowadays...
- mrDmrTmrJ 5y agok-NN methods always love more data and can be embarrassingly competitive with far fancier algorithms. Should always be considered as a classification baseline: https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm
- MaysonL 5y agoA former colleague of mine, Terry Koken, who worked with Rosenblatt at Cornell for a while and is quoted in the article, dropped a comment last year saying pretty much that: https://blogs.umass.edu/comphon/2017/06/15/did-frank-rosenblatt-invent-deep-learning-in-1962/#comment-7831 https://blogs.umass.edu/comphon/2017/06/15/did-frank-rosenbl...
- sjg007 5y agoEmpirical methods/experimentation have classically taken a back seat to mathematical proofs in computer science.
- PeterisP 5y agoI was just reading 1986 book Parallel Distributed Processing (https://mitpress.mit.edu/books/parallel-distributed-processing-volume-1 https://mitpress.mit.edu/books/parallel-distributed-processi...) and was quite surprised by the relevance of their arguments how learning systems should be built and structured, which map quite accurately to how we actually developed them in the following decades after we got more computing power, more data and engineering know-how.
- tmabraham 5y agoThe fastai book actually makes a nice comparison between the systems described in PDP and modern deep learning. > In fact, the approach laid out in PDP is very similar to the approach used in today's neural networks. From: https://github.com/fastai/fastbook/blob/master/01_intro.ipynb https://github.com/fastai/fastbook/blob/master/01_intro.ipyn...