5 ms·

What's hidden in the hidden layers? (1989) [pdf]

124 points by sonabinu 9y ago

realPubkey 9y ago
So was this the first article on how todays AI works?
- smadge 9y ago
  No
- s-macke 9y ago
  The research on artifical neural networks started already in the 50ths. Take a look at this video. https://www.youtube.com/watch?v=cNxadbrN_aI https://www.youtube.com/watch?v=cNxadbrN_aI
  - jve 9y ago
    Did you notice the link ends with an _aI suffix? :)
    - s-macke 9y ago
      Nope ;-)
      - gpvos 9y ago
        But would a neural network have noticed? :)
        russh 9y ago
        Only if trained to.
z3t4 9y ago
Nice. I didn't know they where working on self driving cars back in 1989! There's probably a lot to be learned. Today all the computing power of those racks can probably fit on a respery /nuc. Eg, you do not need a van, you could use an RC!
- s-macke 9y ago
  Take a look at following news and search for the thread started by guicho271828. https://news.ycombinator.com/item?id=15593305 https://news.ycombinator.com/item?id=15593305
- shakna 9y ago
  VaMoRs drove on the Autobahn in '87 [0]. [0] https://www.youtube.com/watch?v=2EH3R6c7Ufg https://www.youtube.com/watch?v=2EH3R6c7Ufg
  - ktpsns 9y ago
    This looks so much fun and was probably unrivalled and academic spirit.
- cr0sh 9y ago
  You need to look deeper into the ALVINN project (and the projects the preceded and came after it); it was one of the earlier attempts at a CNN-like architecture for self-driving vehicles. A good start is here: http://repository.cmu.edu/cgi/viewcontent.cgi?article=2874&context=compsci http://repository.cmu.edu/cgi/viewcontent.cgi?article=2874&c... And yes, that van held a rack-mounted computing system that included a WARP systolic array processor: https://en.wikipedia.org/wiki/WARP_(systolic_array) https://en.wikipedia.org/wiki/WARP_(systolic_array) ...which is very similar to what we have today as a GPU; it was basically a "high-speed" (for its time) vector processing system (in other words, perfect for neural-network processing). The processor on the vehicle ran the pre-trained model; training the model using similar machines took a really long time (weeks, iirc), even with the small number and sizes of layers they were working with. Compare and contrast the ALVINN project (there's a lot of research papers and such on it - video, pictures, etc - dig into it if you are interested) to NVidia's End-to-End CNN self-driving vehicle: https://devblogs.nvidia.com/parallelforall/deep-learning-self-driving-cars/ https://devblogs.nvidia.com/parallelforall/deep-learning-sel... You'll find a lot of parallels (and ALVINN - plus one or two of the other projects, iirc, is mentioned in their paper). I'm familiar with a lot of this mainly thru my self-study of machine learning and deep learning via a combination of courses from Udacity and (now) Coursera. My first impressions about ALVINN actually came from articles and video about it when I was a kid in the 1980s; I didn't understand how it worked at the time, but I was really impressed and intrigued by it. In 2011, I took part in (and ultimately completed) Andrew Ng's "ML Class" MOOC (this became one of Coursera's first courses) - and he made mention of ALVINN as a part of the course - and it inspired this guy, who was part of the class, to re-create it: http://blog.davidsingleton.org/nnrccar/ http://blog.davidsingleton.org/nnrccar/ So you aren't too off-base here; yes, that rack of machinery back in the 1980s has shrunk to pocket-size. Earlier this year, as part of the course for Udacity's "Self-Driving Car Engineer" Nanodegree program, I implemented (using their simulator) a version of the NVidia End-to-End CNN architecture to drive the simulated car around the simulation track based only on what it learned from my own "driving" in the simulation. Finally - if you are really interested in this - and want to compete on somewhat of a budget - there's this (among so many other similar competitions out there): https://diyrobocars.com/ https://diyrobocars.com/
pgeorgi 9y ago
It's from the August 1989 edition of Byte magazine, which has some more articles about neural networks: https://archive.org/stream/byte-magazine-1989-08/1989_08_BYTE_14-08_Neural_Networks https://archive.org/stream/byte-magazine-1989-08/1989_08_BYT...
Bluecobra 9y ago
I like how "Doesn't Look Like IBM" was a feature on the clone PC ad on the last page.
- marcosdumay 9y ago
  It would be interesting to look deeper into IBM's move from "nobody gets fired by buying it" to a poisonous brand.
  - pgeorgi 9y ago
    Back then, IBM was still deep in "nobody gets fired for buying them" territory. It was just the ad style of the day. The same edition of Byte magazine had an Oracle ad that was full of DBase IV references (copies of negative articles about it, mostly) with only a small Oracle logo somewhere near the end. Also stuff like https://78.media.tumblr.com/1fb5edc62a27f8f1c8ef07e611ec4281/tumblr_owhrbtaPvc1u8gbuwo1_1280.jpg https://78.media.tumblr.com/1fb5edc62a27f8f1c8ef07e611ec4281...
eb0la 9y ago
From the article: "According to optimistic predictions, by the year 2000, neural network technology will account for half the total revenue of the robotics and computer markets[]".
- sonabinu 9y ago
  This is an interesting point! We are still only scratching the surface of the possibilities with what this technology can accomplish.
  - gpvos 9y ago
    For as yet unknown values of "can".
wodenokoto 9y ago
Can anybody explain figure 1? As far as I can tell, you can't have hidden unit 1 inactive, while hidden unit 2 is active (as shown in Figure 1c) Also, I can't see how we can make a matrix multiplication that maps (0,0) -> 0 (0,1) -> 0 (1,0) -> 0 (1,1) -> 1 without adding a bias term. At least the weights shown in figure 1 are way off.

124 points by sonabinu 9y ago

realPubkey 9y ago

So was this the first article on how todays AI works?

smadge 9y ago
No
s-macke 9y ago
The research on artifical neural networks started already in the 50ths. Take a look at this video. https://www.youtube.com/watch?v=cNxadbrN_aI https://www.youtube.com/watch?v=cNxadbrN_aI
- jve 9y ago
  Did you notice the link ends with an _aI suffix? :)
  - s-macke 9y ago
    Nope ;-)
    - gpvos 9y ago
      But would a neural network have noticed? :)
      - russh 9y ago
        Only if trained to.

z3t4 9y ago

Nice. I didn't know they where working on self driving cars back in 1989! There's probably a lot to be learned. Today all the computing power of those racks can probably fit on a respery /nuc. Eg, you do not need a van, you could use an RC!

s-macke 9y ago
Take a look at following news and search for the thread started by guicho271828. https://news.ycombinator.com/item?id=15593305 https://news.ycombinator.com/item?id=15593305
shakna 9y ago
VaMoRs drove on the Autobahn in '87 [0]. [0] https://www.youtube.com/watch?v=2EH3R6c7Ufg https://www.youtube.com/watch?v=2EH3R6c7Ufg
- ktpsns 9y ago
  This looks so much fun and was probably unrivalled and academic spirit.
cr0sh 9y ago
You need to look deeper into the ALVINN project (and the projects the preceded and came after it); it was one of the earlier attempts at a CNN-like architecture for self-driving vehicles. A good start is here: http://repository.cmu.edu/cgi/viewcontent.cgi?article=2874&context=compsci http://repository.cmu.edu/cgi/viewcontent.cgi?article=2874&c... And yes, that van held a rack-mounted computing system that included a WARP systolic array processor: https://en.wikipedia.org/wiki/WARP_(systolic_array) https://en.wikipedia.org/wiki/WARP_(systolic_array) ...which is very similar to what we have today as a GPU; it was basically a "high-speed" (for its time) vector processing system (in other words, perfect for neural-network processing). The processor on the vehicle ran the pre-trained model; training the model using similar machines took a really long time (weeks, iirc), even with the small number and sizes of layers they were working with. Compare and contrast the ALVINN project (there's a lot of research papers and such on it - video, pictures, etc - dig into it if you are interested) to NVidia's End-to-End CNN self-driving vehicle: https://devblogs.nvidia.com/parallelforall/deep-learning-self-driving-cars/ https://devblogs.nvidia.com/parallelforall/deep-learning-sel... You'll find a lot of parallels (and ALVINN - plus one or two of the other projects, iirc, is mentioned in their paper). I'm familiar with a lot of this mainly thru my self-study of machine learning and deep learning via a combination of courses from Udacity and (now) Coursera. My first impressions about ALVINN actually came from articles and video about it when I was a kid in the 1980s; I didn't understand how it worked at the time, but I was really impressed and intrigued by it. In 2011, I took part in (and ultimately completed) Andrew Ng's "ML Class" MOOC (this became one of Coursera's first courses) - and he made mention of ALVINN as a part of the course - and it inspired this guy, who was part of the class, to re-create it: http://blog.davidsingleton.org/nnrccar/ http://blog.davidsingleton.org/nnrccar/ So you aren't too off-base here; yes, that rack of machinery back in the 1980s has shrunk to pocket-size. Earlier this year, as part of the course for Udacity's "Self-Driving Car Engineer" Nanodegree program, I implemented (using their simulator) a version of the NVidia End-to-End CNN architecture to drive the simulated car around the simulation track based only on what it learned from my own "driving" in the simulation. Finally - if you are really interested in this - and want to compete on somewhat of a budget - there's this (among so many other similar competitions out there): https://diyrobocars.com/ https://diyrobocars.com/

pgeorgi 9y ago

It's from the August 1989 edition of Byte magazine, which has some more articles about neural networks: https://archive.org/stream/byte-magazine-1989-08/1989_08_BYTE_14-08_Neural_Networks https://archive.org/stream/byte-magazine-1989-08/1989_08_BYT...

Bluecobra 9y ago

I like how "Doesn't Look Like IBM" was a feature on the clone PC ad on the last page.

marcosdumay 9y ago
It would be interesting to look deeper into IBM's move from "nobody gets fired by buying it" to a poisonous brand.
- pgeorgi 9y ago
  Back then, IBM was still deep in "nobody gets fired for buying them" territory. It was just the ad style of the day. The same edition of Byte magazine had an Oracle ad that was full of DBase IV references (copies of negative articles about it, mostly) with only a small Oracle logo somewhere near the end. Also stuff like https://78.media.tumblr.com/1fb5edc62a27f8f1c8ef07e611ec4281/tumblr_owhrbtaPvc1u8gbuwo1_1280.jpg https://78.media.tumblr.com/1fb5edc62a27f8f1c8ef07e611ec4281...

eb0la 9y ago

From the article: "According to optimistic predictions, by the year 2000, neural network technology will account for half the total revenue of the robotics and computer markets[]".

sonabinu 9y ago
This is an interesting point! We are still only scratching the surface of the possibilities with what this technology can accomplish.
- gpvos 9y ago
  For as yet unknown values of "can".

wodenokoto 9y ago

Can anybody explain figure 1? As far as I can tell, you can't have hidden unit 1 inactive, while hidden unit 2 is active (as shown in Figure 1c) Also, I can't see how we can make a matrix multiplication that maps (0,0) -> 0 (0,1) -> 0 (1,0) -> 0 (1,1) -> 1 without adding a bias term. At least the weights shown in figure 1 are way off.