4 ms·

26 years ago... How common was this sentiment back then? How many had serious faith in the neural network approach, or at least were very excited about its pote

26 years ago... How common was this sentiment back then? How many had serious faith in the neural network approach, or at least were very excited about its potential? Also, what are some current technologies and ideas that are unpopular but which have some dedicated fans who insist there's major untapped potential (even if maybe it can't be fully tapped into today)? Specifically, little-known things rather than pretty popular stuff like quantum computing.

jacobush 6y ago
My classmate wrote back-propagating neural nets on his Amiga 1200 in 1994. We had pretty high hopes back then. I and I think many others sort of assumed we would have had General AI by now. Or at least going in that direction. Another classmate said in 1992 that scientists had simulated the neural net equivalent to a worm or something. This topic came up re the neural net featured in the movie Terminator 2. The current use of "neural nets" are both under- and overwhelming. But definitely over-whelmingly boring, to me. It's good, don't get me wrong. But from my viewpoint, it looks like we have found a trick, and now try to apply this same trick to as many new fields as possible.
- tomrod 6y ago
  > now try to apply this same trick to as many new fields as possible. Right, this is normal and expected. While scientific growth looks like an exponential curve, the lateral application path is equally important to maintaining that curve! We may get to AGI one day (I have my doubts) but I have to remind myself that all theses clunky things we see today are v1 -- and virtually everything can improve!
  - jacobush 6y ago
    It is normal, but it's kind of disappointing that from my (uneducated) vantage point, we don't seem to be at all closer to AGI than we were in 1994. It's like the difference between Kitty Hawk and a Boeing airliner. Yes, they are impressive feats of engineering, and the massive training hours tell the story. But the Boeing Airliner is still less clever than a fruit fly, to merge the metaphors. Edit: to expand - it seems to get something as smart as a dog is almost the same as something as smart as a human. 1994 neural net 2020 GPT-3 . . . . . unknown numbers of dots... . . . dog human
2sk21 6y ago
I got one of the earliest PhDs in neural networks (in 1992!), and my dissertation about improving back propagation. Even though I actually saw how well it worked, I somehow had no faith in the technology as it seemed like a toy due to the hopelessly slow processors we had at the time. I did implement back propagation on a Connection Machine CM2 but found it too complex and ultimately wound up running my code on a bunch of IBM RS 6000 workstations instead.
- jacobush 6y ago
  I think my classmate might have read your dissertation in 1994 possibly. Were you cited or inspired magazine articles?
  - 2sk21 6y ago
    Over the years, I have acquired a fair number of citations for my papers which were mostly published in IEEE Transactions in Neural Networks. But I am certain that my work did not inspire any magazine articles - the field was considered too far away from mainstream CS at the time. There was even grumbling from some faculty in my CS department that my dissertation should not have been in CS but rather in statistics :-) I am still getting about a new citation every month for one of my papers that was published 27 years ago!
- whizzter 6y ago
  That's awesome. Have you gotten back to working on them? (seems like you'd have your literal pick of contracts atleast 5-10 years ago)
  - 2sk21 6y ago
    When I graduated, there were absolutely no jobs in machine learning and I used to get blank looks when I told people about my work. Fortunately, I was able to get a job in the systems field and worked there for many years. I was able to get back into machine learning about 8 years ago and am semi-retired now. I'm not complaining as I was always well paid in my career. But sometimes, being too early in a field can be as bad as being too late :-)
xbar 6y ago
I remember Omni magazine articles from the late 80s saying that they were the path.
jcims 6y ago
My dad was a welder and HVAC technician by day and a bit of an intellectual by night. He was all over expert systems in the late 80’s. I can distinctly remember him excitedly explaining them to me and I just couldn’t fathom what was interesting about it at all. Then in the early 90’s he started pivoting to neural networks. Then he got killed at work. I didn’t hear about neural networks again for close to 20 years, but I can attest at least someone was stoked about them back then.
- alex_anglin 6y ago
  Sounds like your dad was an interesting guy. Sorry for your loss.
  - jcims 6y ago
    Thank you. It's one of those things where I loved the guy but it's taken a while to really unpack the layers he was operating at. I'm older now than he was when he passed and I marvel at the diversity of stuff he got into.
bachmeier 6y ago
> How many had serious faith in the neural network approach, or at least were very excited about its potential? There were at least some economists that had hope back then. Here's a paper my advisor published in 1995, but that he had written several years earlier for his dissertation: http://econweb.rutgers.edu/nswanson/papers/selec3.pdf http://econweb.rutgers.edu/nswanson/papers/selec3.pdf Hal White was working with NN in the 1980s. Here's one of his publications from 1988: http://www.machine-learning.martinsewell.com/ann/White1988.pdf http://www.machine-learning.martinsewell.com/ann/White1988.p...
dahart 6y ago
I was in college and interested in neural nets when this was written, and as it happens later graduated and worked with the author of this post. In academia at the time, I believe neural nets were seen as a fading trend, something that was a cute idea but wasn’t destined to work out because they didn’t work well enough and even if they did we didn’t have the compute for anything useful. Personally, I was fascinated, and I know several other people who kept hope as well, one who is now quite a well known AI / neural nets researcher. I didn’t pursue neural nets personally, but I did keep using related genetic algorithms and artificial evolution, which I think was viewed with the same mix of skepticism generally, with pockets of enthusiasts. Gates in the story was probably sharing what he’d been hearing from academics. Lawrence (author of the post) is an incredibly smart person, and I have no doubt was able to see past what many of the researchers were saying. BTW, I recently happened across a paper from 1967 (53 years ago) that mentions neural networks in passing as if it was a popular idea at the time - the original paper on the Medial Axis Transform! Scroll to the 2nd page (“364”) near the top: “Consider a continuous isotropic plane (an idealization of an active granular material or a rudimentary neural net) that has the following properties at each point” http://pageperso.lif.univ-mrs.fr/~edouard.thiel/rech/1967-blum.pdf http://pageperso.lif.univ-mrs.fr/~edouard.thiel/rech/1967-bl...
timonoko 6y ago
In 1984 Teuvo Kohonen told us exactly how neural nets works. Albeit in Finnish. In this video he had matrix of "learning vectors" which change as the net learns. "Meta-learning" happens when several learning matrixes are on top of each others. https://youtu.be/Qy3h7kT3P5I?t=2481 https://youtu.be/Qy3h7kT3P5I?t=2481
kyberias 6y ago
I was in high-school at the time and was really into neural nets.
georgeecollins 6y ago
In the early 90's I worked in a research lab that studied cortical EEG data. I experimented with using neural nets to categorize brain activity, in a way that statistical methods had already shown some success. The professor I worked for felt that neural networks would not perform better than statistical methods. Neural nets could be trained on the data and predict on the untrained set, but not better than the statistical method. So I didn't pursue it. The funny thing to think about for people on this forum was what weak computing power was available to me at that time. As I recall I did this whole experiment in C (using the TLearn libraries) on a Masscomp workstation. The cool thing about the Masscomp was that it had a vector processing board that could do some parallel computations. Well I didn't use that but the lab had the hardware. So I was doing all these things on Motorola 68000 with less RAM and way less CPU than your watch.
dnautics 6y ago
In 03 I was desperately trying to write neural net algorithms for the BeOS, but I had no idea how to train them (much less backprop). I think distributed systems research is enabling personal user-level distributed networks (think like mastodon but to the point where a lay person can own their own node without much trouble). But will the economies of scale for commodities goods like motherboards make sense against behemoths buying up lots of hard drives? Who knows.

jacobush 6y ago

My classmate wrote back-propagating neural nets on his Amiga 1200 in 1994. We had pretty high hopes back then. I and I think many others sort of assumed we would have had General AI by now. Or at least going in that direction. Another classmate said in 1992 that scientists had simulated the neural net equivalent to a worm or something. This topic came up re the neural net featured in the movie Terminator 2. The current use of "neural nets" are both under- and overwhelming. But definitely over-whelmingly boring, to me. It's good, don't get me wrong. But from my viewpoint, it looks like we have found a trick, and now try to apply this same trick to as many new fields as possible.

tomrod 6y ago
> now try to apply this same trick to as many new fields as possible. Right, this is normal and expected. While scientific growth looks like an exponential curve, the lateral application path is equally important to maintaining that curve! We may get to AGI one day (I have my doubts) but I have to remind myself that all theses clunky things we see today are v1 -- and virtually everything can improve!
- jacobush 6y ago
  It is normal, but it's kind of disappointing that from my (uneducated) vantage point, we don't seem to be at all closer to AGI than we were in 1994. It's like the difference between Kitty Hawk and a Boeing airliner. Yes, they are impressive feats of engineering, and the massive training hours tell the story. But the Boeing Airliner is still less clever than a fruit fly, to merge the metaphors. Edit: to expand - it seems to get something as smart as a dog is almost the same as something as smart as a human. 1994 neural net 2020 GPT-3 . . . . . unknown numbers of dots... . . . dog human

2sk21 6y ago

I got one of the earliest PhDs in neural networks (in 1992!), and my dissertation about improving back propagation. Even though I actually saw how well it worked, I somehow had no faith in the technology as it seemed like a toy due to the hopelessly slow processors we had at the time. I did implement back propagation on a Connection Machine CM2 but found it too complex and ultimately wound up running my code on a bunch of IBM RS 6000 workstations instead.

jacobush 6y ago
I think my classmate might have read your dissertation in 1994 possibly. Were you cited or inspired magazine articles?
- 2sk21 6y ago
  Over the years, I have acquired a fair number of citations for my papers which were mostly published in IEEE Transactions in Neural Networks. But I am certain that my work did not inspire any magazine articles - the field was considered too far away from mainstream CS at the time. There was even grumbling from some faculty in my CS department that my dissertation should not have been in CS but rather in statistics :-) I am still getting about a new citation every month for one of my papers that was published 27 years ago!
whizzter 6y ago
That's awesome. Have you gotten back to working on them? (seems like you'd have your literal pick of contracts atleast 5-10 years ago)
- 2sk21 6y ago
  When I graduated, there were absolutely no jobs in machine learning and I used to get blank looks when I told people about my work. Fortunately, I was able to get a job in the systems field and worked there for many years. I was able to get back into machine learning about 8 years ago and am semi-retired now. I'm not complaining as I was always well paid in my career. But sometimes, being too early in a field can be as bad as being too late :-)

xbar 6y ago

I remember Omni magazine articles from the late 80s saying that they were the path.

jcims 6y ago

My dad was a welder and HVAC technician by day and a bit of an intellectual by night. He was all over expert systems in the late 80’s. I can distinctly remember him excitedly explaining them to me and I just couldn’t fathom what was interesting about it at all. Then in the early 90’s he started pivoting to neural networks. Then he got killed at work. I didn’t hear about neural networks again for close to 20 years, but I can attest at least someone was stoked about them back then.

alex_anglin 6y ago
Sounds like your dad was an interesting guy. Sorry for your loss.
- jcims 6y ago
  Thank you. It's one of those things where I loved the guy but it's taken a while to really unpack the layers he was operating at. I'm older now than he was when he passed and I marvel at the diversity of stuff he got into.

bachmeier 6y ago

> How many had serious faith in the neural network approach, or at least were very excited about its potential? There were at least some economists that had hope back then. Here's a paper my advisor published in 1995, but that he had written several years earlier for his dissertation: http://econweb.rutgers.edu/nswanson/papers/selec3.pdf http://econweb.rutgers.edu/nswanson/papers/selec3.pdf Hal White was working with NN in the 1980s. Here's one of his publications from 1988: http://www.machine-learning.martinsewell.com/ann/White1988.pdf http://www.machine-learning.martinsewell.com/ann/White1988.p...

dahart 6y ago

I was in college and interested in neural nets when this was written, and as it happens later graduated and worked with the author of this post. In academia at the time, I believe neural nets were seen as a fading trend, something that was a cute idea but wasn’t destined to work out because they didn’t work well enough and even if they did we didn’t have the compute for anything useful. Personally, I was fascinated, and I know several other people who kept hope as well, one who is now quite a well known AI / neural nets researcher. I didn’t pursue neural nets personally, but I did keep using related genetic algorithms and artificial evolution, which I think was viewed with the same mix of skepticism generally, with pockets of enthusiasts. Gates in the story was probably sharing what he’d been hearing from academics. Lawrence (author of the post) is an incredibly smart person, and I have no doubt was able to see past what many of the researchers were saying. BTW, I recently happened across a paper from 1967 (53 years ago) that mentions neural networks in passing as if it was a popular idea at the time - the original paper on the Medial Axis Transform! Scroll to the 2nd page (“364”) near the top: “Consider a continuous isotropic plane (an idealization of an active granular material or a rudimentary neural net) that has the following properties at each point” http://pageperso.lif.univ-mrs.fr/~edouard.thiel/rech/1967-blum.pdf http://pageperso.lif.univ-mrs.fr/~edouard.thiel/rech/1967-bl...

timonoko 6y ago

In 1984 Teuvo Kohonen told us exactly how neural nets works. Albeit in Finnish. In this video he had matrix of "learning vectors" which change as the net learns. "Meta-learning" happens when several learning matrixes are on top of each others. https://youtu.be/Qy3h7kT3P5I?t=2481 https://youtu.be/Qy3h7kT3P5I?t=2481

kyberias 6y ago

I was in high-school at the time and was really into neural nets.

georgeecollins 6y ago

In the early 90's I worked in a research lab that studied cortical EEG data. I experimented with using neural nets to categorize brain activity, in a way that statistical methods had already shown some success. The professor I worked for felt that neural networks would not perform better than statistical methods. Neural nets could be trained on the data and predict on the untrained set, but not better than the statistical method. So I didn't pursue it. The funny thing to think about for people on this forum was what weak computing power was available to me at that time. As I recall I did this whole experiment in C (using the TLearn libraries) on a Masscomp workstation. The cool thing about the Masscomp was that it had a vector processing board that could do some parallel computations. Well I didn't use that but the lab had the hardware. So I was doing all these things on Motorola 68000 with less RAM and way less CPU than your watch.

dnautics 6y ago

In 03 I was desperately trying to write neural net algorithms for the BeOS, but I had no idea how to train them (much less backprop). I think distributed systems research is enabling personal user-level distributed networks (think like mastodon but to the point where a lay person can own their own node without much trouble). But will the economies of scale for commodities goods like motherboards make sense against behemoths buying up lots of hard drives? Who knows.