4 ms·

Most other scientists dismissed neural networks? Is there some history I am unaware of as that doesn't seem true. Did the article want to push the idea of the l

Most other scientists dismissed neural networks? Is there some history I am unaware of as that doesn't seem true. Did the article want to push the idea of the lone rogue thinker a bit too much?

ZenoArrow 9y ago
I thought this too. Neural networks definitely seemed like a credible research topic in the 90s at the very least (which is when I first became aware of them).
dang 9y ago
Maybe, but let's not get hung up on the title—that tends to keep discussion shallow. We'll change "most" to "many" above.
- deleted 9y ago
  [deleted]
deleted 9y ago
[deleted]
meow1032 9y ago
http://www.dataversity.net/brief-history-deep-learning/ http://www.dataversity.net/brief-history-deep-learning/ Neural Networks weren't really thought of very highly until CNNs started winning image recognition competitions in the early 2010's. I think most people had the feeling that they were interesting tools to learn how the brain worked, but too slow and opaque to be practical statistical tools. I've been following Hinton for a while (because of Hinton and Shallice 1991), and my understanding is that it was really hard for him to get funding especially when he was just starting out. The fact that so much of the work from the mid-80's to 2000 came from just a few labs should tell you how hard it was to get funding for that kind of research.
- _delirium 9y ago
  > The fact that so much of the work from the mid-80's to 2000 came from just a few labs should tell you how hard it was to get funding for that kind of research. If you look through papers from that era I don't think you'll find that's true at all. You could fairly say that only a few of the prominent labs from the first neural-net boom lasted long enough to still be prominent labs now in the second neural-net boom (though even then there are several: Hinton, Bengio, Schmidhuber, LeCun, etc.). Since they're still around doing interviews and putting out new papers, understandably their work has a higher profile now than that of people who aren't in the field anymore. But there have been a ton of others over the years too, just many of them from the first wave have moved on or retired by now. Especially in the '90s the field was hot and reasonably large (and it was pretty easy to get funding, too). If you look at e.g. the NIPS 1992 proceedings, it's definitely not just a handful of labs: https://papers.nips.cc/book/advances-in-neural-information-processing-systems-5-1992 https://papers.nips.cc/book/advances-in-neural-information-p...
  - meow1032 9y ago
    >You could fairly say that only a few of the prominent labs from the first neural-net boom lasted long enough to still be prominent labs now in the second neural-net boom (though even then there are several: Hinton, Bengio, Schmidhuber, LeCun, etc.) I agree, I don't mean to imply that Hinton did all of the work on NN's, but he certainly contributed a lot to the field, even as the popularity waxed and waned. Even if you just click on a few of the names in your link, you can see most of those authors submitted ~1-6 papers. Hinton, Bengio, and Jordan have submitted >50 each. I wouldn't say that NN's were exactly thought of as a joke before 2011, but from the people I talked to about them, they weren't considered very promising as practical statistical systems.
gumby 9y ago
They are referring to Minsky and Papert's 1967 book "Perceptrons". Which said that a single layer NN can't be turing complete because it can't do XOR. The big 1990s breakthrough was having multilayer networks...which actually were also described in the Perceptrons book, though at the time computers weren't powerful enough to implement them.
- deleted 9y ago
  [deleted]
- blt 9y ago
  I don't understand how it took an entire book to show that a linear classifier can't learn the xor function.
  - ehudla 9y ago
    You should check out the book. It is really fun (especially the illustrations).
  - gumby 9y ago
    There’s more to the book than my one-sentence summary. Believe Minsky and Papert have something extensive to say on a subject it’s probably not shallow. There’s an ok, if brief discussion about the book’s influence on NN research in its Wikipedia entry. It’s not a super long book and an excellent, eye opening read even 50 years later.
chriskanan 9y ago
Definitely seemed like that in the 2000s when I was doing my PhD. Yann LeCun even wrote an open letter to the computer vision community around 2012 complaining about them automatically rejecting papers that used neural networks. I've been told that letter led to him launching the conference ICLR. Only a couple years later he was a keynote speaker at CVPR, the big computer vision conference.
olympus 9y ago
I wouldn't wouldn't go so far as to say "dismissed," but they was definitely a period or two when they weren't as shiny. When I did my M.S. reasearch in 2010 SVMs were far more popular than neural networks. Deep learning was just over the horizon and most people had given up on it being useful for learning real world problems. We had proven that NNs could learn anything given enough data and the right structure, but the hard part in real life was having enough data and finding the correct structure. So SVMs seemed more robust with available datasets, and most people used those. There were/are several other popular methods, like nearest neighbors and tree-based methods, and NNs just weren't as sexy as they seemed in the late 80's.
anonytrary 9y ago
As time goes on, higher frequency computers will increase the number of AI tasks which are possible in a fixed period of time. If you took half the apps built today and tried to run them on compatible hardware from 20 years ago, they would all flop. Without the understanding of modern computing, one would have dismissed all such inventions as useless wastes of time.

Most other scientists dismissed neural networks? Is there some history I am unaware of as that doesn't seem true. Did the article want to push the idea of the lone rogue thinker a bit too much?

ZenoArrow 9y ago

I thought this too. Neural networks definitely seemed like a credible research topic in the 90s at the very least (which is when I first became aware of them).

dang 9y ago

Maybe, but let's not get hung up on the title—that tends to keep discussion shallow. We'll change "most" to "many" above.

deleted 9y ago
[deleted]

deleted 9y ago

[deleted]

meow1032 9y ago

http://www.dataversity.net/brief-history-deep-learning/ http://www.dataversity.net/brief-history-deep-learning/ Neural Networks weren't really thought of very highly until CNNs started winning image recognition competitions in the early 2010's. I think most people had the feeling that they were interesting tools to learn how the brain worked, but too slow and opaque to be practical statistical tools. I've been following Hinton for a while (because of Hinton and Shallice 1991), and my understanding is that it was really hard for him to get funding especially when he was just starting out. The fact that so much of the work from the mid-80's to 2000 came from just a few labs should tell you how hard it was to get funding for that kind of research.

_delirium 9y ago
> The fact that so much of the work from the mid-80's to 2000 came from just a few labs should tell you how hard it was to get funding for that kind of research. If you look through papers from that era I don't think you'll find that's true at all. You could fairly say that only a few of the prominent labs from the first neural-net boom lasted long enough to still be prominent labs now in the second neural-net boom (though even then there are several: Hinton, Bengio, Schmidhuber, LeCun, etc.). Since they're still around doing interviews and putting out new papers, understandably their work has a higher profile now than that of people who aren't in the field anymore. But there have been a ton of others over the years too, just many of them from the first wave have moved on or retired by now. Especially in the '90s the field was hot and reasonably large (and it was pretty easy to get funding, too). If you look at e.g. the NIPS 1992 proceedings, it's definitely not just a handful of labs: https://papers.nips.cc/book/advances-in-neural-information-processing-systems-5-1992 https://papers.nips.cc/book/advances-in-neural-information-p...
- meow1032 9y ago
  >You could fairly say that only a few of the prominent labs from the first neural-net boom lasted long enough to still be prominent labs now in the second neural-net boom (though even then there are several: Hinton, Bengio, Schmidhuber, LeCun, etc.) I agree, I don't mean to imply that Hinton did all of the work on NN's, but he certainly contributed a lot to the field, even as the popularity waxed and waned. Even if you just click on a few of the names in your link, you can see most of those authors submitted ~1-6 papers. Hinton, Bengio, and Jordan have submitted >50 each. I wouldn't say that NN's were exactly thought of as a joke before 2011, but from the people I talked to about them, they weren't considered very promising as practical statistical systems.

gumby 9y ago

They are referring to Minsky and Papert's 1967 book "Perceptrons". Which said that a single layer NN can't be turing complete because it can't do XOR. The big 1990s breakthrough was having multilayer networks...which actually were also described in the Perceptrons book, though at the time computers weren't powerful enough to implement them.

deleted 9y ago
[deleted]
blt 9y ago
I don't understand how it took an entire book to show that a linear classifier can't learn the xor function.
- ehudla 9y ago
  You should check out the book. It is really fun (especially the illustrations).
- gumby 9y ago
  There’s more to the book than my one-sentence summary. Believe Minsky and Papert have something extensive to say on a subject it’s probably not shallow. There’s an ok, if brief discussion about the book’s influence on NN research in its Wikipedia entry. It’s not a super long book and an excellent, eye opening read even 50 years later.

chriskanan 9y ago

Definitely seemed like that in the 2000s when I was doing my PhD. Yann LeCun even wrote an open letter to the computer vision community around 2012 complaining about them automatically rejecting papers that used neural networks. I've been told that letter led to him launching the conference ICLR. Only a couple years later he was a keynote speaker at CVPR, the big computer vision conference.

olympus 9y ago

I wouldn't wouldn't go so far as to say "dismissed," but they was definitely a period or two when they weren't as shiny. When I did my M.S. reasearch in 2010 SVMs were far more popular than neural networks. Deep learning was just over the horizon and most people had given up on it being useful for learning real world problems. We had proven that NNs could learn anything given enough data and the right structure, but the hard part in real life was having enough data and finding the correct structure. So SVMs seemed more robust with available datasets, and most people used those. There were/are several other popular methods, like nearest neighbors and tree-based methods, and NNs just weren't as sexy as they seemed in the late 80's.

anonytrary 9y ago

As time goes on, higher frequency computers will increase the number of AI tasks which are possible in a fixed period of time. If you took half the apps built today and tried to run them on compatible hardware from 20 years ago, they would all flop. Without the understanding of modern computing, one would have dismissed all such inventions as useless wastes of time.