7 ms·

Good article, but I still don't understand why they're suddenly popular again. So processing is faster, but is there some development in processing which has im

by anjc 12y ago

Good article, but I still don't understand why they're suddenly popular again. So processing is faster, but is there some development in processing which has improved this domain especially? Developments in "big data" processing? Concepts like mapreduce? What's with the resurgence :S

discardorama 12y ago
There are a myriad reasons why they're popular again. 1. Hardware has caught up, and is cheap. When Backprop was invented back in the 80s, you couldn't train networks with more than a couple of 1000 nodes tops. Today, with GPUs, you can train networks with billions of parameters. 2. More data is available. Back in those days, you had a few dozens (maybe a few 100s) of examples in your training set. Today, people play with sets larges than 1TB. 3. Dramatic successes. For a while, the ImageNet competition was seeing slow and stead progress. Then DL comes along, and there's a 20% jump in performance (I'm too lazy to look up the exact numbers...). If you've ever competed in such competitions, progress is painfully slow (see, for example, the Netflix competition). So a jump of that magnitude in performance in 1 step is mind-blowing. On top of that, every year since then, the performance has increased significantly. These are just 3 that come to mind.
- deleted 12y ago
  [deleted]
nharada 12y ago
Deep neural networks are quite powerful because they don't require someone to design the network specific to the task. Large deep nets will generally be able to learn feature representations by themselves, which makes them extremely powerful for tasks where we aren't quite sure of exactly what features we want. However, deep nets require large amounts of training data and computational power. The fairly recent widespread adoption of general purpose GPUs has allowed much faster training. Combine this with the popularity of "big data", and you've got a perfect storm for deep neural nets. Of course, the hype may be overvaluing deep nets as the future of AI. DNNs work well in practical applications, but they're poorly defined theoretically and the AI community suspects that we're still bad at training them -- a recent paper showed that a simpler shallow net can perform as well as a deep net if a deep net is trained first[1]. We're also fairly certain that deep nets are not how the brain actually works, and thus we'll need a different architecture in order to achieve human level performance on some tasks. [1] http://arxiv.org/abs/1312.6184 http://arxiv.org/abs/1312.6184
deleted 12y ago
[deleted]
chrdlu 12y ago
I feel like cloud technology may be a contributing factor as well. At the same time, once Geoffrey Hinton used a deep neural network and participated in the ImageNet contest (2012: http://www.image-net.org/challenges/LSVRC/2012/results.html http://www.image-net.org/challenges/LSVRC/2012/results.html), his results beat the next best thing by a full 10%. The results were so astounding that many people immediately began re-visiting neural networks. Shortly afterwards, people proved it could beat the current technology for language processing and more. Now a days, it seems like a major leap has been in real-time translation with Skype and now Google launching machine translation applications/functions. Side note, in my opinion, start-ups that are looking to compete with large giants like Google will have a pretty hard time. In the end, implementing deep neural nets that work is still extremely hard. The companies that do it right usually get bought up by one of the giants. Google has some of the leading researchers in academia on its side as well.
- nightski 12y ago
  But don't kid yourself, if you have ever been to CVPR, or one of the machine learning conferences you see all of the papers trying to squeeze a few more % accuracy on a test set out of existing algorithms. Maybe one or two papers actually do something novel. Rarely will their be a new approach altogether. The point I am trying to make is that one shouldn't get star eyed by leading researchers in the field and assume they can't contribute. Simple novel ideas have led to massive changes in the industry.
  - chrdlu 12y ago
    Fair point, Alex Krizhevsky did train the entire neural network on 2 GPUs in his bedroom. I stand corrected
hiddencost 12y ago
Hinton published a much more efficient training algoritm in the last couple decades. They've also shown extremely impressive performance in vision and speech.
hyperbovine 12y ago
They are adept at storing large numbers of patterns and therefore excel at many supervised learning tasks. There indeed was a development in processing, namely the emergence of the GPU, which led to the resurgence. It enabled people to train far larger networks. Large networks = more patterns.

by anjc 12y ago

discardorama 12y ago

There are a myriad reasons why they're popular again. 1. Hardware has caught up, and is cheap. When Backprop was invented back in the 80s, you couldn't train networks with more than a couple of 1000 nodes tops. Today, with GPUs, you can train networks with billions of parameters. 2. More data is available. Back in those days, you had a few dozens (maybe a few 100s) of examples in your training set. Today, people play with sets larges than 1TB. 3. Dramatic successes. For a while, the ImageNet competition was seeing slow and stead progress. Then DL comes along, and there's a 20% jump in performance (I'm too lazy to look up the exact numbers...). If you've ever competed in such competitions, progress is painfully slow (see, for example, the Netflix competition). So a jump of that magnitude in performance in 1 step is mind-blowing. On top of that, every year since then, the performance has increased significantly. These are just 3 that come to mind.

deleted 12y ago
[deleted]

nharada 12y ago

Deep neural networks are quite powerful because they don't require someone to design the network specific to the task. Large deep nets will generally be able to learn feature representations by themselves, which makes them extremely powerful for tasks where we aren't quite sure of exactly what features we want. However, deep nets require large amounts of training data and computational power. The fairly recent widespread adoption of general purpose GPUs has allowed much faster training. Combine this with the popularity of "big data", and you've got a perfect storm for deep neural nets. Of course, the hype may be overvaluing deep nets as the future of AI. DNNs work well in practical applications, but they're poorly defined theoretically and the AI community suspects that we're still bad at training them -- a recent paper showed that a simpler shallow net can perform as well as a deep net if a deep net is trained first[1]. We're also fairly certain that deep nets are not how the brain actually works, and thus we'll need a different architecture in order to achieve human level performance on some tasks. [1] http://arxiv.org/abs/1312.6184 http://arxiv.org/abs/1312.6184

deleted 12y ago

[deleted]

chrdlu 12y ago

I feel like cloud technology may be a contributing factor as well. At the same time, once Geoffrey Hinton used a deep neural network and participated in the ImageNet contest (2012: http://www.image-net.org/challenges/LSVRC/2012/results.html http://www.image-net.org/challenges/LSVRC/2012/results.html), his results beat the next best thing by a full 10%. The results were so astounding that many people immediately began re-visiting neural networks. Shortly afterwards, people proved it could beat the current technology for language processing and more. Now a days, it seems like a major leap has been in real-time translation with Skype and now Google launching machine translation applications/functions. Side note, in my opinion, start-ups that are looking to compete with large giants like Google will have a pretty hard time. In the end, implementing deep neural nets that work is still extremely hard. The companies that do it right usually get bought up by one of the giants. Google has some of the leading researchers in academia on its side as well.

nightski 12y ago
But don't kid yourself, if you have ever been to CVPR, or one of the machine learning conferences you see all of the papers trying to squeeze a few more % accuracy on a test set out of existing algorithms. Maybe one or two papers actually do something novel. Rarely will their be a new approach altogether. The point I am trying to make is that one shouldn't get star eyed by leading researchers in the field and assume they can't contribute. Simple novel ideas have led to massive changes in the industry.
- chrdlu 12y ago
  Fair point, Alex Krizhevsky did train the entire neural network on 2 GPUs in his bedroom. I stand corrected

hiddencost 12y ago

Hinton published a much more efficient training algoritm in the last couple decades. They've also shown extremely impressive performance in vision and speech.

hyperbovine 12y ago

They are adept at storing large numbers of patterns and therefore excel at many supervised learning tasks. There indeed was a development in processing, namely the emergence of the GPU, which led to the resurgence. It enabled people to train far larger networks. Large networks = more patterns.