4 ms·

This is a very welcome read but it hypes neural networks a bit. I've been working with in JavaScript using IndexedDB and while researching I was disappointed to

This is a very welcome read but it hypes neural networks a bit. I've been working with in JavaScript using IndexedDB and while researching I was disappointed to find that some smart people seem to think they are much more limited than made out to be here and elsewhere https://www.youtube.com/watch?v=AyzOUbkUf3M#t=242 https://www.youtube.com/watch?v=AyzOUbkUf3M#t=242 To summarize, people generally abandoned backpropigation trained neural networks for Support Vector Machines because neural nets require labeled and limited datasets, and work slowly and especially so when dealing with multiple layers which is sort of the whole point. In my work in JavaScript, I was able to pull off only a single layer perceptron and it is neat but limited in what it can model.

ma2rten 12y ago
You are linking to a talk by Geoffrey Hinton. A strong advocate for Neural Networks. A single layer perceptron is limited, but a Neural Network with hidden layer(s) is a very powerful tool, and we have now figured out how to deal with multiple hidden layers (given enough data).
jbarrow 12y ago
It's true that the beginning of the talk is about how people abandoned backprop trained neural networks because they often underperformed SVMs, but the rest of the talk is about deep learning, which introduced a new generation of neural nets (since 2006) that are the current state of the art for a lot of problems. In fact, deep neural networks are trained in an unsupervised manner at first, but then back propagation is used to "fine tune" and improve the results. Because they require unlabeled data sets, and can perform so well, research into neural networks has experienced a recent resurgence. By the way, any talk by Geoff Hinton is fantastic. If you are interested in neural networks and their capabilities, and you haven't already seen it, his Coursera course [1] builds from a simple linear perceptron to the current deep learning methods. [1] https://class.coursera.org/neuralnets-2012-001 https://class.coursera.org/neuralnets-2012-001 (You'll have to sign in to see it)
- agibsonccc 12y ago
  I want to add to this, that there's a lot of work that doesn't require pretraining. I recently implemented the more advanced hessian free optimization methods that don't require pretraining in my deep learning framework. The results are amazing. I'm hoping to demonstrate a lot of the tradeoffs of the different methods in a more comprehensive manner here shortly. This was an updated extension by some of hinton's students. The paper I implemented was: http://www.cs.toronto.edu/~rkiros/papers/shf13.pdf http://www.cs.toronto.edu/~rkiros/papers/shf13.pdf
  - taylorbuley 12y ago
    Thanks for the extra resources. Any more are very welcome. Neural nets sometimes feel like quite the black box, despite their ease of implementation and apparent power.
agibsonccc 12y ago
Frankly, the reason there's hype around neural nets again is because of the newer ways they can be trained to augment backprop. Neural nets have made a lot of progress in representation learning in recent years. A major problem that I'm hoping to fix is having the industry catch up to what academia and some of the bigger companies are doing with nets now. Neural nets are from a silver bullet, and shouldn't be used where feature introspection is a huge requirement (this is why decision tree/random forest is popular), but they are far from being what they were in the 90s. Note that I have a commercial interest in this so there's going to be inherent bias in my opinions. To be fair, javascript isn't a scientific computing language. To do most neat training with neural nets, you're going to want to either scale them out, add more layers, and/or use GPUs. That being said, a neat toy example in javascript is convnetjs[1]. [1]: https://github.com/karpathy/convnetjs https://github.com/karpathy/convnetjs

ma2rten 12y ago

You are linking to a talk by Geoffrey Hinton. A strong advocate for Neural Networks. A single layer perceptron is limited, but a Neural Network with hidden layer(s) is a very powerful tool, and we have now figured out how to deal with multiple hidden layers (given enough data).

jbarrow 12y ago

It's true that the beginning of the talk is about how people abandoned backprop trained neural networks because they often underperformed SVMs, but the rest of the talk is about deep learning, which introduced a new generation of neural nets (since 2006) that are the current state of the art for a lot of problems. In fact, deep neural networks are trained in an unsupervised manner at first, but then back propagation is used to "fine tune" and improve the results. Because they require unlabeled data sets, and can perform so well, research into neural networks has experienced a recent resurgence. By the way, any talk by Geoff Hinton is fantastic. If you are interested in neural networks and their capabilities, and you haven't already seen it, his Coursera course [1] builds from a simple linear perceptron to the current deep learning methods. [1] https://class.coursera.org/neuralnets-2012-001 https://class.coursera.org/neuralnets-2012-001 (You'll have to sign in to see it)

agibsonccc 12y ago
I want to add to this, that there's a lot of work that doesn't require pretraining. I recently implemented the more advanced hessian free optimization methods that don't require pretraining in my deep learning framework. The results are amazing. I'm hoping to demonstrate a lot of the tradeoffs of the different methods in a more comprehensive manner here shortly. This was an updated extension by some of hinton's students. The paper I implemented was: http://www.cs.toronto.edu/~rkiros/papers/shf13.pdf http://www.cs.toronto.edu/~rkiros/papers/shf13.pdf
- taylorbuley 12y ago
  Thanks for the extra resources. Any more are very welcome. Neural nets sometimes feel like quite the black box, despite their ease of implementation and apparent power.

agibsonccc 12y ago

Frankly, the reason there's hype around neural nets again is because of the newer ways they can be trained to augment backprop. Neural nets have made a lot of progress in representation learning in recent years. A major problem that I'm hoping to fix is having the industry catch up to what academia and some of the bigger companies are doing with nets now. Neural nets are from a silver bullet, and shouldn't be used where feature introspection is a huge requirement (this is why decision tree/random forest is popular), but they are far from being what they were in the 90s. Note that I have a commercial interest in this so there's going to be inherent bias in my opinions. To be fair, javascript isn't a scientific computing language. To do most neat training with neural nets, you're going to want to either scale them out, add more layers, and/or use GPUs. That being said, a neat toy example in javascript is convnetjs[1]. [1]: https://github.com/karpathy/convnetjs https://github.com/karpathy/convnetjs