4 ms·

I don't know if this the title is right. Beats human in ... what, the speed of neural nets? (In image recognition, by the way, as the paper explains.) I also do

I don't know if this the title is right. Beats human in ... what, the speed of neural nets? (In image recognition, by the way, as the paper explains.) I also don't exactly understand what is the breakthrough here -- there are plenty papers, hardware, etc. that speed up some process significantly. 10x speedup won't bring a huge breakthrough in deep learning in general.

chromakode 12y ago
It accelerates training. From the conclusion: "We have presented a novel mechanism for dramatically accelerating the training of deep networks"
snackyman123 12y ago
It looks like significantly better training accuracy is achieved in a small fraction of the number of training steps previously required. ~14x speedup in training is a big deal; that speedup would enable training many more models (trying new research ideas) in a given period of time.
dang 12y ago
We changed the title from "Googles breakthrough paper shows 10*faster neural nets[pdf]".
karpathy 12y ago
10x speedup in training is a huge deal since modern nets take 2-3 weeks to train on 4 GPUs. Note that the speedup is not actually in the forward-backward raw speed of the computation in the network. The speedup here is a more clever form of the forward function, which makes it converge faster as you train it. In fact, they have made the network slower in this paper because they added the normalization computation into it. At test time (once training is finished), they are very efficient though, on orders of milliseconds per image.
- bradneuberg 12y ago
  I'm looking forward to people applying this result to Recurrent Neural Networks for speech recognition tasks. What do you think?
- mdda 12y ago
  (responding here because your comment addresses the actual new content of the paper). Four things pop out at me from the paper: 1) The whitening (per batch) & rescaling (overall) is a neat new idea. But (as referred to in their p5 comment about the bias term being subsumed) this also points to the idea that the (Wu+b) transformation probably has a better-for-learning 'factorization', since their un-scale/re-scale operation on (Wu+b) is mainly taking out such a factor (while also putting in the minibatch accumulation change). 2) The idea that this could replace Dropout as the go-to trick for speeding up learning is pretty worrying (IMHO), since the gains from Dropout seem to be in a 'meta network' direction, rather than a data-dependency direction. Both approaches seem well worth understanding more thoroughly, even though the 'do what works' ML approach might favour leaving Dropout behind. 3) The publication of this paper, so closely behind the new ReLu+ results from Microsoft, seems too coincidental. One has to wonder what other results each of the companies has in their back-pockets so that they can repeatedly steal the crown from each other. 4) For me, the application to MNIST is attention-grabbing enough. While I appreciate that playing with Inception (etc) sexes-up the paper a lot, it raises the hurdle for others who may not have that quantity of hardware to contribute to the (more interesting) project of improving the learning rates of all projects (which is quite possible to do on the MNIST dataset, except that it's pretty much 'solved' with the error cases being pretty questionable for humans too).
vonnik 12y ago
People would kill for a 10x speed up in training. You have no idea what you're talking about. How well a neural net performs is partly based on how much data it trains on. If you are able to train on 10x as much data in the same period, you're going to have much better nets. This is very important when you start trying to implement real-time solutions.

chromakode 12y ago

It accelerates training. From the conclusion: "We have presented a novel mechanism for dramatically accelerating the training of deep networks"

snackyman123 12y ago

It looks like significantly better training accuracy is achieved in a small fraction of the number of training steps previously required. ~14x speedup in training is a big deal; that speedup would enable training many more models (trying new research ideas) in a given period of time.

dang 12y ago

We changed the title from "Googles breakthrough paper shows 10*faster neural nets[pdf]".

karpathy 12y ago

10x speedup in training is a huge deal since modern nets take 2-3 weeks to train on 4 GPUs. Note that the speedup is not actually in the forward-backward raw speed of the computation in the network. The speedup here is a more clever form of the forward function, which makes it converge faster as you train it. In fact, they have made the network slower in this paper because they added the normalization computation into it. At test time (once training is finished), they are very efficient though, on orders of milliseconds per image.

bradneuberg 12y ago
I'm looking forward to people applying this result to Recurrent Neural Networks for speech recognition tasks. What do you think?
mdda 12y ago
(responding here because your comment addresses the actual new content of the paper). Four things pop out at me from the paper: 1) The whitening (per batch) & rescaling (overall) is a neat new idea. But (as referred to in their p5 comment about the bias term being subsumed) this also points to the idea that the (Wu+b) transformation probably has a better-for-learning 'factorization', since their un-scale/re-scale operation on (Wu+b) is mainly taking out such a factor (while also putting in the minibatch accumulation change). 2) The idea that this could replace Dropout as the go-to trick for speeding up learning is pretty worrying (IMHO), since the gains from Dropout seem to be in a 'meta network' direction, rather than a data-dependency direction. Both approaches seem well worth understanding more thoroughly, even though the 'do what works' ML approach might favour leaving Dropout behind. 3) The publication of this paper, so closely behind the new ReLu+ results from Microsoft, seems too coincidental. One has to wonder what other results each of the companies has in their back-pockets so that they can repeatedly steal the crown from each other. 4) For me, the application to MNIST is attention-grabbing enough. While I appreciate that playing with Inception (etc) sexes-up the paper a lot, it raises the hurdle for others who may not have that quantity of hardware to contribute to the (more interesting) project of improving the learning rates of all projects (which is quite possible to do on the MNIST dataset, except that it's pretty much 'solved' with the error cases being pretty questionable for humans too).

vonnik 12y ago

People would kill for a 10x speed up in training. You have no idea what you're talking about. How well a neural net performs is partly based on how much data it trains on. If you are able to train on 10x as much data in the same period, you're going to have much better nets. This is very important when you start trying to implement real-time solutions.