11 ms·

Batch Normalization: Accelerating Deep Network Training [pdf]

134 points by singularity2001 12y ago

pipeep 12y ago
I get that this is preprint, but the reference on page 8 has an overfull hbox with a rather high badness.
sz4kerto 12y ago
I don't know if this the title is right. Beats human in ... what, the speed of neural nets? (In image recognition, by the way, as the paper explains.) I also don't exactly understand what is the breakthrough here -- there are plenty papers, hardware, etc. that speed up some process significantly. 10x speedup won't bring a huge breakthrough in deep learning in general.
- chromakode 12y ago
  It accelerates training. From the conclusion: "We have presented a novel mechanism for dramatically accelerating the training of deep networks"
- snackyman123 12y ago
  It looks like significantly better training accuracy is achieved in a small fraction of the number of training steps previously required. ~14x speedup in training is a big deal; that speedup would enable training many more models (trying new research ideas) in a given period of time.
- dang 12y ago
  We changed the title from "Googles breakthrough paper shows 10*faster neural nets[pdf]".
- karpathy 12y ago
  10x speedup in training is a huge deal since modern nets take 2-3 weeks to train on 4 GPUs. Note that the speedup is not actually in the forward-backward raw speed of the computation in the network. The speedup here is a more clever form of the forward function, which makes it converge faster as you train it. In fact, they have made the network slower in this paper because they added the normalization computation into it. At test time (once training is finished), they are very efficient though, on orders of milliseconds per image.
  - bradneuberg 12y ago
    I'm looking forward to people applying this result to Recurrent Neural Networks for speech recognition tasks. What do you think?
  - mdda 12y ago
    (responding here because your comment addresses the actual new content of the paper). Four things pop out at me from the paper: 1) The whitening (per batch) & rescaling (overall) is a neat new idea. But (as referred to in their p5 comment about the bias term being subsumed) this also points to the idea that the (Wu+b) transformation probably has a better-for-learning 'factorization', since their un-scale/re-scale operation on (Wu+b) is mainly taking out such a factor (while also putting in the minibatch accumulation change). 2) The idea that this could replace Dropout as the go-to trick for speeding up learning is pretty worrying (IMHO), since the gains from Dropout seem to be in a 'meta network' direction, rather than a data-dependency direction. Both approaches seem well worth understanding more thoroughly, even though the 'do what works' ML approach might favour leaving Dropout behind. 3) The publication of this paper, so closely behind the new ReLu+ results from Microsoft, seems too coincidental. One has to wonder what other results each of the companies has in their back-pockets so that they can repeatedly steal the crown from each other. 4) For me, the application to MNIST is attention-grabbing enough. While I appreciate that playing with Inception (etc) sexes-up the paper a lot, it raises the hurdle for others who may not have that quantity of hardware to contribute to the (more interesting) project of improving the learning rates of all projects (which is quite possible to do on the MNIST dataset, except that it's pretty much 'solved' with the error cases being pretty questionable for humans too).
- vonnik 12y ago
  People would kill for a 10x speed up in training. You have no idea what you're talking about. How well a neural net performs is partly based on how much data it trains on. If you are able to train on 10x as much data in the same period, you're going to have much better nets. This is very important when you start trying to implement real-time solutions.
hyperbovine 12y ago
The phrase "exceeding the accuracy of human raters" is a bit puzzling--if a human misclassifies something whose true class was determined by another human, who is in error?
- dchichkov 12y ago
  Probably exceeding the accuracy of a "single human rater". Note, the definition of a "human rater accuracy" is a bit vague. For example an "Amazon Turk rater" qualifies as a human rater, yet the accuracy of such rater can be easily exceeded by an accuracy of dedicated grad student rater, or an accuracy of a dedicated H1 employee rater. edit: That's my guess from previous (2.5 years back) work in computer vision / imagenet and a quick skim over the article.
- davmre 12y ago
  The comparison is to humans that see the same data as the algorithm. The "true" labels can be assigned using additional data. For example, if I make a dataset for the task of distinguishing cocker spaniels from springer spaniels, then when I collect the pictures I can be quite confident about the labels I assign (eg by DNA-testing the actual dogs). But when someone else tries to identify the dogs based only on the pictures I took, they might get some of them wrong. The 5.1% "human accuracy" number that gets thrown around for ImageNet comes from Andrej Karpathy attempting to manually label part of the ImageNet test set. He wrote a blog post about the experience (http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/ http://karpathy.github.io/2014/09/02/what-i-learned-from-com...), which has some interesting details. For example, they did actually find that 1.7% of images were genuinely misannotated: that's because ImageNet annotations come from a consensus of Mechanical Turkers, which is less reliable than DNA testing. :-)
- karpathy 12y ago
  The TLDR is that to work at scale we sacrifice some correctness in the dataset, and what's really being evaluated here is closer to some notion of human agreement. The way the data was collected was not through a 1000-way classification. Instead, we'd query things like "red fox" from a search engine, and then have turkers do the binary task of YES/NO to clean up the results. This is a messy, noisy process but we do our best. In particular, the turkers saying YES/NO to red fox are completely unaware of what other classes are in the dataset - everything they say YES to becomes the "red fox" class in imagenet, in the final discriminative task among the 1000 categories. Also note that if an image contains a whole bunch of things (e.g. see left-most image here http://karpathy.github.io/assets/ilsvrc2.png http://karpathy.github.io/assets/ilsvrc2.png), the turker during data collection would be correct in saying that there is in fact, e.g. a "ruler" in that image, but at test time when you're seeing everything in that image, you only have 5 possible predictions so you have to guess about what you think the "true" label is.
singularity2001 12y ago
PS: The human being beaten was a guy from Stanford who did the validation set for this specific image recognition task a while ago (he somewhere said that he could reduce the error rate to 3% given a second chance ;) This same week Microsoft released a similar result, but without such an important new approach for networks. The important chart really is on page number seven.
- karpathy 12y ago
  You're referring to my google+ post from yesterday :) https://plus.google.com/+AndrejKarpathy/posts/dwDNcBuWTWf https://plus.google.com/+AndrejKarpathy/posts/dwDNcBuWTWf The short story is that I tried for 500 images and then got 5.1% error on the test set. Looking at my mistakes, I then tried to differentiate two sources of error: a genuine problem with the dataset (e.g. many correct answers in an image, or incorrect label), and errors I felt could be eliminated by an ensemble of very committed humans who were even better than me at classifying dogs :) And that optimistic error rate is approx 3%.
freddealmeida 12y ago
I find it fascinating that deep learning research is no.1 on HN. Not even 6months ago it was nearly unheard of on the top 3-4 pages. Is this an indicator that machine intelligence is the new normal?
- sz4kerto 12y ago
  Nah. Since neural networks are called deep learning it became very fashionable. There's some well-funded startup (SkyMind maybe?) who declares on their homepage that deep learning is sooo much better than machine learning (first wtf). Then they explain that neural nets with less than 3 hidden layers is machine learning, with more than 3 layers it's deep learning.(second) I didn't know if I should laugh or cry.(Not that what they actually seem to be doing is bad, but this newly found hype is just wrong.)
  - mbq 12y ago
    Meh, people just like simple solutions (or at least those which sound simple).
  - deleted 12y ago
    [deleted]
  - agibsonccc 12y ago
    Adam (founder of skymind here). I'd like to say that we don't encourage neural networks as a black box like a lot of the startups out there. I also mainly advocate neural networks on unstructured data where the results are proven to be a significant improvement over other techniques. Many startups in the space would believe that you can have a simple GUI and you're somehow set to go. In my upcoming oreilly book Deep Learning: A Practioner's Approach, I go through a practical applications oriented view of neural networks that I think will help change things in the coming years. I'd also add that I've implemented every neural network architecture and allow people to mix and match. A significant result that many people are familiar with are the image scene description generators done by karpathy et.al[1]. Either way, unlike most our code is open source :). I don't claim things are ideal or perfect, but it's out there for you to play with. I focus on integrations and packaging and providing real value rather than pretending that some algorithm is going to be my edge. Many startups in the space will tell you they have awesome algorithms that are cutting edge when in reality you should be providing a solution for people. [1]: https://github.com/karpathy/neuraltalk/tree/master/imagernn https://github.com/karpathy/neuraltalk/tree/master/imagernn
deleted 12y ago
[deleted]
lars 12y ago
Since this paper along with [1] both beat human level performance on ImageNet, using seemingly unrelated techniques, it seems like there's a clear path to achieve the next state of the art result: Implement and train a network that uses both Batch Normalization and PReLU's at the same time. Right? 1: http://arxiv.org/pdf/1502.01852v1.pdf http://arxiv.org/pdf/1502.01852v1.pdf
- dougabug 12y ago
  Exactly my thoughts. Performance of both incremental improvements seems similar, would be interesting to see what the combined effect would be. It's pretty clear that they're reaching the limitations of the data set, though.

I get that this is preprint, but the reference on page 8 has an overfull hbox with a rather high badness.

I don't know if this the title is right. Beats human in ... what, the speed of neural nets? (In image recognition, by the way, as the paper explains.) I also don't exactly understand what is the breakthrough here -- there are plenty papers, hardware, etc. that speed up some process significantly. 10x speedup won't bring a huge breakthrough in deep learning in general.

chromakode 12y ago
It accelerates training. From the conclusion: "We have presented a novel mechanism for dramatically accelerating the training of deep networks"
snackyman123 12y ago
It looks like significantly better training accuracy is achieved in a small fraction of the number of training steps previously required. ~14x speedup in training is a big deal; that speedup would enable training many more models (trying new research ideas) in a given period of time.
dang 12y ago
We changed the title from "Googles breakthrough paper shows 10*faster neural nets[pdf]".
karpathy 12y ago
10x speedup in training is a huge deal since modern nets take 2-3 weeks to train on 4 GPUs. Note that the speedup is not actually in the forward-backward raw speed of the computation in the network. The speedup here is a more clever form of the forward function, which makes it converge faster as you train it. In fact, they have made the network slower in this paper because they added the normalization computation into it. At test time (once training is finished), they are very efficient though, on orders of milliseconds per image.
- bradneuberg 12y ago
  I'm looking forward to people applying this result to Recurrent Neural Networks for speech recognition tasks. What do you think?
- mdda 12y ago
  (responding here because your comment addresses the actual new content of the paper). Four things pop out at me from the paper: 1) The whitening (per batch) & rescaling (overall) is a neat new idea. But (as referred to in their p5 comment about the bias term being subsumed) this also points to the idea that the (Wu+b) transformation probably has a better-for-learning 'factorization', since their un-scale/re-scale operation on (Wu+b) is mainly taking out such a factor (while also putting in the minibatch accumulation change). 2) The idea that this could replace Dropout as the go-to trick for speeding up learning is pretty worrying (IMHO), since the gains from Dropout seem to be in a 'meta network' direction, rather than a data-dependency direction. Both approaches seem well worth understanding more thoroughly, even though the 'do what works' ML approach might favour leaving Dropout behind. 3) The publication of this paper, so closely behind the new ReLu+ results from Microsoft, seems too coincidental. One has to wonder what other results each of the companies has in their back-pockets so that they can repeatedly steal the crown from each other. 4) For me, the application to MNIST is attention-grabbing enough. While I appreciate that playing with Inception (etc) sexes-up the paper a lot, it raises the hurdle for others who may not have that quantity of hardware to contribute to the (more interesting) project of improving the learning rates of all projects (which is quite possible to do on the MNIST dataset, except that it's pretty much 'solved' with the error cases being pretty questionable for humans too).
vonnik 12y ago
People would kill for a 10x speed up in training. You have no idea what you're talking about. How well a neural net performs is partly based on how much data it trains on. If you are able to train on 10x as much data in the same period, you're going to have much better nets. This is very important when you start trying to implement real-time solutions.

hyperbovine 12y ago

The phrase "exceeding the accuracy of human raters" is a bit puzzling--if a human misclassifies something whose true class was determined by another human, who is in error?

dchichkov 12y ago
Probably exceeding the accuracy of a "single human rater". Note, the definition of a "human rater accuracy" is a bit vague. For example an "Amazon Turk rater" qualifies as a human rater, yet the accuracy of such rater can be easily exceeded by an accuracy of dedicated grad student rater, or an accuracy of a dedicated H1 employee rater. edit: That's my guess from previous (2.5 years back) work in computer vision / imagenet and a quick skim over the article.
davmre 12y ago
The comparison is to humans that see the same data as the algorithm. The "true" labels can be assigned using additional data. For example, if I make a dataset for the task of distinguishing cocker spaniels from springer spaniels, then when I collect the pictures I can be quite confident about the labels I assign (eg by DNA-testing the actual dogs). But when someone else tries to identify the dogs based only on the pictures I took, they might get some of them wrong. The 5.1% "human accuracy" number that gets thrown around for ImageNet comes from Andrej Karpathy attempting to manually label part of the ImageNet test set. He wrote a blog post about the experience (http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/ http://karpathy.github.io/2014/09/02/what-i-learned-from-com...), which has some interesting details. For example, they did actually find that 1.7% of images were genuinely misannotated: that's because ImageNet annotations come from a consensus of Mechanical Turkers, which is less reliable than DNA testing. :-)
karpathy 12y ago
The TLDR is that to work at scale we sacrifice some correctness in the dataset, and what's really being evaluated here is closer to some notion of human agreement. The way the data was collected was not through a 1000-way classification. Instead, we'd query things like "red fox" from a search engine, and then have turkers do the binary task of YES/NO to clean up the results. This is a messy, noisy process but we do our best. In particular, the turkers saying YES/NO to red fox are completely unaware of what other classes are in the dataset - everything they say YES to becomes the "red fox" class in imagenet, in the final discriminative task among the 1000 categories. Also note that if an image contains a whole bunch of things (e.g. see left-most image here http://karpathy.github.io/assets/ilsvrc2.png http://karpathy.github.io/assets/ilsvrc2.png), the turker during data collection would be correct in saying that there is in fact, e.g. a "ruler" in that image, but at test time when you're seeing everything in that image, you only have 5 possible predictions so you have to guess about what you think the "true" label is.

singularity2001 12y ago

PS: The human being beaten was a guy from Stanford who did the validation set for this specific image recognition task a while ago (he somewhere said that he could reduce the error rate to 3% given a second chance ;) This same week Microsoft released a similar result, but without such an important new approach for networks. The important chart really is on page number seven.

karpathy 12y ago
You're referring to my google+ post from yesterday :) https://plus.google.com/+AndrejKarpathy/posts/dwDNcBuWTWf https://plus.google.com/+AndrejKarpathy/posts/dwDNcBuWTWf The short story is that I tried for 500 images and then got 5.1% error on the test set. Looking at my mistakes, I then tried to differentiate two sources of error: a genuine problem with the dataset (e.g. many correct answers in an image, or incorrect label), and errors I felt could be eliminated by an ensemble of very committed humans who were even better than me at classifying dogs :) And that optimistic error rate is approx 3%.

freddealmeida 12y ago

I find it fascinating that deep learning research is no.1 on HN. Not even 6months ago it was nearly unheard of on the top 3-4 pages. Is this an indicator that machine intelligence is the new normal?

sz4kerto 12y ago
Nah. Since neural networks are called deep learning it became very fashionable. There's some well-funded startup (SkyMind maybe?) who declares on their homepage that deep learning is sooo much better than machine learning (first wtf). Then they explain that neural nets with less than 3 hidden layers is machine learning, with more than 3 layers it's deep learning.(second) I didn't know if I should laugh or cry.(Not that what they actually seem to be doing is bad, but this newly found hype is just wrong.)
- mbq 12y ago
  Meh, people just like simple solutions (or at least those which sound simple).
- deleted 12y ago
  [deleted]
- agibsonccc 12y ago
  Adam (founder of skymind here). I'd like to say that we don't encourage neural networks as a black box like a lot of the startups out there. I also mainly advocate neural networks on unstructured data where the results are proven to be a significant improvement over other techniques. Many startups in the space would believe that you can have a simple GUI and you're somehow set to go. In my upcoming oreilly book Deep Learning: A Practioner's Approach, I go through a practical applications oriented view of neural networks that I think will help change things in the coming years. I'd also add that I've implemented every neural network architecture and allow people to mix and match. A significant result that many people are familiar with are the image scene description generators done by karpathy et.al[1]. Either way, unlike most our code is open source :). I don't claim things are ideal or perfect, but it's out there for you to play with. I focus on integrations and packaging and providing real value rather than pretending that some algorithm is going to be my edge. Many startups in the space will tell you they have awesome algorithms that are cutting edge when in reality you should be providing a solution for people. [1]: https://github.com/karpathy/neuraltalk/tree/master/imagernn https://github.com/karpathy/neuraltalk/tree/master/imagernn

deleted 12y ago

[deleted]

lars 12y ago

Since this paper along with [1] both beat human level performance on ImageNet, using seemingly unrelated techniques, it seems like there's a clear path to achieve the next state of the art result: Implement and train a network that uses both Batch Normalization and PReLU's at the same time. Right? 1: http://arxiv.org/pdf/1502.01852v1.pdf http://arxiv.org/pdf/1502.01852v1.pdf

dougabug 12y ago
Exactly my thoughts. Performance of both incremental improvements seems similar, would be interesting to see what the combined effect would be. It's pretty clear that they're reaching the limitations of the data set, though.