7 ms·

Why Are Eight Bits Enough for Deep Neural Networks?

94 points by aburan28 11y ago

taliesinb 11y ago
I had wondered this myself -- it seems reasonable to see limiting the precision of activation as a form of regularization, as the author alludes to. For me, the place we'll eventually end up is obviously custom deep learning / evaluation chips that perform analogue operations using transistors in their linear regime (like how op-amps work). These chips would be programmed merely to express the tensor operation graph, essentially analogue tensor FPGAs. This should bring multiple order-of-magnitude reductions in power consumption and increases in evaluation speed. And when you don't have a clock there might also be interesting ways of dealing with time in which you don't discretize and unroll, like one currently does with GRUs or LSTMs.
- fchollet 11y ago
  Let the laws of physics do the recurrent math for you. Analog RNN computers would be very interesting, but they would first setting in stone the basics of the algorithms we use. We are still only beginning to explore the algorithm space, and this requires a flexibility that analog computers (or even ASICs or FPGAs) don't provide. It's still not clear whether the future of AI will even involve neural networks at all. Intuitively, they seem so inefficient.
- darkmighty 11y ago
  I agree that this kind of naive analog computing sounds very attractive with those simple linear operations (linear networks have been exhaustively studied, as you noted you essentially need only resistors and amplifiers). But it's not entirely obvious to me they ought to be better than digital electronics for comparable precision (considering their noise) and power consumption. I think you may get into trouble in the small current regime due to quantum mechanics: while you can do digital electronics with only a few electrons, you may need a large number to be able to maintain good linearity. An then there's the fact you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential, so ~n^k vs ~exp(2n) power consumption doesn't look good from this pov. But who knows, as the article points out the nonlinearities of the network may miraculously make it work even with very poor linearity and poor precision. It remains to be tested.
  - taliesinb 11y ago
    > n then there's the fact you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential, so ~n^k vs ~exp(2n) power consumption doesn't look good from this pov. That's true, I feel stupid for not having thought of that! I'm not an electrical engineer, but with the FETs that modern Intel chips are using, what fraction of their power consumption comes from parasitic gate capacitance, versus other losses? And if you operated in the linear region, what's the ballpark steady-state I_SD current you'd need on one FET to drive the gate of the next FET? I think that's what this comes down to: if gate capacitance dominates other losses even in the linear regime, you still win by not having a clock and lots of digital transitions. You could even imagine exploiting that: apply 'slow' augmentations of the input data that get you the equivalent of a bunch of iterations on a single example batch, while incurring a much smaller fraction of that initial cost because activations aren't going to change nearly as much as switching to a whole new example batch.
    - p1esk 11y ago
      "parasitic gate capacitance" - not sure if you want to call it "parasitic", after all, a gate capacitance is what makes everything work! Power is mainly lost via leakage (the smaller the transistor, the more it leaks), and via interconnect capacitance, which dominates all other capacitances in modern circuits.
      - taliesinb 11y ago
        > "parasitic gate capacitance" - not sure if you want to call it "parasitic", after all, a gate capacitance is what makes everything work! Of course, but the 'ideal' FET has zero gate capacitance, despite that being the way they work. > Power is mainly lost via leakage (the smaller the transistor, the more it leaks), and via interconnect capacitance, which dominates all other capacitances in modern circuits. Interconnect meaning things like the buses? There's no reason to want a von Neumann architecture for an analog chip. If that leaves leakage, I suppose an analog chip would be the beneficiary of needing a lot fewer transistors per op.
- rndn 11y ago
  What are your thoughts on using memristors for neural networks? They appear to have pretty good properties for that.
  - taliesinb 11y ago
    "Training and Operation of an Integrated Neuromorphic Network Based on Metal-Oxide Memristors", which appeared in Nature recently: http://arxiv.org/ftp/arxiv/papers/1412/1412.0611.pdf http://arxiv.org/ftp/arxiv/papers/1412/1412.0611.pdf
- nightski 11y ago
  Wouldn't noise be a huge issue?
  - taliesinb 11y ago
    Deep nets love noise, because with so many parameters they are very vulnerable to overfitting. For example, dropout, which is almost ubiquitous for deep learning, basically makes activations 'wrong' 50% of the time during training.
    - p1esk 11y ago
      That's not true. NNs don't like noise, there have been a lot of research done about effect of noise on NNs in the 90s. Random noise over a certain threshold will progressively degrade the performance of NNs, and below the threshold will have no effect. Dropout is not the same as random noise. By using dropout you eliminate some neurons from making contribution. As a result, you effectively train many smaller nets, each one adjusting its available weights to perform the same task. During testing, there's no noise - all neurons are back in business and contributing.
      - srean 11y ago
        I am pretty sure that parent meant DNNs love to fit noise. So you two are in strong agreement.
        p1esk 11y ago
        No, we are talking about a random electrical circuit noise in the analog NN hardware. Of course, if the noise is known and fixed, the net could learn to compensate (to a certain extent). The noise we are talking about is like when you put your finger on the chip, and raise its temperature by 10 degrees, the whole thing needs to be retrained.
        taliesinb 11y ago
        > The noise we are talking about is like when you put your finger on the chip, and raise its temperature by 10 degrees, the whole thing needs to be retrained. What would change with temperature that would require retraining? Are you saying the output of an op could depend sensitively on temperature, or that higher temperatures would increase things like thermal or shot noise? Why would the latter require retraining?
jsprogrammer 11y ago
It's not just eight bits. It's 8 bits * # of nodes. The bits per node just determine the 'resolution' of your individual nodes; while the network as a whole determines how many states can be represented.
Animats 11y ago
That's fascinating, especially since very slow training, where the weights don't change much per cycle, is in fashion. One would think that would result in changes rounding down to zero and nothing happening, but apparently it doesn't.
hyperion2010 11y ago
Point of interest, if you do the fundamental physics on neuronal membranes, the number of levels that are actually distinguishable give the noise in the system is only about 1000. So even in a biological system there are only 4x the the number of discrete levels. I realize this isn't a good match to what is mentioned in the article but it does put some constraints on the maximum dynamic range that biological sensors have to work within.
- tajen 11y ago
  1024 levels = 10 bits. The article mentions 8 bits, which is 256 levels. Now I get what you mean with 4x.
jokoon 11y ago
Wouldn't that mean that using 8 bit cores would be enough to simulate neuron networks ? That might significantly reduce the amount of transistors, thus increasing the amount of cores and parallelism.
LoSboccacc 11y ago
did my thesis on this topic (at that time we were searching the lower bound of ALU needed to have them running in zero power devices) it's interesting, NN degrade at about 6bit, and that's mostly because the transfer function become stable and the training gets stuck more often in local minimums. we built a training methodology in two step, first you trained them in 16bit precision, finding the absolute minimum, then retrain them with 6bit precision, and the NN basically learned to cope with the precision loss on its own. funny part is, the less bit you have, the more robust the network became, because error correcting became a normal part of its transfer function. we couldn't make the network solution converge on 4bit however. we tried using different transfer function, but then ran out of time before getting meaningful results (Each function needs it's own back propagation adjustment and things like that take time, I'm not a mathematician :D)
- fgimenez 11y ago
  I had similar empirical results on one of my PhD projects for medical image classification. With small data sets, we got better results on 8-bit data sets compared to 16-bit. We viewed it as a form of regularization that was extremely effective on smaller data sets with a lot of noise (x-rays in this case).
  - tachyonbeam 11y ago
    When using 8-bit weights, what kind of mapping do you do? Do you map the 8-bit range into -10 to 10? Do you have more precision near zero or is it a linear mapping?
    - LoSboccacc 11y ago
      Don't know about him but I was working with -8 8 for input and -4 4 for weights, using atan function for transfer maps quite well and there is no need to oversaturate the next layer.
afsina 11y ago
Funny that I just finished initial implementation of the code that uses the techniques from the paper (Vanhoucke et al.) mentioned in the post. https://github.com/ahmetaa/fast-dnn https://github.com/ahmetaa/fast-dnn
dnautics 11y ago
Agreed. I'm going working on an 8bit floating point that is optimized for learning algos, and optimized to be easy to soft emulate and also very efficient in hardware. One of the cool things about this float is that transfer functions (like the logistic) basically becomes a lookup table for really good performance. Also, there is no strong need for "zero".
TD-Linux 11y ago
>On the general CPU side, modern SIMD instruction sets are often geared towards float, and so eight bit calculations don’t offer a massive computational advantage on recent x86 or ARM chips. This isn't true, modern SIMD instruction sets have tons of operations for smaller fixed point numbers, as used heavily in video codecs. Unless the author meant some sort of weird 8 bit float?
Houshalter 11y ago
The problem with using digital calculations is that they are deterministic. If a result is really small, it is just rounded down to zero. So if you add a bunch of small numbers, you get zero. Even if the result should be large. Stochastic rounding can fix this. You round each step with the probability so it's expected value is the same. Usually it will round down to 0, but sometimes it will round up to 1. Relevant paper, using stochastic rounding. Without it the results get worse and worse before you even get to 8 bits. With stochastic rounding, there is no performance degradation. You could probably even reduce the bits even further. I think it may even be possible to get it down to 1 or 2 bits: http://arxiv.org/abs/1502.02551 http://arxiv.org/abs/1502.02551 The relevant graph: https://i.imgur.com/cOZ4fn3.jpg https://i.imgur.com/cOZ4fn3.jpg
rdlecler1 11y ago
These networks ought to be robust to minor changes in W. It's the topology that maters and frankly most of the W_ij != 0 are spurious connections -- meaning perturbation analysis will show that they play no causal role in the computation. I wrote a paper on this which has >100 citation (Survival of The Sparsest: Robust Gene Networks are Parsimonious). I used gene networks, but this is just a special case of neural networks. In fact there been a bunch of papers published on gene regulatory networks that show that topology is the main driver of function -- not surprising, if you show the circuit diagram of an 8-bit adder to an EE, they'll know exactly the function. Logically it has to be so. In fact you can model the gene network of the drosophila segmentation pattern with Boolean (1-bit) networks. The problem with ANN research is that no few take the time to understand why things function as they do. We should be reverse engineering these from biology. Every time a major advancement is made in ANNs neurobiologist say "yes, we could have told you that ten years ago" deep learning is just the latest example. It will hit its asymptote soon, then people will say that AI failed to live up to its expectation, then someone will make a new discovery. It's very frustrating to sit on the sidelines and watch this happen again and again.
- JuliaLang 11y ago
  Care to do it yourself?
IgorPartola 11y ago
I guess this is why rowing crews that regularly practice on choppy water end up doing better in an average competition (no citation, just something my coach once told me). Training in adverse conditions results in better built in corrections.
Tobu 11y ago
Eight bits are enough for me … https://www.youtube.com/watch?v=GoGpLl-SUfk https://www.youtube.com/watch?v=GoGpLl-SUfk

94 points by aburan28 11y ago

I had wondered this myself -- it seems reasonable to see limiting the precision of activation as a form of regularization, as the author alludes to. For me, the place we'll eventually end up is obviously custom deep learning / evaluation chips that perform analogue operations using transistors in their linear regime (like how op-amps work). These chips would be programmed merely to express the tensor operation graph, essentially analogue tensor FPGAs. This should bring multiple order-of-magnitude reductions in power consumption and increases in evaluation speed. And when you don't have a clock there might also be interesting ways of dealing with time in which you don't discretize and unroll, like one currently does with GRUs or LSTMs.

fchollet 11y ago
Let the laws of physics do the recurrent math for you. Analog RNN computers would be very interesting, but they would first setting in stone the basics of the algorithms we use. We are still only beginning to explore the algorithm space, and this requires a flexibility that analog computers (or even ASICs or FPGAs) don't provide. It's still not clear whether the future of AI will even involve neural networks at all. Intuitively, they seem so inefficient.
darkmighty 11y ago
I agree that this kind of naive analog computing sounds very attractive with those simple linear operations (linear networks have been exhaustively studied, as you noted you essentially need only resistors and amplifiers). But it's not entirely obvious to me they ought to be better than digital electronics for comparable precision (considering their noise) and power consumption. I think you may get into trouble in the small current regime due to quantum mechanics: while you can do digital electronics with only a few electrons, you may need a large number to be able to maintain good linearity. An then there's the fact you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential, so ~n^k vs ~exp(2n) power consumption doesn't look good from this pov. But who knows, as the article points out the nonlinearities of the network may miraculously make it work even with very poor linearity and poor precision. It remains to be tested.
- taliesinb 11y ago
  > n then there's the fact you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential, so ~n^k vs ~exp(2n) power consumption doesn't look good from this pov. That's true, I feel stupid for not having thought of that! I'm not an electrical engineer, but with the FETs that modern Intel chips are using, what fraction of their power consumption comes from parasitic gate capacitance, versus other losses? And if you operated in the linear region, what's the ballpark steady-state I_SD current you'd need on one FET to drive the gate of the next FET? I think that's what this comes down to: if gate capacitance dominates other losses even in the linear regime, you still win by not having a clock and lots of digital transitions. You could even imagine exploiting that: apply 'slow' augmentations of the input data that get you the equivalent of a bunch of iterations on a single example batch, while incurring a much smaller fraction of that initial cost because activations aren't going to change nearly as much as switching to a whole new example batch.
  - p1esk 11y ago
    "parasitic gate capacitance" - not sure if you want to call it "parasitic", after all, a gate capacitance is what makes everything work! Power is mainly lost via leakage (the smaller the transistor, the more it leaks), and via interconnect capacitance, which dominates all other capacitances in modern circuits.
    - taliesinb 11y ago
      > "parasitic gate capacitance" - not sure if you want to call it "parasitic", after all, a gate capacitance is what makes everything work! Of course, but the 'ideal' FET has zero gate capacitance, despite that being the way they work. > Power is mainly lost via leakage (the smaller the transistor, the more it leaks), and via interconnect capacitance, which dominates all other capacitances in modern circuits. Interconnect meaning things like the buses? There's no reason to want a von Neumann architecture for an analog chip. If that leaves leakage, I suppose an analog chip would be the beneficiary of needing a lot fewer transistors per op.
rndn 11y ago
What are your thoughts on using memristors for neural networks? They appear to have pretty good properties for that.
- taliesinb 11y ago
  "Training and Operation of an Integrated Neuromorphic Network Based on Metal-Oxide Memristors", which appeared in Nature recently: http://arxiv.org/ftp/arxiv/papers/1412/1412.0611.pdf http://arxiv.org/ftp/arxiv/papers/1412/1412.0611.pdf
nightski 11y ago
Wouldn't noise be a huge issue?
- taliesinb 11y ago
  Deep nets love noise, because with so many parameters they are very vulnerable to overfitting. For example, dropout, which is almost ubiquitous for deep learning, basically makes activations 'wrong' 50% of the time during training.
  - p1esk 11y ago
    That's not true. NNs don't like noise, there have been a lot of research done about effect of noise on NNs in the 90s. Random noise over a certain threshold will progressively degrade the performance of NNs, and below the threshold will have no effect. Dropout is not the same as random noise. By using dropout you eliminate some neurons from making contribution. As a result, you effectively train many smaller nets, each one adjusting its available weights to perform the same task. During testing, there's no noise - all neurons are back in business and contributing.
    - srean 11y ago
      I am pretty sure that parent meant DNNs love to fit noise. So you two are in strong agreement.
      - p1esk 11y ago
        No, we are talking about a random electrical circuit noise in the analog NN hardware. Of course, if the noise is known and fixed, the net could learn to compensate (to a certain extent). The noise we are talking about is like when you put your finger on the chip, and raise its temperature by 10 degrees, the whole thing needs to be retrained.
        taliesinb 11y ago
        > The noise we are talking about is like when you put your finger on the chip, and raise its temperature by 10 degrees, the whole thing needs to be retrained. What would change with temperature that would require retraining? Are you saying the output of an op could depend sensitively on temperature, or that higher temperatures would increase things like thermal or shot noise? Why would the latter require retraining?

jsprogrammer 11y ago

It's not just eight bits. It's 8 bits * # of nodes. The bits per node just determine the 'resolution' of your individual nodes; while the network as a whole determines how many states can be represented.

Animats 11y ago

That's fascinating, especially since very slow training, where the weights don't change much per cycle, is in fashion. One would think that would result in changes rounding down to zero and nothing happening, but apparently it doesn't.

hyperion2010 11y ago

Point of interest, if you do the fundamental physics on neuronal membranes, the number of levels that are actually distinguishable give the noise in the system is only about 1000. So even in a biological system there are only 4x the the number of discrete levels. I realize this isn't a good match to what is mentioned in the article but it does put some constraints on the maximum dynamic range that biological sensors have to work within.

tajen 11y ago
1024 levels = 10 bits. The article mentions 8 bits, which is 256 levels. Now I get what you mean with 4x.

jokoon 11y ago

Wouldn't that mean that using 8 bit cores would be enough to simulate neuron networks ? That might significantly reduce the amount of transistors, thus increasing the amount of cores and parallelism.

LoSboccacc 11y ago

did my thesis on this topic (at that time we were searching the lower bound of ALU needed to have them running in zero power devices) it's interesting, NN degrade at about 6bit, and that's mostly because the transfer function become stable and the training gets stuck more often in local minimums. we built a training methodology in two step, first you trained them in 16bit precision, finding the absolute minimum, then retrain them with 6bit precision, and the NN basically learned to cope with the precision loss on its own. funny part is, the less bit you have, the more robust the network became, because error correcting became a normal part of its transfer function. we couldn't make the network solution converge on 4bit however. we tried using different transfer function, but then ran out of time before getting meaningful results (Each function needs it's own back propagation adjustment and things like that take time, I'm not a mathematician :D)

fgimenez 11y ago
I had similar empirical results on one of my PhD projects for medical image classification. With small data sets, we got better results on 8-bit data sets compared to 16-bit. We viewed it as a form of regularization that was extremely effective on smaller data sets with a lot of noise (x-rays in this case).
- tachyonbeam 11y ago
  When using 8-bit weights, what kind of mapping do you do? Do you map the 8-bit range into -10 to 10? Do you have more precision near zero or is it a linear mapping?
  - LoSboccacc 11y ago
    Don't know about him but I was working with -8 8 for input and -4 4 for weights, using atan function for transfer maps quite well and there is no need to oversaturate the next layer.

afsina 11y ago

Funny that I just finished initial implementation of the code that uses the techniques from the paper (Vanhoucke et al.) mentioned in the post. https://github.com/ahmetaa/fast-dnn https://github.com/ahmetaa/fast-dnn

dnautics 11y ago

Agreed. I'm going working on an 8bit floating point that is optimized for learning algos, and optimized to be easy to soft emulate and also very efficient in hardware. One of the cool things about this float is that transfer functions (like the logistic) basically becomes a lookup table for really good performance. Also, there is no strong need for "zero".

TD-Linux 11y ago

>On the general CPU side, modern SIMD instruction sets are often geared towards float, and so eight bit calculations don’t offer a massive computational advantage on recent x86 or ARM chips. This isn't true, modern SIMD instruction sets have tons of operations for smaller fixed point numbers, as used heavily in video codecs. Unless the author meant some sort of weird 8 bit float?

Houshalter 11y ago

The problem with using digital calculations is that they are deterministic. If a result is really small, it is just rounded down to zero. So if you add a bunch of small numbers, you get zero. Even if the result should be large. Stochastic rounding can fix this. You round each step with the probability so it's expected value is the same. Usually it will round down to 0, but sometimes it will round up to 1. Relevant paper, using stochastic rounding. Without it the results get worse and worse before you even get to 8 bits. With stochastic rounding, there is no performance degradation. You could probably even reduce the bits even further. I think it may even be possible to get it down to 1 or 2 bits: http://arxiv.org/abs/1502.02551 http://arxiv.org/abs/1502.02551 The relevant graph: https://i.imgur.com/cOZ4fn3.jpg https://i.imgur.com/cOZ4fn3.jpg

rdlecler1 11y ago

These networks ought to be robust to minor changes in W. It's the topology that maters and frankly most of the W_ij != 0 are spurious connections -- meaning perturbation analysis will show that they play no causal role in the computation. I wrote a paper on this which has >100 citation (Survival of The Sparsest: Robust Gene Networks are Parsimonious). I used gene networks, but this is just a special case of neural networks. In fact there been a bunch of papers published on gene regulatory networks that show that topology is the main driver of function -- not surprising, if you show the circuit diagram of an 8-bit adder to an EE, they'll know exactly the function. Logically it has to be so. In fact you can model the gene network of the drosophila segmentation pattern with Boolean (1-bit) networks. The problem with ANN research is that no few take the time to understand why things function as they do. We should be reverse engineering these from biology. Every time a major advancement is made in ANNs neurobiologist say "yes, we could have told you that ten years ago" deep learning is just the latest example. It will hit its asymptote soon, then people will say that AI failed to live up to its expectation, then someone will make a new discovery. It's very frustrating to sit on the sidelines and watch this happen again and again.

JuliaLang 11y ago
Care to do it yourself?

IgorPartola 11y ago

I guess this is why rowing crews that regularly practice on choppy water end up doing better in an average competition (no citation, just something my coach once told me). Training in adverse conditions results in better built in corrections.

Tobu 11y ago

Eight bits are enough for me … https://www.youtube.com/watch?v=GoGpLl-SUfk https://www.youtube.com/watch?v=GoGpLl-SUfk