4 ms·

Or, you could split your 64 bit computation into 8 bit computations, which could be done with analog circuits, and still save a lot of power! :-) But yes, I un

by p1esk 11y ago

Or, you could split your 64 bit computation into 8 bit computations, which could be done with analog circuits, and still save a lot of power! :-) But yes, I understand your point. Both analog and digital implementations have their strengths and weaknesses. If you value power over precision, go with analog. If the opposite - go with digital.

darkmighty 11y ago
Right, but note you can't even split it, if you are thinking of linear circuits. Precision necessarily means how your signal compared to the thermal noise floor. It is possible to show can't compose 8-bit precision linear units to get a >8-bit precision value. What happens is actually the opposite, if the noise of the units are uncorrelated noise will propagate and increase to the tune of sqrt(number of operations). Avoiding error propagation is another advantage of digital operations. The reason NNs don't exhibit strong error propagation is because of the non-linearities between linear layers that perform operations analogous to threshold/majority voting or the like, which have error correction properties.
- p1esk 11y ago
  Interesting, but then how do you explain that rectified linear operations between layers work better than sigmoids? According to your logic, ReLU should have worse error propagating quality than squashing functions?
darkmighty 11y ago
I'm going to reply to your question below here since HN is preventing a reply (anti-flaming/long threads I guess). Be careful with jumping to conclusions: I never even cited ReLUs or Sigmoids in my post! I don't have any opinion on which non-linearity is better, I only know both are dramatic non-linearities. My claims were about linear circuits. You should use whatever nonlinear element works best in your Neural Network, of course (and I've heard ReLUs have good advantages).

4 ms·

Or, you could split your 64 bit computation into 8 bit computations, which could be done with analog circuits, and still save a lot of power! :-) But yes, I un

by p1esk 11y ago

darkmighty 11y ago
Right, but note you can't even split it, if you are thinking of linear circuits. Precision necessarily means how your signal compared to the thermal noise floor. It is possible to show can't compose 8-bit precision linear units to get a >8-bit precision value. What happens is actually the opposite, if the noise of the units are uncorrelated noise will propagate and increase to the tune of sqrt(number of operations). Avoiding error propagation is another advantage of digital operations. The reason NNs don't exhibit strong error propagation is because of the non-linearities between linear layers that perform operations analogous to threshold/majority voting or the like, which have error correction properties.
- p1esk 11y ago
  Interesting, but then how do you explain that rectified linear operations between layers work better than sigmoids? According to your logic, ReLU should have worse error propagating quality than squashing functions?
darkmighty 11y ago
I'm going to reply to your question below here since HN is preventing a reply (anti-flaming/long threads I guess). Be careful with jumping to conclusions: I never even cited ReLUs or Sigmoids in my post! I don't have any opinion on which non-linearity is better, I only know both are dramatic non-linearities. My claims were about linear circuits. You should use whatever nonlinear element works best in your Neural Network, of course (and I've heard ReLUs have good advantages).