5 ms·

I agree that this kind of naive analog computing sounds very attractive with those simple linear operations (linear networks have been exhaustively studied, as

I agree that this kind of naive analog computing sounds very attractive with those simple linear operations (linear networks have been exhaustively studied, as you noted you essentially need only resistors and amplifiers). But it's not entirely obvious to me they ought to be better than digital electronics for comparable precision (considering their noise) and power consumption. I think you may get into trouble in the small current regime due to quantum mechanics: while you can do digital electronics with only a few electrons, you may need a large number to be able to maintain good linearity. An then there's the fact you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential, so ~n^k vs ~exp(2n) power consumption doesn't look good from this pov. But who knows, as the article points out the nonlinearities of the network may miraculously make it work even with very poor linearity and poor precision. It remains to be tested.

taliesinb 11y ago
> n then there's the fact you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential, so ~n^k vs ~exp(2n) power consumption doesn't look good from this pov. That's true, I feel stupid for not having thought of that! I'm not an electrical engineer, but with the FETs that modern Intel chips are using, what fraction of their power consumption comes from parasitic gate capacitance, versus other losses? And if you operated in the linear region, what's the ballpark steady-state I_SD current you'd need on one FET to drive the gate of the next FET? I think that's what this comes down to: if gate capacitance dominates other losses even in the linear regime, you still win by not having a clock and lots of digital transitions. You could even imagine exploiting that: apply 'slow' augmentations of the input data that get you the equivalent of a bunch of iterations on a single example batch, while incurring a much smaller fraction of that initial cost because activations aren't going to change nearly as much as switching to a whole new example batch.
- p1esk 11y ago
  "parasitic gate capacitance" - not sure if you want to call it "parasitic", after all, a gate capacitance is what makes everything work! Power is mainly lost via leakage (the smaller the transistor, the more it leaks), and via interconnect capacitance, which dominates all other capacitances in modern circuits.
  - taliesinb 11y ago
    > "parasitic gate capacitance" - not sure if you want to call it "parasitic", after all, a gate capacitance is what makes everything work! Of course, but the 'ideal' FET has zero gate capacitance, despite that being the way they work. > Power is mainly lost via leakage (the smaller the transistor, the more it leaks), and via interconnect capacitance, which dominates all other capacitances in modern circuits. Interconnect meaning things like the buses? There's no reason to want a von Neumann architecture for an analog chip. If that leaves leakage, I suppose an analog chip would be the beneficiary of needing a lot fewer transistors per op.
    - p1esk 11y ago
      'ideal' FET has zero gate capacitance, despite that being the way they work. I don't understand this statement. What do you mean? A FET is a capacitor (gate to channel). If a gate has no capacitance, you have no transistor. Interconnect means wire. This has nothing to do with von Neumann architecture. If you have wires in your circuit, then you have wire capacitance. As transistors get smaller, that capacitance starts to dominate internal transistor capacitances.
p1esk 11y ago
you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential This does not make sense to me. Can you explain? I think there might be misunderstanding of how analog computing is used to build a neural network. First, a weight is stored as some analog physical property, typically as charge on a floating gate, or on a capacitor in a DRAM type cell. Second, the multiplication operation is performed by modulating the analog input signal going through the floating gate transistor by the charge on the floating gate (weight). Third, the summation is done via simple summation of the currents. Finally, activation function is performed by an opamp. Regarding power consumption: 1. A digital computer needs a thousand of transistors to perform multiplication, analog circuit can do it with a single one. 2. Analog NN stores parameters (weights) locally, right where they are needed to perform computation. Digital NN will need lots of memory transfers to bring weights from RAM to ALU, and to store intermediate results. That's why a properly implemented analog NN will always consume much less power.
- taliesinb 11y ago
  > 1. A digital computer needs a thousand of transistors to perform multiplication, analog circuit can do it with a single one. That's interesting. What would the circuit be? > Digital NN will need lots of memory transfers to bring weights from RAM to ALU, and to store intermediate results. That's not necessarily the case. Cellular neural networks were proposed long ago, for example, and they're digital -- how multiplication happens is independent from the data flow architecture. > That's why a properly implemented analog NN will always consume much less power. How do you know that the I^2 cost of operating in the linear regime isn't excessive? I'm totally ignorant on the matter -- I'd love to see a ballpark calculation to understand why it isn't important.
  - p1esk 11y ago
    As I described above: "the multiplication operation is performed by modulating the analog input signal going through the floating gate transistor by the charge on the floating gate (weight)." The circuit is the single transistor in this case. Cellular neural networks were proposed long ago, for example, and they're digital What is so inherently digital about cellular networks? Can you provide a link to an implementation of a cellular net in digital hardware? How the weights are stored? Where the multiplication happens?

taliesinb 11y ago

> n then there's the fact you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential, so ~n^k vs ~exp(2n) power consumption doesn't look good from this pov. That's true, I feel stupid for not having thought of that! I'm not an electrical engineer, but with the FETs that modern Intel chips are using, what fraction of their power consumption comes from parasitic gate capacitance, versus other losses? And if you operated in the linear region, what's the ballpark steady-state I_SD current you'd need on one FET to drive the gate of the next FET? I think that's what this comes down to: if gate capacitance dominates other losses even in the linear regime, you still win by not having a clock and lots of digital transitions. You could even imagine exploiting that: apply 'slow' augmentations of the input data that get you the equivalent of a bunch of iterations on a single example batch, while incurring a much smaller fraction of that initial cost because activations aren't going to change nearly as much as switching to a whole new example batch.

p1esk 11y ago
"parasitic gate capacitance" - not sure if you want to call it "parasitic", after all, a gate capacitance is what makes everything work! Power is mainly lost via leakage (the smaller the transistor, the more it leaks), and via interconnect capacitance, which dominates all other capacitances in modern circuits.
- taliesinb 11y ago
  > "parasitic gate capacitance" - not sure if you want to call it "parasitic", after all, a gate capacitance is what makes everything work! Of course, but the 'ideal' FET has zero gate capacitance, despite that being the way they work. > Power is mainly lost via leakage (the smaller the transistor, the more it leaks), and via interconnect capacitance, which dominates all other capacitances in modern circuits. Interconnect meaning things like the buses? There's no reason to want a von Neumann architecture for an analog chip. If that leaves leakage, I suppose an analog chip would be the beneficiary of needing a lot fewer transistors per op.
  - p1esk 11y ago
    'ideal' FET has zero gate capacitance, despite that being the way they work. I don't understand this statement. What do you mean? A FET is a capacitor (gate to channel). If a gate has no capacitance, you have no transistor. Interconnect means wire. This has nothing to do with von Neumann architecture. If you have wires in your circuit, then you have wire capacitance. As transistors get smaller, that capacitance starts to dominate internal transistor capacitances.

p1esk 11y ago

you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential This does not make sense to me. Can you explain? I think there might be misunderstanding of how analog computing is used to build a neural network. First, a weight is stored as some analog physical property, typically as charge on a floating gate, or on a capacitor in a DRAM type cell. Second, the multiplication operation is performed by modulating the analog input signal going through the floating gate transistor by the charge on the floating gate (weight). Third, the summation is done via simple summation of the currents. Finally, activation function is performed by an opamp. Regarding power consumption: 1. A digital computer needs a thousand of transistors to perform multiplication, analog circuit can do it with a single one. 2. Analog NN stores parameters (weights) locally, right where they are needed to perform computation. Digital NN will need lots of memory transfers to bring weights from RAM to ALU, and to store intermediate results. That's why a properly implemented analog NN will always consume much less power.

taliesinb 11y ago
> 1. A digital computer needs a thousand of transistors to perform multiplication, analog circuit can do it with a single one. That's interesting. What would the circuit be? > Digital NN will need lots of memory transfers to bring weights from RAM to ALU, and to store intermediate results. That's not necessarily the case. Cellular neural networks were proposed long ago, for example, and they're digital -- how multiplication happens is independent from the data flow architecture. > That's why a properly implemented analog NN will always consume much less power. How do you know that the I^2 cost of operating in the linear regime isn't excessive? I'm totally ignorant on the matter -- I'd love to see a ballpark calculation to understand why it isn't important.
- p1esk 11y ago
  As I described above: "the multiplication operation is performed by modulating the analog input signal going through the floating gate transistor by the charge on the floating gate (weight)." The circuit is the single transistor in this case. Cellular neural networks were proposed long ago, for example, and they're digital What is so inherently digital about cellular networks? Can you provide a link to an implementation of a cellular net in digital hardware? How the weights are stored? Where the multiplication happens?