4 ms·

I think what they mean is that classical systems use more than a single packet of energy to represent a state. Our “digital systems” are actually analog systems

I think what they mean is that classical systems use more than a single packet of energy to represent a state. Our “digital systems” are actually analog systems in saturation states. Each time you want to set a bit in memory, you need enough energy to fill up a capacitor (or similar related concept) which is far more than a single electron. This methodology from the viewpoint of quantum systems is repetition/redundancy which provides robustness against stray electrons (eg: from induced current/interference.) As we build systems that require less power, we are also building systems that use less electrons to represent a single bit. Another fun tidbit in this space comes down to signal propagation in a lossy medium: we don’t actually use square waves for our clocks since a square wave is composed of many different frequencies. Since different frequencies propagate at different speeds, a square wave looks less and less like a square and also uses more power than a simple sine wave. If you remember your Fourier/Laplace knowledge, you probably remember that a sine is a single frequency and thus it will be coherent through a conductor and use the least amount of power to generate. Edit: I’m talking about electrons here but the concept of many packets for a single state extends to most of our communication channels today … eg radio communications like we see to/from Voyager.

eru 2y ago
Going off on a bit of a tangent: Some people suggest that digital computing and neural networks are a bit fit, and that would should be using analog devices. That sounds very appealing at first. But we have (at least) two problems: First, our transistors dissipate almost no energy when they are either 'fully open' or 'fully closed'. Because either there's approximately no current, or approximately no resistance. Holding them partially open, like you'd do in analog processing, would produce a lot of heat. The second problem: electrons are discrete, and thanks to miniaturisation and faster and faster clockspeeds, we are actually getting into realms where that makes a difference. So either you have to accept that the maximum resolution of activation of your analog neuron is fairly small (perhaps 10 bits or so?), which is not that much better than using your transistors in binary only; or you'll have to use much larger transistors in your neural chips. Both problems together mean that analog computing for neural networks isn't really competitive with digital computing. (Outside of some very niche applications, perhaps.)
- imoverclocked 2y ago
  I love tangents! I think all of your points are valid. I also think that we have optimized heavily for this state of technology. If we figured out that analog computing was somehow superior in a big way, I bet we would find ways of reducing power etc in analog designs. One way that analog computing would be really neat for neural networks is in speed. The way it might not be so great is in reliability (or repeatability, specifically.) Analog systems are more susceptible to noise as well as variation from fabrication processes. Running things at saturation makes them easier to design, test and mass produce.
  - akdom 2y ago
    Specifically to the point of the comparatively low reliability / high variation of analog systems: an interesting property of neural nets is that they can be robust relative to noise when trained with the same type of noise witnessed under inference. Whether or not speed/etc. would be better in the digital vs. analog design-space, it's an interesting thing to consider that neural nets can automatically account for the encoding-medium's variability. This perhaps makes neural solutions a good fit for low power analog media which otherwise aren't useful for classical computing. See https://arxiv.org/abs/2104.13386 https://arxiv.org/abs/2104.13386 for an exploration of physically encoding neural architectures.

eru 2y ago

Going off on a bit of a tangent: Some people suggest that digital computing and neural networks are a bit fit, and that would should be using analog devices. That sounds very appealing at first. But we have (at least) two problems: First, our transistors dissipate almost no energy when they are either 'fully open' or 'fully closed'. Because either there's approximately no current, or approximately no resistance. Holding them partially open, like you'd do in analog processing, would produce a lot of heat. The second problem: electrons are discrete, and thanks to miniaturisation and faster and faster clockspeeds, we are actually getting into realms where that makes a difference. So either you have to accept that the maximum resolution of activation of your analog neuron is fairly small (perhaps 10 bits or so?), which is not that much better than using your transistors in binary only; or you'll have to use much larger transistors in your neural chips. Both problems together mean that analog computing for neural networks isn't really competitive with digital computing. (Outside of some very niche applications, perhaps.)

imoverclocked 2y ago
I love tangents! I think all of your points are valid. I also think that we have optimized heavily for this state of technology. If we figured out that analog computing was somehow superior in a big way, I bet we would find ways of reducing power etc in analog designs. One way that analog computing would be really neat for neural networks is in speed. The way it might not be so great is in reliability (or repeatability, specifically.) Analog systems are more susceptible to noise as well as variation from fabrication processes. Running things at saturation makes them easier to design, test and mass produce.
- akdom 2y ago
  Specifically to the point of the comparatively low reliability / high variation of analog systems: an interesting property of neural nets is that they can be robust relative to noise when trained with the same type of noise witnessed under inference. Whether or not speed/etc. would be better in the digital vs. analog design-space, it's an interesting thing to consider that neural nets can automatically account for the encoding-medium's variability. This perhaps makes neural solutions a good fit for low power analog media which otherwise aren't useful for classical computing. See https://arxiv.org/abs/2104.13386 https://arxiv.org/abs/2104.13386 for an exploration of physically encoding neural architectures.