3 ms·

Impressive. Am I right that what is really going on here is that the network is implementing the "+" operator by actually having the CPU of the host carrying ou

• Filligree 4y ago
  That's an accurate enough description of what's going on here, yeah, but I'm very curious if this could be implemented in hardware. What we've got is an inexact not-quite-binary adder, but one that's potentially smaller and faster than the regular binary ones.
  • garganzol 4y ago
    It can be implemented in hardware but the implementation would be more complex than a digital adder based on logical gates.
    • em3rgent0rdr 4y ago
      Analog addition is actually really easy if you can tolerate noise and heat and can convert to/from representing the signal as an analog current. Using Kirchhoff's Current Law, addition of N currents from current sources is achieved by joining those N wires to a shared exit wire which contains the summed current.
      • garganzol 4y ago
        Addition is easy, DAC is relatively easy, but ADC is not.
  • vletal 4y ago
    Why it works here is that the "analog" representation is not influenced by noise, because it's simulated on digital hardware. On the other hand, why we use digital hardware precisely because it's robust against the noise present in the hardware analog circuits.
    • TeMPOraL 4y ago
      So in other words, this is analog computation on a digital hardware on analog substrate - the analog-to-digital step eliminates the noise of our physical reality, and the subsequent digital-to-analog reintroduces a certain flexibility of design thinking. I wonder, is there ever a case analog-on-digital is better to work with as an abstraction layer, or is it always easier to work with digital signals directly?
      • chaorace 4y ago
        Yes, at least I think so. That's why we so often try to approximate analog with psuedo-continuous data patterns like samples and floats. Even going beyond electronic computing, we invented calculus due to similar limitations with discrete data in mathematics. Of course, these are all just approximations of analog. Much like emulation, there are limitations, penalties, and inaccuracies that will inevitably kneecap applications when compared to a native implementation running on bare metal (though, it seems that humans lack a proper math coprocessor, so calculus could be considered no more "native" than traditional algebra). We do sometimes see specialized hardware emerge (DSPs in the case of audio signal processing, FPUs in the case of floats, NPUs/TPUs in the case of neural networks), but these are almost always highly optimized for operations specific to analog-like data patterns (e.g.: fourier transforms) rather than true analog operations. This is probably because scalable/reliable/fast analog memory remains an unsolved problem (semiconductors are simply too useful).
      • imtringued 4y ago
        Sorry but those signals aren't analog. Floating point numbers are still digital.
    • Filligree 4y ago
  • Someone 4y ago
    Quad-level flash memory cells (https://en.wikipedia.org/wiki/Multi-level_cell#Quad-level_cell https://en.wikipedia.org/wiki/Multi-level_cell#Quad-level_ce...) can store 4 bits for months. Because of that, I would (hand-wavingly) expect it can be made to work for a three-bit adder (with four bits of output)
  • em3rgent0rdr 4y ago
    There is valid use case for when designers simply need "fuzzy logic" or "fuzzy math" which doesn't need exact results, but which can tolerate inaccurate results. If using fuzzy hardware saves resources (time, energy, die space, etc.) then it might be a valid tradeoff to use inaccurate fuzzy hardware instead of exact hardware. For instance instead of evaluating an exponential function in digital logic, it might be quicker and more energy-efficient to just evaluate it using a diode as an analog voltage, if the value is available as an analog voltage and if some noise is tolerable, as done with analog computers.
• greenbit 4y ago
  You know how the basic node does a weighted sum of inputs, and feeds the result to a non linear function for output? This network exploited the addition operation implicit in that first part, the weighted sum.
  • xyzzy_plugh 4y ago
    Fascinating! Is it really exploitation? It seems to me that the node should optimize to the minimal inputs (weights) in the happy path. I.e. it's optimal for all non-operand inputs to be weighted to zero. I'm curious how reproducible this is. I'm also curious how networks form abstractions and if we can verify those abstractions in isolation (or have other networks verify them, like this) then the building blocks become far less opaque.
• TeMPOraL 4y ago
  I'd say it's more that the network "discovered" that addition is something its own structure does "naturally", and reduced the problem into decoding binary input, and encoding binary output. In particular, I don't think it's specifically about the CPU having an addition operator. My intuition is as follows: if I were to train this network with pencil, paper and a slide rule, I'd expect the same result. Addition (or maybe rather integration) is embedded in the abstract structure of a neural network as a computation artifact. Sure, specifics of the substrate may "leak through" - e.g. in the pen-and-paper case, were I to round everything to first decimal space, or in the computer case, was the network implemented with 4-bit floats, I'd expect it not to converge because of loss of precision range (or maybe figure out the logic gate solution). But if the substrate can execute the mathematical model of a neural network to sufficient precision, I'd expect the same result to occur regardless of whether the network is run on paper, on a CPU, an fluid-based analog computer, or a beam of light and a clever arrangement of semi-transparent plastic plates.