I found this paragraph from the paper very interesting: > 7 The relevance of FF to analog hardware > An energy efficient way to multiply an activity vector by

I found this paragraph from the paper very interesting: > 7 The relevance of FF to analog hardware > An energy efficient way to multiply an activity vector by a weight matrix is to implement activities as voltages and weights as conductances. Their products, per unit time, are charges which add themselves. This seems a lot more sensible than driving transistors at high power to model the individual bits in the digital representation of a number and then performing O(n^2) single bit operations to multiply two n-bit numbers together. Unfortunately, it is difficult to implement the backpropagation procedure in an equally efficient way, so people have resorted to using A-to-D converters and digital computations for computing gradients (Kendall et al., 2020). The use of two forward passes instead of a forward and a backward pass should make these A-to-D converters unnecessary. It was my impression that it is difficult to properly isolate an electronic system to use voltages in this way (hence computers sort of "cut" voltages into bits 0/1 using a step function). Have these limitations been overcome or do they not matter as much, as neural networks can work with more fuzzy data? Interesting to imagine such a processor though.