3 ms·

Therein lies your problem, I've been telling anyone who'll listen that data movement, even on chip is our energy hog, not computation or even reading from the m

Therein lies your problem, I've been telling anyone who'll listen that data movement, even on chip is our energy hog, not computation or even reading from the memory banks. How does analog hardware deal with that?

elcritch 9y ago
In the case I listed above, the data (i.e. the training weights) only needs to be read once at the end of training. It'd require move expensive instrumenting of the memristor's which is expensive and bad for general purpose computation unit. However, comparing the energy needed to modify the neural net data in-situ via analog signals vs shuffling that same data in digital form back-and-forth repeatedly to simulate the analog process seems to provide a viable use case. A quick mental check seems to lean toward back-propagation being "cheaper" in the analog processes as the data doesn't need to be moved while the calculation is performed as part of the same signal propagation via the properties of the analog circuit. In other words its cheaper to move the computation "units" to the data than to move the data to the computation engine for this particular case. Performing the training in digital form requires repeatedly shuffling all the weights for _each_ training iteration. That process is expensive. Luckily the inherent nature of the back-propagation algorithm adapts to the "sloppiness" of analog circuitry. Transferring the final weights to digital form could require final but light post-processing training to remove particulars of the underlying analog circuits. But replicating and distributing the final trained model would be more efficient since it only requires a single shuffling of the weight data to apply the neural net and get an answer. Applying the trained model via standard digital means should be cheaper/easier for all the reasons you mentioned previously. [edit: grammar & clarity]

elcritch 9y ago

In the case I listed above, the data (i.e. the training weights) only needs to be read once at the end of training. It'd require move expensive instrumenting of the memristor's which is expensive and bad for general purpose computation unit. However, comparing the energy needed to modify the neural net data in-situ via analog signals vs shuffling that same data in digital form back-and-forth repeatedly to simulate the analog process seems to provide a viable use case. A quick mental check seems to lean toward back-propagation being "cheaper" in the analog processes as the data doesn't need to be moved while the calculation is performed as part of the same signal propagation via the properties of the analog circuit. In other words its cheaper to move the computation "units" to the data than to move the data to the computation engine for this particular case. Performing the training in digital form requires repeatedly shuffling all the weights for _each_ training iteration. That process is expensive. Luckily the inherent nature of the back-propagation algorithm adapts to the "sloppiness" of analog circuitry. Transferring the final weights to digital form could require final but light post-processing training to remove particulars of the underlying analog circuits. But replicating and distributing the final trained model would be more efficient since it only requires a single shuffling of the weight data to apply the neural net and get an answer. Applying the trained model via standard digital means should be cheaper/easier for all the reasons you mentioned previously. [edit: grammar & clarity]