3 ms·
Yeah, and on analog platforms you have serious problems with accumulating. Good luck doing a 100x100 matrix times a 100x500 matrix on an analog system, which i
by dnautics 10y ago
Yeah, and on analog platforms you have serious problems with accumulating. Good luck doing a 100x100 matrix times a 100x500 matrix on an analog system, which is the sort of thing you need for machine learning.
Automatic differentiation is usually very little overhead. Think "putting a second set of values through a compiled slightly differently function".
Although there are calculus-related issues like saturation, smoothness of transfer functions, etc. "solving calculus" is not the major problem with machine learning efficiency and throughput. Linear algebra is the main bottleneck.
- p1esk 10y agoActually, matrix-matrix multiplication is where analog computing really shines. A multiplication operation can be done with a single transistor: one operand is gate voltage, another one is source to drain voltage. The resulting current through the transistor is proportional to the product of the two. An addition operation is even simpler: it's just connecting two wires carrying two different currents - the resulting current on the output wire will be the sum (due to Kirchhoff laws). Of course, you pay for this efficiency with precision, which for some applications (e.g. neural networks) can be a reasonable trade off.