2 ms·
This isn't about language support or library support, this is about hardware support. How many instructions does it take to derive/integrate a function on x86?
by adamnemecek 10y ago
This isn't about language support or library support, this is about hardware support. How many instructions does it take to derive/integrate a function on x86? On an analog platform this is very close to "single instruction". What if you could solve calculus idk, 1000x times faster. It could be even more though.
- dnautics 10y agoYeah, and on analog platforms you have serious problems with accumulating. Good luck doing a 100x100 matrix times a 100x500 matrix on an analog system, which is the sort of thing you need for machine learning. Automatic differentiation is usually very little overhead. Think "putting a second set of values through a compiled slightly differently function". Although there are calculus-related issues like saturation, smoothness of transfer functions, etc. "solving calculus" is not the major problem with machine learning efficiency and throughput. Linear algebra is the main bottleneck.
- p1esk 10y agoActually, matrix-matrix multiplication is where analog computing really shines. A multiplication operation can be done with a single transistor: one operand is gate voltage, another one is source to drain voltage. The resulting current through the transistor is proportional to the product of the two. An addition operation is even simpler: it's just connecting two wires carrying two different currents - the resulting current on the output wire will be the sum (due to Kirchhoff laws). Of course, you pay for this efficiency with precision, which for some applications (e.g. neural networks) can be a reasonable trade off.