4 ms·
I've been trying to find an answer to this for some time. Is it possible that ML could be implemented much better on analog computers as they have a "native sup
by adamnemecek 10y ago
I've been trying to find an answer to this for some time. Is it possible that ML could be implemented much better on analog computers as they have a "native support" for differential calculus?
- kelvin0 10y agoOK, so how do you store your model? In analog or digital format? If it's digital you might have to convert the analog signals into digital, with all the constraints involved with such a conversion. Also when you replay your model to process some data, you need to convert back from digital to analog. Please correct me if I misunderstood your point. Otherwise you might need to clarify your idea a bit more.
- adamnemecek 10y agoNote that my arguments are pretty likely wrong in some fundamental ways but here it goes. > OK, so how do you store your model? In analog or digital format? This is a good question. Do you think that conversion back and forth would diminish the gains from analog computing?
- kelvin0 10y agoConversion from analog to digital will necessarily involve the loss of information. Now the question is, is the data lost crucial to the operation of your system or not? If losing some data is not that big a deal, you might as well build it digitally from the ground up? On the other hand if you can't afford to lose any original information from your analog system, you might have quite a few obstacles to overcome and design an entire system which would be analog based (kinda like the human brain on some level). But then again, if you can build a perfect functioning human brain a nobel prize awaits you.
- adamnemecek 10y ago> Conversion from analog to digital will necessarily involve the loss of information. Now the question is, is the data lost crucial to the operation of your system or not? If losing some data is not that big a deal, you might as well build it digitally from the ground up? I think of it as the output being the solution of an equation/dynamic system. It's ok you lose the equations, you care about the solution. I've been doing some light internet research into this not too long ago and came across this title https://www.amazon.com/Neuromorphic-Photonics-Paul-R-Prucnal/dp/1498725228/ref=as_li_ss_tl?ie=UTF8&qid=1484086290&sr=8-3&keywords=photonics+neural+networks&linkCode=ll1&tag=akhn-20&linkId=32c9eac37d5cf1c5815e59f1098bbfb9 https://www.amazon.com/Neuromorphic-Photonics-Paul-R-Prucnal... that seems to suggest that analog photonics might be the way forward. But I can't really comment on the state of this. Photonics is definitely becoming more and more viable it appears.
- dnautics 10y ago"native support" for differential calculus is not a problem for ML. If you have a sufficiently powerful programming language, you can trivially do forward differentiation/automatic differentiation instead of backprop which is just about as "native support" as it gets (AD can also be done in, say, C, or python, it's just trickier). Granularity is also not really that much of a problem, you can reduce your bit precision to ~12 bits and still get good results with ML.
- adamnemecek 10y agoThis isn't about language support or library support, this is about hardware support. How many instructions does it take to derive/integrate a function on x86? On an analog platform this is very close to "single instruction". What if you could solve calculus idk, 1000x times faster. It could be even more though.
- dnautics 10y agoYeah, and on analog platforms you have serious problems with accumulating. Good luck doing a 100x100 matrix times a 100x500 matrix on an analog system, which is the sort of thing you need for machine learning. Automatic differentiation is usually very little overhead. Think "putting a second set of values through a compiled slightly differently function". Although there are calculus-related issues like saturation, smoothness of transfer functions, etc. "solving calculus" is not the major problem with machine learning efficiency and throughput. Linear algebra is the main bottleneck.
- p1esk 10y agoActually, matrix-matrix multiplication is where analog computing really shines. A multiplication operation can be done with a single transistor: one operand is gate voltage, another one is source to drain voltage. The resulting current through the transistor is proportional to the product of the two. An addition operation is even simpler: it's just connecting two wires carrying two different currents - the resulting current on the output wire will be the sum (due to Kirchhoff laws). Of course, you pay for this efficiency with precision, which for some applications (e.g. neural networks) can be a reasonable trade off.