3 ms·
We are using the wrong computational paradigm. We have to abandon bits and go back to analog computing in the form of analog photonic computing that gives you f
by adamnemecek 9y ago
We are using the wrong computational paradigm. We have to abandon bits and go back to analog computing in the form of analog photonic computing that gives you fast Calculus. This is painfully obvious in the case of neural networks, which run faster on an analog computer and are also easier to program.
- sgt101 9y agoI think you are out of date. Rectification is pretty good at removing a lot of the vanishing gradient issues that nn's used to face, and the overwhelming power of modern digital computers (50k cores is common) make this all moot as far as I can see.
- adamnemecek 9y agoNope, they aren't even in the same category. E.g. Notice that there is a cpu size limit because of heat. Do you know what doesn't overheat? A photonic computer. You can build a cpu the size of a house. Also how does the number of cores constitute "overwhelming power".
- sgt101 9y ago:o) well, due to the wavelength of light a photonic cpu has to be 50->100 times the size of a current gen electronic one. When I were a young 'un we had one core, and it ran at 25Mhz and about 130 of us shared it. Now I have 50,000 cores that run at 2 Ghz and five people share it. Things aren't quite directly comparable but the speed up is at least 100,000x I am overwhelmed by this, things that would have taken 1000 days; approximately 3 years, can be achieved in ten or twenty minutes. In reality the use of these infrastructures has enabled (in neural net land) the development of techniques that improve performance by several more orders of magnitude - so things that would have taken several years are now done in a minute or so. I believe that there is plenty more headroom to be had.
- adamnemecek 9y agoWhere is that estimate coming from? I mean year there has been progress by why do you think photonic would be even better?
- sgt101 9y ago1989->2017 Photonics are promising QC technologies - especially Phonons, but we are a long way off!
- dimatura 9y agoHow is an analog computer "easier to program" than a digital computer? Making neural networks do what you want is hard enough with the help of tons of libraries, decent scripting languages, the ability to dump the weights into a file and inspect them, etc. Programming with an analog computer, which I'm guessing would be something like programming with FPGAs, sounds like a nightmare in comparison.
- adamnemecek 9y agoBecause neural networks are fundamentally dynamic systems that are much easier to model with continuous signals than discrete bits. A lot of the hardness comes from the fact that you are discretizing fundamentally continuous signals.
- PeterisP 9y agoQuite the opposite, neural network research & experiments show that discreteness isn't a problem - in particular, there's no benefit on having a model with more fine-grained values and that even extremely discrete models (e.g 8 bits or less) work quite well.
- adamnemecek 9y agoThere are advantages to having Calculus operations as first class citizens.
- HarHarVeryFunny 9y agoWell, yes, that's where some of the hardness comes from... but how do you build a predictive model capable of transfer learning without disentangling the latent factors? How do you perform one-shot learning without being able to lay down discrete episodic memories? I'd say that rather than a continuous analog data stream needing an analog model, the real problem is that the causality (hence predictability - the goal) of this data stream is due to discrete actors and actions and therefore we need to discretize the stream into objects and spatio-temporal events. Anyhow, we're making great strides with ANNs on the perceptual side to the point where it's almost a solved problem... What's lacking (outside of DeepMind) is more of a focus on intelligent embedded agents, complete with lifetime continuous learning, and adaptive behavior. IMO we're focusing too much on artificial isolated problems rather than the embedded systems/agents that are the real goal! Just as ImageNet - and human competitiveness - drove vision research, what could accelerate AI research would be a similar annual competition for embedded agents (either in a simulated environment or maybe robots in a competition space), which would at least focus efforts on building systems and addressing the goal of AI rather than breaking it down into someone's (maybe incorrect) notions of the piece-parts necessary to get there. Some people shy away from robotics as an unwelcome added complexity, but that never stopped the popular micromouse competitions, and these sorts of competition could go a very long way with simple robots/vehicles (e.g. based on Lego mindstorms or R/C vehicles) with remote compute.
- Nimitz14 9y agoCan you provide any links for this, I'm interested but know nothing about it.
- adamnemecek 9y agoIt's kind of hard cause I can't say I fully understand all of it but googling around for analog computing neural networks is a good start. Check also real computation https://en.m.wikipedia.org/wiki/Real_computation https://en.m.wikipedia.org/wiki/Real_computation and work of Lenore Blum. Im super pumped about the research into computational metamaterials http://users.ece.utexas.edu/~aalu/research%20-%20page%203.htm http://users.ece.utexas.edu/~aalu/research%20-%20page%203.ht...