4 ms·
> spiking neural networks Move along, nothing to see here. Just a continuation of an old research direction that has nothing at all to do with the current revo
by modeless 10y ago
> spiking neural networks
Move along, nothing to see here. Just a continuation of an old research direction that has nothing at all to do with the current revolution in machine learning, except in name. Human designed spiking neural networks simply do not work well at solving any real world problems.
The important research in hardware for machine learning is focused on taking designs that work, such as convolutional nets and LSTM RNNs, and mapping them to silicon in the most efficient way possible. There's a lot of really exciting stuff going on there, and none of it has anything to do with spiking neural networks.
- vadansky 10y ago>The important research in hardware for machine learning is focused on taking designs that work I'm sorry but that is a really glib remark. The best working neural net is the human brain, and the current direction of neural networks have completely abandoned resembling a brain. While backpropogation has led to impressive results we shouldn't forget that it was never found in neurobiology, and it's pretty antithesis to how the brain operates. In my opinion machine learning should reconcile with neurobiology, but it's too obsessed with the results backprop is giving them. Frankly all "machine learning" right now is impressive exercises in high dimensional differentiation using backprop. I even remember a talk by a University of Toronto professors saying that even though neurobiologists have never found any support for large scale backprop at the heart of learning, maybe they should look again because neural nets are working so well with it. I would say they actually abandoned empiricism at this point. Keep in mind that if we assume the brain to be akin to an evolutionary system like DNA then backprop is even more agrecious because it's like saying the sunlight and the organism is conspiring to optimise. That the sun, or sperm/egg are getting feedback from the organism and it's fitness to finetune how they mutate the offspring.
- modeless 10y agoGiven the choice between algorithms that work and algorithms that crudely mimic the brain but don't work, I'll choose algorithms that work every time. Why waste millions of dollars fabbing chips for algorithms that we know don't work very well? It's cargo cult science, thinking that if we just build brain imitations without even understanding how the brain works then it will magically produce AI. Spiking proponents should focus on simulation and brain measurement until they figure out how to simulate something that works. At that point we can start making chips to improve efficiency. Meanwhile, the machine learning people may end up arriving at an AI that works as well as the brain or better despite operating on different principles, and that would be just fine!
- threeseed 10y ago> Given the choice Why do we have to choose ? Surely it is better to attempt both. Throughout the history of science knowing when something does not work is just as important as knowing when something does work.
- modeless 10y agoOf course we should investigate both. But when spending finite research dollars, we do have to choose. Should we fab chips for both? No, fabbing chips is very expensive. We should only fab chips for the one that works. The other can be investigated just fine in simulation.
- blueprint 10y agoFor the most part I completely agree. However, while it might seem reasonable to presume the brain is generating consciousness, it remains a presumption - and an ill-defined one, at that.
- emcq 10y agoThere are thousands of people invested in the next CNN, GPU, CPU, TPU, or FPGA architecture. Having a few smart people with demonstrated experience try an idea that doesn't fit the current biases in the field is perfectly reasonable funding allocation and great for science. If there weren't neural net holdouts getting funding doing something that didn't seem like it would work we'd be still using SVMs or arguing about probablistic graphical models. If you think this approach isn't warranted, I suggest that your example of an LSTM chip is also not sensible. First why not something simpler like GRU, secondly it's not compute efficient for the memory stored as compute scales poorly due to the matrix multiply for the size of your hidden state vector. If in a couple months we have a new differentiable memory architecture (a la neural Turing machine) all the work you did building a chip is outdated! Starting with first principles of what you can efficiently do with current process technology combined with an understanding of computational neuroscience is what Kwabenas lab excels at. It's actually very difficult to simulate the mixed analog digital asynchronous chips coming out of Kwabenas lab or elsewhere in the industry. There are plenty of other academic chips that fail after tapeout. Kwabenas lab has a proven track record of taping out chips that does what he claims. So from a hardware perspective I'd rather bet on a guy who has successfully taped out dozens of chips in his lab, and have the rest stick to simulation.
- sgift 10y agoThe comment above yours reminds me of all the times deep learning had been dismissed in the past as useless until computers finally were fast enough to make it actually usable. Now they are heralded as the best thing in the world. People always dismiss everything which doesn't fit the current favorite of the month. Short term thinking, the bane of the world.
- Seanny123 10y agoAs I mention in my comment [1], there are already a lot of designs that do work on very real world problems. Also, we've already started mapping CNNs onto the silicon for greater power efficiency. But you're right to say there's still a lot of work to do. [1] https://news.ycombinator.com/item?id=14084846 https://news.ycombinator.com/item?id=14084846
- bobsil1 10y agoRealtime video object recognition at 10^-5 the power draw of CNNs. Roughly simulating the # of neurons in a human brain for $700 rather than $70M.