4 ms·

Reservoir computing using dynamic memristors for temporal information processing

75 points by lainon 9y ago

dnewms 9y ago
For those without a technical background: https://news.engin.umich.edu/2017/12/new-quick-learning-neural-network-powered-by-memristors/ https://news.engin.umich.edu/2017/12/new-quick-learning-neur...
Seanny123 9y ago
Whelp. I was super wrong. I had predicted I wouldn't see memristor-based neuromorphic hardware in the next 10 years. That being said, I feel like they're over-selling the capabilities of reservoir computing. Yeah, [you can stack them](https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2016-175.pdf https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2016...) and get pretty high accuracy on a few tasks, but it's still not competitive with traditional Deep Learning.
- p1esk 9y ago
  https://www.nature.com/articles/nature14441?WT.ec_id=NATURE-20150507 https://www.nature.com/articles/nature14441?WT.ec_id=NATURE-...
- captaindiego 9y ago
  I think one of the potential advantages is energy consumption of this compared to running deep learning on a conventional CPU/GPU would be dramatically less. If the memristor chips could be mass-produced, this could lead to things like basic voice commands and face/image recognition being added very cheaply and being added to cheap embedded/IoT stuff (for better or worse).
cs702 9y ago
For deep learning and ML practitioners who are unfamiliar with this, the big deal about memristors is that they have memory and therefore produce different readout outputs for different input sequences over time -- out-of-the-box, without requiring any training. These researchers took a network of memristors, called the "reservoir," added a linear layer with a SoftMax on top of the reservoir, trained this hybrid network on a lower-resolution variant of MNIST (feeding pixel values over time, as varying voltages), and achieved classification accuracy superior to a tiny neural net despite having only 1/90th the number of neurons.[1] Note that they only trained the added layer; they did NOT have to train the reservoir. Figure (a) in this image has a simplified diagram of the reservoir + added layer architecture: https://www.nature.com/articles/s41467-017-02337-y/figures/1 https://www.nature.com/articles/s41467-017-02337-y/figures/1 -- only the matrix Θ had to be learned. The potential here, over time, is for having highly scalable hardware components that can be plugged into neural net architectures as and where needed for learning to recognize and work with sequences out of the box. PS. For clarity's sake, I'm ignoring a lot of important details and playing fast and loose with language. If you're really curious about this, I highly recommend you read at least the abstract and introduction of the Nature paper, which is well-written and straightforward to follow. [1] https://news.engin.umich.edu/2017/12/new-quick-learning-neural-network-powered-by-memristors/ https://news.engin.umich.edu/2017/12/new-quick-learning-neur...

75 points by lainon 9y ago

For those without a technical background: https://news.engin.umich.edu/2017/12/new-quick-learning-neural-network-powered-by-memristors/ https://news.engin.umich.edu/2017/12/new-quick-learning-neur...

Seanny123 9y ago

Whelp. I was super wrong. I had predicted I wouldn't see memristor-based neuromorphic hardware in the next 10 years. That being said, I feel like they're over-selling the capabilities of reservoir computing. Yeah, [you can stack them](https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2016-175.pdf https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2016...) and get pretty high accuracy on a few tasks, but it's still not competitive with traditional Deep Learning.

p1esk 9y ago
https://www.nature.com/articles/nature14441?WT.ec_id=NATURE-20150507 https://www.nature.com/articles/nature14441?WT.ec_id=NATURE-...
captaindiego 9y ago
I think one of the potential advantages is energy consumption of this compared to running deep learning on a conventional CPU/GPU would be dramatically less. If the memristor chips could be mass-produced, this could lead to things like basic voice commands and face/image recognition being added very cheaply and being added to cheap embedded/IoT stuff (for better or worse).

cs702 9y ago

For deep learning and ML practitioners who are unfamiliar with this, the big deal about memristors is that they have memory and therefore produce different readout outputs for different input sequences over time -- out-of-the-box, without requiring any training. These researchers took a network of memristors, called the "reservoir," added a linear layer with a SoftMax on top of the reservoir, trained this hybrid network on a lower-resolution variant of MNIST (feeding pixel values over time, as varying voltages), and achieved classification accuracy superior to a tiny neural net despite having only 1/90th the number of neurons.[1] Note that they only trained the added layer; they did NOT have to train the reservoir. Figure (a) in this image has a simplified diagram of the reservoir + added layer architecture: https://www.nature.com/articles/s41467-017-02337-y/figures/1 https://www.nature.com/articles/s41467-017-02337-y/figures/1 -- only the matrix Θ had to be learned. The potential here, over time, is for having highly scalable hardware components that can be plugged into neural net architectures as and where needed for learning to recognize and work with sequences out of the box. PS. For clarity's sake, I'm ignoring a lot of important details and playing fast and loose with language. If you're really curious about this, I highly recommend you read at least the abstract and introduction of the Nature paper, which is well-written and straightforward to follow. [1] https://news.engin.umich.edu/2017/12/new-quick-learning-neural-network-powered-by-memristors/ https://news.engin.umich.edu/2017/12/new-quick-learning-neur...