5 ms·

So do memristors exist or not? Wiki claims they are still 'hypothetical'.

by kovrik 9y ago

smalley 9y ago
They 100% exist. We had a wafer worth of cutup die of the TiO_2-x type devices all the way back in 2007. HP officially published results of a physical device as well (http://www.hpl.hp.com/news/2008/apr-jun/memristor.html http://www.hpl.hp.com/news/2008/apr-jun/memristor.html)
- aidenn0 9y ago
  My understanding of the TiO2-x devices is that they are strongly non-linear; which makes them very useful for binary storage, but less so for analogue memristor-y uses.
  - elcritch 9y ago
    Wouldn't non-linear mem-resistors make pretty ideal neural nets? Most activation functions are modeled as non-linear functions anyway. Edit: Appears some HP associated labs had success with just such a chip! Surprised it hasn't been more success in that field. https://www.technologyreview.com/s/537211/a-better-way-to-build-brain-inspired-chips/ https://www.technologyreview.com/s/537211/a-better-way-to-bu...
    - deepnotderp 9y ago
      Because no one wants neuormorphic hardware, we want 16-bit hardware that allows us to do math. Contrary to popular opinion, neural networks is more about math than biology. Having 1-bit or analog weights/activations is a royal pain.
      - elcritch 9y ago
        That's a good point. Most of our current CS / Hardware tech and experience relies on digital computing. Makes me wonder if instead of using neuormorphic hardware as the deployment vehicle, it'd make sense to use it as a specialized high speed trainer. Whereas most of these research projects seem to be assuming that they'd be deployed as endpoint/client devices. The alternative would be to use the annoying/tedious analog hardware to significantly (possibly) reduce the amount of time/energy costs to train a neural net. Then have special equipment to measure the learned weights and convert them to 16-bit weights that'd be easier to deploy. Facebook/Google are showing that custom chips for deep learning make economic sense. That market feature could tip the scales for specialized neuormorphic training chips which deal with analog circuitry but offset those costs & complexities by driving efficiency and/or speed while still deploying via traditional digital chips. Of course, this path relies on the assumption that an analog circuit could be much faster or energy efficient. It's not too far fetched though based on the current numbers and time for training vs biological system energy costs. [edit: grammar]
        deepnotderp 9y ago
        Therein lies your problem, I've been telling anyone who'll listen that data movement, even on chip is our energy hog, not computation or even reading from the memory banks. How does analog hardware deal with that?
        elcritch 9y ago
        In the case I listed above, the data (i.e. the training weights) only needs to be read once at the end of training. It'd require move expensive instrumenting of the memristor's which is expensive and bad for general purpose computation unit. However, comparing the energy needed to modify the neural net data in-situ via analog signals vs shuffling that same data in digital form back-and-forth repeatedly to simulate the analog process seems to provide a viable use case. A quick mental check seems to lean toward back-propagation being "cheaper" in the analog processes as the data doesn't need to be moved while the calculation is performed as part of the same signal propagation via the properties of the analog circuit. In other words its cheaper to move the computation "units" to the data than to move the data to the computation engine for this particular case. Performing the training in digital form requires repeatedly shuffling all the weights for _each_ training iteration. That process is expensive. Luckily the inherent nature of the back-propagation algorithm adapts to the "sloppiness" of analog circuitry. Transferring the final weights to digital form could require final but light post-processing training to remove particulars of the underlying analog circuits. But replicating and distributing the final trained model would be more efficient since it only requires a single shuffling of the weight data to apply the neural net and get an answer. Applying the trained model via standard digital means should be cheaper/easier for all the reasons you mentioned previously. [edit: grammar & clarity]
        aidenn0 9y ago
        Analog circuits are usually less accurate and more expensive to build than the equivalent digital. They also usually can't be as dense, as they are usually more prone to crosstalk.
WaxProlix 9y ago
My understanding is that the current 'memristor' technologies are essentially combinations of materials which satisfy the linkage between flux and charge, thereby meeting the definition. From the same wikipedia article: > There is no such thing as a standard memristor. Instead, each device implements a particular function, wherein the integral of voltage determines the integral of current, and vice versa. A linear time-invariant memristor, with a constant value for M, is simply a conventional resistor.[1] Manufactured devices are never purely memristors (ideal memristor), but also exhibit some capacitance and resistance. Just reading the 'Background' and following 'Memristor definition and criticism' sections gives a super clear and lucid answer that even a goober like me can mostly degest. https://en.wikipedia.org/wiki/Memristor#Background https://en.wikipedia.org/wiki/Memristor#Background
igk 9y ago
Basically, in the science world, there is a lot of debate going on about exact definitions (memristive device vs. memristor). I can see if I can find the relevant papers if there is interest. Prof. Chua has updated and loosened his criteria a bit in response to the criticisms. In the engineering world, if it talks like a memristor, and walks like a memristor if you squint a bit, we are damn well going to use it like a memristor. Hence we are happily building and testing nonvolatile storage (MRAM, ReRAM, PCRAM) and other applications - I'm gonna start research on applications as ML accelerators in May :-)
deepnotderp 9y ago
They absolutely exist. The question is whether they can be reliably fabricated in volume without having loads of faulty units and hopefully, to not be ridiculously expensive.

So do memristors exist or not? Wiki claims they are still 'hypothetical'.

by kovrik 9y ago

So do memristors exist or not? Wiki claims they are still 'hypothetical'.

They 100% exist. We had a wafer worth of cutup die of the TiO_2-x type devices all the way back in 2007. HP officially published results of a physical device as well (http://www.hpl.hp.com/news/2008/apr-jun/memristor.html http://www.hpl.hp.com/news/2008/apr-jun/memristor.html)

aidenn0 9y ago
My understanding of the TiO2-x devices is that they are strongly non-linear; which makes them very useful for binary storage, but less so for analogue memristor-y uses.
- elcritch 9y ago
  Wouldn't non-linear mem-resistors make pretty ideal neural nets? Most activation functions are modeled as non-linear functions anyway. Edit: Appears some HP associated labs had success with just such a chip! Surprised it hasn't been more success in that field. https://www.technologyreview.com/s/537211/a-better-way-to-build-brain-inspired-chips/ https://www.technologyreview.com/s/537211/a-better-way-to-bu...
  - deepnotderp 9y ago
    Because no one wants neuormorphic hardware, we want 16-bit hardware that allows us to do math. Contrary to popular opinion, neural networks is more about math than biology. Having 1-bit or analog weights/activations is a royal pain.
    - elcritch 9y ago
      That's a good point. Most of our current CS / Hardware tech and experience relies on digital computing. Makes me wonder if instead of using neuormorphic hardware as the deployment vehicle, it'd make sense to use it as a specialized high speed trainer. Whereas most of these research projects seem to be assuming that they'd be deployed as endpoint/client devices. The alternative would be to use the annoying/tedious analog hardware to significantly (possibly) reduce the amount of time/energy costs to train a neural net. Then have special equipment to measure the learned weights and convert them to 16-bit weights that'd be easier to deploy. Facebook/Google are showing that custom chips for deep learning make economic sense. That market feature could tip the scales for specialized neuormorphic training chips which deal with analog circuitry but offset those costs & complexities by driving efficiency and/or speed while still deploying via traditional digital chips. Of course, this path relies on the assumption that an analog circuit could be much faster or energy efficient. It's not too far fetched though based on the current numbers and time for training vs biological system energy costs. [edit: grammar]
      - deepnotderp 9y ago
        Therein lies your problem, I've been telling anyone who'll listen that data movement, even on chip is our energy hog, not computation or even reading from the memory banks. How does analog hardware deal with that?
        elcritch 9y ago
        In the case I listed above, the data (i.e. the training weights) only needs to be read once at the end of training. It'd require move expensive instrumenting of the memristor's which is expensive and bad for general purpose computation unit. However, comparing the energy needed to modify the neural net data in-situ via analog signals vs shuffling that same data in digital form back-and-forth repeatedly to simulate the analog process seems to provide a viable use case. A quick mental check seems to lean toward back-propagation being "cheaper" in the analog processes as the data doesn't need to be moved while the calculation is performed as part of the same signal propagation via the properties of the analog circuit. In other words its cheaper to move the computation "units" to the data than to move the data to the computation engine for this particular case. Performing the training in digital form requires repeatedly shuffling all the weights for _each_ training iteration. That process is expensive. Luckily the inherent nature of the back-propagation algorithm adapts to the "sloppiness" of analog circuitry. Transferring the final weights to digital form could require final but light post-processing training to remove particulars of the underlying analog circuits. But replicating and distributing the final trained model would be more efficient since it only requires a single shuffling of the weight data to apply the neural net and get an answer. Applying the trained model via standard digital means should be cheaper/easier for all the reasons you mentioned previously. [edit: grammar & clarity]
      - aidenn0 9y ago
        Analog circuits are usually less accurate and more expensive to build than the equivalent digital. They also usually can't be as dense, as they are usually more prone to crosstalk.

WaxProlix 9y ago

My understanding is that the current 'memristor' technologies are essentially combinations of materials which satisfy the linkage between flux and charge, thereby meeting the definition. From the same wikipedia article: > There is no such thing as a standard memristor. Instead, each device implements a particular function, wherein the integral of voltage determines the integral of current, and vice versa. A linear time-invariant memristor, with a constant value for M, is simply a conventional resistor.[1] Manufactured devices are never purely memristors (ideal memristor), but also exhibit some capacitance and resistance. Just reading the 'Background' and following 'Memristor definition and criticism' sections gives a super clear and lucid answer that even a goober like me can mostly degest. https://en.wikipedia.org/wiki/Memristor#Background https://en.wikipedia.org/wiki/Memristor#Background

igk 9y ago

Basically, in the science world, there is a lot of debate going on about exact definitions (memristive device vs. memristor). I can see if I can find the relevant papers if there is interest. Prof. Chua has updated and loosened his criteria a bit in response to the criticisms. In the engineering world, if it talks like a memristor, and walks like a memristor if you squint a bit, we are damn well going to use it like a memristor. Hence we are happily building and testing nonvolatile storage (MRAM, ReRAM, PCRAM) and other applications - I'm gonna start research on applications as ML accelerators in May :-)

deepnotderp 9y ago

They absolutely exist. The question is whether they can be reliably fabricated in volume without having loads of faulty units and hopefully, to not be ridiculously expensive.