6 ms·

If memristors act like neurons, put them in neural networks

83 points by hypomnemata 6y ago

dvh 6y ago
Digikey doesn't sell memristors
peatmoss 6y ago
Memristors are the technology that seemed poised to usher in a new era of computing. The promise of being able to redraw the current computer architectural hierarchies is tantalizing. If machine learning applications are what finally get memristors out into the world, I wish them godspeed.
- theamk 6y ago
  What do you think of memristors vs FPGAs? They have many similarities -- they both redrew the current computer architectures, they integrate memory and computing, they can have randomness built-in, and they both take less power than mainstream GPUs. What does memristor provide that specially designed "neural FPGA" can not?
  - sgtnoodle 6y ago
    They aren't really directly comparable like you imply. A memristor is basically a resistor that changes resistance when you run current through it. An FPGA is a clever layout of many thousands of logic gates that can be programmed to form arbitrarily complex digital logic circuits. It's like comparing apples to skyscrapers because they're both associated with New York. Presumably, a memristor based neural network would have the advantage over an FPGA of requiring significantly less silicon area to achieve the same function. I imagine an FPGA based neural network would approximate analog signals digitally, perhaps using floating point "half's" or something. Memristors would directly operate on analog signals, encoding information as amplitudes or pulses of currents and voltages. Notably, FPGAs and GPUs can't really be directly compared to each other in terms of power consumption unless you specify specific use cases. You can't build the equivalent of a mainstream GPU out of FPGAs without severely limiting the clock speed (because of how physically large it would be), and if you did anyway, it would use many orders of magnitude more power to function. So, a GPU is way more power efficient than an FPGA for rendering graphics. There are certainly problems that a GPU isn't good at solving, and so there's a good chance that an FPGA solution would be more power efficient.
    - TeMPOraL 6y ago
      Correct me if I'm wrong, but isn't it a good metaphor to say that FPGAs are just ASIC emulators? I.e. much less efficient than an equivalent ASIC, but good for prototyping and something to use in production if you can't afford to manufacture your own chip (which is most low-volume use cases). Under this metaphor, memristor-based ASICs will obviously be more efficient than FPGAs emulating them, but that's predicated on there being a high-volume use case justifying the creation of a memristor chip in the first place.
      - sesuximo 6y ago
        Wow there... sometimes digital emulation of an analog circuit is faster than the analog circuit since you can solve for equilibria, easily repeat calculations, use well researched programming tools, etc
  - Grambo 6y ago
    My understanding of the use of FPGAs and ASICs that are used to speed up neural networks (such as those in phones) is that they are simply designed to do the types of calculations used for NNs more quickly (matrix operations) and generally at a reduced level of precision. This is very different from a memristor approach where the structure of the network itself would be represented in the silicon. I also think it's unfair to compare the two because it took decades of work to get CMOS transistors to where they are today. I imagine that once commercial applications for memristors appear many optimizations/improvements will present themselves.
    - theamk 6y ago
      "the structure of the network itself would be represented in the silicon" -- ASICs then? perhaps even hybrid analog/digital one, where fixed coefficients are stored in digital memory, while input data is analog. I believe there is a great value in being able to "snapshot" the state and later load exactly the same state into millions of devices. And I cannot see how this will easily work with memristors.
theamk 6y ago
I am surprised people have high expectations from memristors. They are just another way to build an analog computer -- better for machine learning, worse for classical ODEs. But we have not used analog computers for 50 years, and for a good reason -- they are not reproducible, their accuracy is very process dependent and has a hard upper limit, and they are often tuned for a single function. Would people want a chip which is basically unpredictable -- the performance can vary up by tens of %, they have to be re-trained periodically to prevent data loss, and there is no way to load pre-trained network? I doubt it. Maybe there is an extremely narrow use case, but I do not see it in the mainstream devices.
- waste_monk 6y ago
  >Would people want a chip which is basically unpredictable -- the performance can vary up by tens of %, they have to be re-trained periodically to prevent data loss, and there is no way to load pre-trained network? I doubt it. Maybe there is an extremely narrow use case, but I do not see it in the mainstream devices. Human brains have the same problems and seem to be fairly popular.
  - amelius 6y ago
    True, but they're hard to do experiments with, which is useful if you're building such a brain.
    - jokethrowaway 6y ago
      If the trade-off between energy-cheap human-like processing and our computers is reliability, it's definitely worth doing. If we had unlimited human like processing, we would basically have unlimited slaves without affecting biological organism similar to us. Not to mention, it's worth doing it just for having 100000 times cheaper machine learning.
      - birdyrooster 6y ago
        Oh boy I can’t wait for the unlimited slaves
gioscarab 6y ago
Isn't the coherer https://en.wikipedia.org/wiki/Coherer https://en.wikipedia.org/wiki/Coherer the first memristor? If so it was invented in 1890.
3JPLW 6y ago
I encourage folks to actually read the linked article instead of basing their commentary on the shoddy title. https://www.nature.com/articles/s41928-020-00523-3 https://www.nature.com/articles/s41928-020-00523-3 Abstract: > Resistive memory technologies could be used to create intelligent systems that learn locally at the edge. However, current approaches typically use learning algorithms that cannot be reconciled with the intrinsic non-idealities of resistive memory, particularly cycle-to-cycle variability. Here, we report a machine learning scheme that exploits memristor variability to implement Markov chain Monte Carlo sampling in a fabricated array of 16,384 devices configured as a Bayesian machine learning model. We apply the approach experimentally to carry out malignant tissue recognition and heart arrhythmia detection tasks, and, using a calibrated simulator, address the cartpole reinforcement learning task. Our approach demonstrates robustness to device degradation at ten million endurance cycles, and, based on circuit and system-level simulations, the total energy required to train the models is estimated to be on the order of microjoules, which is notably lower than in complementary metal– oxide–semiconductor (CMOS)-based approaches.
YeGoblynQueenne 6y ago
>> The devices could also work well within neural networks, which are machine learning systems that use synthetic versions of synapses and neurons to mimic the process of learning in the human brain. Yann LeCun disagrees: IEEE Spectrum: We read about Deep Learning in the news a lot these days. What’s your least favorite definition of the term that you see in these stories? Yann LeCun: My least favorite description is, “It works just like the brain.” I don’t like people saying this because, while Deep Learning gets an inspiration from biology, it’s very, very far from what the brain actually does. And describing it like the brain gives a bit of the aura of magic to it, which is dangerous. It leads to hype; people claim things that are not true. AI has gone through a number of AI winters because people claimed things they couldn’t deliver. https://spectrum.ieee.org/automaton/artificial-intelligence/machine-learning/facebook-ai-director-yann-lecun-on-deep-learning https://spectrum.ieee.org/automaton/artificial-intelligence/...
nobodyandproud 6y ago
Basic question, but why are transistors not considered fundamental?
Cyder 6y ago
The movie 'ex machina' is a great example of this discussion... must see

83 points by hypomnemata 6y ago

dvh 6y ago

Digikey doesn't sell memristors

Memristors are the technology that seemed poised to usher in a new era of computing. The promise of being able to redraw the current computer architectural hierarchies is tantalizing. If machine learning applications are what finally get memristors out into the world, I wish them godspeed.

theamk 6y ago
What do you think of memristors vs FPGAs? They have many similarities -- they both redrew the current computer architectures, they integrate memory and computing, they can have randomness built-in, and they both take less power than mainstream GPUs. What does memristor provide that specially designed "neural FPGA" can not?
- sgtnoodle 6y ago
  They aren't really directly comparable like you imply. A memristor is basically a resistor that changes resistance when you run current through it. An FPGA is a clever layout of many thousands of logic gates that can be programmed to form arbitrarily complex digital logic circuits. It's like comparing apples to skyscrapers because they're both associated with New York. Presumably, a memristor based neural network would have the advantage over an FPGA of requiring significantly less silicon area to achieve the same function. I imagine an FPGA based neural network would approximate analog signals digitally, perhaps using floating point "half's" or something. Memristors would directly operate on analog signals, encoding information as amplitudes or pulses of currents and voltages. Notably, FPGAs and GPUs can't really be directly compared to each other in terms of power consumption unless you specify specific use cases. You can't build the equivalent of a mainstream GPU out of FPGAs without severely limiting the clock speed (because of how physically large it would be), and if you did anyway, it would use many orders of magnitude more power to function. So, a GPU is way more power efficient than an FPGA for rendering graphics. There are certainly problems that a GPU isn't good at solving, and so there's a good chance that an FPGA solution would be more power efficient.
  - TeMPOraL 6y ago
    Correct me if I'm wrong, but isn't it a good metaphor to say that FPGAs are just ASIC emulators? I.e. much less efficient than an equivalent ASIC, but good for prototyping and something to use in production if you can't afford to manufacture your own chip (which is most low-volume use cases). Under this metaphor, memristor-based ASICs will obviously be more efficient than FPGAs emulating them, but that's predicated on there being a high-volume use case justifying the creation of a memristor chip in the first place.
    - sesuximo 6y ago
      Wow there... sometimes digital emulation of an analog circuit is faster than the analog circuit since you can solve for equilibria, easily repeat calculations, use well researched programming tools, etc
- Grambo 6y ago
  My understanding of the use of FPGAs and ASICs that are used to speed up neural networks (such as those in phones) is that they are simply designed to do the types of calculations used for NNs more quickly (matrix operations) and generally at a reduced level of precision. This is very different from a memristor approach where the structure of the network itself would be represented in the silicon. I also think it's unfair to compare the two because it took decades of work to get CMOS transistors to where they are today. I imagine that once commercial applications for memristors appear many optimizations/improvements will present themselves.
  - theamk 6y ago
    "the structure of the network itself would be represented in the silicon" -- ASICs then? perhaps even hybrid analog/digital one, where fixed coefficients are stored in digital memory, while input data is analog. I believe there is a great value in being able to "snapshot" the state and later load exactly the same state into millions of devices. And I cannot see how this will easily work with memristors.

theamk 6y ago

I am surprised people have high expectations from memristors. They are just another way to build an analog computer -- better for machine learning, worse for classical ODEs. But we have not used analog computers for 50 years, and for a good reason -- they are not reproducible, their accuracy is very process dependent and has a hard upper limit, and they are often tuned for a single function. Would people want a chip which is basically unpredictable -- the performance can vary up by tens of %, they have to be re-trained periodically to prevent data loss, and there is no way to load pre-trained network? I doubt it. Maybe there is an extremely narrow use case, but I do not see it in the mainstream devices.

waste_monk 6y ago
>Would people want a chip which is basically unpredictable -- the performance can vary up by tens of %, they have to be re-trained periodically to prevent data loss, and there is no way to load pre-trained network? I doubt it. Maybe there is an extremely narrow use case, but I do not see it in the mainstream devices. Human brains have the same problems and seem to be fairly popular.
- amelius 6y ago
  True, but they're hard to do experiments with, which is useful if you're building such a brain.
  - jokethrowaway 6y ago
    If the trade-off between energy-cheap human-like processing and our computers is reliability, it's definitely worth doing. If we had unlimited human like processing, we would basically have unlimited slaves without affecting biological organism similar to us. Not to mention, it's worth doing it just for having 100000 times cheaper machine learning.
    - birdyrooster 6y ago
      Oh boy I can’t wait for the unlimited slaves

gioscarab 6y ago

Isn't the coherer https://en.wikipedia.org/wiki/Coherer https://en.wikipedia.org/wiki/Coherer the first memristor? If so it was invented in 1890.

3JPLW 6y ago

I encourage folks to actually read the linked article instead of basing their commentary on the shoddy title. https://www.nature.com/articles/s41928-020-00523-3 https://www.nature.com/articles/s41928-020-00523-3 Abstract: > Resistive memory technologies could be used to create intelligent systems that learn locally at the edge. However, current approaches typically use learning algorithms that cannot be reconciled with the intrinsic non-idealities of resistive memory, particularly cycle-to-cycle variability. Here, we report a machine learning scheme that exploits memristor variability to implement Markov chain Monte Carlo sampling in a fabricated array of 16,384 devices configured as a Bayesian machine learning model. We apply the approach experimentally to carry out malignant tissue recognition and heart arrhythmia detection tasks, and, using a calibrated simulator, address the cartpole reinforcement learning task. Our approach demonstrates robustness to device degradation at ten million endurance cycles, and, based on circuit and system-level simulations, the total energy required to train the models is estimated to be on the order of microjoules, which is notably lower than in complementary metal– oxide–semiconductor (CMOS)-based approaches.

YeGoblynQueenne 6y ago

>> The devices could also work well within neural networks, which are machine learning systems that use synthetic versions of synapses and neurons to mimic the process of learning in the human brain. Yann LeCun disagrees: IEEE Spectrum: We read about Deep Learning in the news a lot these days. What’s your least favorite definition of the term that you see in these stories? Yann LeCun: My least favorite description is, “It works just like the brain.” I don’t like people saying this because, while Deep Learning gets an inspiration from biology, it’s very, very far from what the brain actually does. And describing it like the brain gives a bit of the aura of magic to it, which is dangerous. It leads to hype; people claim things that are not true. AI has gone through a number of AI winters because people claimed things they couldn’t deliver. https://spectrum.ieee.org/automaton/artificial-intelligence/machine-learning/facebook-ai-director-yann-lecun-on-deep-learning https://spectrum.ieee.org/automaton/artificial-intelligence/...

nobodyandproud 6y ago

Basic question, but why are transistors not considered fundamental?

Cyder 6y ago

The movie 'ex machina' is a great example of this discussion... must see