4 ms·

12 years ago, on HN: "The memristor—the functional equivalent of a synapse—could revolutionize circuit design" https://news.ycombinator.com/item?id=397335 http

12 years ago, on HN: "The memristor—the functional equivalent of a synapse—could revolutionize circuit design" https://news.ycombinator.com/item?id=397335 https://news.ycombinator.com/item?id=397335

k__ 6y ago
Didn't HP want to build a new type of computer on it?
- vbezhenar 6y ago
  It was called The Machine: computer with unified memory. They released it without memristors in the end: HPE unveils The Machine, a single-memory computer with 160 terabytes of memory: https://news.ycombinator.com/item?id=14350396 https://news.ycombinator.com/item?id=14350396
mywittyname 6y ago
It's fascinating that none of the comments mention neural network applications. Seems like these people came a bit too early to the party.
- freeone3000 6y ago
  We've left biomimicry in the dust a fair time ago - neurons do not work like relus after all (nor do they work like memristors), but relus are great at solving problems. None of our current NN applications would even want something that behaves like a neuron. You'd essentially have to start from scratch. It's not even clear if behaves-like-a-neuron is better or not to be worth the effort. (And that's disregarding the entire fact that memristors do not behave like neurons either!)
  - andy_ppp 6y ago
    Do we even understand how a neuron works well enough to say that either thing is correct. I thought a relu was just the activation function that decides what weight to triggers the next part of the network with?
    - freeone3000 6y ago
      We understand enough about how a biological neuron works to know that single signals do not provoke a diminished response after the hysterisis period, which is not modeled by either a memristor or a sigmoid. Sigmoids don't have hysterisis, and you have to backflow a memristor for it to recover which is not true of biological nuerons. (The activation function is the neuron('s mathematical model) - we elide everything that happens internal to the neuron and model it as only its inputs and outputs.)
  - bnegreve 6y ago
    I don't think Relus are the proplem. Relus are good, but artificial neural networks can work with sigmoid activation fonctions instead. The neurones in an ANN with sigmoids are somewhat close to what is described in this article. The main problem is that the neurons in this article are not trainable (you can't update the weights dynamically) and not differentiable (you can't use the usual gradient-based training procedures to update the weights). They could be used for inference only though.
  - sgt101 6y ago
    Wait! Some people have left biomimicry in the dust and are manipulating matrices to sets find optimal parameter sets representing concepts. Other people (https://www.humanbrainproject.eu/en/ https://www.humanbrainproject.eu/en/) haven't! There is a big community of people building novel types of neural network using spikes and so on. And you are right - it isn't clear who is right. On one side the "no nature" folks might say : we don't even understand neurons to the point where we can definitely categorize them in a mouse brain, let alone explain their behaviour, or actually properly measure their behaviour, and so on and so on - this before understanding any of it really. On the other hand the "lets mimic things we don't understand" folks would say - we need to understand this because the human brain out performs all AI today on a 20 watt budget. My view is the understand stuff agenda is right - but not now and probably not for 20 years (in a big way). We need small scale capability and direction finding work - and we will need to do that for a long time.
  - unishark 6y ago
    I thin it's more that we're operating in a position of extreme simplification, and don't understand most biological details well enough to apply them in our own systems. But biological motivation is quite common in neural networks. Name a popular technique (backprop, dropout, etc.) and I can probably find you a source arguing it's biological relationship. It's just at an engineering-style level of approximation that scientists may abhor. The ReLU was originally introduced by computational neuroscientists as a more biologically accurate approximation compared to the other activation functions. E.g. https://www.nature.com/articles/35016072 https://www.nature.com/articles/35016072 I would expect similar history behind convolutional neural networks, given how the early stages of the vision system have been understood to operate.
  - orbifold 6y ago
    Research in this area has always been cyclical. The Neural ODE paper that won a best paper award at Neurips two years ago just implements a technique that is common knowledge in the optimal control theory community since the 60s. I wouldn’t dismiss an approach because it isn’t popular right now. Biological neurons force you to think about constraints you normally wouldn’t and crucially have solved temporal credit assignment, memory etc. in interesting ways that already have inspired some breakthroughs. As an example the replay memory idea was directly inspired by the rough understanding neuroscientist developed of the role of the hypocampus in learning.
- dfischer 6y ago
  These guys seem to be working on it in that way: https://knowm.org/ https://knowm.org/

k__ 6y ago

Didn't HP want to build a new type of computer on it?

vbezhenar 6y ago
It was called The Machine: computer with unified memory. They released it without memristors in the end: HPE unveils The Machine, a single-memory computer with 160 terabytes of memory: https://news.ycombinator.com/item?id=14350396 https://news.ycombinator.com/item?id=14350396

mywittyname 6y ago

It's fascinating that none of the comments mention neural network applications. Seems like these people came a bit too early to the party.

freeone3000 6y ago
We've left biomimicry in the dust a fair time ago - neurons do not work like relus after all (nor do they work like memristors), but relus are great at solving problems. None of our current NN applications would even want something that behaves like a neuron. You'd essentially have to start from scratch. It's not even clear if behaves-like-a-neuron is better or not to be worth the effort. (And that's disregarding the entire fact that memristors do not behave like neurons either!)
- andy_ppp 6y ago
  Do we even understand how a neuron works well enough to say that either thing is correct. I thought a relu was just the activation function that decides what weight to triggers the next part of the network with?
  - freeone3000 6y ago
    We understand enough about how a biological neuron works to know that single signals do not provoke a diminished response after the hysterisis period, which is not modeled by either a memristor or a sigmoid. Sigmoids don't have hysterisis, and you have to backflow a memristor for it to recover which is not true of biological nuerons. (The activation function is the neuron('s mathematical model) - we elide everything that happens internal to the neuron and model it as only its inputs and outputs.)
- bnegreve 6y ago
  I don't think Relus are the proplem. Relus are good, but artificial neural networks can work with sigmoid activation fonctions instead. The neurones in an ANN with sigmoids are somewhat close to what is described in this article. The main problem is that the neurons in this article are not trainable (you can't update the weights dynamically) and not differentiable (you can't use the usual gradient-based training procedures to update the weights). They could be used for inference only though.
- sgt101 6y ago
  Wait! Some people have left biomimicry in the dust and are manipulating matrices to sets find optimal parameter sets representing concepts. Other people (https://www.humanbrainproject.eu/en/ https://www.humanbrainproject.eu/en/) haven't! There is a big community of people building novel types of neural network using spikes and so on. And you are right - it isn't clear who is right. On one side the "no nature" folks might say : we don't even understand neurons to the point where we can definitely categorize them in a mouse brain, let alone explain their behaviour, or actually properly measure their behaviour, and so on and so on - this before understanding any of it really. On the other hand the "lets mimic things we don't understand" folks would say - we need to understand this because the human brain out performs all AI today on a 20 watt budget. My view is the understand stuff agenda is right - but not now and probably not for 20 years (in a big way). We need small scale capability and direction finding work - and we will need to do that for a long time.
- unishark 6y ago
  I thin it's more that we're operating in a position of extreme simplification, and don't understand most biological details well enough to apply them in our own systems. But biological motivation is quite common in neural networks. Name a popular technique (backprop, dropout, etc.) and I can probably find you a source arguing it's biological relationship. It's just at an engineering-style level of approximation that scientists may abhor. The ReLU was originally introduced by computational neuroscientists as a more biologically accurate approximation compared to the other activation functions. E.g. https://www.nature.com/articles/35016072 https://www.nature.com/articles/35016072 I would expect similar history behind convolutional neural networks, given how the early stages of the vision system have been understood to operate.
- orbifold 6y ago
  Research in this area has always been cyclical. The Neural ODE paper that won a best paper award at Neurips two years ago just implements a technique that is common knowledge in the optimal control theory community since the 60s. I wouldn’t dismiss an approach because it isn’t popular right now. Biological neurons force you to think about constraints you normally wouldn’t and crucially have solved temporal credit assignment, memory etc. in interesting ways that already have inspired some breakthroughs. As an example the replay memory idea was directly inspired by the rough understanding neuroscientist developed of the role of the hypocampus in learning.
dfischer 6y ago
These guys seem to be working on it in that way: https://knowm.org/ https://knowm.org/