4 ms·

I’m in no way an expert on this stuff, but I’ve long wondered if AI is really a hardware problem. That is, the way that conventional computers work is nothing l

I’m in no way an expert on this stuff, but I’ve long wondered if AI is really a hardware problem. That is, the way that conventional computers work is nothing like the way brains work, and if we’re trying to model brain-like intelligence, perhaps we need brain-like hardware to do it. It’s fascinating stumbling across more and more articles describing neuromorphic hardware like this recently. I guess this idea has occurred to a lot of other people.

onemoresoop 6y ago
We are trying everything I guess and the reason it’s happening in silicon is because thats whats available mainstream and it’s possible the wrong road. Regardless, at this point Im sure we’ll get there sooner or later. But there are serious questions like what will we do with it or will we, as humans, become completely irrelevant at that point? Lots of questions worth pondering about
ken47 6y ago
Perhaps progress requires a move away from human-like brains. There are infinitely many kinds of potential intelligence. It's very unlikely ours is the pinnacle.
- telmo 6y ago
  I know of almost no human-like brain inspiration in current AI models. Current ANNs are similar to biological brains, but only in the sense that they can be understood as networks of threshold units. One can also understand most modern ANN models purely in terms of linear algebra. They are super-abstract, super-generic models of computation, and biological brains happen to fit this category (at a very high level of simplification and abstraction). Then, gradient descent is a centralized and supervised learning algorithm that has nothing to do with all the decentralized and emergent processes that take place in biological brains. For example, consider all the complex interactions of the various neurotransmitters and neuroreceptors. Biological brains (let alone human) are actual physical realizations of networks with very complex and specific topologies. New connections grow and are formed in a very real sense, and this process interacts with the environment in a variety of ways, plus it is spatially embedded.
  - JoeAltmaier 6y ago
    Human brains (neural networks) are little like networks of threshold units, as you observe. Neurons can process time-dependent integrations and differentiations with dynamic thresholds. Not the tiny static networks that AI uses. More like an analogue computer but with more types of computing elements.
d_tr 6y ago
I am also in no way an expert, but I would not say it is "really a hardware problem". There is already serious research going on neuromorphic computing, but you have to know what hardware to build and how to actually use / train / evolve it. We are trying to imitate our brains and each single neuron has very rich dynamics. Then you have a general graph with 80 billions of them, and this network also interacts with and is regulated by another network of chemical reactions... One could maybe give up understanding and build a fairly generalized evolutionary process, but that hardware could end up needing a degree of flexibility that would make it either closer to a biological brain grown in a lab or something much closer to a conventional computer...
hombre_fatal 6y ago
Is there hardware that we cannot model in software given enough knowledge of the domain?
- TeMPOraL 6y ago
  Depends on the values of "enough". Organic chemistry comes to mind - we can tell what proteins are floating around your body, but we can't tell how exactly they're interacting, as it's computationally intractable. Perhaps that qualifies as "not enough knowledge", but it seems to me that this is a kind of problem where you'll only have enough knowledge once you can do a molecular-scale simulation, and there isn't much to simplify there.
mantap 6y ago
It's actually the reverse. The hardware we have is way more efficient than our brains. One TPU can simulate billions of neurons a second in a tiny area. Computer hardware has a significant time advantage. Electrical impulses in the brain travel very slowly. The problem is we don't understand how the brain is architected at anything other than a very coarse level. That architecture is the product of millions of years of evolution so we have some catching up to do.
- d_tr 6y ago
  Except that the TPU does not simulate real neurons in any way. It just performs simple arithmetic using an architecture that is suited to the ANNs we use today. If you managed to understand how the brain works and tried to run the model on a cluster of TPUs, you would need way more energy to simulate one second of brain activity than the 20J a real brain uses in a second. Only by some magical coincidence would any of the hardware we use today turn out to be efficient.
  - darkerside 6y ago
    Are neurons binary in the way they operate, in that they either fire or they don't? Or are they analog in nature, where a signal can be either strong, weak, nonexistent or anywhere in between?
    - qubex 6y ago
      For starters, they use a pulse-frequency based signalling system. And more generally, it’s pretty clear that there is a mixture of context-sensitive inhibitory and excitatory signals both deriving from the dendrites (inputs) and from the chemical stew of hormones and so forth they find themselves immersed in. There are neurobiologists who claim (controversially, I concede) that a single neurone is a fully self-contained ‘processor’[1], and not the equivalent of single logic gate. Those who somehow confuse our “neural network architectures” with the actual biology in our skulls are really misguided. Based on the evidence, one can at best opine that a TPU-based matrix add/multiply unit is approximating a very clunky and un-nuanced neural network. To assert that they are one and the same is... indefensible, when one goes beyond even the coarsest detail. A neural network as we implement it technologically is nothing but a weighted network of threshold units. That is all. [1] To clarify: some kind of highly context-aware DSP with both long- and short-term storage facilities that allow it to react not only to the inputs that it is being fed, but also to inputs it was fed previously (and possibly outputs it produced previously as well).

onemoresoop 6y ago

We are trying everything I guess and the reason it’s happening in silicon is because thats whats available mainstream and it’s possible the wrong road. Regardless, at this point Im sure we’ll get there sooner or later. But there are serious questions like what will we do with it or will we, as humans, become completely irrelevant at that point? Lots of questions worth pondering about

ken47 6y ago

Perhaps progress requires a move away from human-like brains. There are infinitely many kinds of potential intelligence. It's very unlikely ours is the pinnacle.

telmo 6y ago
I know of almost no human-like brain inspiration in current AI models. Current ANNs are similar to biological brains, but only in the sense that they can be understood as networks of threshold units. One can also understand most modern ANN models purely in terms of linear algebra. They are super-abstract, super-generic models of computation, and biological brains happen to fit this category (at a very high level of simplification and abstraction). Then, gradient descent is a centralized and supervised learning algorithm that has nothing to do with all the decentralized and emergent processes that take place in biological brains. For example, consider all the complex interactions of the various neurotransmitters and neuroreceptors. Biological brains (let alone human) are actual physical realizations of networks with very complex and specific topologies. New connections grow and are formed in a very real sense, and this process interacts with the environment in a variety of ways, plus it is spatially embedded.
- JoeAltmaier 6y ago
  Human brains (neural networks) are little like networks of threshold units, as you observe. Neurons can process time-dependent integrations and differentiations with dynamic thresholds. Not the tiny static networks that AI uses. More like an analogue computer but with more types of computing elements.

d_tr 6y ago

I am also in no way an expert, but I would not say it is "really a hardware problem". There is already serious research going on neuromorphic computing, but you have to know what hardware to build and how to actually use / train / evolve it. We are trying to imitate our brains and each single neuron has very rich dynamics. Then you have a general graph with 80 billions of them, and this network also interacts with and is regulated by another network of chemical reactions... One could maybe give up understanding and build a fairly generalized evolutionary process, but that hardware could end up needing a degree of flexibility that would make it either closer to a biological brain grown in a lab or something much closer to a conventional computer...

hombre_fatal 6y ago

Is there hardware that we cannot model in software given enough knowledge of the domain?

TeMPOraL 6y ago
Depends on the values of "enough". Organic chemistry comes to mind - we can tell what proteins are floating around your body, but we can't tell how exactly they're interacting, as it's computationally intractable. Perhaps that qualifies as "not enough knowledge", but it seems to me that this is a kind of problem where you'll only have enough knowledge once you can do a molecular-scale simulation, and there isn't much to simplify there.

mantap 6y ago

It's actually the reverse. The hardware we have is way more efficient than our brains. One TPU can simulate billions of neurons a second in a tiny area. Computer hardware has a significant time advantage. Electrical impulses in the brain travel very slowly. The problem is we don't understand how the brain is architected at anything other than a very coarse level. That architecture is the product of millions of years of evolution so we have some catching up to do.

d_tr 6y ago
Except that the TPU does not simulate real neurons in any way. It just performs simple arithmetic using an architecture that is suited to the ANNs we use today. If you managed to understand how the brain works and tried to run the model on a cluster of TPUs, you would need way more energy to simulate one second of brain activity than the 20J a real brain uses in a second. Only by some magical coincidence would any of the hardware we use today turn out to be efficient.
- darkerside 6y ago
  Are neurons binary in the way they operate, in that they either fire or they don't? Or are they analog in nature, where a signal can be either strong, weak, nonexistent or anywhere in between?
  - qubex 6y ago
    For starters, they use a pulse-frequency based signalling system. And more generally, it’s pretty clear that there is a mixture of context-sensitive inhibitory and excitatory signals both deriving from the dendrites (inputs) and from the chemical stew of hormones and so forth they find themselves immersed in. There are neurobiologists who claim (controversially, I concede) that a single neurone is a fully self-contained ‘processor’[1], and not the equivalent of single logic gate. Those who somehow confuse our “neural network architectures” with the actual biology in our skulls are really misguided. Based on the evidence, one can at best opine that a TPU-based matrix add/multiply unit is approximating a very clunky and un-nuanced neural network. To assert that they are one and the same is... indefensible, when one goes beyond even the coarsest detail. A neural network as we implement it technologically is nothing but a weighted network of threshold units. That is all. [1] To clarify: some kind of highly context-aware DSP with both long- and short-term storage facilities that allow it to react not only to the inputs that it is being fed, but also to inputs it was fed previously (and possibly outputs it produced previously as well).