3 ms·

It's not clear we have enough hardware. I've been trying to get a handle on the processing power needed for human level AI. If you take the naive approach and a

It's not clear we have enough hardware. I've been trying to get a handle on the processing power needed for human level AI. If you take the naive approach and assume it's just a matter of getting models with the same number of parameters as the brain has synapses, our biggest models are about a million times too small. If you assume Moore's law will remain dead, then it may be a long while. Then again if modern deep learning is amenable to some sort of ASIC it could happen much quicker than one would expect, depending on how much investment is put into such things. And who knows how algorithmic advances will change things. If we ever get quantum computers, they're likely to be very useful in the design of conventional computers and materials technology in general, so another age of exponential advancement isn't out of the question. It's very, very hard to be certain about these things.

kaveh_h 11y ago
As I've come to understand it when implementing deep learning machine learning the hardware requirement for running a trained neural network model efficiently is much lower than the one required to train it. I've also read somewhere that if an instance of AlphaGo run on a modern powerful desktop machine would compete with the cluster used in Lee See-dol match which consisted of something close to 2000 of CPUs and more than 100 gpus then the ratio in wins would be only be 3:1 in favor of the cluster. Much less than if you would compare it to the ratio of computational power so that gives you hint of how much of the intelligence in it is about previous knowledge gained vs brute computing power. To me that also implies that a lot of the innovation done at DeepMind is more about improving the speed of which a neural net learns and aquires knowledge and less about the running time of applying the model. And that's probably an important distinction in the discussion of this technology.
aab0 11y ago
> If you take the naive approach and assume it's just a matter of getting models with the same number of parameters as the brain has synapses, our biggest models are about a million times too small. And if you had that hardware, what would you run on it, exactly? There is no off the shelf algorithm which we could plop on that futuristic GPU and say, yes, this will definitely yield a general human-level intelligence. That is my point. We may or may not have the necessary hardware, but we sure don't have the necessary software right now.
- Moshe_Silnorin 11y ago
  I agree. But I often here the claim that we have enough hardware and now it's just a software problem. I don't think that's at all clear.

kaveh_h 11y ago

As I've come to understand it when implementing deep learning machine learning the hardware requirement for running a trained neural network model efficiently is much lower than the one required to train it. I've also read somewhere that if an instance of AlphaGo run on a modern powerful desktop machine would compete with the cluster used in Lee See-dol match which consisted of something close to 2000 of CPUs and more than 100 gpus then the ratio in wins would be only be 3:1 in favor of the cluster. Much less than if you would compare it to the ratio of computational power so that gives you hint of how much of the intelligence in it is about previous knowledge gained vs brute computing power. To me that also implies that a lot of the innovation done at DeepMind is more about improving the speed of which a neural net learns and aquires knowledge and less about the running time of applying the model. And that's probably an important distinction in the discussion of this technology.

aab0 11y ago

> If you take the naive approach and assume it's just a matter of getting models with the same number of parameters as the brain has synapses, our biggest models are about a million times too small. And if you had that hardware, what would you run on it, exactly? There is no off the shelf algorithm which we could plop on that futuristic GPU and say, yes, this will definitely yield a general human-level intelligence. That is my point. We may or may not have the necessary hardware, but we sure don't have the necessary software right now.

Moshe_Silnorin 11y ago
I agree. But I often here the claim that we have enough hardware and now it's just a software problem. I don't think that's at all clear.