5 ms·

This paper builds off of DeepMind's previous work on differentiable computation: Neural Turing Machines. That paper generated a lot of enthusiasm when it came o

by rkaplan 10y ago

This paper builds off of DeepMind's previous work on differentiable computation: Neural Turing Machines. That paper generated a lot of enthusiasm when it came out in 2014, but not many researchers use NTMs today. The feeling among researchers I've spoken to is not that NTMs aren't useful. DeepMind is simply operating on another level. Other researchers don't understand the intuitions behind the architecture well enough to make progress with it. But it seems like DeepMind, and specifically Alex Graves (first author on NTMs and now this), can.

wiz21c 10y ago
>> DeepMind is simply operating on another level. Would you be so kind as to to explain what you mean here ? Thanks !
- outsideline 10y ago
  They're taking features that are present in the brain that aren't modeled and are making computational models for them. They're not a gold standard. You can create your own in under an hour. It's not another level. It's bio-inspired computing. Here.. take the 'Axon Hillock' https://en.wikipedia.org/wiki/Axon_hillock https://en.wikipedia.org/wiki/Axon_hillock code up a function for it, attach it to present day neuron models, make it do something fancy, write a white-paper and kazaam you're operating on another level.. Get it?
  - wiz21c 10y ago
    ok I get it :) Nice little sarcasm, I'm loving it :-)
visarga 10y ago
Deep mind is breaking new ground in number of directions. For example, "Decoupled Neural Interfaces using Synthetic Gradients" is simply amazing - they can make training a net async and run individual layers on separate machines by approximating the gradients with a local net. It's the kind of thing that sounds crazy on paper, but they proved it works. Another amazing thing they did was to generate audio by direct synthesis from a neural net, beating all previous benchmarks. If they can make it work in real time, it would be a huge upgrade in our TTS technology. We're still waiting for the new and improved AlphaGo. I hope they don't bury that project.
- outsideline 10y ago
  Decoupled Neural Interfaces using Synthetic Gradients is a fancy name for the electro-chemical gradient that lies outside the cell wall of neurons : https://en.wikipedia.org/wiki/Electrochemical_gradient https://en.wikipedia.org/wiki/Electrochemical_gradient It's decoupled yet stores transient local information regarding previous neuron activity. Another bio-inspired copy-pasta.
  - vintermann 10y ago
    You should absolutely get a job doing it, if you think bio-inspired copy-pasta is all it takes. May I recommend Numenta?
  - dj-wonk 10y ago
    Please choose derogatory phrases like 'copy pasta' intentionally and carefully. Many algorithms are bio-inspired -- good artists borrow, the best steal.
  - deleted 10y ago
    [deleted]
- shostack 10y ago
  I'm not super knowledgeable about the space, but would the audio generation you mentioned be what is needed to let their Assistant communicate verbally in any language, any voice, add inflections, emotion, etc. without needing to pre-record all the chunks/combinations?
usgroup 10y ago
Any chance you could fix this statement: Input = Data Process = Optimisation to create an automata. Output = Automata Computer power means much larger variable spaces can be handled in optimisation problems. NN are a means to prune the variable space during optimisation in a domain unspecific way.
deleted 10y ago
[deleted]
modeless 10y ago
The reason other researchers haven't jumped on NTMs may be that, unlike commonly-researched types of neural nets such as CNNs or RNNs, NTMs are not currently the best way to solve any real-world problem. The problems they have solved so far are relatively trivial, and they are very inefficient, inaccurate, and complex relative to traditional CS methods (e.g. Dijkstra's algorithm coded in C). That's not to say that NTMs are bad or uninteresting! They are super cool and I think have huge potential in natural language understanding, reasoning, and planning. However, I do think that DeepMind will have to prove that they can be used to solve some non-trivial task, one that can't be solved much more efficiently with traditional CS methods, before people will join in to their research. Also, I think there's a possibility that solving non-trivial problems with NTMs may require more computing power than Moore's law has given us so far. In the same way that NNs didn't really take off until GPU implementations became available, we may have to wait for the next big hardware breakthrough for NTMs to come into their own.
- svantana 10y ago
  They sure put a lot of focus on "toy" problems such as sorting and path planning in their papers - perhaps because they are easy to understand and show a major improvement over other ML approaches. IMHO they should focus more on "real" problems - e.g. in Table 1 of this paper it seems to be state of the art on the bAbl tasks, which is amazing.
  - sherjilozair 10y ago
    bAbI isn't really a "real" problem either, although somewhat better than sorting and the like. bAbI works with extremely restrictive worlds and grammar. In contrast, current speech recognition, language modeling, and object detection do quite well with actual audio, text, and pictures. I think the strength of NTMs will be best demonstrated by putting it to work on a long-range language modeling task where you need to organize what you read so that you can use it to predict better a paragraph or two later. Current language models based on LSTM are not really able to do this.
  - TeeWEE 10y ago
    Once you have a learning machine that can solve simple problems. You can scale it up to solve very complex problems. Its a first step to true AI imho. Al lot of small steps are needed to go towards this goal. Integrating Memory & Neural Nets is a big step imho.
outsideline 10y ago
Alan Turing's tape machine + neuron model. In the human brain, Neurons store an incredible amount of information. Neuron models in neural networks only did so with weights. There is still a lack of understanding on how the human brain does it. Deep Mind grabbed a proven memory model from Alan Turing's work and applied it to the feature barren neuron models in use. Sprinkle magic ... They are not operating on another level, they're bringing over features that are well documented in the human brain and in white papers from a past period when people actually thought deeply about this problem and applying it. https://en.wikipedia.org/wiki/Bio-inspired_computing https://en.wikipedia.org/wiki/Bio-inspired_computing There is no 'intuition' about the architecture. Study the human brain and copy pasta into the computing realm. Others are doing this as well. If anyone bothered to read the white papers people publish, you'll see that many people have presented similar ideas over the years. You can come up with your own neural Turing machine. Take a featureless neuron model, slap a memory module on it and you have a neural turing machine.
- vintermann 10y ago
  In order to use a turing machine in a neural network - or at least to train it, in any way that isn't impractical and/or cheating - you need to make it differentiable somehow. Graves and co. have been really creative in overcoming problems in their ongoing program to differentiate ALL the things.
  - igravious 10y ago
    In this context what does differentiable mean?
    - ebalit 10y ago
      It means it can be trained by backpropagating the error gradient through the network. To train a neural network, you want to know how much each component contributed to an error. We do that by propagating the error through each component in reverse, using the partial derivatives of the corresponding function.
    - 10y ago

by rkaplan 10y ago

wiz21c 10y ago

>> DeepMind is simply operating on another level. Would you be so kind as to to explain what you mean here ? Thanks !

outsideline 10y ago
They're taking features that are present in the brain that aren't modeled and are making computational models for them. They're not a gold standard. You can create your own in under an hour. It's not another level. It's bio-inspired computing. Here.. take the 'Axon Hillock' https://en.wikipedia.org/wiki/Axon_hillock https://en.wikipedia.org/wiki/Axon_hillock code up a function for it, attach it to present day neuron models, make it do something fancy, write a white-paper and kazaam you're operating on another level.. Get it?
- wiz21c 10y ago
  ok I get it :) Nice little sarcasm, I'm loving it :-)

visarga 10y ago

Deep mind is breaking new ground in number of directions. For example, "Decoupled Neural Interfaces using Synthetic Gradients" is simply amazing - they can make training a net async and run individual layers on separate machines by approximating the gradients with a local net. It's the kind of thing that sounds crazy on paper, but they proved it works. Another amazing thing they did was to generate audio by direct synthesis from a neural net, beating all previous benchmarks. If they can make it work in real time, it would be a huge upgrade in our TTS technology. We're still waiting for the new and improved AlphaGo. I hope they don't bury that project.

outsideline 10y ago
Decoupled Neural Interfaces using Synthetic Gradients is a fancy name for the electro-chemical gradient that lies outside the cell wall of neurons : https://en.wikipedia.org/wiki/Electrochemical_gradient https://en.wikipedia.org/wiki/Electrochemical_gradient It's decoupled yet stores transient local information regarding previous neuron activity. Another bio-inspired copy-pasta.
- vintermann 10y ago
  You should absolutely get a job doing it, if you think bio-inspired copy-pasta is all it takes. May I recommend Numenta?
- dj-wonk 10y ago
  Please choose derogatory phrases like 'copy pasta' intentionally and carefully. Many algorithms are bio-inspired -- good artists borrow, the best steal.
- deleted 10y ago
  [deleted]
shostack 10y ago
I'm not super knowledgeable about the space, but would the audio generation you mentioned be what is needed to let their Assistant communicate verbally in any language, any voice, add inflections, emotion, etc. without needing to pre-record all the chunks/combinations?

usgroup 10y ago

Any chance you could fix this statement: Input = Data Process = Optimisation to create an automata. Output = Automata Computer power means much larger variable spaces can be handled in optimisation problems. NN are a means to prune the variable space during optimisation in a domain unspecific way.

deleted 10y ago

[deleted]

modeless 10y ago

The reason other researchers haven't jumped on NTMs may be that, unlike commonly-researched types of neural nets such as CNNs or RNNs, NTMs are not currently the best way to solve any real-world problem. The problems they have solved so far are relatively trivial, and they are very inefficient, inaccurate, and complex relative to traditional CS methods (e.g. Dijkstra's algorithm coded in C). That's not to say that NTMs are bad or uninteresting! They are super cool and I think have huge potential in natural language understanding, reasoning, and planning. However, I do think that DeepMind will have to prove that they can be used to solve some non-trivial task, one that can't be solved much more efficiently with traditional CS methods, before people will join in to their research. Also, I think there's a possibility that solving non-trivial problems with NTMs may require more computing power than Moore's law has given us so far. In the same way that NNs didn't really take off until GPU implementations became available, we may have to wait for the next big hardware breakthrough for NTMs to come into their own.

svantana 10y ago
They sure put a lot of focus on "toy" problems such as sorting and path planning in their papers - perhaps because they are easy to understand and show a major improvement over other ML approaches. IMHO they should focus more on "real" problems - e.g. in Table 1 of this paper it seems to be state of the art on the bAbl tasks, which is amazing.
- sherjilozair 10y ago
  bAbI isn't really a "real" problem either, although somewhat better than sorting and the like. bAbI works with extremely restrictive worlds and grammar. In contrast, current speech recognition, language modeling, and object detection do quite well with actual audio, text, and pictures. I think the strength of NTMs will be best demonstrated by putting it to work on a long-range language modeling task where you need to organize what you read so that you can use it to predict better a paragraph or two later. Current language models based on LSTM are not really able to do this.
- TeeWEE 10y ago
  Once you have a learning machine that can solve simple problems. You can scale it up to solve very complex problems. Its a first step to true AI imho. Al lot of small steps are needed to go towards this goal. Integrating Memory & Neural Nets is a big step imho.

outsideline 10y ago

Alan Turing's tape machine + neuron model. In the human brain, Neurons store an incredible amount of information. Neuron models in neural networks only did so with weights. There is still a lack of understanding on how the human brain does it. Deep Mind grabbed a proven memory model from Alan Turing's work and applied it to the feature barren neuron models in use. Sprinkle magic ... They are not operating on another level, they're bringing over features that are well documented in the human brain and in white papers from a past period when people actually thought deeply about this problem and applying it. https://en.wikipedia.org/wiki/Bio-inspired_computing https://en.wikipedia.org/wiki/Bio-inspired_computing There is no 'intuition' about the architecture. Study the human brain and copy pasta into the computing realm. Others are doing this as well. If anyone bothered to read the white papers people publish, you'll see that many people have presented similar ideas over the years. You can come up with your own neural Turing machine. Take a featureless neuron model, slap a memory module on it and you have a neural turing machine.

vintermann 10y ago
In order to use a turing machine in a neural network - or at least to train it, in any way that isn't impractical and/or cheating - you need to make it differentiable somehow. Graves and co. have been really creative in overcoming problems in their ongoing program to differentiate ALL the things.
- igravious 10y ago
  In this context what does differentiable mean?
  - ebalit 10y ago
    It means it can be trained by backpropagating the error gradient through the network. To train a neural network, you want to know how much each component contributed to an error. We do that by propagating the error through each component in reverse, using the partial derivatives of the corresponding function.
  - 10y ago