5 ms·

It is already succeeding on language tasks, see https://research.facebook.com/research/babi/ https://research.facebook.com/research/babi/ It is funny how every

It is already succeeding on language tasks, see https://research.facebook.com/research/babi/ https://research.facebook.com/research/babi/ It is funny how every AI post on HN turns into a speculative discussion forum full of words "I think", "likely", "I suspect", "My guess" etc, when all the research is available for free and everyone is free to download and read it to get a real understanding of what's going on in the field. >what I see is the fact that networks have huge capacity to fit to data and are deep (rely on a hierarchy of features). Actually recurrent neural networks like LSTM are turing-complete, i.e. for every halting algorithm it is trivial to implement an RNN that computes it. It is non-trivial to learn these parameters from algorithm IO data, but for many tasks it is possible too. >I suspect that most of what is being learned (and solved) is perception of useful features from the raw game images. It is not this simple, deep enough convnets can represent computations, the consensus is that middle and upper layers of convnets represent some useful computation steps. Also note that human brain can only do so much computation steps to answer questions when in dialogue, due to time and speed limits. >My guess is we may need a fundamental breakthrough in a newfangled hierarchical learning system that is better suited for language to “solve” NLP. This is being worked on, see the first link for Memory Networks and Stack RNNs, DeQue RNNs, Tree RNNs. Deep learning is a very generic term, there are dozens of various feedforward and recurrent architectures that are fully differentiable. The full potential of such models has not been nearly reached yet and maybe language understanding will be solved in the coming years (again, the first link shows that it is in process of being solved).

brianchu 10y ago
I specifically and mindfully added those words because everything is really an open research question. Would you rather I dissembled a false sense of confidence? If anything, you're stating your vague case way over-confidently. Turing-completeness is broad and nonspecific. Doing "some computation" is an obvious statement that doesn't add any information. The human brain does not seem to have time limits when it comes to thinking about what to say, and further we don't understand enough about neuroscience to make statements like that. Like I said, these are all active areas of research; the jury is still out on whether any specific approach will be the breakthrough. EDIT (reply to below): in general these statements are either vague and nonspecific, or perfectly correct and non-informative, comments that don't have much to do with my original point.
- sapphireblue 10y ago
  I agree to your points. Your comment is a quality one, I mostly talked about other ones. >Turing-completeness is quite broad and nonspecific, like I said. It is, but feedforward models (and almost every Bayesian/statistical model) don't possess it even in theory, while RNNs do. >Doing "some computation" is an obvious statement that doesn't add any information. Let me be more specific: currently researchers think that later stages of CNNs do something that is more interpretable as computation than as mere pattern matching. Our world doesn't require 50-level hierarchy, but resnets with 50+ layers do good, looks like because they learn some non-trivial computation. >the jury is still out on whether any of those RNN approaches will be the needed breakthrough. Sure, we'll see. Maybe there won't be need in any breakthrough, just incremental improvement of models. And even current models when scaled up to next-gen hardware (see nervana) can surprise us again with their performance.
  - deleted 10y ago
    [deleted]
PaulHoule 10y ago
My skepticism is not about "succeeding" academically in the sense that research groups get better and better scores on Kaggle competitions. My skepticism is about success in the sense of commercially useful systems that can process language and function "off the leash" of human supervision without the results being dominated by unacceptably bad results. Look at the XBOX ONE Kinect vs the XBOX 360 Kinect. On paper the newer product is much better than the old product, but neither one is any easier or fun to use than picking up the gamepad. In the current paradigm, researchers can keep putting up better and better numbers without ever crossing the threshold to something anybody can make a living off.
ifdefdebug 10y ago
> It is funny how every AI post on HN turns into a speculative discussion forum full of words "I think", "likely", "I suspect", "My guess" etc This is probably due to the fact that the field is very interesting and has lots of undefined boundaries, so people like to take educated guesses based on the knowledge they might have and on their intuition. Fair enough for this discussion. > maybe language understanding will be solved in the coming years maybe? :) OK, here comes my guess: I think reasoning about and producing computer programs should be easier than reasoning about and producing natural language. So if that's possible (big if), then it should come first. And then maybe the NLP will be solved with the help of code writing computers. Or maybe just by code writing computers, and nobody here has a job anymore :)

brianchu 10y ago

I specifically and mindfully added those words because everything is really an open research question. Would you rather I dissembled a false sense of confidence? If anything, you're stating your vague case way over-confidently. Turing-completeness is broad and nonspecific. Doing "some computation" is an obvious statement that doesn't add any information. The human brain does not seem to have time limits when it comes to thinking about what to say, and further we don't understand enough about neuroscience to make statements like that. Like I said, these are all active areas of research; the jury is still out on whether any specific approach will be the breakthrough. EDIT (reply to below): in general these statements are either vague and nonspecific, or perfectly correct and non-informative, comments that don't have much to do with my original point.

sapphireblue 10y ago
I agree to your points. Your comment is a quality one, I mostly talked about other ones. >Turing-completeness is quite broad and nonspecific, like I said. It is, but feedforward models (and almost every Bayesian/statistical model) don't possess it even in theory, while RNNs do. >Doing "some computation" is an obvious statement that doesn't add any information. Let me be more specific: currently researchers think that later stages of CNNs do something that is more interpretable as computation than as mere pattern matching. Our world doesn't require 50-level hierarchy, but resnets with 50+ layers do good, looks like because they learn some non-trivial computation. >the jury is still out on whether any of those RNN approaches will be the needed breakthrough. Sure, we'll see. Maybe there won't be need in any breakthrough, just incremental improvement of models. And even current models when scaled up to next-gen hardware (see nervana) can surprise us again with their performance.
- deleted 10y ago
  [deleted]

PaulHoule 10y ago

My skepticism is not about "succeeding" academically in the sense that research groups get better and better scores on Kaggle competitions. My skepticism is about success in the sense of commercially useful systems that can process language and function "off the leash" of human supervision without the results being dominated by unacceptably bad results. Look at the XBOX ONE Kinect vs the XBOX 360 Kinect. On paper the newer product is much better than the old product, but neither one is any easier or fun to use than picking up the gamepad. In the current paradigm, researchers can keep putting up better and better numbers without ever crossing the threshold to something anybody can make a living off.

ifdefdebug 10y ago

> It is funny how every AI post on HN turns into a speculative discussion forum full of words "I think", "likely", "I suspect", "My guess" etc This is probably due to the fact that the field is very interesting and has lots of undefined boundaries, so people like to take educated guesses based on the knowledge they might have and on their intuition. Fair enough for this discussion. > maybe language understanding will be solved in the coming years maybe? :) OK, here comes my guess: I think reasoning about and producing computer programs should be easier than reasoning about and producing natural language. So if that's possible (big if), then it should come first. And then maybe the NLP will be solved with the help of code writing computers. Or maybe just by code writing computers, and nobody here has a job anymore :)