7 ms·

The article doesn't mention "embedding" even once. How can an argument about discontinuities in language space leave out things like word2vec, etc, which are de

The article doesn't mention "embedding" even once. How can an argument about discontinuities in language space leave out things like word2vec, etc, which are designed to make things continuous?

empath75 10y ago
He sort of does, and then says that the information loss from that process will make it ineffective for NLP. I don't think anyone deny that 'true' translation would require some kind of general intelligence that somehow understands what is being translated, but it seems to be the case that a 'dumb' translation works well enough for a great many use cases, regardless. He's really just making the Chinese room argument. We have a computer shuffling symbols around according to some rule set, that doesn't know what they mean. I don't think it really matters, though, if it produces a reasonably accurate translation.
- deong 10y ago
  In a way, he's making an even stronger argument than Searle did in his Chinese Room. Even Searle would probably admit that it's possible in principle for the computer in the chinese room argument to fool people into thinking it's intelligent. Searle just objects to the idea that it could ever really be intelligent in the way a human is. To Searle, the human obviously isn't just running some algorithms on the input text to produce output. I think the counter to Searle's argument isn't really that it doesn't matter as long as the result is close enough. The counter to that is that we don't understand how human intelligence works either. Searle is simply assuming that it's "magic" (or less condescendingly, some sort of metaphysical process) that can't be simulated by algorithmic machine. I think it's far more likely that intelligence is physical and we just don't understand the machinery than it is that it's mystical and cannot in principle ever be understood. For this article, all that is seemingly unnecessary. He's just saying they won't work well enough to even fake it convincingly. Which is very nearly falsifiable just by running today's algorithms.
  - westoncb 10y ago
    Searle's Chinese Room is addressing something different from intelligence; it's concerned with the what happens /within/ the intelligent mind, whereas inputs and outputs are the only things that matter here. To be more specific, in the question of whether deep learning can be used to generate and/or comprehend natural language, we are not concerned with whether the algorithm is conscious of what it's doing as longs as the results are good.
    - visarga 10y ago
      In order to be conscious it has to be more than a reactive or feedforward system. It has to loop back on itself, like RNNs, and hold internal state.
- rvense 10y ago
  Accuracy is what matters if you're trying to build something that translates. You can judge if it's good or not based on (subjective or objective) measures of accuracy. But if you're at 75% per cent and want to get to 100%, you need to understand what the problem is with the rest. And if it's 75% of "perfect translation of single written sentences from newspapers or technical litterature", how far along is that towards something "being part of an everyday conversation"? I studied linguistics at a university where focus was very much spoken language, sociolinguistics, language in context, before moving into (or through) NLP, and the distance between what a statistical machine translation system is able to handle and the stuff I used to work with is very large. A lot of NLP work now seems to focus on algorithms, but intuitively it seems to me that a much larger issue is the quality of the data, in the sense that humans don't learn language from piles of isolated text and somehow we're expecting machines to do it.. Rext is a lossy encoding of spoken language, even if you try your best to mimic it, but more seriously it does not include the physical context that children encounter language in. The learning situations aren't the same, I don't know why we're expecting the results to be.
  - empath75 10y ago
    What's the purpose of a translation? I think 'non-intelligent' machine translation should eventually be fairly effective at producing a grammatically correct gloss of basically any sentence as long as there isn't too much ambiguity. What machine translation isn't going to do is capture emotion or style or understand what someone is saying without them really saying it and so on. I would be very surprised if a machine translated novel ever hits the best seller charts. But I bet translators are going to be working from machine translated glosses if they aren't already.
- visarga 10y ago
  The Chinese room argument is just a modern rehash of incredulity that physical systems could give rise to consciousness - which is absurd because that's what we know we are. The argument goes - "when you look inside, it's just things pushing at each other. How could that produce perception and the conscious mind?" So, a failure of imagination and incredulity based on how they understand the world and the mind makes them reject AI. They feel that the special place of the soul was traded for "information processing" which is dry and mechanical - a form of dualism creeping up in our day and age. I would have felt the same if I didn't learn and use neural networks such as CNNs, RNNs and MLPs. Now I know how simple mechanical systems can recognize patterns and process information to generate complex behavior and I don't feel that "explanatory gap" any more. Reinforcement learning is a good base for consciousness research - much more precise and with scientific results, not just p-zombies and bat based armchair experimentation. There's a limit where you can go with just pure thinking and then you need to start direct implementation.
sprobertson 10y ago
And bases the argument on a single feed forward network, nothing about RNNs which most good NN NLP results are based on.
- habeanf 10y ago
  Google's SyntaxNet (as of Andor et al 2016) achieves state-of-the-art accuracy for dependency parsing without RNNs.

The article doesn't mention "embedding" even once. How can an argument about discontinuities in language space leave out things like word2vec, etc, which are designed to make things continuous?

empath75 10y ago

He sort of does, and then says that the information loss from that process will make it ineffective for NLP. I don't think anyone deny that 'true' translation would require some kind of general intelligence that somehow understands what is being translated, but it seems to be the case that a 'dumb' translation works well enough for a great many use cases, regardless. He's really just making the Chinese room argument. We have a computer shuffling symbols around according to some rule set, that doesn't know what they mean. I don't think it really matters, though, if it produces a reasonably accurate translation.

deong 10y ago
In a way, he's making an even stronger argument than Searle did in his Chinese Room. Even Searle would probably admit that it's possible in principle for the computer in the chinese room argument to fool people into thinking it's intelligent. Searle just objects to the idea that it could ever really be intelligent in the way a human is. To Searle, the human obviously isn't just running some algorithms on the input text to produce output. I think the counter to Searle's argument isn't really that it doesn't matter as long as the result is close enough. The counter to that is that we don't understand how human intelligence works either. Searle is simply assuming that it's "magic" (or less condescendingly, some sort of metaphysical process) that can't be simulated by algorithmic machine. I think it's far more likely that intelligence is physical and we just don't understand the machinery than it is that it's mystical and cannot in principle ever be understood. For this article, all that is seemingly unnecessary. He's just saying they won't work well enough to even fake it convincingly. Which is very nearly falsifiable just by running today's algorithms.
- westoncb 10y ago
  Searle's Chinese Room is addressing something different from intelligence; it's concerned with the what happens /within/ the intelligent mind, whereas inputs and outputs are the only things that matter here. To be more specific, in the question of whether deep learning can be used to generate and/or comprehend natural language, we are not concerned with whether the algorithm is conscious of what it's doing as longs as the results are good.
  - visarga 10y ago
    In order to be conscious it has to be more than a reactive or feedforward system. It has to loop back on itself, like RNNs, and hold internal state.
rvense 10y ago
Accuracy is what matters if you're trying to build something that translates. You can judge if it's good or not based on (subjective or objective) measures of accuracy. But if you're at 75% per cent and want to get to 100%, you need to understand what the problem is with the rest. And if it's 75% of "perfect translation of single written sentences from newspapers or technical litterature", how far along is that towards something "being part of an everyday conversation"? I studied linguistics at a university where focus was very much spoken language, sociolinguistics, language in context, before moving into (or through) NLP, and the distance between what a statistical machine translation system is able to handle and the stuff I used to work with is very large. A lot of NLP work now seems to focus on algorithms, but intuitively it seems to me that a much larger issue is the quality of the data, in the sense that humans don't learn language from piles of isolated text and somehow we're expecting machines to do it.. Rext is a lossy encoding of spoken language, even if you try your best to mimic it, but more seriously it does not include the physical context that children encounter language in. The learning situations aren't the same, I don't know why we're expecting the results to be.
- empath75 10y ago
  What's the purpose of a translation? I think 'non-intelligent' machine translation should eventually be fairly effective at producing a grammatically correct gloss of basically any sentence as long as there isn't too much ambiguity. What machine translation isn't going to do is capture emotion or style or understand what someone is saying without them really saying it and so on. I would be very surprised if a machine translated novel ever hits the best seller charts. But I bet translators are going to be working from machine translated glosses if they aren't already.
visarga 10y ago
The Chinese room argument is just a modern rehash of incredulity that physical systems could give rise to consciousness - which is absurd because that's what we know we are. The argument goes - "when you look inside, it's just things pushing at each other. How could that produce perception and the conscious mind?" So, a failure of imagination and incredulity based on how they understand the world and the mind makes them reject AI. They feel that the special place of the soul was traded for "information processing" which is dry and mechanical - a form of dualism creeping up in our day and age. I would have felt the same if I didn't learn and use neural networks such as CNNs, RNNs and MLPs. Now I know how simple mechanical systems can recognize patterns and process information to generate complex behavior and I don't feel that "explanatory gap" any more. Reinforcement learning is a good base for consciousness research - much more precise and with scientific results, not just p-zombies and bat based armchair experimentation. There's a limit where you can go with just pure thinking and then you need to start direct implementation.

sprobertson 10y ago

And bases the argument on a single feed forward network, nothing about RNNs which most good NN NLP results are based on.

habeanf 10y ago
Google's SyntaxNet (as of Andor et al 2016) achieves state-of-the-art accuracy for dependency parsing without RNNs.