5 ms·

Superb reminder of the difficulty of true NLP. Google Translate is an amazing and very useful tool, but machines have a long way to go before they really unders

Superb reminder of the difficulty of true NLP. Google Translate is an amazing and very useful tool, but machines have a long way to go before they really understand language. Example sentence used by article: > In their house, everything comes in pairs. There’s his car and her car, his towels and her towels, and his library and hers

danso 9y ago
I agree with the thrust of the article, that Google Translate is fundamentally limited (as most systems are) by its inputs and ostensible goals. But what does it mean to "really understand language"? What is the recognizable benchmark for "true NLP"? At some point, how "good" comprehension/translation is very subjective, because it intersects so heavily with culture and history. Think of how debates still rage over interpreting Shakespeare among Western readers and academics, even though there isn't a language barrier.
- eesmith 9y ago
  I notice you quoted "really understand language." That is not a quote from the essay. Is it an indicator that you are taking the conversation in a different direction? Hofstadter has written a lot about the question of what it means to "really understand a language", and the related more relevant question of what it means to translate from one language to another. Several of his essays in "Metamagical Themas" in the 1980s covered those topics, and they are quite interesting. He gives an example "Mary was sick yesterday", with different levels of 'understanding'. Call it "sentence M" (quoting from https://archive.org/stream/MetamagicalThemas/Metamagical%20Themas%2C%20Hofstadter#page/n29/mode/2up/search/street https://archive.org/stream/MetamagicalThemas/Metamagical%20T... , including apparent typos) > 1. Sentence M contains twenty characters. > 2. Sentence M contains four English words. > 3. Sentence M contains one proper noun, one verb, one one adverb, in that order. > 4. Sentence M contains one human's name, one linking verb, one adjective describing a potential health state of a living being, and one temporal adverb, in that order. > 5. The subject of Sentence M is a pointer to an individual named 'Mary', the predicate is an ascription of ill health to the individual so indicated, on the day preceding the statement's utterance. > 6. Sentence M asserts that the health of an individual named 'Mary' was not good the day before today. > 7. Sentence M says that Mary was sick yesterday and > Just where is the boundary line that says, "You can't do that much processing!"? A machine that could go as far as version 7 would have actually understood-at least in some rudimentary sense-the content of Sentence M. Elsewhere in the same book he (like in this Atlantic essay) talks about the problems of translating gender across languages. I think it was in Godel, Escher, Bach where he talks about the problem of translating a line from Russian, along the line of "He lived on B____ street." It's possible from the story, which takes place in a real city, to figure out which street that was. Let's say it was "Main Street". Does the English translation keep the original Russian word and initial? Does it translate to "Main Street" and replace the "B" with an "M"? Or consider his book "Le Ton beau de Marot: In Praise of the Music of Language", which contains 88 different translations of a 16th-century French poem. I think it would be difficult to incorporate all of those thousands of pages of writing on the topic into a single essay for a lay public magazine, and that your expectations are too high. While it's true that "good" is subjective, this is a solved problem in the Turing test or Chinese room sense. We judge professional translators, like those who work at the UN. We judge students learning a foreign language. There's no reason to believe that we can't apply similar techniques to judge machine translation. Indeed, he gives many concrete examples of a minimum level of translation competency which should be expected for a good-enough system.
  - danso 9y ago
    I agree that the OP gives good and reasonable examples of what should be doable by a competent translator. I think he is entirely on point about how people fail to realize how shallow Google Translate in terms of what we understand as "understanding" language. My quote about "really understand language" is from the GP -- I was asking him what he meant by that phrase. I don't think I disagree with the OP much at all. But I was confused because much of his essay shows how Google Translate is getting the most basic things wrong. But then he ends with discussions about what it means for a computer to have true understanding, the type of understanding that can't be achieved with just more data. It seems to me that the obvious screw ups that Google Translate is demonstrated to make could be alleviated through better algorithm design and data -- I.e. without achieving what Hofstadler argues is true understanding. Just like a self-driving car could be very safe despite having no more deep understanding of driving than Google Translate does of words.
    - eesmith 9y ago
      My apologizes, I missed up on the thread context. This sort of conversation of "really understand" is Searle's Chinese Room thought experiment. Hofstadter and many others have written a lot on the topic.
  - MarkMMullin 9y ago
    The classic joke in the '80s was that a systran project meant to translate Russian into English wasn't too good outside of purely technical work - The proverb 'The spirit is willing but the flesh is weak' became 'The vodka is good but the meat has gone bad.'
    - eesmith 9y ago
      The joke dates from the 1950s. See https://www.google.com/search?tbm=bks&q=%22The+whiskey+is+agreeable%22 https://www.google.com/search?tbm=bks&q=%22The+whiskey+is+ag... . For example, https://books.google.com/books?id=90Y8AAAAMAAJ&q=%22The+whiskey+is+agreeable%22&dq=%22The+whiskey+is+agreeable%22&hl=en&sa=X&ved=0ahUKEwiF996j2IfZAhWMyaQKHYoDAY8Q6AEIRDAG https://books.google.com/books?id=90Y8AAAAMAAJ&q=%22The+whis... show the snippet from a 1958 publication: "Univac cannot anthologize, though it has all sorts of language tricks, and can translate "The spirit is willing but the flesh is weak" to "The whiskey is agreeable but the ..." and here's a 1965 reference of the same joke: https://archive.org/stream/journalofkentuck6319kent#page/272/mode/2up/search/%22The+whisky+is+agreeable%22 https://archive.org/stream/journalofkentuck6319kent#page/272... .

danso 9y ago

I agree with the thrust of the article, that Google Translate is fundamentally limited (as most systems are) by its inputs and ostensible goals. But what does it mean to "really understand language"? What is the recognizable benchmark for "true NLP"? At some point, how "good" comprehension/translation is very subjective, because it intersects so heavily with culture and history. Think of how debates still rage over interpreting Shakespeare among Western readers and academics, even though there isn't a language barrier.

eesmith 9y ago
I notice you quoted "really understand language." That is not a quote from the essay. Is it an indicator that you are taking the conversation in a different direction? Hofstadter has written a lot about the question of what it means to "really understand a language", and the related more relevant question of what it means to translate from one language to another. Several of his essays in "Metamagical Themas" in the 1980s covered those topics, and they are quite interesting. He gives an example "Mary was sick yesterday", with different levels of 'understanding'. Call it "sentence M" (quoting from https://archive.org/stream/MetamagicalThemas/Metamagical%20Themas%2C%20Hofstadter#page/n29/mode/2up/search/street https://archive.org/stream/MetamagicalThemas/Metamagical%20T... , including apparent typos) > 1. Sentence M contains twenty characters. > 2. Sentence M contains four English words. > 3. Sentence M contains one proper noun, one verb, one one adverb, in that order. > 4. Sentence M contains one human's name, one linking verb, one adjective describing a potential health state of a living being, and one temporal adverb, in that order. > 5. The subject of Sentence M is a pointer to an individual named 'Mary', the predicate is an ascription of ill health to the individual so indicated, on the day preceding the statement's utterance. > 6. Sentence M asserts that the health of an individual named 'Mary' was not good the day before today. > 7. Sentence M says that Mary was sick yesterday and > Just where is the boundary line that says, "You can't do that much processing!"? A machine that could go as far as version 7 would have actually understood-at least in some rudimentary sense-the content of Sentence M. Elsewhere in the same book he (like in this Atlantic essay) talks about the problems of translating gender across languages. I think it was in Godel, Escher, Bach where he talks about the problem of translating a line from Russian, along the line of "He lived on B____ street." It's possible from the story, which takes place in a real city, to figure out which street that was. Let's say it was "Main Street". Does the English translation keep the original Russian word and initial? Does it translate to "Main Street" and replace the "B" with an "M"? Or consider his book "Le Ton beau de Marot: In Praise of the Music of Language", which contains 88 different translations of a 16th-century French poem. I think it would be difficult to incorporate all of those thousands of pages of writing on the topic into a single essay for a lay public magazine, and that your expectations are too high. While it's true that "good" is subjective, this is a solved problem in the Turing test or Chinese room sense. We judge professional translators, like those who work at the UN. We judge students learning a foreign language. There's no reason to believe that we can't apply similar techniques to judge machine translation. Indeed, he gives many concrete examples of a minimum level of translation competency which should be expected for a good-enough system.
- danso 9y ago
  I agree that the OP gives good and reasonable examples of what should be doable by a competent translator. I think he is entirely on point about how people fail to realize how shallow Google Translate in terms of what we understand as "understanding" language. My quote about "really understand language" is from the GP -- I was asking him what he meant by that phrase. I don't think I disagree with the OP much at all. But I was confused because much of his essay shows how Google Translate is getting the most basic things wrong. But then he ends with discussions about what it means for a computer to have true understanding, the type of understanding that can't be achieved with just more data. It seems to me that the obvious screw ups that Google Translate is demonstrated to make could be alleviated through better algorithm design and data -- I.e. without achieving what Hofstadler argues is true understanding. Just like a self-driving car could be very safe despite having no more deep understanding of driving than Google Translate does of words.
  - eesmith 9y ago
    My apologizes, I missed up on the thread context. This sort of conversation of "really understand" is Searle's Chinese Room thought experiment. Hofstadter and many others have written a lot on the topic.
- MarkMMullin 9y ago
  The classic joke in the '80s was that a systran project meant to translate Russian into English wasn't too good outside of purely technical work - The proverb 'The spirit is willing but the flesh is weak' became 'The vodka is good but the meat has gone bad.'
  - eesmith 9y ago
    The joke dates from the 1950s. See https://www.google.com/search?tbm=bks&q=%22The+whiskey+is+agreeable%22 https://www.google.com/search?tbm=bks&q=%22The+whiskey+is+ag... . For example, https://books.google.com/books?id=90Y8AAAAMAAJ&q=%22The+whiskey+is+agreeable%22&dq=%22The+whiskey+is+agreeable%22&hl=en&sa=X&ved=0ahUKEwiF996j2IfZAhWMyaQKHYoDAY8Q6AEIRDAG https://books.google.com/books?id=90Y8AAAAMAAJ&q=%22The+whis... show the snippet from a 1958 publication: "Univac cannot anthologize, though it has all sorts of language tricks, and can translate "The spirit is willing but the flesh is weak" to "The whiskey is agreeable but the ..." and here's a 1965 reference of the same joke: https://archive.org/stream/journalofkentuck6319kent#page/272/mode/2up/search/%22The+whisky+is+agreeable%22 https://archive.org/stream/journalofkentuck6319kent#page/272... .