9 ms·

Using AI to match human performance in translating news from Chinese to English

218 points by Maimedpuppet 9y ago

rvense 9y ago
Translate "sentences of news" is very different to translating an entire article, which is obviously what's interesting. Is anybody in MT or text comprehension/generation really working on systems that construct a model/"understanding" of the bigger narrative in a longer-running text? Even just to be able to do correct anaphora resolution across sentence and paragraph boundaries, but intuitively also WSD seems easier if you've got some sort of abstract context over more than just a sentence.
- roel_v 9y ago
  I think Google translate already has this. I was translating some text into German a few days ago, and after a few sentences I used a word that made it clear that I was talking about a specific type of contract appointment, and it went back and adjusted earlier sentences to use more precise terminology. You only notice this when you a) speak the language you're translating into somewhat; b) actually type/compose the message in the Google Translate text box; and c) are typing something idiomatic enough that such specific phrases can be inferred. So I guess it's just something you wouldn't normally notice. Either way, I was mightily impressed, to the point where my wife had to roll her eyes and say 'yeah yeah I understand it now' to get me to drop it. (I'm just easily excited I guess.)
  - simias 9y ago
    I've found that google translate works decently (as far as these automated translations can be expected to work) from/to English, however translating from a different pair of languages the results are often very off in my experience. In particular it seems it has some sort of internal bias, as if it always an english-like intermediary representation. For instance, if you ask google to translate the Portuguese "báculo" into French it gives you "personnel". It's nonsense as far as I can tell, a báculo is a "crosier of a bishop"[1]. So what's going on here? Well if you translate it from PT to EN it gives you "staff" and suddenly it starts making sense, because while staff means "A long, straight, thick wooden rod or stick, especially one used to assist in walking" (which fits báculo) it can also mean "The employees of a business" which is an accurate definition for french "personnel". And I believe that's how you end up with the nonsensical PT -> FR translation. Similarly Google used to be confused by the tu/vous (informal/formal) distinction that exists in many languages but not in English. At some point the portuguese "tu és" would be translated in french by the formal "vous êtes" instead of the informal "tu es". This appears to have been fixed however, I can't reproduce it at the moment. Conjugations don't fare so well however, for instance imperfect past french "je chantais" is translated into portuguese preterite "eu cantei" even though "eu cantava" would make more sense I think. Obviously with such small phrases I can't really be too harsh on google's bad grammar, they're probably not optimizing for that case. [1] https://en.wiktionary.org/wiki/b%C3%A1culo https://en.wiktionary.org/wiki/b%C3%A1culo
  - rerx 9y ago
    That sounds extremely interesting. I had not noticed that feature before. Do you happen to have some example input at hand that triggers such an adjustment?
    - londons_explore 9y ago
      It's likely this tech is released to only a small percentage of users, and at off-peak times. Parsing an entire paragraph for context is expensive...
- anewhnaccount2 9y ago
  People have certainly worked on moving beyond sentence boundaries, although what is meant by understanding is always a bit nebulous. Certainly we need to make sure whatever process we are using has a sufficiently rich internal knowledge representation. One piece of work is this: https://github.com/chardmeier/docent/wiki https://github.com/chardmeier/docent/wiki which is a document level phrases-based statistical machine translation decoder. There have also been special purpose evaluation tasks which include correct pronoun resolution e.g. https://lindat.mff.cuni.cz/repository/xmlui/handle/11372/LRT-1611 https://lindat.mff.cuni.cz/repository/xmlui/handle/11372/LRT... . If you're interested, probably the DiscoMT workshops are a good starting point for some things people have tried.
anewhnaccount2 9y ago
Compare this to one of Google's blog post promoting their MT research: https://research.googleblog.com/2016/09/a-neural-network-for-machine.html https://research.googleblog.com/2016/09/a-neural-network-for... It is: 1) More accurate, compared to hyperbole like e.g. "Bridging the Gap between Human and Machine Translation" we have right there in the title the domain: news. 2) A more impressive result. This result is on an independently set up evaluation framework, compared to Google's which used their own framework. Compare further the papers: https://arxiv.org/pdf/1609.08144.pdf https://arxiv.org/pdf/1609.08144.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/03/final-achieving-human.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0... These researcher appear to have been much clearer about what they're actually claiming, and also used more standard evaluation tools (Appraise) and methodology rather than something haphazardly hacked together.
- snovv_crash 9y ago
  The outputs of Microsoft Research are really good. At least in my field, it is one of the few places where if they published something you can be sure of being able to reproduce the results using only what is described in the paper, no secret sauce required.
- londons_explore 9y ago
  Googles was nearly 2 years ago though... Thats a long time in this field
- golfer 9y ago
  If you are claiming that Microsoft is pure as the driven snow with regard to making exaggerated and hyperbolic claims, then you clearly know little about Microsoft.
  - gumby 9y ago
    I don't read that claim being made. Microsoft Research has been generally left alone to do good computer science, in which "left alone" has included "by the marketing department"
  - 9y ago
lima 9y ago
The most impressive ML translation tool I've seen so far is DeepL[0]. Sometimes, it manages to translate whole articles without errors. [0]: https://www.deepl.com/translator https://www.deepl.com/translator
- realusername 9y ago
  I can also confirm, it's very impressive. Not sure how their setup differs from Google Translate but I've read articles translated with it and only knew it was translated when it was written at the end of the article that it was using DeepL.
  - mrec 9y ago
    Word choice is sometimes very slightly off, but grammar appears to be flawless. As you say, very very impressive.
- shoshin23 9y ago
  DeepL is what I use for European language translations. I wish they added more languages and maybe a nice app like Google Translate. It blows every other translation service out of the water.
- narag 9y ago
  Wow! That's very good.
- deleted 9y ago
  [deleted]
- foldr 9y ago
  Impressive, but there's an easy formula for getting these systems to make mistakes. Just input a sentence with some kind of long distance dependency. For example, DeepL gets agreement right in English to Spanish translations when the two things that agree are close together: I like soup -> COMO sopa They eat soup -> COMEN sopa Impressively, it can even get agreement correct across clause boundaries in many cases. But if you do wh-movement through two or more clauses, you're usually out of luck: Which boys does he say he believes eat soup? -> ¿Qué chicos dice que cree que COME sopa? [should be COMEN] It doesn't really matter very much in practice if an MT system makes mistakes like this, but they are mistakes that you can rely on humans not to make systematically.
Quanttek 9y ago
Be careful when reading such claims: https://www.theatlantic.com/technology/archive/2018/01/the-shallowness-of-google-translate/551570/?single_page=true https://www.theatlantic.com/technology/archive/2018/01/the-s...
- deleted 9y ago
  [deleted]
- jimbokun 9y ago
  That was much better than I expected. And it wasn't until I looked at the byline at the end when I realized, yes, it is that Hofstadter (Godel, Escher, Bach).
d--b 9y ago
As impressive as it may be, these people should refrain from claiming 'human-like' translation from a system that has no way of 'knowing' anything about context, other than statistical occurrences. It is certain that, on occasion, the system will make such mistakes as stating the opposite of what is being said in the first place, or attribute one action to the wrong person, and what not. Perhaps on average it's as good as a person, but this system will make mistakes that disqualifies it from being used without a bucket of salt.
- azag0 9y ago
  Certainly true. I'd just point out that exactly these kinds of mistakes happen to people too.
- vixen99 9y ago
  This was much the point that John Searle made in his criticism of AI prognostications. Ultimately I think he'll be shown to be wrong but the time frame for this will be (I suggest) much longer than is currently touted. https://en.wikipedia.org/wiki/John_Searle https://en.wikipedia.org/wiki/John_Searle
  - avgarrison 9y ago
    Are you referring to the Chinese Room? I've always had an issue with that argument. Instructions are immutable, but neural networks certainly are not.
    - foldr 9y ago
      Any finite computational system can be implemented as a lookup table in the manner that Searle suggests. But that's not essential to the argument. You can imagine that the man in the room is following instructions for simulating a (finite approximation of) a Turing machine, or any other computational device that you like.
    - Chathamization 9y ago
      I agree, the argument seems to come down to saying that human consciousness can't be replicated because human consciousness stems from a "soul" (a non-physical and undetectable element of someone that's at the root of their consciousness). The amount of attention this argument has received has made me wonder whether the "rigor" used by philosophy departments is mostly just a way to obfuscate bad arguments.
fouc 9y ago
Would be nice to have improved MTL performance for Wuxia/Xanxia webnovels.
jakecrouch 9y ago
It's obvious that there are limits to how well machine translation can work unless the models have sensory grounding. I wonder if the problem is that people haven't figured out how to do sensory grounding or that the hardware is still too slow for it to work.
- jimbokun 9y ago
  This article, posted by Quanttek above, is very relevant to your question: https://www.theatlantic.com/technology/archive/2018/01/the-shallowness-of-google-translate/551570/?single_page=true https://www.theatlantic.com/technology/archive/2018/01/the-s...
iliketosleep 9y ago
I find these types of "match human performance" claims to be ridiculous, especially when it comes to Chinese -> English translations. Translation is both an art and a science, requiring nuanced understanding of the languages, cultures, and context. It also demands quite a bit of creativity. No translation tool I've tried has come even close to matching human performance of a good human translator, including microsoft's tools. AI will need to reach the point where its understanding of language, culture, context, and creative ability matches that of humans to truly be capable of "human performance" in translation.
- yorwba 9y ago
  After reading the paper, my takeaway is that humans aren't really very good at translation either. None of the methods scores higher than 70% in the evaluation and that includes several different human translations (whose performance varies greatly depending on how they were sourced). So while matching the quality of the average human translator is a great milestone, there's still lots of room to improve.
  - iliketosleep 9y ago
    Most humans who attempt to translate are not actually translators in the proper sense. Specifically, they are not fully competent in both the source and target language and usually have no formal training in translation. Translation is hard, but there are competent people out there who do it extremely well. Sadly, as an industry it's not taken as seriously as it should be, and most of the people who are actually doing translations do not have the appropriate skill set.
- londons_explore 9y ago
  I think there are a lot of bad human translators out there... The number of papers I've read that have very poor grammar, to such an extent it's barely understandable...
  - iliketosleep 9y ago
    I can offer some insight on this, as I've dealt with translations of papers before (Chinese to English). There have been countless times where I've been asked to check the translation of a paper and ended up retranslating it completely. I was often curious as to why the original translation was bad - it was typically either of the following two reasons: 1. The so-called translator was just an overworked research assistant who knows a bit of English. 2. The translation was outsourced to a dodgy company that spends more on marketing than on their translators, usually hiring college grads who did English-related majors.
baybal2 9y ago
About translators solely reliant on NN. The thing is, while 70% of output can be well passable, some of the rest can be very weird if original input was not learned. Like a string of gibberish turning into 10 full sentences. You have to score the extent of wrongness too.
blennon 9y ago
What I find most interesting is the multiple training methods used to get the network to improve its performance. They name a few in the article: - dual learning - deliberation networks - joint training - agreement regularization I haven't read the paper to see how these are combined but it makes intuitive sense that using multiple training methods can lead to better performance. That is to say, to more effectively search the weight space of the network.
abacate 9y ago
I'd suggest addressing non-English to non-English translations first, which is usually limited in most engines out there compared to translations to/from English.
trisimix 9y ago
Amazing when can i start reading chinese cs boards

218 points by Maimedpuppet 9y ago

Translate "sentences of news" is very different to translating an entire article, which is obviously what's interesting. Is anybody in MT or text comprehension/generation really working on systems that construct a model/"understanding" of the bigger narrative in a longer-running text? Even just to be able to do correct anaphora resolution across sentence and paragraph boundaries, but intuitively also WSD seems easier if you've got some sort of abstract context over more than just a sentence.

roel_v 9y ago
I think Google translate already has this. I was translating some text into German a few days ago, and after a few sentences I used a word that made it clear that I was talking about a specific type of contract appointment, and it went back and adjusted earlier sentences to use more precise terminology. You only notice this when you a) speak the language you're translating into somewhat; b) actually type/compose the message in the Google Translate text box; and c) are typing something idiomatic enough that such specific phrases can be inferred. So I guess it's just something you wouldn't normally notice. Either way, I was mightily impressed, to the point where my wife had to roll her eyes and say 'yeah yeah I understand it now' to get me to drop it. (I'm just easily excited I guess.)
- simias 9y ago
  I've found that google translate works decently (as far as these automated translations can be expected to work) from/to English, however translating from a different pair of languages the results are often very off in my experience. In particular it seems it has some sort of internal bias, as if it always an english-like intermediary representation. For instance, if you ask google to translate the Portuguese "báculo" into French it gives you "personnel". It's nonsense as far as I can tell, a báculo is a "crosier of a bishop"[1]. So what's going on here? Well if you translate it from PT to EN it gives you "staff" and suddenly it starts making sense, because while staff means "A long, straight, thick wooden rod or stick, especially one used to assist in walking" (which fits báculo) it can also mean "The employees of a business" which is an accurate definition for french "personnel". And I believe that's how you end up with the nonsensical PT -> FR translation. Similarly Google used to be confused by the tu/vous (informal/formal) distinction that exists in many languages but not in English. At some point the portuguese "tu és" would be translated in french by the formal "vous êtes" instead of the informal "tu es". This appears to have been fixed however, I can't reproduce it at the moment. Conjugations don't fare so well however, for instance imperfect past french "je chantais" is translated into portuguese preterite "eu cantei" even though "eu cantava" would make more sense I think. Obviously with such small phrases I can't really be too harsh on google's bad grammar, they're probably not optimizing for that case. [1] https://en.wiktionary.org/wiki/b%C3%A1culo https://en.wiktionary.org/wiki/b%C3%A1culo
- rerx 9y ago
  That sounds extremely interesting. I had not noticed that feature before. Do you happen to have some example input at hand that triggers such an adjustment?
  - londons_explore 9y ago
    It's likely this tech is released to only a small percentage of users, and at off-peak times. Parsing an entire paragraph for context is expensive...
anewhnaccount2 9y ago
People have certainly worked on moving beyond sentence boundaries, although what is meant by understanding is always a bit nebulous. Certainly we need to make sure whatever process we are using has a sufficiently rich internal knowledge representation. One piece of work is this: https://github.com/chardmeier/docent/wiki https://github.com/chardmeier/docent/wiki which is a document level phrases-based statistical machine translation decoder. There have also been special purpose evaluation tasks which include correct pronoun resolution e.g. https://lindat.mff.cuni.cz/repository/xmlui/handle/11372/LRT-1611 https://lindat.mff.cuni.cz/repository/xmlui/handle/11372/LRT... . If you're interested, probably the DiscoMT workshops are a good starting point for some things people have tried.

anewhnaccount2 9y ago

Compare this to one of Google's blog post promoting their MT research: https://research.googleblog.com/2016/09/a-neural-network-for-machine.html https://research.googleblog.com/2016/09/a-neural-network-for... It is: 1) More accurate, compared to hyperbole like e.g. "Bridging the Gap between Human and Machine Translation" we have right there in the title the domain: news. 2) A more impressive result. This result is on an independently set up evaluation framework, compared to Google's which used their own framework. Compare further the papers: https://arxiv.org/pdf/1609.08144.pdf https://arxiv.org/pdf/1609.08144.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/03/final-achieving-human.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0... These researcher appear to have been much clearer about what they're actually claiming, and also used more standard evaluation tools (Appraise) and methodology rather than something haphazardly hacked together.

snovv_crash 9y ago
The outputs of Microsoft Research are really good. At least in my field, it is one of the few places where if they published something you can be sure of being able to reproduce the results using only what is described in the paper, no secret sauce required.
londons_explore 9y ago
Googles was nearly 2 years ago though... Thats a long time in this field
golfer 9y ago
If you are claiming that Microsoft is pure as the driven snow with regard to making exaggerated and hyperbolic claims, then you clearly know little about Microsoft.
- gumby 9y ago
  I don't read that claim being made. Microsoft Research has been generally left alone to do good computer science, in which "left alone" has included "by the marketing department"
- 9y ago

lima 9y ago

The most impressive ML translation tool I've seen so far is DeepL[0]. Sometimes, it manages to translate whole articles without errors. [0]: https://www.deepl.com/translator https://www.deepl.com/translator

realusername 9y ago
I can also confirm, it's very impressive. Not sure how their setup differs from Google Translate but I've read articles translated with it and only knew it was translated when it was written at the end of the article that it was using DeepL.
- mrec 9y ago
  Word choice is sometimes very slightly off, but grammar appears to be flawless. As you say, very very impressive.
shoshin23 9y ago
DeepL is what I use for European language translations. I wish they added more languages and maybe a nice app like Google Translate. It blows every other translation service out of the water.
narag 9y ago
Wow! That's very good.
deleted 9y ago
[deleted]
foldr 9y ago
Impressive, but there's an easy formula for getting these systems to make mistakes. Just input a sentence with some kind of long distance dependency. For example, DeepL gets agreement right in English to Spanish translations when the two things that agree are close together: I like soup -> COMO sopa They eat soup -> COMEN sopa Impressively, it can even get agreement correct across clause boundaries in many cases. But if you do wh-movement through two or more clauses, you're usually out of luck: Which boys does he say he believes eat soup? -> ¿Qué chicos dice que cree que COME sopa? [should be COMEN] It doesn't really matter very much in practice if an MT system makes mistakes like this, but they are mistakes that you can rely on humans not to make systematically.

Quanttek 9y ago

Be careful when reading such claims: https://www.theatlantic.com/technology/archive/2018/01/the-shallowness-of-google-translate/551570/?single_page=true https://www.theatlantic.com/technology/archive/2018/01/the-s...

deleted 9y ago
[deleted]
jimbokun 9y ago
That was much better than I expected. And it wasn't until I looked at the byline at the end when I realized, yes, it is that Hofstadter (Godel, Escher, Bach).

d--b 9y ago

As impressive as it may be, these people should refrain from claiming 'human-like' translation from a system that has no way of 'knowing' anything about context, other than statistical occurrences. It is certain that, on occasion, the system will make such mistakes as stating the opposite of what is being said in the first place, or attribute one action to the wrong person, and what not. Perhaps on average it's as good as a person, but this system will make mistakes that disqualifies it from being used without a bucket of salt.

azag0 9y ago
Certainly true. I'd just point out that exactly these kinds of mistakes happen to people too.
vixen99 9y ago
This was much the point that John Searle made in his criticism of AI prognostications. Ultimately I think he'll be shown to be wrong but the time frame for this will be (I suggest) much longer than is currently touted. https://en.wikipedia.org/wiki/John_Searle https://en.wikipedia.org/wiki/John_Searle
- avgarrison 9y ago
  Are you referring to the Chinese Room? I've always had an issue with that argument. Instructions are immutable, but neural networks certainly are not.
  - foldr 9y ago
    Any finite computational system can be implemented as a lookup table in the manner that Searle suggests. But that's not essential to the argument. You can imagine that the man in the room is following instructions for simulating a (finite approximation of) a Turing machine, or any other computational device that you like.
  - Chathamization 9y ago
    I agree, the argument seems to come down to saying that human consciousness can't be replicated because human consciousness stems from a "soul" (a non-physical and undetectable element of someone that's at the root of their consciousness). The amount of attention this argument has received has made me wonder whether the "rigor" used by philosophy departments is mostly just a way to obfuscate bad arguments.

fouc 9y ago

Would be nice to have improved MTL performance for Wuxia/Xanxia webnovels.

jakecrouch 9y ago

It's obvious that there are limits to how well machine translation can work unless the models have sensory grounding. I wonder if the problem is that people haven't figured out how to do sensory grounding or that the hardware is still too slow for it to work.

jimbokun 9y ago
This article, posted by Quanttek above, is very relevant to your question: https://www.theatlantic.com/technology/archive/2018/01/the-shallowness-of-google-translate/551570/?single_page=true https://www.theatlantic.com/technology/archive/2018/01/the-s...

iliketosleep 9y ago

I find these types of "match human performance" claims to be ridiculous, especially when it comes to Chinese -> English translations. Translation is both an art and a science, requiring nuanced understanding of the languages, cultures, and context. It also demands quite a bit of creativity. No translation tool I've tried has come even close to matching human performance of a good human translator, including microsoft's tools. AI will need to reach the point where its understanding of language, culture, context, and creative ability matches that of humans to truly be capable of "human performance" in translation.

yorwba 9y ago
After reading the paper, my takeaway is that humans aren't really very good at translation either. None of the methods scores higher than 70% in the evaluation and that includes several different human translations (whose performance varies greatly depending on how they were sourced). So while matching the quality of the average human translator is a great milestone, there's still lots of room to improve.
- iliketosleep 9y ago
  Most humans who attempt to translate are not actually translators in the proper sense. Specifically, they are not fully competent in both the source and target language and usually have no formal training in translation. Translation is hard, but there are competent people out there who do it extremely well. Sadly, as an industry it's not taken as seriously as it should be, and most of the people who are actually doing translations do not have the appropriate skill set.
londons_explore 9y ago
I think there are a lot of bad human translators out there... The number of papers I've read that have very poor grammar, to such an extent it's barely understandable...
- iliketosleep 9y ago
  I can offer some insight on this, as I've dealt with translations of papers before (Chinese to English). There have been countless times where I've been asked to check the translation of a paper and ended up retranslating it completely. I was often curious as to why the original translation was bad - it was typically either of the following two reasons: 1. The so-called translator was just an overworked research assistant who knows a bit of English. 2. The translation was outsourced to a dodgy company that spends more on marketing than on their translators, usually hiring college grads who did English-related majors.

baybal2 9y ago

About translators solely reliant on NN. The thing is, while 70% of output can be well passable, some of the rest can be very weird if original input was not learned. Like a string of gibberish turning into 10 full sentences. You have to score the extent of wrongness too.

blennon 9y ago

What I find most interesting is the multiple training methods used to get the network to improve its performance. They name a few in the article: - dual learning - deliberation networks - joint training - agreement regularization I haven't read the paper to see how these are combined but it makes intuitive sense that using multiple training methods can lead to better performance. That is to say, to more effectively search the weight space of the network.

abacate 9y ago

I'd suggest addressing non-English to non-English translations first, which is usually limited in most engines out there compared to translations to/from English.

trisimix 9y ago

Amazing when can i start reading chinese cs boards