4 ms·

I wonder about those samples. The Chinese I would expect to be kind of garbage going off BLEUs from published papers, but the French and German should be great.

by gwern 9y ago

I wonder about those samples. The Chinese I would expect to be kind of garbage going off BLEUs from published papers, but the French and German should be great. Am I imagining it, or do they just not sound like RNN translations? The RNN translations in my experience tend to flow well and are comprehensible, albeit sometimes just wrong because they picked a wrong semantic interpretation. But these samples sound like the older phrase-based translations, where they immediately descend into word salad. As people noted at the time, the jump in quality for French/German was so large that people knew when the new translations went live without Google publishing anything... (I assume Hofstadter at least was careful enough to do his sampling after the new RNN translations were opened up.) Google doesn't discuss how it rolls out the translation upgrades, exactly, and it's an uneven deployment. Can any Googlers comment on the possibility Hofstadter was using the old translations? Or can any NMT researchers compare and contrast his examples with current SOTA models?

irq11 9y ago
“Am I imagining it, or do they just not sound like RNN translations?” You’re imagining it. Google translate is about as bad as he’s saying. The translations are usually servicable for short phrases, but quickly fall apart for anything with depth or nuance. The Google paper was an interesing technical description, but pretty weak in terms of evaluation: it had a single experiment, based on simple, short phrases, evaluated in a subjective manner by “experts”. It shouldn’t be surprising at all to find out that the method was a bit overhyped. It’s quite possible that Google did greatly improve their performance...it just wasn’t very good to begin with.
GuiA 9y ago
The French translation in the article matches what Google returns today (ie garbage), albeit for one difference (it now translates "in pairs" to "par deux" instead of "en paires" (which is a slight improvement, but doesn't address the fundamental errors of the translation). The garbage it gives is consistent with the garbage statistical approaches tend to give; particularly when translating from a language without gendered pronouns to a language with them. My favorite, most concise example to demonstrate this is the sentence "my cousin and her wife". Anyone with basic understanding of English grammar would infer that my cousin is a woman married to a woman; Google Translate gives me back a French sentence where suddenly my cousin has become a man. This is a great example of something that a rules based translation system would never get wrong (of course, rules based translation systems have plenty of other shortcomings) and that statistical approaches have a hard time dealing with. See also: https://twitter.com/seyyedreza/status/935291317252493312 https://twitter.com/seyyedreza/status/935291317252493312 https://twitter.com/taikadahlbom/status/935612093906194432 https://twitter.com/taikadahlbom/status/935612093906194432

by gwern 9y ago

irq11 9y ago

“Am I imagining it, or do they just not sound like RNN translations?” You’re imagining it. Google translate is about as bad as he’s saying. The translations are usually servicable for short phrases, but quickly fall apart for anything with depth or nuance. The Google paper was an interesing technical description, but pretty weak in terms of evaluation: it had a single experiment, based on simple, short phrases, evaluated in a subjective manner by “experts”. It shouldn’t be surprising at all to find out that the method was a bit overhyped. It’s quite possible that Google did greatly improve their performance...it just wasn’t very good to begin with.

GuiA 9y ago

The French translation in the article matches what Google returns today (ie garbage), albeit for one difference (it now translates "in pairs" to "par deux" instead of "en paires" (which is a slight improvement, but doesn't address the fundamental errors of the translation). The garbage it gives is consistent with the garbage statistical approaches tend to give; particularly when translating from a language without gendered pronouns to a language with them. My favorite, most concise example to demonstrate this is the sentence "my cousin and her wife". Anyone with basic understanding of English grammar would infer that my cousin is a woman married to a woman; Google Translate gives me back a French sentence where suddenly my cousin has become a man. This is a great example of something that a rules based translation system would never get wrong (of course, rules based translation systems have plenty of other shortcomings) and that statistical approaches have a hard time dealing with. See also: https://twitter.com/seyyedreza/status/935291317252493312 https://twitter.com/seyyedreza/status/935291317252493312 https://twitter.com/taikadahlbom/status/935612093906194432 https://twitter.com/taikadahlbom/status/935612093906194432