4 ms·

I find that machine translation is getting better at producing what reads as grammatically correct text, but that it still misses a lot of what is said, even wh

I find that machine translation is getting better at producing what reads as grammatically correct text, but that it still misses a lot of what is said, even when the idioms and expressions used are common. Often what is produced is grammatical, but opposite/orthogonal of the initial intent. In a sense, I feel that this is an even worse situation, because the word-salad signal that it isn't an accurate-meaning translation isn't as strong.

xbmcuser 6y ago
I think the problem is when direct translation is done. Netflix paper using simplify then translating to me seems to be the better approach when you want to convey the meaning rather than just translating the exact words https://arxiv.org/abs/2005.11197 https://arxiv.org/abs/2005.11197
userbinator 6y ago
That's what MT based on neural networks will do: their output can look great, but the correspondence with the input can be highly inaccurate. On the other hand, traditional methods based on syntactic analysis tend to yield less grammatically correct and more literal translations, but the literalness can also result in better accuracy.
- MiroF 6y ago
  Mm... I think you'd be hard pressed at this point to find any advantage that traditional methods have over deep methods. Maybe this was true 4-6 years ago, but no longer.
  - _-___________-_ 6y ago
    Depends what you optimise for. Neural translation fairly consistently produces incorrect results that read like they could be correct, for me. If your goal is "text that makes sense" they seem to be doing great. If your goal is "text that conveys the same meaning as the source" they're universally rubbish, at least for the language pairs I need (mostly english <-> east Asian languages).
    - MiroF 6y ago
      > english <-> east Asian languages I'm curious what the domain is and what languages you're talking about, it's hard for me to evaluate your claims otherwise. I would buy it for English -> Tibetan for instance.
      - _-___________-_ 6y ago
        Mostly Chinese, Japanese, and Vietnamese, and mostly business and lightly-technical text. My observation of the failure rate (if you define failure as "meaning has been lost") is well above 80% with Google Translate for these languages and this type of text. It's literally unusable because you have no confidence that the translation has not completely reversed or garbled the meaning of the text, and the failure mode is particularly unpleasant because you may be unaware that it has failed to translate correctly, since it produces text that appears to have meaning - just the wrong meaning.
        numpad0 6y ago
        Kinda funny when people say "East Asia" it mostly means Japan and its isolationism, with China mixed in just to change the tone a bit. I don't find it offensive, just kind of funny/interesting.
        _-___________-_ 6y ago
        Well, in the context of language it's a useful term because of the close relationship many east Asian languages have.
        numpad0 6y ago
        Do those languages really have that relationship? When I scratched surface of Chinese language in university it felt almost English written in Kanji. No resemblance to Japanese, and it even made the classical Chinese interpretation methods taught in Japanese education rather dubious to me.
  - 082349872349872 6y ago
    I strongly prefer the disfluent but more likely accurate results from the old days to the fluent-appearing but of unknown accuracy of today. https://news.ycombinator.com/item?id=24302564 https://news.ycombinator.com/item?id=24302564 Bonus clip: https://www.youtube.com/watch?v=0E4aYgKmIko https://www.youtube.com/watch?v=0E4aYgKmIko
    - MiroF 6y ago
      "the disfluent but more likely accurate results" -> Human evaluation does not agree that PBSMT is more accurate than NMT. Sure, the fluency/"sounds good" gains from NMT are much larger than the accuracy gains, but evaluations I've seen put NMT ahead on both metrics.
      - 082349872349872 6y ago
        Fair enough, this is just my sentiment, where accurate was not a term of art, but meant "honest where it is weak and where it is confident." Before, I could skim outputs, and get a fair idea of which areas were poorly translated by where it was garbage. Now, I get the impression that (like with GPT or style transfer) something always comes out, whether it's related (or even necessarily has the right negation parity) or not.
        MiroF 6y ago
        You're absolutely right with your impression of GPT/these models - they will generally generate plausible text. And it is also worth noting (as you've sort of mentioned) that appearance of fluency can bias translators who are instructed to just evaluate accuracy, or to evaluate accuracy and fluency separately. There is ongoing research into that problem.

xbmcuser 6y ago

I think the problem is when direct translation is done. Netflix paper using simplify then translating to me seems to be the better approach when you want to convey the meaning rather than just translating the exact words https://arxiv.org/abs/2005.11197 https://arxiv.org/abs/2005.11197

userbinator 6y ago

That's what MT based on neural networks will do: their output can look great, but the correspondence with the input can be highly inaccurate. On the other hand, traditional methods based on syntactic analysis tend to yield less grammatically correct and more literal translations, but the literalness can also result in better accuracy.

MiroF 6y ago
Mm... I think you'd be hard pressed at this point to find any advantage that traditional methods have over deep methods. Maybe this was true 4-6 years ago, but no longer.
- _-___________-_ 6y ago
  Depends what you optimise for. Neural translation fairly consistently produces incorrect results that read like they could be correct, for me. If your goal is "text that makes sense" they seem to be doing great. If your goal is "text that conveys the same meaning as the source" they're universally rubbish, at least for the language pairs I need (mostly english <-> east Asian languages).
  - MiroF 6y ago
    > english <-> east Asian languages I'm curious what the domain is and what languages you're talking about, it's hard for me to evaluate your claims otherwise. I would buy it for English -> Tibetan for instance.
    - _-___________-_ 6y ago
      Mostly Chinese, Japanese, and Vietnamese, and mostly business and lightly-technical text. My observation of the failure rate (if you define failure as "meaning has been lost") is well above 80% with Google Translate for these languages and this type of text. It's literally unusable because you have no confidence that the translation has not completely reversed or garbled the meaning of the text, and the failure mode is particularly unpleasant because you may be unaware that it has failed to translate correctly, since it produces text that appears to have meaning - just the wrong meaning.
      - numpad0 6y ago
        Kinda funny when people say "East Asia" it mostly means Japan and its isolationism, with China mixed in just to change the tone a bit. I don't find it offensive, just kind of funny/interesting.
        _-___________-_ 6y ago
        Well, in the context of language it's a useful term because of the close relationship many east Asian languages have.
        numpad0 6y ago
        Do those languages really have that relationship? When I scratched surface of Chinese language in university it felt almost English written in Kanji. No resemblance to Japanese, and it even made the classical Chinese interpretation methods taught in Japanese education rather dubious to me.
- 082349872349872 6y ago
  I strongly prefer the disfluent but more likely accurate results from the old days to the fluent-appearing but of unknown accuracy of today. https://news.ycombinator.com/item?id=24302564 https://news.ycombinator.com/item?id=24302564 Bonus clip: https://www.youtube.com/watch?v=0E4aYgKmIko https://www.youtube.com/watch?v=0E4aYgKmIko
  - MiroF 6y ago
    "the disfluent but more likely accurate results" -> Human evaluation does not agree that PBSMT is more accurate than NMT. Sure, the fluency/"sounds good" gains from NMT are much larger than the accuracy gains, but evaluations I've seen put NMT ahead on both metrics.
    - 082349872349872 6y ago
      Fair enough, this is just my sentiment, where accurate was not a term of art, but meant "honest where it is weak and where it is confident." Before, I could skim outputs, and get a fair idea of which areas were poorly translated by where it was garbage. Now, I get the impression that (like with GPT or style transfer) something always comes out, whether it's related (or even necessarily has the right negation parity) or not.
      - MiroF 6y ago
        You're absolutely right with your impression of GPT/these models - they will generally generate plausible text. And it is also worth noting (as you've sort of mentioned) that appearance of fluency can bias translators who are instructed to just evaluate accuracy, or to evaluate accuracy and fluency separately. There is ongoing research into that problem.