4 ms·

These NMTs seem tragically bad at translating simple straightforward Chinese e.g.: 我要下班了。下班了再说。 Google: I have to get off work. Off to say. OpenNMT: I am g

by vicaya 10y ago

These NMTs seem tragically bad at translating simple straightforward Chinese e.g.: 我要下班了。下班了再说。 Google: I have to get off work. Off to say. OpenNMT: I am going to work. After work, again. If this is the state of art for NLP. We have a long way to go.

srush 10y ago
It's not there yet, but the improvement has been quite significant in aggregate. Key table from the GoogleNMT paper, empirically showing a 60% relative improvement on this task: PBMT GNMT Human Relative Improvement English → Spanish 4.885 5.428 5.504 87% English → French 4.932 5.295 5.496 64% English → Chinese 4.035 4.594 4.987 58% Spanish → English 4.872 5.187 5.372 63% French → English 5.046 5.343 5.404 83% Chinese → English 3.694 4.263 4.636 60%
- vicaya 10y ago
  This is obviously data dependent. I suspect that the advantage of human is much higher in colloquial content vs written (esp. news) content. "Universal Adversarial Perturbations"[1] last year showed that you can easily generate reasonable (to human) perturbations to completely fool the state of the art DNNs for images. I suspect that the same is true for the current batch of NMTs as well. As a simple demo, I change the example Chinese a little (就要下班了。下班了再说吧。Only aux characters changes with the same meaning) and all NMTs failed spectacularly in different ways. Google: It is necessary to get off work. To say it again. OpenNMT: it's going to work. Go back to work again. Baidu: It's going to work. After work. [1] https://arxiv.org/abs/1610.08401 https://arxiv.org/abs/1610.08401
  - srush 10y ago
    Yeah this is a nice connection. Note however that there has been much less success in using perturbation in language. The fact that inputs are discrete makes it harder to apply some of the tricks in the adverserial image work.
eagletusk 10y ago
Could be the inane target language. http://pinyin.info/readings/texts/moser.html http://pinyin.info/readings/texts/moser.html

by vicaya 10y ago

srush 10y ago

It's not there yet, but the improvement has been quite significant in aggregate. Key table from the GoogleNMT paper, empirically showing a 60% relative improvement on this task: PBMT GNMT Human Relative Improvement English → Spanish 4.885 5.428 5.504 87% English → French 4.932 5.295 5.496 64% English → Chinese 4.035 4.594 4.987 58% Spanish → English 4.872 5.187 5.372 63% French → English 5.046 5.343 5.404 83% Chinese → English 3.694 4.263 4.636 60%

vicaya 10y ago
This is obviously data dependent. I suspect that the advantage of human is much higher in colloquial content vs written (esp. news) content. "Universal Adversarial Perturbations"[1] last year showed that you can easily generate reasonable (to human) perturbations to completely fool the state of the art DNNs for images. I suspect that the same is true for the current batch of NMTs as well. As a simple demo, I change the example Chinese a little (就要下班了。下班了再说吧。Only aux characters changes with the same meaning) and all NMTs failed spectacularly in different ways. Google: It is necessary to get off work. To say it again. OpenNMT: it's going to work. Go back to work again. Baidu: It's going to work. After work. [1] https://arxiv.org/abs/1610.08401 https://arxiv.org/abs/1610.08401
- srush 10y ago
  Yeah this is a nice connection. Note however that there has been much less success in using perturbation in language. The fact that inputs are discrete makes it harder to apply some of the tricks in the adverserial image work.

eagletusk 10y ago

Could be the inane target language. http://pinyin.info/readings/texts/moser.html http://pinyin.info/readings/texts/moser.html