4 ms·
We tried solving this by analysing language for decades; the reason we're doing this now is because it actually works a lot better.
by Eridrus 10y ago
We tried solving this by analysing language for decades; the reason we're doing this now is because it actually works a lot better.
- timr 10y agoThe article directly suggests that isn't the case. There's a (debatable) improvement over Google's existing phrase-based system on these benchmarks, but the whole paper is about the huge effort they've had to put in to get past the shortcomings of the method. A more realistic assessment is that state-of-the-art neural translation is maybe getting as good as phrase-based systems.
- Houshalter 10y agoThey claim 60% fewer translation errors than the phrase based system. That's not nothing.
- Houshalter 10y agoAlso, see this graph: https://1.bp.blogspot.com/-jOLa-LdidQU/V-qV2oJn1aI/AAAAAAAABPg/-6OhKKPhxT89Vs9HhyKMEnyG_0ncWGjJQCLcB/s1600/image00.png https://1.bp.blogspot.com/-jOLa-LdidQU/V-qV2oJn1aI/AAAAAAAAB... On French > English and English > Spanish it's almost indistinguishable from human translations. On the rest it still gets greater than half the distance between the previous best method and human quality translations. This is a massive step forward.
- timr 10y ago"This is a massive step forward." Take it easy. The graph you're linking to is for a limited survey of 500 isolated sentences, and it's a comparison of mean scores without confidence intervals. The authors themselves admit that it has serious problems as an evaluative tool: "Note that we have observed that human raters, even though fluent in both languages, do not necessarily fully understand each randomly sampled sentence sufficiently and hence cannot necessarily generate the best possible translation or rate a given translation accurately. Also note that, although the scale for the scores goes from 0 (complete nonsense) to 6 (perfect translation) the human translations get an imperfect score of only around 5 in Table 10, which shows possible ambiguities in the translations and also possibly non-calibrated raters and translators with a varying level of proficiency. Testing our GNMT system on particularly difficult translation cases and longer inputs than just single sentences is the subject of future work." When you look at the other evaluations in the paper, it's clear that it's about as good as the current best phrase-based tools. In other words, this may be a massive step forward for neural nets, but it puts them at about par for the field.
- Houshalter 10y agoI don't see any problem with this evaluation. Both human translators and the machine were subject to the same conditions and evaluators, making it possible to do an objective comparison. 500 samples is more than enough to get an accurate score. If phrase based was really so good, it should have gotten approximately equal scores on this simple task. Instead in every case it did way worse than the neural translations.
- Eridrus 10y agoUnless you think Google's existing phrase based system is significantly behind the state of the art (possible, but I doubt it's too far behind), 60% better is a pretty great improvement. I only skimmed the paper briefly, but it seems like they don't depart significantly from the NMT literature, but they do need to tweak a few things to get their results.
- timr 10y agoLike I said, it's debatable. It was honestly the weakest part of the paper. The specific claim comes from a vaguely specified "human evaluation" (i.e. how many evaluators? based on what criterion? did you just ask folks at google to compare stuff?) on a tiny sample of 500 basic sentences, and the results were presented as differences between mean scores without so much as a confidence interval or a power calculation to back them. Moreover, from the paper: "In some cases human and GNMT translations are nearly indistinguishable on the relatively simplistic and isolated sentences sampled from Wikipedia and news articles for this experiment. Note that we have observed that human raters, even though fluent in both languages, do not necessarily fully understand each randomly sampled sentence sufficiently and hence cannot necessarily generate the best possible translation or rate a given translation accurately. Also note that, although the scale for the scores goes from 0 (complete nonsense) to 6 (perfect translation) the human translations get an imperfect score of only around 5 in Table 10, which shows possible ambiguities in the translations and also possibly non-calibrated raters and translators with a varying level of proficiency. Testing our GNMT system on particularly difficult translation cases and longer inputs than just single sentences is the subject of future work." Basically, you should ignore the claims of improvement, and replace them with a big question mark.
- nl 10y agoI think you are overestimating human performance here. The weakness in the evaluation system is mostly because humans are imperfect translators.
- nomailing 10y agoThat is why there are metrics like inter rater reliability and significance tests and confidence intervals, which seem to be all missing in paper...
- scottlocklin 10y agoGoogle translate is quite good, and uses KNN. Does this beat KNN?
- eva1984 10y agoDidn't work well for Chinese though, or Asian languages in general.