2 ms·
> BLEU and ROUGE scores are almost useless for what they are trying to do (measure how good translation or summarization systems are). The author shows a great
by thaumasiotes 6y ago
> BLEU and ROUGE scores are almost useless for what they are trying to do (measure how good translation or summarization systems are). The author shows a great example of an excellent translation getting a 0 score.
The author also (perhaps unintentionally) shows a great example of being unable to tell whether a translation is excellent or awful.
猫坐在垫子上 is not a great correspondence to either reference sentence. Going purely by grammar, the equivalent of "the cat is on the mat" would be 猫在垫子上 (note the missing 坐, which is the verb "to sit", not present in the english references), and "there is a cat on the mat" would be 垫子上有猫 [literally "the top of the mat has a cat"].
猫坐在垫子上 could be a good translation of "the cat is on the mat". But it could also be a good translation of "cats sit on mats". Those two sentences are radically different in English; to judge the translation, you need to be aware of whether e.g. someone just asked "where's the cat?" or "what do cats do?"