5 ms·
The point of BLEU (and ROUGE and METEOR) is to correlate with human grading. It's unusual to find cases where a MT model change increases BLEU but hurts human r
by gok 6y ago
The point of BLEU (and ROUGE and METEOR) is to correlate with human grading. It's unusual to find cases where a MT model change increases BLEU but hurts human ratings.
- Der_Einzige 6y agoMaybe that was the case when we were using DUC datasets with several gold standard summaries or translations. It certainly is not the case on classic datasets like CNN/Dailymail where there is only one gold standard summary.
- gas9S9zw3P9c 6y agoIt's not so unusual in my experience. I used to do research in this area and we ran side-by-side human evals with our experiments. Yes, huge jumps in BLEU typically imply better human ratings, but the variance is big. Many times, significant jumps in BLEU had no significant effect on human ratings. ROUGE is even more useless. You just don't see these things reported in the literature because 1. human evals are a pain to run and can be expensive 2. researchers need to publish or perish.