It's not so unusual in my experience. I used to do research in this area and we ran side-by-side human evals with our experiments. Yes, huge jumps in BLEU typic

It's not so unusual in my experience. I used to do research in this area and we ran side-by-side human evals with our experiments. Yes, huge jumps in BLEU typically imply better human ratings, but the variance is big. Many times, significant jumps in BLEU had no significant effect on human ratings. ROUGE is even more useless. You just don't see these things reported in the literature because 1. human evals are a pain to run and can be expensive 2. researchers need to publish or perish.