I don't see any problem with this evaluation. Both human translators and the machine were subject to the same conditions and evaluators, making it possible to d

I don't see any problem with this evaluation. Both human translators and the machine were subject to the same conditions and evaluators, making it possible to do an objective comparison. 500 samples is more than enough to get an accurate score. If phrase based was really so good, it should have gotten approximately equal scores on this simple task. Instead in every case it did way worse than the neural translations.