The researchers did the comparison in their control datasets. There is a significant and detectable difference between generated garbage and failed translation

The researchers did the comparison in their control datasets. There is a significant and detectable difference between generated garbage and failed translation. Speaking of which, just out of curiosity I tried and translated one of their examples to simplified Chinese and back into English using Google translate. The results were shockingly bad, but until I discovered that it was the virtual line breaks introduced by the UI (i.e. the textbox) that caused it. Turns out it's a horrible idea to introduce line breaks in the middle of Chinese character sequences as the meaning of individual symbols varies depending on adjacent symbols. I'd be willing to bet that a lot of translation mistakes stem from that horrible design decision.