Nice technique. Another idea might be to select some standard text (eg: some news article) that you predetermine, and run it through several online translators,

Nice technique. Another idea might be to select some standard text (eg: some news article) that you predetermine, and run it through several online translators, and compare what you get back from the humans to those. You might also include a 'turing test' sentence in there like: "obama talked to the bushes about their time in office".