They wouldn't be so surprised if they'd used Mechanical Turk before. There, task approval was capricious and too ill-defined to make the resulting data meaningf

They wouldn't be so surprised if they'd used Mechanical Turk before. There, task approval was capricious and too ill-defined to make the resulting data meaningful. (No wonder Twitter sentiment analysis is so bad.) These systems obviously descend from those, with Amazon's own operators instead of third-party task buyers. Like Mturk, inflexible and vague rules protect their secret automation and its operators from gaming the system to the company's detriment.