Because mechanical turk MUST be supervised. The reward is disassociated from performance - it's the agent problem, writ large, since workers are effectively ano

Because mechanical turk MUST be supervised. The reward is disassociated from performance - it's the agent problem, writ large, since workers are effectively anonymous and paid piecework. In any paper using mechanical turk, half or more is trying to get the humans to do what they're told correctly. (They behave more like mischevious djinn than computers if allowed to act freely.) Just dropping in mechanical turk without a feedback system would lead to arbitrary responses, unhandled queries, and unsolvable inconsistencies.