Or pay better like other people are suggesting. If you have to "average over" 3 data sets why not just pay $15 instead of $5 and save the computation if $15 or

Or pay better like other people are suggesting. If you have to "average over" 3 data sets why not just pay $15 instead of $5 and save the computation if $15 or so seemed to be the threshold for getting humans to be good data labelers?