Depends what it is. I've had reasonable success with "validating" ASR transcripts by loading up the annotator's transcript, running the audio at 2x speed and ju

Depends what it is. I've had reasonable success with "validating" ASR transcripts by loading up the annotator's transcript, running the audio at 2x speed and just clicking "yes" or "no" to keep the good ones and bin the bad ones. It's roughly 5x faster to do this than to come up with transcripts from scratch, so if the annotators are 5x cheaper, then you come out ahead. You can go even further and pre-filter labels to discard any where inter-annotator agreement falls below some threshold (i.e. 3 people label the same piece of data, and you only include a sample when at least 2 annotators give the same label). You can also use that to discard all annotations from annotators who regularly disagree with the majority. This is just the reality of outsourced data labelling. One thing I think is really important is to structure the compensation well, so that labellers get paid more when they do a better job. Paying per sample is a terrible idea, and even I was guilty of this - back in university I was paid $20 or so to hand-write 500 words on a resistive touchscreen to train a handwriting recognition model. I won't say I half-assed it, but I remember trying to get through it as as quickly as possible to get my money and go for beer (I think I also justified it to myself on the basis that sloppy samples would help make the bounds of the dataset distribution more robust!).