3 ms·
> How do you assess labelling quality? You have multiple people to label the same piece of data. This increases accuracy of labeling and help to spot bad labele
by BitJockey 4y ago
> How do you assess labelling quality?
You have multiple people to label the same piece of data. This increases accuracy of labeling and help to spot bad labelers.
- layer8 4y agoIt only spots labellers that are not in the middle of the curve of those who you employ. My point was, if factors like salary affect labelling quality, how do you steer that? You don’t know whether if you’d pay double, you’d get much better fact-checkers, for example.
- kelseyfrog 4y agoInterater and intrarater reliability are two general approaches. You could even regress reliability with salary and make a case that reliability would improve by a certain amount(within a prediction interval), but that's a business decision to act on, not a statistics problem.