4 ms·

It is hard for me to compute this. AI engineers know very well the value of good training data. They all get to label some, they get close and personal with it.

It is hard for me to compute this. AI engineers know very well the value of good training data. They all get to label some, they get close and personal with it. But when it comes to job titles and perks, it's like the labelling people are inferior. I have worked as a ML engineer close with internal labelling teams and let me tell you, not everyone could do it. It is a hard job, and often requires deep thinking to solve. Many ML engineers themselves are actually bad at labelling. And the contribution of the labelling team can be equal sized or more important than that of the ML engineers. In normal conditions you need at least one labelling person / engineer, all projects need labelling, there's never a gap, labelling is never finished. Finding a good labeller is also hard - not everyone on the street could do it, contrary to public expectations.

lifeisstillgood 4y ago
Like a really good tester ?
layer8 4y ago
How do you assess labelling quality? What I’m wondering is, how would the feedback loop work, in the longer term, between quality of AI output and paying/selecting labellers?
- BitJockey 4y ago
  > How do you assess labelling quality? You have multiple people to label the same piece of data. This increases accuracy of labeling and help to spot bad labelers.
  - layer8 4y ago
    It only spots labellers that are not in the middle of the curve of those who you employ. My point was, if factors like salary affect labelling quality, how do you steer that? You don’t know whether if you’d pay double, you’d get much better fact-checkers, for example.
    - kelseyfrog 4y ago
      Interater and intrarater reliability are two general approaches. You could even regress reliability with salary and make a case that reliability would improve by a certain amount(within a prediction interval), but that's a business decision to act on, not a statistics problem.

lifeisstillgood 4y ago

Like a really good tester ?

layer8 4y ago

How do you assess labelling quality? What I’m wondering is, how would the feedback loop work, in the longer term, between quality of AI output and paying/selecting labellers?

BitJockey 4y ago
> How do you assess labelling quality? You have multiple people to label the same piece of data. This increases accuracy of labeling and help to spot bad labelers.
- layer8 4y ago
  It only spots labellers that are not in the middle of the curve of those who you employ. My point was, if factors like salary affect labelling quality, how do you steer that? You don’t know whether if you’d pay double, you’d get much better fact-checkers, for example.
  - kelseyfrog 4y ago
    Interater and intrarater reliability are two general approaches. You could even regress reliability with salary and make a case that reliability would improve by a certain amount(within a prediction interval), but that's a business decision to act on, not a statistics problem.