5 ms·

For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal

For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal the performance of “crowd-sourced” solutions are (eg Amazon Mechanical Turk), and how incredibly important the quality of these labels are for training a model, it made the most sense. I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options? I’m still shocked how low the quality of Mechanical Turk was for just sentiment (positive/negative/neutral/unsure), 99% of the classifications were just random. We narrowed our section for workers to higher-qualified ones, for that matter. What a giant waste of money and time that was, because supposedly it’s the canonical use case for it.

arein3 4y ago
Why don't they label using the same method as captcha Show the same image to 10 people, and keep only those who have a high confidence
- pueblito 4y ago
  Because that literally costs 10x
echen 4y ago
I'd love to chat. Want to reach out to the email in my profile? I'm the founder of a startup solving this exact problem (https://www.surgehq.ai https://www.surgehq.ai), and previously built the human computation platforms at a couple FAANGs (precisely because this was a huge issue I always faced internally). We work with a lot of the top AI/NLP companies and research labs, and do both the "typical" data labeling work (sentiment analysis, text categorization, etc), but also a lot more advanced stuff (e.g., search evaluation, training the new wave of large language models, adversarial labeling, etc -- so not just distinguishing cats and dogs, but rather making full use of the power of the human mind!).
- stingraycharles 4y ago
  Good news for you: being your target audience, we actually did have you guys on our radar.I For the scale of our project, however, the price point was prohibitive. We ended up building a small cli tool that interactively trained the model, and allowed us to focus on the most important messages (eg those where positive/negative sentiment was closest, the labels with the smallest volume, etc). EDIT: If I now look at your website, it seems like you’ve also just provide good tooling for doing these types of things yourself? If that were the case, I wouldn’t mind having paid $50-$100 for a week of access to such a tool. But $20/hr to hire someone who classifies data which we would still need to audit afterwards was too much for us.
  - Aeolun 4y ago
    $20/hour to classify data sounds reasonable though? If you have more time than money it might not make sense, but at that price point I could save myself a lot of time by just working a few extra hours doing SE and let someone else do 3x that amount of labelling.
    - stingraycharles 4y ago
      I fully agree $20/hr is reasonable, it just was too expensive for us at that time. So in the end the whole problem boils down to “quality is (more) expensive”; but MTurk is a special case since they’re so heavily positioning themselves as “the” solution for this and they’re terrible.
weird-eye-issue 4y ago
I haven't worked in this space specifically, but next time you are thinking of outsourcing this type of work I would suggest giving some Filipino VAs a shot. You can hire fluent, sometimes native English speakers who are motivated at $4-7/hr. Actually even less but I stick to the top of that range personally. (I use OnlineJobs.ph to find people)
shreyshnaccount 4y ago
you have experience with this, so you're probably the perfect person to ask this to- why didn't you just use the old (inaccurate) labels, perform some clustering based op and re-label then clusters? does that even make sense?
- theK 4y ago
  Doesn't this bias the labels?
  - shreyshnaccount 4y ago
    probably does, but idk how much.. if its mostly okay it's gonna be easier to correct it id assume
- stingraycharles 4y ago
  The old (inaccurate) labels were so completely utter shit (pardon my French) that it may as wel have been random. I honestly believe most MTurkers just clicked random BS in order to complete the tasks as soon as possible. What I ended up doing was make some Python CLI-based took that made it extremely fast for us to classify messages; after seeding it with about 1000 classifications, we would then focus on the messages based on certain dimensions: eg “contradictions” (“positive” and “negative” being closest as possible, or “angry” and “happy”), “least” (it was surprisingly difficult to find positive and uplifting tweets, and you don’t want a dataset with 90% negative messages!), etc. That way we worked our way through the dataset and were able to get a pretty decent dataset in about a week time. No idea how others approach this type of problem, but it’s what I came up with.
  - bragr 4y ago
    >I honestly believe most MTurkers just clicked random BS in order to complete the tasks as soon as possible. This is correct based on my (limited) mechanical turk experience. Most tasks pay peanuts (minimum payout can be as low as $0.01) so the only reasonable way to make an income is to complete as many tasks as humanly possible, and doing anything but clicking random buttons would slow them down. I doubt paying more could overcome that because so many people engage with the platform in bad faith.
jackbrookes 4y ago
> Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options? The way I think Google does it with reCAPTCHA is to request classification for each item multiple times. If they differ, keep sending them out until you get a consensus on those items. It weeds out those responses that were just basically random clicks.
codedokode 4y ago
> I’m still shocked how low the quality of Mechanical Turk was I don't know about Mechanical Turk, but there is a crowdourcing platform by Yandex. The pay is so low that the reasonable way to earn something is to find a task that is not properly validated and put random answers there from multiple accounts (because there are speed limits). Usually those are tasks by naive foreign companies not knowing about validation. So if you want high quality you need to implement proper validation, triple check every label by different people and do not expect that someone is going to do it for $5/hour. And maybe you should learn how a crowdsourcing service looks from the worker's side, for example, by registering and trying to do some tasks yourself or by reading forums for workers.
- Tycho 4y ago
  Why shouldn’t someone do it for five dollars per hour, and do it properly lest they get fired? Seems like a very easy job and easy to supervise.
  - alxlaz 4y ago
    If it's such an easy job, why outsource it instead of doing it yourself?
    - Tycho 4y ago
      Because there’s other stuff you need to do that you can’t outsource.
      - alxlaz 4y ago
        Surely if it's such an easy job, you can fit it along your other tasks. If you can't then perhaps it's worth re-examining if it's as easy as you think it is.
        tqkxzugoaupvwqr 4y ago
        It’s easy but requires time. Like turning a page is easy but turning a million pages takes time.
maest 4y ago
> I wonder how others do this kind of thing. We did the exact same for a text classification project. The multi-week grind was awful, but it meant 1. we had a really good understanding of our data 2. we discovered surprising edge cases that we would have missed otherwise. There is a very large fixed overhead you need to pay when you start outsourcing that work, so doing it yourself is cheaper at scales beyond what you'd normally expect.
alxlaz 4y ago
> I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options? Others who successfully do it have exactly the same secret sauce that you do: they assign it to someone who is well-compensated and competent. The one time I needed anything remotely like this I just took the old "nobody said programming was gonna be glamorous" adage and ran with it for two weeks. Money-wise, two weeks of programmer time sorting data manually sure beats twelve weeks of programmer time debugging models to cope with inaccurate data, and the results are orders of magnitude better. Given the widespread understanding of how critical training data is, it's mind-boggling to me that businesses in this field try to outsource it to the lowest-paid external company they can find, thus offering the lowest possible performance incentives and pretty much losing any control they have over quality assurance. Then they proceed to spend humongous amounts of money on clean-up and further refining training data sets, with which they could've hired English Lit majors in the first place, who would've given them a perfectly-classified data set from the very beginning.
- EricMausler 4y ago
  But then STEM would have to admit that English Lit Majors didn't waste their money /s
  - drewcoo 4y ago
    English Lit Majors can't find work? Try MTurk!
  - belfalas 4y ago
    One of my university professors liked to quip: next time someone tells you that you're arguing semantics, ask them what semantics means. That'll teach 'em.
sireat 4y ago
I was using MTurk for labeling about 10 years ago. To see the other side I also did a 1 Month stint as a MTurk worker, earning about $300. It is absolutely horrible work, and I used MTurk subreddit to find the "decent" jobs. I had the special firefox extension which ranked the job givers etc. All jobs were below 1st world minimum wage and were incredibly depressing. I think the adult content ones were the worst. "Jian Yang: No! That's very boring work" Worst thing was that you could not stop to think, you had to keep going if you wanted to at least make a few bucks an hour. There are two solutions to the labeling issue: Pay well (at least $20 an hour no matter the location). If the workers feel exploited you do not get good results no matter the sanctions. Even better solution to labeling is to find people who give a shit about your task. That is how I've done 50k lines of labeling training our OCR for a rare font for Tesseract. We have a few volunteers and they know this is important work (preserving cultural heritage, non-profit, national library, etc). Old reCaptcha had a similar "feel good" element. Compare it to newest Google reCaptcha - you know those labels are going to be used for evil at some point in the future.
- MonkeyMalarky 4y ago
  I'm eager to see what nefarious things Google will do once they've mastered the art of identifying all the busses in a photo!
  - stjohnswarts 4y ago
    their motorcycle, hills, signal light, and traffic light AI will break the world when it's over!
  - htrp 4y ago
    https://xkcd.com/1897/ https://xkcd.com/1897/
  - a9h74j 4y ago
    What tells me to start worrying if they get very good at identifying paperclips?
- drewcoo 4y ago
Aeolun 4y ago
I was going to recommend https://gengo.com/sentiment-analysis/ https://gengo.com/sentiment-analysis/ But it seems they’ve been acquired a few years ago, so I have no idea if the quality is still the same.
cptskippy 4y ago
> I’m still shocked how low the quality of Mechanical Turk was for just sentiment Given that MT workers earn based on how quickly they complete a task, and not how accurate. I'm struggling to understand how anyone would expect quality.
amelius 4y ago
> I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options? Use the consensus of 3 or more annotators (or median).
- wongarsu 4y ago
  But is paying 3x $5/h getting better results than hiring a local collage student for $15/h?
  - sgc 4y ago
    Won't you just wind up with a collage from either source? Answer is yes, because those $5/h workers are likely just as educated, but from a less affluent part of the world.
mdorazio 4y ago
> I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options? Fwiw, in a similar situation I did about half myself and farmed out the other half to my retired parents. Using trusted people was the only way I found I could get high accuracy without spending thousands of dollars. But, as I frequently tell people, the hard part of ML isn’t the model/code, it’s the training data.

arein3 4y ago

Why don't they label using the same method as captcha Show the same image to 10 people, and keep only those who have a high confidence

pueblito 4y ago
Because that literally costs 10x

echen 4y ago

I'd love to chat. Want to reach out to the email in my profile? I'm the founder of a startup solving this exact problem (https://www.surgehq.ai https://www.surgehq.ai), and previously built the human computation platforms at a couple FAANGs (precisely because this was a huge issue I always faced internally). We work with a lot of the top AI/NLP companies and research labs, and do both the "typical" data labeling work (sentiment analysis, text categorization, etc), but also a lot more advanced stuff (e.g., search evaluation, training the new wave of large language models, adversarial labeling, etc -- so not just distinguishing cats and dogs, but rather making full use of the power of the human mind!).

stingraycharles 4y ago
Good news for you: being your target audience, we actually did have you guys on our radar.I For the scale of our project, however, the price point was prohibitive. We ended up building a small cli tool that interactively trained the model, and allowed us to focus on the most important messages (eg those where positive/negative sentiment was closest, the labels with the smallest volume, etc). EDIT: If I now look at your website, it seems like you’ve also just provide good tooling for doing these types of things yourself? If that were the case, I wouldn’t mind having paid $50-$100 for a week of access to such a tool. But $20/hr to hire someone who classifies data which we would still need to audit afterwards was too much for us.
- Aeolun 4y ago
  $20/hour to classify data sounds reasonable though? If you have more time than money it might not make sense, but at that price point I could save myself a lot of time by just working a few extra hours doing SE and let someone else do 3x that amount of labelling.
  - stingraycharles 4y ago
    I fully agree $20/hr is reasonable, it just was too expensive for us at that time. So in the end the whole problem boils down to “quality is (more) expensive”; but MTurk is a special case since they’re so heavily positioning themselves as “the” solution for this and they’re terrible.

weird-eye-issue 4y ago

I haven't worked in this space specifically, but next time you are thinking of outsourcing this type of work I would suggest giving some Filipino VAs a shot. You can hire fluent, sometimes native English speakers who are motivated at $4-7/hr. Actually even less but I stick to the top of that range personally. (I use OnlineJobs.ph to find people)

shreyshnaccount 4y ago

you have experience with this, so you're probably the perfect person to ask this to- why didn't you just use the old (inaccurate) labels, perform some clustering based op and re-label then clusters? does that even make sense?

theK 4y ago
Doesn't this bias the labels?
- shreyshnaccount 4y ago
  probably does, but idk how much.. if its mostly okay it's gonna be easier to correct it id assume
stingraycharles 4y ago
The old (inaccurate) labels were so completely utter shit (pardon my French) that it may as wel have been random. I honestly believe most MTurkers just clicked random BS in order to complete the tasks as soon as possible. What I ended up doing was make some Python CLI-based took that made it extremely fast for us to classify messages; after seeding it with about 1000 classifications, we would then focus on the messages based on certain dimensions: eg “contradictions” (“positive” and “negative” being closest as possible, or “angry” and “happy”), “least” (it was surprisingly difficult to find positive and uplifting tweets, and you don’t want a dataset with 90% negative messages!), etc. That way we worked our way through the dataset and were able to get a pretty decent dataset in about a week time. No idea how others approach this type of problem, but it’s what I came up with.
- bragr 4y ago
  >I honestly believe most MTurkers just clicked random BS in order to complete the tasks as soon as possible. This is correct based on my (limited) mechanical turk experience. Most tasks pay peanuts (minimum payout can be as low as $0.01) so the only reasonable way to make an income is to complete as many tasks as humanly possible, and doing anything but clicking random buttons would slow them down. I doubt paying more could overcome that because so many people engage with the platform in bad faith.

jackbrookes 4y ago

> Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options? The way I think Google does it with reCAPTCHA is to request classification for each item multiple times. If they differ, keep sending them out until you get a consensus on those items. It weeds out those responses that were just basically random clicks.

codedokode 4y ago

> I’m still shocked how low the quality of Mechanical Turk was I don't know about Mechanical Turk, but there is a crowdourcing platform by Yandex. The pay is so low that the reasonable way to earn something is to find a task that is not properly validated and put random answers there from multiple accounts (because there are speed limits). Usually those are tasks by naive foreign companies not knowing about validation. So if you want high quality you need to implement proper validation, triple check every label by different people and do not expect that someone is going to do it for $5/hour. And maybe you should learn how a crowdsourcing service looks from the worker's side, for example, by registering and trying to do some tasks yourself or by reading forums for workers.

Tycho 4y ago
Why shouldn’t someone do it for five dollars per hour, and do it properly lest they get fired? Seems like a very easy job and easy to supervise.
- alxlaz 4y ago
  If it's such an easy job, why outsource it instead of doing it yourself?
  - Tycho 4y ago
    Because there’s other stuff you need to do that you can’t outsource.
    - alxlaz 4y ago
      Surely if it's such an easy job, you can fit it along your other tasks. If you can't then perhaps it's worth re-examining if it's as easy as you think it is.
      - tqkxzugoaupvwqr 4y ago
        It’s easy but requires time. Like turning a page is easy but turning a million pages takes time.

maest 4y ago

> I wonder how others do this kind of thing. We did the exact same for a text classification project. The multi-week grind was awful, but it meant 1. we had a really good understanding of our data 2. we discovered surprising edge cases that we would have missed otherwise. There is a very large fixed overhead you need to pay when you start outsourcing that work, so doing it yourself is cheaper at scales beyond what you'd normally expect.

alxlaz 4y ago

> I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options? Others who successfully do it have exactly the same secret sauce that you do: they assign it to someone who is well-compensated and competent. The one time I needed anything remotely like this I just took the old "nobody said programming was gonna be glamorous" adage and ran with it for two weeks. Money-wise, two weeks of programmer time sorting data manually sure beats twelve weeks of programmer time debugging models to cope with inaccurate data, and the results are orders of magnitude better. Given the widespread understanding of how critical training data is, it's mind-boggling to me that businesses in this field try to outsource it to the lowest-paid external company they can find, thus offering the lowest possible performance incentives and pretty much losing any control they have over quality assurance. Then they proceed to spend humongous amounts of money on clean-up and further refining training data sets, with which they could've hired English Lit majors in the first place, who would've given them a perfectly-classified data set from the very beginning.

EricMausler 4y ago
But then STEM would have to admit that English Lit Majors didn't waste their money /s
- drewcoo 4y ago
  English Lit Majors can't find work? Try MTurk!
- belfalas 4y ago
  One of my university professors liked to quip: next time someone tells you that you're arguing semantics, ask them what semantics means. That'll teach 'em.

sireat 4y ago

I was using MTurk for labeling about 10 years ago. To see the other side I also did a 1 Month stint as a MTurk worker, earning about $300. It is absolutely horrible work, and I used MTurk subreddit to find the "decent" jobs. I had the special firefox extension which ranked the job givers etc. All jobs were below 1st world minimum wage and were incredibly depressing. I think the adult content ones were the worst. "Jian Yang: No! That's very boring work" Worst thing was that you could not stop to think, you had to keep going if you wanted to at least make a few bucks an hour. There are two solutions to the labeling issue: Pay well (at least $20 an hour no matter the location). If the workers feel exploited you do not get good results no matter the sanctions. Even better solution to labeling is to find people who give a shit about your task. That is how I've done 50k lines of labeling training our OCR for a rare font for Tesseract. We have a few volunteers and they know this is important work (preserving cultural heritage, non-profit, national library, etc). Old reCaptcha had a similar "feel good" element. Compare it to newest Google reCaptcha - you know those labels are going to be used for evil at some point in the future.

MonkeyMalarky 4y ago
I'm eager to see what nefarious things Google will do once they've mastered the art of identifying all the busses in a photo!
- stjohnswarts 4y ago
  their motorcycle, hills, signal light, and traffic light AI will break the world when it's over!
- htrp 4y ago
  https://xkcd.com/1897/ https://xkcd.com/1897/
- a9h74j 4y ago
  What tells me to start worrying if they get very good at identifying paperclips?
drewcoo 4y ago

Aeolun 4y ago

I was going to recommend https://gengo.com/sentiment-analysis/ https://gengo.com/sentiment-analysis/ But it seems they’ve been acquired a few years ago, so I have no idea if the quality is still the same.

cptskippy 4y ago

> I’m still shocked how low the quality of Mechanical Turk was for just sentiment Given that MT workers earn based on how quickly they complete a task, and not how accurate. I'm struggling to understand how anyone would expect quality.

amelius 4y ago

> I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options? Use the consensus of 3 or more annotators (or median).

wongarsu 4y ago
But is paying 3x $5/h getting better results than hiring a local collage student for $15/h?
- sgc 4y ago
  Won't you just wind up with a collage from either source? Answer is yes, because those $5/h workers are likely just as educated, but from a less affluent part of the world.

mdorazio 4y ago

> I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options? Fwiw, in a similar situation I did about half myself and farmed out the other half to my retired parents. Using trusted people was the only way I found I could get high accuracy without spending thousands of dollars. But, as I frequently tell people, the hard part of ML isn’t the model/code, it’s the training data.