I'm on the team at Surge AI (data labeling platform + workforce), so this article hits home. We started Surge AI precisely because our team has always been agai

I'm on the team at Surge AI (data labeling platform + workforce), so this article hits home. We started Surge AI precisely because our team has always been against the adversarial, penny labor design of crowdwork systems: systems dependent on multi-annotator consensus lead to poor quality data (and ignore the inherent subjectivity in the rich, language-based tasks I love), they lead to suboptimal outcomes and treatment for all parties, and you get get what you pay for! For example: 1. Most data labeling systems don't allow you to communicate meaningfully with your workforce. In contrast, we prize two-way communication; your data labelers are the ones going through tens of thousands examples, so they often have amazing feedback for how to improve your data and design your tasks better. And of course, you often have questions for them as well. 2. Context matters. I can't label Spanish hate speech; someone from Mexico City often can't label Madrid slang either. 3. The majority vote isn't always the best one. Real-world data is often personalized and subjective; your opinion on a funny or angry story may not match mine, and that's okay. Our training sets and AI models should reflect that. We just wrote a blog post on the subtle nuances when considering majority votes and inter-rater reliability metrics: https://www.surgehq.ai/blog/the-pitfalls-of-inter-rater-reliability-in-data-labeling-and-machine-learning https://www.surgehq.ai/blog/the-pitfalls-of-inter-rater-reli... 4. Curating annotator pools. Our product is designed around helping you build custom labeling teams that you trust, who learn the nuances of your domain and stay with you over time.