3 ms·

Greetings from Japan. Thank you for sharing such an insightful point. This really resonates, speaking from my experience as an annotator on crowdsourcing plat

Greetings from Japan. Thank you for sharing such an insightful point. This really resonates, speaking from my experience as an annotator on crowdsourcing platforms. I also found that a genuine commitment to quality from fellow annotators can be quite rare. This makes me curious about a few things: 1. What are some concrete examples of the "unintended consequences" you ran into? 2. When you initially considered outsourcing, what was the main benefit you were hoping for (e.g., speed, cost)? 3. On the flip side, what have been the biggest frustrations or challenges with the in-house approach? Would love to hear your thoughts on any of these. Thanks!

fzwang 1y ago
1) RE: Unintended consequences - It was usually some mix of willful or accidental misinterpretation of what we wanted. I can't go into details, but in many cases the annotators are really aiming for maximizing billable activities. In situations where there are some ambiguities, they would pick one interpretation and just go with it without really making the effort to verify. In some ways, I understand their perspective in the sense that they know their work is a commodity and would just do the minimally-viable job to get paid. 2) RE: Benefits of outsourcing - The primary benefit was usually speed to get to a certain dataset scale. These vendor had existing pools of workers, which we can access immediately. There were potential cost-savings but it was never as good as we had projected. The quality of labeling would be less than ideal, which would trigger interventions to verify or improve annotations, which then adds to cost and complexity. 3) RE: In-house ops - Essentially, moving things in-house doesn't magically solve the issues we had. It's a lot of work to recruit and organize data labeling teams. They are still subject to the same incentive-misalignment problems as outsourcing, but we obviously have a closer relationship with them and that seems to help. We try to communicate to them the importance of their work, especially early on, where their feedback and "feel" for the data is very valuable. And it's much much more expensive, but all things considered still the "right" approach in many cases. In some scenarios, we can amplify some of their work by using synthetic data generators etc.
Fastlane2013 1y ago
[dead]

fzwang 1y ago

1) RE: Unintended consequences - It was usually some mix of willful or accidental misinterpretation of what we wanted. I can't go into details, but in many cases the annotators are really aiming for maximizing billable activities. In situations where there are some ambiguities, they would pick one interpretation and just go with it without really making the effort to verify. In some ways, I understand their perspective in the sense that they know their work is a commodity and would just do the minimally-viable job to get paid. 2) RE: Benefits of outsourcing - The primary benefit was usually speed to get to a certain dataset scale. These vendor had existing pools of workers, which we can access immediately. There were potential cost-savings but it was never as good as we had projected. The quality of labeling would be less than ideal, which would trigger interventions to verify or improve annotations, which then adds to cost and complexity. 3) RE: In-house ops - Essentially, moving things in-house doesn't magically solve the issues we had. It's a lot of work to recruit and organize data labeling teams. They are still subject to the same incentive-misalignment problems as outsourcing, but we obviously have a closer relationship with them and that seems to help. We try to communicate to them the importance of their work, especially early on, where their feedback and "feel" for the data is very valuable. And it's much much more expensive, but all things considered still the "right" approach in many cases. In some scenarios, we can amplify some of their work by using synthetic data generators etc.

Fastlane2013 1y ago

[dead]