1) RE: Unintended consequences - It was usually some mix of willful or accidental misinterpretation of what we wanted. I can't go into details, but in many case

1) RE: Unintended consequences - It was usually some mix of willful or accidental misinterpretation of what we wanted. I can't go into details, but in many cases the annotators are really aiming for maximizing billable activities. In situations where there are some ambiguities, they would pick one interpretation and just go with it without really making the effort to verify. In some ways, I understand their perspective in the sense that they know their work is a commodity and would just do the minimally-viable job to get paid. 2) RE: Benefits of outsourcing - The primary benefit was usually speed to get to a certain dataset scale. These vendor had existing pools of workers, which we can access immediately. There were potential cost-savings but it was never as good as we had projected. The quality of labeling would be less than ideal, which would trigger interventions to verify or improve annotations, which then adds to cost and complexity. 3) RE: In-house ops - Essentially, moving things in-house doesn't magically solve the issues we had. It's a lot of work to recruit and organize data labeling teams. They are still subject to the same incentive-misalignment problems as outsourcing, but we obviously have a closer relationship with them and that seems to help. We try to communicate to them the importance of their work, especially early on, where their feedback and "feel" for the data is very valuable. And it's much much more expensive, but all things considered still the "right" approach in many cases. In some scenarios, we can amplify some of their work by using synthetic data generators etc.