3 ms·

What the heck is data turking?

clay_to_n 10y ago
I think they mean manually labeling data en-masse. Maybe tagging photos with a description of what's in it, as a common example. "Turking" is probably a reference to Amazon's Mechanical Turk service, where you can to get lots of low-effort work done by people.
bradneuberg 10y ago
Using something like Mechanical Turk to manually add ground truth data to something, though we don't use Mechanical Turk but rather other providers.
- prats226 10y ago
  What are other providers that you use? Since you are using it as input to your training set, it should have fairly high accuracy (even more accurate than manual errors in tagging)
- rtkaleta 10y ago
  Ditto, are there any providers you are allowed to mention that you have had good experience with?
dharma1 10y ago
I guess crowd sourcing ground truthing/generating more validated data with mechanical turk?
- jxramos 10y ago
  The crowd sourcing aspect is accurate, but I think the turk half is used more generically past the actual Amazon browser GUI experience tool to whatever annotation tool one has available or has invested development in. Stuff like https://www.crowdflower.com/ https://www.crowdflower.com/ One other thing worth pointing out is why you can't always lean on the Amazon Turk per se because some of the tasks you need to train from require specialist knowledge that takes years to hone. It is not a task for the common layperson who can spot a cat in an image or anything else that exploits "common knowledge". In the case of the company I'm at we had to pay contract work for these specialists to come in house and operate our annotation tools to build up our ground truth data.
  - dharma1 10y ago
    Yep, it's just a historical reference to Mechanical Turk since that was one of the first of such services. You are probably not going to be able to get minimum wage online workers in 3rd world countries to annotate medical images. What was your data set like?
    - jxramos 10y ago
      It was basically a bunch of microscope images of a certain kind of tissue.
  - rtkaleta 10y ago
    Ditto on not being able to rely on some mechanical turk services - although the quality of answers/labels you get back inevitably depends on how well you have laid out the task at hand, we have additionally witnessed poor quality output that was outright rogue - tasks being completed in sub seconds or same answers given by a user no matter the question - both pointing to tasks being completed by bots not humans. In fact we had resolved to building a turker bot detector and started rejecting tasks completed in suspicious ways. Only once we have built ourselves a trusted turkers population did we start to get quality data back. I suspect most people don't bother to go back and reject poor quality answers and that is why the bots survive.
    - jxramos 10y ago
      Wow, I believe it. We didn't have so much a rogue situation, but you really do have to constrain their actions to just what you wanted. I tried to find the source but there was some YouTube video I watched where the guy made this good comment about creating GUIs where you have to put a real emphasis on preventing users from doing things you do not want them to do. You can't always focus on features but also constraints. I really took that message to heart after experiencing some of the human unpredictability found in building up training data. It made for some interesting payment debates to ask them to redo some work that was incomplete. Fun stuff.

I think they mean manually labeling data en-masse. Maybe tagging photos with a description of what's in it, as a common example. "Turking" is probably a reference to Amazon's Mechanical Turk service, where you can to get lots of low-effort work done by people.

bradneuberg 10y ago

Using something like Mechanical Turk to manually add ground truth data to something, though we don't use Mechanical Turk but rather other providers.

prats226 10y ago
What are other providers that you use? Since you are using it as input to your training set, it should have fairly high accuracy (even more accurate than manual errors in tagging)
rtkaleta 10y ago
Ditto, are there any providers you are allowed to mention that you have had good experience with?

dharma1 10y ago

I guess crowd sourcing ground truthing/generating more validated data with mechanical turk?

jxramos 10y ago
The crowd sourcing aspect is accurate, but I think the turk half is used more generically past the actual Amazon browser GUI experience tool to whatever annotation tool one has available or has invested development in. Stuff like https://www.crowdflower.com/ https://www.crowdflower.com/ One other thing worth pointing out is why you can't always lean on the Amazon Turk per se because some of the tasks you need to train from require specialist knowledge that takes years to hone. It is not a task for the common layperson who can spot a cat in an image or anything else that exploits "common knowledge". In the case of the company I'm at we had to pay contract work for these specialists to come in house and operate our annotation tools to build up our ground truth data.
- dharma1 10y ago
  Yep, it's just a historical reference to Mechanical Turk since that was one of the first of such services. You are probably not going to be able to get minimum wage online workers in 3rd world countries to annotate medical images. What was your data set like?
  - jxramos 10y ago
    It was basically a bunch of microscope images of a certain kind of tissue.
- rtkaleta 10y ago
  Ditto on not being able to rely on some mechanical turk services - although the quality of answers/labels you get back inevitably depends on how well you have laid out the task at hand, we have additionally witnessed poor quality output that was outright rogue - tasks being completed in sub seconds or same answers given by a user no matter the question - both pointing to tasks being completed by bots not humans. In fact we had resolved to building a turker bot detector and started rejecting tasks completed in suspicious ways. Only once we have built ourselves a trusted turkers population did we start to get quality data back. I suspect most people don't bother to go back and reject poor quality answers and that is why the bots survive.
  - jxramos 10y ago
    Wow, I believe it. We didn't have so much a rogue situation, but you really do have to constrain their actions to just what you wanted. I tried to find the source but there was some YouTube video I watched where the guy made this good comment about creating GUIs where you have to put a real emphasis on preventing users from doing things you do not want them to do. You can't always focus on features but also constraints. I really took that message to heart after experiencing some of the human unpredictability found in building up training data. It made for some interesting payment debates to ask them to redo some work that was incomplete. Fun stuff.