9 ms·

Human Workers, Managed by an Algorithm

80 points by prayag 14y ago

rdl 14y ago
Interesting that MIT Tech Review wrote an article on this, since a lot of great research (at various universities, including MIT) is now a lot easier due to easily available human task workers. Hopefully this will make research easier, maybe reduce grad-student drudgery, and speed up scientific progress for everyone. (Task work is useful for business, too, but it seems particularly useful in the same ways cloud computing is useful -- the ability to rapidly spin up a short-term task to handle load, without persistent staffing costs.)
- prayag 14y ago
  >since a lot of great research (at various universities, including MIT) is now a lot easier due to easily available human task workers This is very true. A lot of researchers today are using MobileWorks and other crowdsourcing platforms. It has a lot of implication in a number of research areas including AI, HCI, Databases, Bio-sciences, Earth-Sciences and a lot more. These are all fields that have used crowdsourcing and have published papers about crowdsourcing. In fact, MobileWorks came out of UC Berkeley where the founders were doing research in HCI using crowdsourcing before we started building our own platform because we thought the existing competitors were a pain to use.
- dchichkov 14y ago
  Unlike turkers, grad students produce much cleaner data and also see the data first hand. Which is important. I consider mturk/offshore as a necessary evil, rather than a panacea that will "make research easier and speed it up". If data quality matters and you can avoid mturk - avoid it.
  - prayag 14y ago
    >Unlike turkers, grad students produce much cleaner data and also see the data first hand. Which is important. I consider mturk/offshore as a necessary evil, rather than a panacea that will "make research easier and speed it up". This is what we are trying to solve. You can get good clean data from crowdsourcing too if you give it out to the most suitable workers and have them work collaboratively.
  - anandkulkarni 14y ago
    >If data quality matters and you can avoid mturk - avoid it. You've identified a great reason to use MobileWorks instead. Crowd work doesn't have to be low-quality; Turk just hasn't done a terrific job of it. We've achieved substantially higher quality than MTurk by treating workers better, paying fairer wages, and giving workers tools to collaborate.
  - huahaiy 14y ago
    You get what you pay for. If you pay them pennies, you get garbage, but if you pay them equitable wages, you get real work done. It's that simple.
  - aantix 14y ago
    Seems like you're painting with a broad brush. Design of HITs on Mechanical Turk is crucial. Implement gold standard questions per assigment to ensure that you're not receiving spam from a bot.
    - anandkulkarni 14y ago
      It turns out this is one of the core problems with Mechanical Turk that drives folks to MobileWorks instead. In Turk, users spend so much time implementing protection against malicious workers and bots that they need to develop real expertise in crowdsourcing in order to be able to outsource their work effectively. We've filtered workers more effectively up front and reduced the incentive to cheat substantially, so things largely just work.
nobbis 14y ago
Add a couple more layers and you get... algorithms managing humans who control robots to clean for humans: http://www.youtube.com/watch?v=0aqghgoeCWk&hd=1 http://www.youtube.com/watch?v=0aqghgoeCWk&hd=1
kghose 14y ago
Full circle. Remember that the original 'computers' were people who performed computations. If you recall Feynman's self indulgent book 'Surely you are joking' he has a chapter where he describes how they would write algorithms which were then run by humans operating adding machines while working on atomic bomb related problems.
- danso 14y ago
  Is this related to the claim that Feynman was one of the original forefathers of parallel computing?
  - kghose 14y ago
    Hmm, perhaps: there is a passage somewhere about how they set up a scheme where instead of the one person doing all the different computations they set up sequences where each person would only do one computation but for different problems which would circulate through the pool in a staggered fashion of some such.
  - jbester 14y ago
    I think that has to do with feynman's time at Thinking Machines: http://longnow.org/essays/richard-feynman-connection-machine/ http://longnow.org/essays/richard-feynman-connection-machine...
EvilTerran 14y ago
I'm immediately reminded of Manna: http://marshallbrain.com/manna1.htm http://marshallbrain.com/manna1.htm
- anandkulkarni 14y ago
  Let's hope it doesn't end up that way! The article only briefly mentions it, but we've worked hard to make sure that even if tasks are algorithmically generated for MobileWorkers, they have the opportunity to collaborate on task execution in a way that's much more similar to a virtual office than to being a cog (persistent worker-to-worker chat, managers to support and explain tasks, and more). This makes a material difference in the quality of the final results, lets people enjoy their work more, and lets individuals take ownership over what they're doing.
  - EvilTerran 14y ago
    Oh, I'm not saying you're heralding some kind of dystopian future, it was the headline more than the article that reminded me of the story. It does sound like you're doing good work. Incidentally, much as that story does drag on a little in places, it does eventually present a more utopian part of that hypothetical future world -- one where post-scarcity has been embraced, and the technology is used to increase freedom instead of restricting it.
huahaiy 14y ago
It's all about how much you pay the turkers. I have done many studies on mturk and the quality of results have been very good so far. My secret? Pay people equitable wages. I always do a small pilot study to find out how long on average does my task take people to do. Then price the HIT so that people can get at least $6 an hour, sometimes much more. This way, I get high quality results very fast. For example, I posted 100 assignments at 11am today, each took about 10 minutes to do, and now it is 1:40pm, and I have already gotten all the results I need. Compared with having to pay $20 for people to come to my lab for an hour, mturk is heaven.
- anandkulkarni 14y ago
  That's a terrific strategy, and that's one of the strategies we apply internally here at MobileWorks. Treating folks fairly is the right way to get good results. We tie price to task time explicitly, to prevent folks from underpaying MobileWorkers. This makes sure results come back right. You can read our IEEE paper for details: http://www.cs.berkeley.edu/~bjoern/papers/kulkarni-mobileworks-ieee2012.pdf http://www.cs.berkeley.edu/~bjoern/papers/kulkarni-mobilewor...
  - technotony 14y ago
    That is great. When I was working in the Philippines for a microfinance bank I had some of my staff work on mTurk during a quiet period and they were barely earning local minimum wage from the work. Croudsourcing work like this offers great potential to get income into poor, under-employed regions of the world but we can't let it become the new sweatshop.
DenisM 14y ago
Frederick Taylor would be proud: http://en.wikipedia.org/wiki/Scientific_management http://en.wikipedia.org/wiki/Scientific_management
stfu 14y ago
Interesting article, but somehow the comment section turned into a MobileWorkers commercial?
orangethirty 14y ago
Whilst waiting for my turn to order on a local subs restaurant, I saw something familiar. I was watching Big O live and in action. There were two cashiers. Each one greeted the customer differently. The lady on the right was faster, because she would ask yes or no questions. Then she would ask for payment type up front and prepare the ATM/credit card terminal in advance if the client said ATM/CC. The other cashier would just ask random questions in no particular order, and would wait until the last minute to prepare everything for the payment part of the transaction. She never had the pen at hand. Ever. Two people. Same training ( I assume ), and two vastly different algorithms operating for the same goal. Imagine if a computer would take in the order. It would closely match the lady on the left POS terminal. She was as close as a computer anyway. Her routine went a bit like this: (start transaction (greeting (choose sub type (type of bread (white or wheat) (any deviation from the STD ingredients) (? combo meal)) (? change chips for cookie) (? change soda for bottled water) (type of payment (prepare pen & adjust terminal) (? ATM (get card ( swipe ( punch in numbers ( ? accepted || declined (goto start <-- ) give customer card back ))))) (end transaction))) I just realized how lispy this came out to be. :)
patdennis 14y ago
According to company cofounder Anand Kulkarni, the aim is to get the crowd of workers to "behave much more like an automatic resource than like individual and unreliable human beings." Rationally, this makes sense. Emotionally, I find it repulsive. I'm not sure which impulse is stronger. I feel like if I were involved in this company, I'd have to focus to some extent on providing a leg up to people in developing countries. Turning people into (potentially miserable) cogs in a machine would weigh on me after a while.
- patio11 14y ago
  I'd have to focus to some extent on providing a leg up to people in developing countries. Get them paying jobs which involve nothing worse than sitting at a computer screen and you're already more effective than, oh, approximately all charity and foreign aid. After you have the jobs you can even try to make them fun jobs, but un-fun jobs beat 3rd world poverty by quite a lot.
  - patdennis 14y ago
    I don't think fun jobs are the goal. I think merit based opportunity is the goal.
    - deleted 14y ago
      [deleted]
- anandkulkarni 14y ago
  Our goal is to get the crowd to behave like an automatic resource, taken as a whole, while providing a decidedly non-cog-like experience for individual workers. We want users to be able to trust the crowd, but we want people to behave, well, like people. In MobileWork, workers work together in small teams under real names, have a dedicated manager and persistent chat, and talk and collaborate closely on work. Even if there is an algorithmic component to the work, the outcome is much more of a virtual office than the anonymous task machine that you're describing. This worker-centered approach is also a point of departure from just about every other crowd platform out there. To tell the truth, we've invested considerable effort into developing an environment where we'd actually be happy working ourselves. Our studies show that this approach works: far from feeling like cogs, folks really do enjoy being MobileWorkers. See, for instance, our upcoming paper in IEEE, cited below. There's a rational reason for this strategy. Our starting premise is that happy, motivated, and fairly-paid workers working in a collaborative environment will outperform anonymous crowd agents. Helping people with a leg up is the other half of that mission.

80 points by prayag 14y ago

rdl 14y ago

Interesting that MIT Tech Review wrote an article on this, since a lot of great research (at various universities, including MIT) is now a lot easier due to easily available human task workers. Hopefully this will make research easier, maybe reduce grad-student drudgery, and speed up scientific progress for everyone. (Task work is useful for business, too, but it seems particularly useful in the same ways cloud computing is useful -- the ability to rapidly spin up a short-term task to handle load, without persistent staffing costs.)

prayag 14y ago
>since a lot of great research (at various universities, including MIT) is now a lot easier due to easily available human task workers This is very true. A lot of researchers today are using MobileWorks and other crowdsourcing platforms. It has a lot of implication in a number of research areas including AI, HCI, Databases, Bio-sciences, Earth-Sciences and a lot more. These are all fields that have used crowdsourcing and have published papers about crowdsourcing. In fact, MobileWorks came out of UC Berkeley where the founders were doing research in HCI using crowdsourcing before we started building our own platform because we thought the existing competitors were a pain to use.
dchichkov 14y ago
Unlike turkers, grad students produce much cleaner data and also see the data first hand. Which is important. I consider mturk/offshore as a necessary evil, rather than a panacea that will "make research easier and speed it up". If data quality matters and you can avoid mturk - avoid it.
- prayag 14y ago
  >Unlike turkers, grad students produce much cleaner data and also see the data first hand. Which is important. I consider mturk/offshore as a necessary evil, rather than a panacea that will "make research easier and speed it up". This is what we are trying to solve. You can get good clean data from crowdsourcing too if you give it out to the most suitable workers and have them work collaboratively.
- anandkulkarni 14y ago
  >If data quality matters and you can avoid mturk - avoid it. You've identified a great reason to use MobileWorks instead. Crowd work doesn't have to be low-quality; Turk just hasn't done a terrific job of it. We've achieved substantially higher quality than MTurk by treating workers better, paying fairer wages, and giving workers tools to collaborate.
- huahaiy 14y ago
  You get what you pay for. If you pay them pennies, you get garbage, but if you pay them equitable wages, you get real work done. It's that simple.
- aantix 14y ago
  Seems like you're painting with a broad brush. Design of HITs on Mechanical Turk is crucial. Implement gold standard questions per assigment to ensure that you're not receiving spam from a bot.
  - anandkulkarni 14y ago
    It turns out this is one of the core problems with Mechanical Turk that drives folks to MobileWorks instead. In Turk, users spend so much time implementing protection against malicious workers and bots that they need to develop real expertise in crowdsourcing in order to be able to outsource their work effectively. We've filtered workers more effectively up front and reduced the incentive to cheat substantially, so things largely just work.

nobbis 14y ago

Add a couple more layers and you get... algorithms managing humans who control robots to clean for humans: http://www.youtube.com/watch?v=0aqghgoeCWk&hd=1 http://www.youtube.com/watch?v=0aqghgoeCWk&hd=1

kghose 14y ago

Full circle. Remember that the original 'computers' were people who performed computations. If you recall Feynman's self indulgent book 'Surely you are joking' he has a chapter where he describes how they would write algorithms which were then run by humans operating adding machines while working on atomic bomb related problems.

danso 14y ago
Is this related to the claim that Feynman was one of the original forefathers of parallel computing?
- kghose 14y ago
  Hmm, perhaps: there is a passage somewhere about how they set up a scheme where instead of the one person doing all the different computations they set up sequences where each person would only do one computation but for different problems which would circulate through the pool in a staggered fashion of some such.
- jbester 14y ago
  I think that has to do with feynman's time at Thinking Machines: http://longnow.org/essays/richard-feynman-connection-machine/ http://longnow.org/essays/richard-feynman-connection-machine...

EvilTerran 14y ago

I'm immediately reminded of Manna: http://marshallbrain.com/manna1.htm http://marshallbrain.com/manna1.htm

anandkulkarni 14y ago
Let's hope it doesn't end up that way! The article only briefly mentions it, but we've worked hard to make sure that even if tasks are algorithmically generated for MobileWorkers, they have the opportunity to collaborate on task execution in a way that's much more similar to a virtual office than to being a cog (persistent worker-to-worker chat, managers to support and explain tasks, and more). This makes a material difference in the quality of the final results, lets people enjoy their work more, and lets individuals take ownership over what they're doing.
- EvilTerran 14y ago
  Oh, I'm not saying you're heralding some kind of dystopian future, it was the headline more than the article that reminded me of the story. It does sound like you're doing good work. Incidentally, much as that story does drag on a little in places, it does eventually present a more utopian part of that hypothetical future world -- one where post-scarcity has been embraced, and the technology is used to increase freedom instead of restricting it.

huahaiy 14y ago

It's all about how much you pay the turkers. I have done many studies on mturk and the quality of results have been very good so far. My secret? Pay people equitable wages. I always do a small pilot study to find out how long on average does my task take people to do. Then price the HIT so that people can get at least $6 an hour, sometimes much more. This way, I get high quality results very fast. For example, I posted 100 assignments at 11am today, each took about 10 minutes to do, and now it is 1:40pm, and I have already gotten all the results I need. Compared with having to pay $20 for people to come to my lab for an hour, mturk is heaven.

anandkulkarni 14y ago
That's a terrific strategy, and that's one of the strategies we apply internally here at MobileWorks. Treating folks fairly is the right way to get good results. We tie price to task time explicitly, to prevent folks from underpaying MobileWorkers. This makes sure results come back right. You can read our IEEE paper for details: http://www.cs.berkeley.edu/~bjoern/papers/kulkarni-mobileworks-ieee2012.pdf http://www.cs.berkeley.edu/~bjoern/papers/kulkarni-mobilewor...
- technotony 14y ago
  That is great. When I was working in the Philippines for a microfinance bank I had some of my staff work on mTurk during a quiet period and they were barely earning local minimum wage from the work. Croudsourcing work like this offers great potential to get income into poor, under-employed regions of the world but we can't let it become the new sweatshop.

DenisM 14y ago

Frederick Taylor would be proud: http://en.wikipedia.org/wiki/Scientific_management http://en.wikipedia.org/wiki/Scientific_management

stfu 14y ago

Interesting article, but somehow the comment section turned into a MobileWorkers commercial?

orangethirty 14y ago

Whilst waiting for my turn to order on a local subs restaurant, I saw something familiar. I was watching Big O live and in action. There were two cashiers. Each one greeted the customer differently. The lady on the right was faster, because she would ask yes or no questions. Then she would ask for payment type up front and prepare the ATM/credit card terminal in advance if the client said ATM/CC. The other cashier would just ask random questions in no particular order, and would wait until the last minute to prepare everything for the payment part of the transaction. She never had the pen at hand. Ever. Two people. Same training ( I assume ), and two vastly different algorithms operating for the same goal. Imagine if a computer would take in the order. It would closely match the lady on the left POS terminal. She was as close as a computer anyway. Her routine went a bit like this: (start transaction (greeting (choose sub type (type of bread (white or wheat) (any deviation from the STD ingredients) (? combo meal)) (? change chips for cookie) (? change soda for bottled water) (type of payment (prepare pen & adjust terminal) (? ATM (get card ( swipe ( punch in numbers ( ? accepted || declined (goto start <-- ) give customer card back ))))) (end transaction))) I just realized how lispy this came out to be. :)

patdennis 14y ago

According to company cofounder Anand Kulkarni, the aim is to get the crowd of workers to "behave much more like an automatic resource than like individual and unreliable human beings." Rationally, this makes sense. Emotionally, I find it repulsive. I'm not sure which impulse is stronger. I feel like if I were involved in this company, I'd have to focus to some extent on providing a leg up to people in developing countries. Turning people into (potentially miserable) cogs in a machine would weigh on me after a while.

patio11 14y ago
I'd have to focus to some extent on providing a leg up to people in developing countries. Get them paying jobs which involve nothing worse than sitting at a computer screen and you're already more effective than, oh, approximately all charity and foreign aid. After you have the jobs you can even try to make them fun jobs, but un-fun jobs beat 3rd world poverty by quite a lot.
- patdennis 14y ago
  I don't think fun jobs are the goal. I think merit based opportunity is the goal.
  - deleted 14y ago
    [deleted]
anandkulkarni 14y ago
Our goal is to get the crowd to behave like an automatic resource, taken as a whole, while providing a decidedly non-cog-like experience for individual workers. We want users to be able to trust the crowd, but we want people to behave, well, like people. In MobileWork, workers work together in small teams under real names, have a dedicated manager and persistent chat, and talk and collaborate closely on work. Even if there is an algorithmic component to the work, the outcome is much more of a virtual office than the anonymous task machine that you're describing. This worker-centered approach is also a point of departure from just about every other crowd platform out there. To tell the truth, we've invested considerable effort into developing an environment where we'd actually be happy working ourselves. Our studies show that this approach works: far from feeling like cogs, folks really do enjoy being MobileWorkers. See, for instance, our upcoming paper in IEEE, cited below. There's a rational reason for this strategy. Our starting premise is that happy, motivated, and fairly-paid workers working in a collaborative environment will outperform anonymous crowd agents. Helping people with a leg up is the other half of that mission.