4 ms·

Ground truth basic dynamics are similar to the basic dynamics of Prediction Markets. Having experimented with creating Prediction Markets from scratch I have w

Ground truth basic dynamics are similar to the basic dynamics of Prediction Markets. Having experimented with creating Prediction Markets from scratch I have witnessed first hand how the bias of the participants will skew or even nullify the wisdom of the crowd, with wisdom-free results. > ‘[A] large majority of crowdworkers (94%) have had work that was rejected or for which they were not paid. Yet, requesters retain full rights over the data they receive regardless of whether they accept or reject it; Roberts (2016) describes this system as one that “enables wage theft”. So, what was the source of wisdom in rejecting the results?

visarga 5y ago
> So, what was the source of wisdom in rejecting the results? You can run the same task 3 or 5 times and take the consensus option rejecting the others. But from experience unless the taggers have had special training in the current task the quality of annotations is crap. Then you have to do it all over again with someone in-house and gain nothing from the whole crowd sourcing ordeal. Outside people tend to think anyone could label examples, but practice shows that it takes a special kind of person to do it well. Probably this is why many get their work rejected.
- SQueeeeeL 5y ago
  >Probably this is why many get their work rejected. It's interesting we've built a system where people are allowed to be given a job and then told they're unqualified to perform it. If a construction site hired someone as a heavy machine operator and then immediately decided to fire them, they'd still be owed some form of wages.
  - bjornsing 5y ago
    The owed wages wouldn’t be the major problem though…
    - SQueeeeeL 5y ago
      A days work at least, definitely isn't free
  - visarga 5y ago
    Some tasks are only given to people who have completed a training course. But not all customers do it.
    - willcipriano 5y ago
      If you want to bring workers into a store and train them you have to pay them for the time they spent even if they leave halfway though.

visarga 5y ago

> So, what was the source of wisdom in rejecting the results? You can run the same task 3 or 5 times and take the consensus option rejecting the others. But from experience unless the taggers have had special training in the current task the quality of annotations is crap. Then you have to do it all over again with someone in-house and gain nothing from the whole crowd sourcing ordeal. Outside people tend to think anyone could label examples, but practice shows that it takes a special kind of person to do it well. Probably this is why many get their work rejected.

SQueeeeeL 5y ago
>Probably this is why many get their work rejected. It's interesting we've built a system where people are allowed to be given a job and then told they're unqualified to perform it. If a construction site hired someone as a heavy machine operator and then immediately decided to fire them, they'd still be owed some form of wages.
- bjornsing 5y ago
  The owed wages wouldn’t be the major problem though…
  - SQueeeeeL 5y ago
    A days work at least, definitely isn't free
- visarga 5y ago
  Some tasks are only given to people who have completed a training course. But not all customers do it.
  - willcipriano 5y ago
    If you want to bring workers into a store and train them you have to pay them for the time they spent even if they leave halfway though.