6 ms·

I have an anecdotal story on the other side. I had a small startup and we had a list of URLs. We loaded them into mTurk and asked turkers to visit site and fi

by giarc 8y ago

I have an anecdotal story on the other side. I had a small startup and we had a list of URLs. We loaded them into mTurk and asked turkers to visit site and find an email address (probably could have automated this ourselves, but we didn't). Most of the responses were just info@<<websitename>>.com and we could see they completed the task in <10 seconds. So although info@ is a legit answer, after checking a few we found out the turker just guessed it would be a suitable answer and get paid for it. We basically couldn't trust any of the answers and just walked away from that service.

jchw 8y ago
Amusing. Couldn't someone on the other end automate this, then cash in on it? I wonder if there's some potential in this thought: paying for automation as a service, based on the volume of output...
- turtlegrids 8y ago
  So... Fiverr or Upwork?
  - jchw 8y ago
    Those are paid on a per job basis. Different incentive structure. Here you would basically be defining how much you'd be willing to pay per work item, regardless of how it's done; you wouldn't get the secret sauce, just the black box output. It's not really a great idea, but in fairness I don't think it's much worse than what's out there either.
    - jjeaff 8y ago
      There are quite a few "scraping" companies out there that do this. They generally focus on getting you the data in an automated fashion, but it is black box and could be manual if they wanted.
      - turtlegrids 8y ago
        I sense an opportunity to arbitrage mTurk -> Automation -> mTurk. Repeat as many times as necessary for a successful exit (IPO vs. Acquisition doesn't matter, either are acceptable)
turtlegrids 8y ago
What's even worse is the people who do that simple guessing vs. actually doing the task ruins it for others -- eventually the Amazon mTurk AI thinks that your correct answers are 'wrong' due to a mass of others garbage responses. Thus your accuracy rate goes down and you can't qualify for many or most of the new mTurk tasks.
notafraudster 8y ago
You set up the HIT the wrong way. The correct way to do this is to assign the same URL to multiple Turkers and look for cases where people disagree in order to verify the response quality. Granted it's still possible that both Turkers assigned to the HIT could commit fraud in the same way, but surely just as a matter of chance this bounds your error rate pretty substantially.
- ceejayoz 8y ago
  You can also use qualification tests (https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMechanicalTurkRequester/Concepts_QualificationsArticle.html#qualification-tests https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMechanical...) to require Turkers to demonstrate competence before doing tasks. I'd imagine you could inject a certain percentage of known-answer tasks into the queue as well, as ongoing quality control.
- turtlegrids 8y ago
  > Granted it's still possible that both Turkers assigned to the HIT could commit fraud in the same way, but surely just as a matter of chance this bounds your error rate pretty substantially. I wonder if there's been research into the effort in time or persons to perpetuate this fraud. Knowing Amazon, I'm certain they've analyzed this internally if not constantly at least a few times over the years. I wonder if there's a public attempt at the same analysis.
varelse 8y ago
Seems to me the answer to this is a sliding scale of compensation that rises for consistently valid responses and decays to zero for these sorts. Of course, this can be gamed as well, but it seems like everything's a Red Queen's Race these days so the only way to avoid^H^H^H^H^Hminimize one's losses is to race.
- paulryanrogers 8y ago
  Did you mean "... is to [not] race"?
  - varelse 8y ago
    No, I meant exactly what I wrote. For example, if you were to try to duplicate Google today, you would drown in content farms and other sorts of BlackHat SEO because Google has been engaged in just such a Red Queen's Race with the BlackHats for over a decade. So even if you come up with a scheme that rewards WhiteHat Turks, the BlackHats will inevitably crack it unless you insure it's a moving target.
josephjrobison 8y ago
Have seen similar things, as it sounds like a great way to use Mturk on the surface. Perhaps the answer is more direction and qualifiers, such as "must be a name, cannot be info@ or editor@" Side note, if anyone has a way to automate the above in a way better than Hunter.io, there's big money in that.
- giarc 8y ago
  Had we known about Hunter.io we would have used it. We had a small list of URL (~500). Would have been the small price to simply have it automated by a specific program rather than humans.
haditab 8y ago
I tried MTurk as well out of curiosity. I only made a few dollars before giving up, but my observation was that like anything else in life, you get what you pay for. High paying jobs on MTurk require a impeccable record of a high number of accepted jobs and a near perfect acceptance ratio. You slowly (I can't emphasize how slow this is) build up your reputation as you complete jobs for pennies until you get to the category where you can accept jobs that pay $100 dollars or more. Turkers who work hard to get the reputation to do high paying jobs won't risk losing the reputation by cheating. So if you want good results you need to pay up to get those Turkers on your job.
- testvox 8y ago
  Seems like there should be other ways to boost your reputation. Maybe putting a deposit down which you can lose if your acceptance ratio gets too bad.
  - thatoneuser 8y ago
    Wtf is with people suggesting contractors getting paid less than minimum wage should be required to put deposits that can be stolen from them if they don't work? It's this just schilling or...?
    - testvox 8y ago
      No it's the standard solution for quickly establishing some reputation, prove that you have something to lose.

by giarc 8y ago

jchw 8y ago

Amusing. Couldn't someone on the other end automate this, then cash in on it? I wonder if there's some potential in this thought: paying for automation as a service, based on the volume of output...

turtlegrids 8y ago
So... Fiverr or Upwork?
- jchw 8y ago
  Those are paid on a per job basis. Different incentive structure. Here you would basically be defining how much you'd be willing to pay per work item, regardless of how it's done; you wouldn't get the secret sauce, just the black box output. It's not really a great idea, but in fairness I don't think it's much worse than what's out there either.
  - jjeaff 8y ago
    There are quite a few "scraping" companies out there that do this. They generally focus on getting you the data in an automated fashion, but it is black box and could be manual if they wanted.
    - turtlegrids 8y ago
      I sense an opportunity to arbitrage mTurk -> Automation -> mTurk. Repeat as many times as necessary for a successful exit (IPO vs. Acquisition doesn't matter, either are acceptable)

turtlegrids 8y ago

What's even worse is the people who do that simple guessing vs. actually doing the task ruins it for others -- eventually the Amazon mTurk AI thinks that your correct answers are 'wrong' due to a mass of others garbage responses. Thus your accuracy rate goes down and you can't qualify for many or most of the new mTurk tasks.

notafraudster 8y ago

You set up the HIT the wrong way. The correct way to do this is to assign the same URL to multiple Turkers and look for cases where people disagree in order to verify the response quality. Granted it's still possible that both Turkers assigned to the HIT could commit fraud in the same way, but surely just as a matter of chance this bounds your error rate pretty substantially.

ceejayoz 8y ago
You can also use qualification tests (https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMechanicalTurkRequester/Concepts_QualificationsArticle.html#qualification-tests https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMechanical...) to require Turkers to demonstrate competence before doing tasks. I'd imagine you could inject a certain percentage of known-answer tasks into the queue as well, as ongoing quality control.
turtlegrids 8y ago
> Granted it's still possible that both Turkers assigned to the HIT could commit fraud in the same way, but surely just as a matter of chance this bounds your error rate pretty substantially. I wonder if there's been research into the effort in time or persons to perpetuate this fraud. Knowing Amazon, I'm certain they've analyzed this internally if not constantly at least a few times over the years. I wonder if there's a public attempt at the same analysis.

varelse 8y ago

Seems to me the answer to this is a sliding scale of compensation that rises for consistently valid responses and decays to zero for these sorts. Of course, this can be gamed as well, but it seems like everything's a Red Queen's Race these days so the only way to avoid^H^H^H^H^Hminimize one's losses is to race.

paulryanrogers 8y ago
Did you mean "... is to [not] race"?
- varelse 8y ago
  No, I meant exactly what I wrote. For example, if you were to try to duplicate Google today, you would drown in content farms and other sorts of BlackHat SEO because Google has been engaged in just such a Red Queen's Race with the BlackHats for over a decade. So even if you come up with a scheme that rewards WhiteHat Turks, the BlackHats will inevitably crack it unless you insure it's a moving target.

josephjrobison 8y ago

Have seen similar things, as it sounds like a great way to use Mturk on the surface. Perhaps the answer is more direction and qualifiers, such as "must be a name, cannot be info@ or editor@" Side note, if anyone has a way to automate the above in a way better than Hunter.io, there's big money in that.

giarc 8y ago
Had we known about Hunter.io we would have used it. We had a small list of URL (~500). Would have been the small price to simply have it automated by a specific program rather than humans.

haditab 8y ago

I tried MTurk as well out of curiosity. I only made a few dollars before giving up, but my observation was that like anything else in life, you get what you pay for. High paying jobs on MTurk require a impeccable record of a high number of accepted jobs and a near perfect acceptance ratio. You slowly (I can't emphasize how slow this is) build up your reputation as you complete jobs for pennies until you get to the category where you can accept jobs that pay $100 dollars or more. Turkers who work hard to get the reputation to do high paying jobs won't risk losing the reputation by cheating. So if you want good results you need to pay up to get those Turkers on your job.

testvox 8y ago
Seems like there should be other ways to boost your reputation. Maybe putting a deposit down which you can lose if your acceptance ratio gets too bad.
- thatoneuser 8y ago
  Wtf is with people suggesting contractors getting paid less than minimum wage should be required to put deposits that can be stolen from them if they don't work? It's this just schilling or...?
  - testvox 8y ago
    No it's the standard solution for quickly establishing some reputation, prove that you have something to lose.