11 ms·

How much money can you make on Mechanical Turk?

151 points by Steven-Clarke 7y ago

Ajs1 7y ago
Surprising stats. I’ve been aware of mech Turk a long time, but did not realize average pay per task is quite that low
senectus1 7y ago
wow, those number suck hard. I wonder what Nationality these "Turker" are from...
- Thorrez 7y ago
  > Despite an uptick in foreign users over the past few years, the majority of them (75%) are based in the US.
  - bb2018 7y ago
    It is a complete waste of time in many ways but every once in a while I will log on and take a couple surveys while watching a rerun of something before bed. I used to do some survey design/analysis in academia and sometimes it is interesting to see what or how weird the surveys are. I would guess my hourly rate is about $3 and I used that very small amount to buy splurge on cheap useless stuff on Amazon.
  - anovikov 7y ago
    This is because it's near impossible to register unless you are from the U.S., now, i heard. Was a reply to me when i suggessted bs 'programmers' who had no idea what they were doing, to go there instead of trying to hopelessly beg for projects on Upwork - 'i'd love to, but they locked everyone outside the U.S. out'.
  - ForHackernews 7y ago
    America is becoming like Brazil: a third world nation with some wealthy enclaves.
    - meddlepal 7y ago
      Except it's not. Most Americans enjoy a very high standard of living and our poor are still much better off still than Brazilian poor.
    - mruts 7y ago
      As a person who actually lives in a 3rd world country and grew up in America, what you’re saying could not be more false. Even the poorest Americans live a life here that would be considered one of great luxury. We’re talking about living on a dirt floor in collapsing mud houses. We’re talking about babies dying from dirty water as a regular occurrence. No running water, intermittent electricity and definitely no sewage. Most people here haven’t eaten meat in months, occasionally years. If you actually think America is a 3rd world country you need to get out more.
      - ForHackernews 7y ago
        Americans in Flint, MI have poisonous water, and hookworm is on the rise from sewage contaminated water in Alabama.[0] In 2017, millions of Americans went without electricity or running water for months.[1] > If you actually think America is a 3rd world country you need to get out more. I said it's becoming a 3rd world country. There's still some legacy 20th century infrastructure that hasn't collapsed, yet. [0] https://www.theguardian.com/us-news/2017/sep/05/hookworm-lowndes-county-alabama-water-waste-treatment-poverty https://www.theguardian.com/us-news/2017/sep/05/hookworm-low... [1] https://www.theverge.com/2017/9/25/16362410/hurricane-maria-puerto-rico-power-outages-electrical-grid-destroyed https://www.theverge.com/2017/9/25/16362410/hurricane-maria-...
        ataturk 7y ago
        Not sure why the downvote when it is abundantly clear that in every corner of the USA there is this general trend towards decay with very little offsetting it. All our infrastructure is falling apart. I blame the pension crisis. All available monies are going towards pension obligations. Not much is left over for anything else.
gitgud 7y ago
I want to know who is paying $0.01 USD to get someone to manually transrcibe a reciept to a table. It seems the Mechanical Turk ecconomy could be dwindling due to the accessiblity of AI these days.
- cameronbrown 7y ago
  Well, AI can't handle the edge cases yet. For some businesses it makes more economical sense to invest in Turk for transcription rather than training a ML model, which obviously requires a sizeable investment. Either way, you'll need a decent training set and Turk is great for that.
- FluffyKitty 7y ago
  Possibly Expensify? There was that news that came out a bit ago that their "SmartScan" actually fell back on Mechanical Turk [1] even though it was sold as a AI-type thing. [1] https://arstechnica.com/information-technology/2017/11/expensify-acknowledges-potential-privacy-problem-by-calling-it-a-feature/ https://arstechnica.com/information-technology/2017/11/expen...
  - Thorrez 7y ago
    Yep >“Most people’s first mistake is they go on there and accept any task they see,” says Naab. “They’ll do these transcriptions — $0.01 to transcribe a whole receipt for Expensify. That’s a terrible ROI.”
    - jmnicolas 7y ago
      It depends, $0.01 might be worth something in a very poor country. But on the other hand if you have access to a computer a poor country, 1 cent might not be worth your time anyway.
      - mruts 7y ago
        You can use MTurk on phones I assume? Even very poor people often have smartphones and data where I am living (Tanzania).
dsissitka 7y ago
If you'd like to see what this looks like in practice see: https://www.youtube.com/watch?v=h9Qt6bVc5mY&list=PLuh2SVJZ17yC2-32cCfOCw_CIRFsVdNF5&index=2&t=0s https://www.youtube.com/watch?v=h9Qt6bVc5mY&list=PLuh2SVJZ17... The hourly rate will flash every few seconds.
- bob_theslob646 7y ago
  That video looks to show someone who built an automated script that spams random answers. Seems rather terrible to me to be fair.
probably_wrong 7y ago
I had to do some digging into crowdsourcing for my PhD thesis, and these numbers are well reported in the literature. Our group made the calculations to ensure that our task would pay at least minimum wage. As a result, we had to throttle our participants because they would hit our servers pretty hard. That's not to say that the experience was easy - we had to implement every possible sanity check, and even then we ended up throwing away half our data. For us it was still worth it, since twice the rate was still cheap and we knew not to trust the internet. But a less internet-savy researcher blindly trusting their data would have gotten some surprising results.
- namanyayg 7y ago
  What kind of sanity checks did you implement, or how did you know what data to throw?
  - probably_wrong 7y ago
    We had an interactive task (hosted in our own server) and a questionnaire afterwards. Sanity checks off the top of my head: * Control words: our server gave them two (unique) control words (one for joining, one for winning), and we asked for them in the post-task questionnaire. Some of these words ended up being reused among several participants, even from participants that never even started the task. * IP checks: we had an experiment that you could "win" (and earn a bonus), but you only were allowed to play once. Some people restarted the task several times, so we only used the first attempt in a sequence (as reported by their IP and timestamp). * Data thrown away: we further removed data where we had more than one player per IP (to control for both multiple accounts per person and use of proxies), experiments that were way too fast, and experiments with unsupported browsers (which we explictly mentioned in the description). Regarding money, we were not allowed by the TOS to withold payment to anyone that filled the questionnaire, even if we knew they did it in bad faith. We therefore implemented the "winning" bonus, and also gave bonus to people who lost but really tried. I want to point out that a LARGE percentage of participants played honestly, and some of their data was thrown away only out of an abundance of caution. Once you keep the first bad apples out, they simply move to other, easily-exploitable tasks.
closetOperator 7y ago
Unless you have a clear strategy, MTurk work is a complete waste of time. True. I perused Mechanical Turk tasks some months ago, and was confronted by an array of plaintive, useless demands to perform what amounts to unproductive garbage picking for listless and disinterested college grad students running unimaginative projects without a clue as to whether the requested task is even possible. It would be a request like: Gather phone numbers from this queue of web pages. And they'd seem to have paid for a "database" of "leads" which was more than likely an excel spreadsheet of "hyperlinks" categorized according to a search query of keywords from the data provider. You get into the queue, and start pulling up each URL in series, and they're all these expired domains with parked registrar pages for GoDaddy and Tucows or whatever, Along with some Geocities, Angelfile, Tripod and AOL home pages thrown in. Quickly, you get a sense that some fool of data science masters program enrollee paid good money for a dusty, mouldering text file, didn't even look at it, and dropped it right the fuck into a template for a Mechanical Turk task. Now, there are three immediately obvious courses of action. One, abort and never again consider Mechanical Turk as a useful platform for operating an exchange of effort for rewards. Two, plead with the task owner by reporting feedback to them and ask them to stop for a moment and consider the flaws inherent to this framing of a human activity deemed worthy of compensation. Three, obey the letter of the law, and not the spirit, hold your nose, bellow the words "you asked for it!" and proceed to fill the task with the tech support numbers for all of the domain registrars, hoping that you'll not only get paid for grifting on the task owner, but also possibly inundate all these domain squatting registrars with robocalls trawling for psyche student surveys and questionnaires that will attempt to publish similarly terrible research papers designed with the intent to ostensibly "prove" a flawed hypothesis of human behavior with results that couldn't possibly be replicated because the hypothesis itself begs its own question. Valuing my time, I just logged out, and haven't looked back since. They need moderators to mechanical turk the quality of each task, because it benefits no one and wastes people's time, to even propose fruitless, unredeeming tasks. Unless things have changed since winter, from what I witnessed, there is perhaps zero review of tasks to assess whether a request fits the profile of anything even remotely possible or worth trying.
TrackerFF 7y ago
The problem with these gigs is that the tasks are extremely generalized. So while they attract more workers, it is always going to be a sellers market. The only way a worker can scale his income, is by increasing volume - and picking the right assignments. I've been doing side-hustles since I was a teen, and continue today, even though I have a well-paying professional job. My observations are that domain knowledge / specialized expertise always pays better. You're much better off learning some hobby, almost any hobby, and monetizing it. How do you monetize it? Do content writing. Start flipping on Ebay / Craigslist / etc. Perform / create if you can. etc. Now - I understand that these (MTurk) kinds of jobs are for desperate people, that want/need the money now - not 4 years down the road. Building expert knowledge will take time, and you will more likely than not get burnt a few times in the start - but the long-term payoff is much better. Last month, I made $800 in selling two musical instruments. Time invested in it all was around 2-3 hours, mostly on picking 'em up, and shipping them. Maybe not the answer for everyone, but it's extremely hard for me to imagine any other way now. I currently have three different hobbies which I enjoy very much (and have for years), and I'm lucky enough to have so much knowledge in that I also make a decent side-income off. Best of all? It doesn't feel like you're working - though you need to treat it like work, if you're gonna make money.
- arvinsim 7y ago
  Interesting. May I ask what hobbies are these?
s_Hogg 7y ago
Related to this, Stanford created something so you can use Mechanical Turk without being overly exploitative (ensures you pay at least US$15/hr): https://fairwork.stanford.edu/ https://fairwork.stanford.edu/
- s_Hogg 7y ago
  I particularly like this part of the FAQ: I have an even cleverer point suggesting that I might not be able to trust workers' self reports. Might my clever rebuttal be right? Yes. But you are using this tool because you want to help ensure fair treatment to workers, right? Fair treatment starts with trusting workers. They will make good faith estimates.
- mruts 7y ago
  I like the idea of having the market decide what is fair, instead of some Stanford website.
Mediterraneo10 7y ago
I signed up for Mechanical Turk soon after its creation. I transcribed podcasts and made about $10/hour, which was not bad money for a student, especially considering that these podcasts were on interesting subjects and it did not feel like hard work. After a few months I moved on to other things, and when I looked at MTurk again, I saw that it had quickly become a race to the bottom. No freelancing site can withstand those capitalist pressures for long.
ptero 7y ago
The person described is a "senior business analyst", reportedly making good money, but decides to work on MTurk, doing menial tasks, to help offset extra costs (diapers, etc.) of a baby that was born. This sounds insane to me. Almost anything else sounds better: reduce expenses, figure out if he can get more responsibilities and a raise, even get a low % credit to pay for those diapers for a year, etc. . Is this just me? This is an honest question.
- londons_explore 7y ago
  Some MTurk tasks are amenable to sub-contracting out or writing bots to mostly or fully automate. If he does that, I could see it being competitive on an hourly basis to a regular job.
  - dalbasal 7y ago
    How open are Amazon to this? Ooh, the idea of mTurk is "human powered." OTOH, a market for tasks that can be do e via a combination of automations and humans could be powerful.
    - jwieczorek 7y ago
      Such a combination is what Unbabel does in translation. ML-driven automated translation followed by crowd-sourced human review and corrections. It's a neat idea and could be the future of translation (as opposed to full automation).
    - dahfizz 7y ago
      I would assume that Amazon already does this. If a task is trivial enough for an AI, they would do it themselves and get the posters money before the job gets forwarded to the Turks.
      - dalbasal 7y ago
        Sure, Amazon have all sorts of ml APIs. But a task platform where anyone can complete the tasks using ai, humans or a combination that f both could be substantial impressive. Say you need to look through 20k images and determine if they contain hipsters. There may not be an off-the-shelf hipster detector. Maybe you could make one, or use humans or a combination by training an NN.. etc.
Havoc 7y ago
Only way I can see this working is if you're in a CoL area that is low end 3rd world. Else pretty much anything else will be more profitable
- rjf72 7y ago
  I think many people do this not out of need but as a time waster that pays a bit. As the article mentions, 75% of 'Turkers' are American. The article's subject says, "I’d literally be sitting around bored half the day without it. Making money beats doing nothing." Of course he wouldn't be doing literally nothing. Instead he'd be just mindlessly wasting time browsing Facebook, checking out the latest clickbait, posting on forums, and engaging in the sort of activities that studies are increasingly showing are destroying our collective mental health and social cohesion. Instead, he's using that time to make an extra thousand a month.
  - Havoc 7y ago
    >I think many people do this not out of need but as a time waster that pays a bit. As the article mentions, 75% of 'Turkers' are American. Been tempted myself. But then I crunch the numbers vs my actual job and go clean my apartment instead. That plus a very real fear of RSI from all the mouse clicking. I mean I already click at work all day then on hn/reddit in afternoons...
sschueller 7y ago
I wonder how many task on Mechanical Turk can actually be automated. By that I don't mean building some fancy AI but tasks that can easily be automated and have just been posted on Mechanical Turk.
- kuu 7y ago
  I guess automating is not always the cheapest option, and for "once in a life" experiment/test/dataset it can be sometimes cheaper to hire people than develop an algorithm. I guess XKCD said it before: https://xkcd.com/974/ https://xkcd.com/974/
  - taneq 7y ago
    XKCD gave me a 503 "Backend unavailable" and I thought it was some advanced meta-joke for a second.
- H8crilA 7y ago
  Much of what's happening in such markets is collecting data to build ML systems.
Yuval_Halevi 7y ago
MTurk is the closest there is for a modern slave.
- shawabawa3 7y ago
  Slaves are the closest thing to a modern slave
- systemtest 7y ago
  The workers in Qatar beg to differ.
lordnacho 7y ago
Is there anything stopping people from just randomly filling in the surveys? It seems like the kind of thing where someone would have made a bot. How many people who are paying a couple of bucks for something like that are going to complain at you?
- jonchang 7y ago
  Survey instruments generally have a few ways to check for this, usually via attention checks which require you to answer in a specific way as stated in the question text (defeating your bot). Similarly a survey could be designed to ask the same or similar question multiple ways, and a survey response that lacks the expected correlation can also be rejected.
Phemist 7y ago
Fun anecdote about some of my own dabblings in online psych experiments and MTurk. 7 or 8 years ago, my then supervisor and I experimented with running psychological (behavioural) experiments on MTurk. We had created a method that would run in browsers in native javascript and were looking to validate it. Coming from a testing-1st-year-psych-students-in-the-basement-of-the-faculty-building kind of thing, we naively took what the going wage was for that (something like 8 euros/hour), and put an experiment online that would take around 20 min to complete, and paid 2 euros. My supervisor used his own credit card and set the spend limit to around 1000 euros; thinking that we'd never hit it. Boy how wrong were we. Apparently this 6 euro/hour wage was _much_ _much_ higher than the going rate, and we hit the spend limit in around 2 hours. Even though we had to throw out around 70% of the completions, we ended up with usable data from around 150 participants. We went in expecting to run the experiment for a week or 2, and get maybe 50 participants, but came out with 500 in a span of 2 hours. Safe to say, we celebrated a job well done that night over one or two drinks. People were commenting on how nice of a change of pace it was compared to the then-usual MTurk tasks and that they would have done it for free. Some even left their e-mail addresses should we run another online experiment. I've since been out of the field of online behavioural experimentation, but it seems to have taken off quite a bit.
- bob_theslob646 7y ago
  Holy smokes that is an amazing story. Do you have any more? I am always fascinated to hear how researchers design studies to prevent people from just spamming in any old answer.
  - Phemist 7y ago
    I'm not sure what the state of the art is nowadays. Back in the day. The method at the time was so new, we just assumed that there would be no bots that would be able to complete the experiment successfully, or that they would stand out like a sore thumb upon analysing the data. Modern frameworks will eventually have to deal with this I'm sure. The structured way in which experiments are defined also make it easy to develop tools to automate this. Unfortunately I'm not into the field enough anymore to know how people are dealing with it. The biggest problem we had was with people with high latency connections. The effects we were looking for were measured in 10s of milliseconds. In order to tease out these effects, we had to be very particular about the timing of when certain stimuli were presented to the participant. High latency page reloads (which were unavoidable in the system we built our method on) would mess with this high-precision presentation requirement. We measured the latency, but did not pre-emptively exclude anyone based on their latency, hence the high % of people with unusable data in our initial validation experiment. For subsequent experiments I built a "loader" screen that would pretend to be loading the experiment. What it in fact was doing was refreshing the webpage several times (ofcourse, while progressing a progress bar) to measure the latency of the connection of the participant. High average latency + high variance latency connections were excluded. The tresholds were based on what we found in the early validation study. Surprisingly, after throwing out the high-latency data, there were no other exclusions necessary. It seemed that for our validation study, all participants were very attentive during the experiment. In "in-person" cases, researchers would add attention checks to their experiments with the logic that failing these attention checks by itself is no indication of "spamming", but seeing weird quirks in the data + failed attention checks would be. One that my supervisor was fond of was throwing in instruction screens, in the middle of runs of trials that required you to press a very specific button to continue. E.g., the experiment has you press 'A' and 'J' constantly, and to continue you have to press 'N'. Secretly though, A and J were also valid ways to continue with the experiment. The thought being that if you were hammering one of those buttons to quickly get through the experiment, you would also skip past the instruction screen very quickly with it.
blintz 7y ago
How do people feel about the research ethics of this? Many academic studies involve the use of MTurk participants, but it seems as though they are not being paid close to a living or even fair wage ($2 an hour is abysmal!). Should universities require that studies using MTurk pay some minimum pro-rated hourly wage? That is, if a Stanford HCI study (which seem to use MTurk often) wants workers to fill out a 15-minute survey, they would need to pay at least $8 an hour (so $2). I’ve been to thesis defenses at Stanford where researchers have explicitly stated that they chose to use MTurk because they can pay workers far less than locally recruited participants.
- wsc981 7y ago
  2 USD per hour can be decent depending the location. Where I live in Thailand people on average make 7 USD per day.
  - dsr_ 7y ago
    Do the people in Thailand who make $7/day generally have reliable access to the Internet for 8 hours a day, with a cost for network access and smartphone under $8/day? And a ready source of loans to cover the initial purchase? And can they read English?
    - nl 7y ago
      Yes. You will probably get better English skills in Philippines or India, but the economics are similar.
    - wsc981 7y ago
      Yeah, internet in our remote village is quite excellent. I've worked from home with on mobile 4G internet, was good enough for video calling. And now have proper broadband in home of course. English skills aren't great for most people, though if people really wanted to could learn for free using apps like Duolingo (there's English lessons for Thai people). A lot of people have decent smartphones. Good 4G internet for a month cost me around 600 Baht per month (~15 USD). Sure could be expensive if people make 300 Baht a day, but if they could work through Mechanical Turk, they could quickly earn back the cost, since they would likely earn more compared to working on a farm. Thai people like to lend money a lot. They're the most indebted of SEA when looking at loans compared to their incomes [0]. The use these loans to buy new cars, houses and phones as well. --- [0]: https://www.chiangmaicitylife.com/citylife-articles/drowning-in-debt/ https://www.chiangmaicitylife.com/citylife-articles/drowning...
dgivney 7y ago
A somewhat interesting article which strongly turns into a puff piece of MT. I started scrolling to the bottom to find out who was paying for this content and there it was "In his book, Side Hustle From Home: How To Make Money Online With Amazon Mechanical Turk".. I wonder just how does he manage a new baby, a side hustle that fills every minute with productive money-making and writing a book!.. The author really should be writing a book about his exceptional time management skills instead.
- mikehollinger 7y ago
  Yeah I noticed as well. It seems like what this is really saying is that the Mechanical Turk marketplace is an interesting study in economics. Buyers are offering to buy pieces of information for too high of a price and a select few people have found out how to nab those contracts. I halfway expected to see a secondary market emerge where you “complete” a top expensive task like “translate this text” by bundling it with five other translation tasks and posting it for three times the price.
- Traster 7y ago
  This seems like an almost constant feature of ways to make money online - the people who are making the money are almost uniformly making money by selling tutorials on how to make money online. Want to make money online playing poker? Pay $800 for Upswing Poker's masterclass! Want to make money online trading stocks? Check out the ultimate Penny Stock Playbook for only $50! Want to make money online drop shipping? For only $1600 you can buy the ProfitableOnlineStore course and have all your questions answered! So it makes you wonder doesn't it: If people are so good at making millions making money online, why are they offering to sell you the secret rather than actually making the millions themselves. The answer is: It's easier to sell gullible people tutorials than it is to actually make a living exploiting arbitrage opportunities in one of the most efficient markets ever known to man.
  - tpkj 7y ago
    Bingo. "Yo dawg, I heard you're thinking of a new way to make money...so I wrote a book about thinking of a new way to make money so you can think about a new way to make money while I make money from you thinking about my new way to think about making money."
RocketSyntax 7y ago
You could totally automate commonly requested tasks and then apply your algorithm to many other tasks to print $
mhh__ 7y ago
In my brief experience, bursts of actually quite a lot (My favourite being one where you got £1-£2 for "Does this sentence make sense" Yes/No) surrounded by almost nothing. I've seen some people on Reddit who are effectively gaming it well enough to actually make consistent money, but that's clearly the exception not the rule.
PaulHoule 7y ago
Every time I have looked at Turk lately the HITs I've seen have been awful. For instance there used to be a lot of HITs that involves transcribing printed receipts. I wouldn't mind doing this so much if the receipts weren't smudged and creased or otherwise illegible, but I think these people have an OCR that can handle the easy ones and you're left with a residuum of hard cases which sometimes can't be transcribed at all, or for which you'll probably make enough mistakes to get in trouble. I find that mentally fatiguing on top of the extra time.
anm89 7y ago
What would the argument be that some workers in the economy but not these people specifically deserve a minimum wage? I get that they are designated contractors but it seems pretty arbitrary. People would lose their minds if they found out Amazon was paying warehouse workers $2.50 an hour but couldn't they just arbitrarily assign prices per package and call them contractors as well?
sct202 7y ago
I tried mTurk after reading an article like this years ago, and it was the biggest waste of time ever. It was ungodly tedious for like pennies per task. I don't know how anyone could be doing so well at getting tasks that average out to $0.47 ($45k/95k tasks) like this guy is claiming; this is an absurd outlier.
neilv 7y ago
A few years ago, a friend of mine, in her 20s, worked full-time as a technician in a university-affiliated research lab, she managed to find a steal of a studio apartment (half the price of a bottom-end one-bedroom in the neighborhood), and she bought awful cheap bulk food... but money was still so tight that, in the evening, when she was too tired to do anything else, she did these online "gigs" that paid only $1-$2 an hour. Since she could write, she mostly did writing assignments (which I suspected were for SEO Web sites), in which they tell you a topic and how many words to write, you research and write the article, and you get paid a pittance. But $2 will buy steel-cut oats for the day. Her time should've been worth more than $1-$2/hour, and I don't like the idea of companies arguably exploiting desperate people this way. Though it's not just companies: university researchers sometimes use Mechanical Turk workers to process data, and as research subjects.
- alexashka 7y ago
  Every time I hear these stories, I don't know if they've done the math. Let's say I live in Toronto (an expensive city to live in) and make 10$/h after taxes working at Starbucks. 40h/week, 4.5 weeks/mo = 1800$ Bedroom in apartment = 700$ Metropass = 150$ Phone = 50$ Internet = 50$ Food = 350$ Total = 1300$ That's 500$ left for things like going out, clothes, etc. Nobody is exploiting your friends - it sounds like she complained to you, you took her words at face value and concluded that those other people are at fault.
  - ABCLAW 7y ago
    I live in Toronto, those numbers are hilarious. You cannot live in this city on a yearly wage of 21600. Not only are your numbers wrong individually, but they're also woefully under representative because your fake individual isn't buying toiletries, isn't paying for dental, doesn't purchase clothing, doesn't have any medical consumables to purchase, only eats a meal per day or so, etc. I've done the math. I've lived here. I've financed years of study in the city on loans and been on Ontario Works at times at the start of my career. Your model is poorly constructed.
    - 7y ago
simonebrunozzi 7y ago
Working as a "turker" on Mechanical Turk, or as a Tasker on Taskrabbit, or as a Lyft/Uber driver, all have one thing in common: the switching cost to/from that particular job is almost zero. In my view, this is why you can get paid less than what you deserve, but that option is still attractive to you. In other words, if you have 34 minutes between things, one of the few paying things you can do is MTurk, even if it doesn't pay well.
manubeauty 7y ago
Have you ever searched for a way to create a side income? If so, then you want to check out this free training how to make 6 figure income in 2019 https://pst.cr/CVC1J https://pst.cr/CVC1J https://pst.cr/CVC1J https://pst.cr/CVC1J
manubeauty 7y ago
Have you ever searched for a way to create a side income? If so, then you want to check out this free training how to make 6 figure income in 2019 https://pst.cr/CVC1J https://pst.cr/CVC1J https://pst.cr/CVC1J https://pst.cr/CVC1J

151 points by Steven-Clarke 7y ago

Ajs1 7y ago

Surprising stats. I’ve been aware of mech Turk a long time, but did not realize average pay per task is quite that low

senectus1 7y ago

wow, those number suck hard. I wonder what Nationality these "Turker" are from...

Thorrez 7y ago
> Despite an uptick in foreign users over the past few years, the majority of them (75%) are based in the US.
- bb2018 7y ago
  It is a complete waste of time in many ways but every once in a while I will log on and take a couple surveys while watching a rerun of something before bed. I used to do some survey design/analysis in academia and sometimes it is interesting to see what or how weird the surveys are. I would guess my hourly rate is about $3 and I used that very small amount to buy splurge on cheap useless stuff on Amazon.
- anovikov 7y ago
  This is because it's near impossible to register unless you are from the U.S., now, i heard. Was a reply to me when i suggessted bs 'programmers' who had no idea what they were doing, to go there instead of trying to hopelessly beg for projects on Upwork - 'i'd love to, but they locked everyone outside the U.S. out'.
- ForHackernews 7y ago
  America is becoming like Brazil: a third world nation with some wealthy enclaves.
  - meddlepal 7y ago
    Except it's not. Most Americans enjoy a very high standard of living and our poor are still much better off still than Brazilian poor.
  - mruts 7y ago
    As a person who actually lives in a 3rd world country and grew up in America, what you’re saying could not be more false. Even the poorest Americans live a life here that would be considered one of great luxury. We’re talking about living on a dirt floor in collapsing mud houses. We’re talking about babies dying from dirty water as a regular occurrence. No running water, intermittent electricity and definitely no sewage. Most people here haven’t eaten meat in months, occasionally years. If you actually think America is a 3rd world country you need to get out more.
    - ForHackernews 7y ago
      Americans in Flint, MI have poisonous water, and hookworm is on the rise from sewage contaminated water in Alabama.[0] In 2017, millions of Americans went without electricity or running water for months.[1] > If you actually think America is a 3rd world country you need to get out more. I said it's becoming a 3rd world country. There's still some legacy 20th century infrastructure that hasn't collapsed, yet. [0] https://www.theguardian.com/us-news/2017/sep/05/hookworm-lowndes-county-alabama-water-waste-treatment-poverty https://www.theguardian.com/us-news/2017/sep/05/hookworm-low... [1] https://www.theverge.com/2017/9/25/16362410/hurricane-maria-puerto-rico-power-outages-electrical-grid-destroyed https://www.theverge.com/2017/9/25/16362410/hurricane-maria-...
      - ataturk 7y ago
        Not sure why the downvote when it is abundantly clear that in every corner of the USA there is this general trend towards decay with very little offsetting it. All our infrastructure is falling apart. I blame the pension crisis. All available monies are going towards pension obligations. Not much is left over for anything else.

gitgud 7y ago

I want to know who is paying $0.01 USD to get someone to manually transrcibe a reciept to a table. It seems the Mechanical Turk ecconomy could be dwindling due to the accessiblity of AI these days.

cameronbrown 7y ago
Well, AI can't handle the edge cases yet. For some businesses it makes more economical sense to invest in Turk for transcription rather than training a ML model, which obviously requires a sizeable investment. Either way, you'll need a decent training set and Turk is great for that.
FluffyKitty 7y ago
Possibly Expensify? There was that news that came out a bit ago that their "SmartScan" actually fell back on Mechanical Turk [1] even though it was sold as a AI-type thing. [1] https://arstechnica.com/information-technology/2017/11/expensify-acknowledges-potential-privacy-problem-by-calling-it-a-feature/ https://arstechnica.com/information-technology/2017/11/expen...
- Thorrez 7y ago
  Yep >“Most people’s first mistake is they go on there and accept any task they see,” says Naab. “They’ll do these transcriptions — $0.01 to transcribe a whole receipt for Expensify. That’s a terrible ROI.”
  - jmnicolas 7y ago
    It depends, $0.01 might be worth something in a very poor country. But on the other hand if you have access to a computer a poor country, 1 cent might not be worth your time anyway.
    - mruts 7y ago
      You can use MTurk on phones I assume? Even very poor people often have smartphones and data where I am living (Tanzania).

dsissitka 7y ago

If you'd like to see what this looks like in practice see: https://www.youtube.com/watch?v=h9Qt6bVc5mY&list=PLuh2SVJZ17yC2-32cCfOCw_CIRFsVdNF5&index=2&t=0s https://www.youtube.com/watch?v=h9Qt6bVc5mY&list=PLuh2SVJZ17... The hourly rate will flash every few seconds.

bob_theslob646 7y ago
That video looks to show someone who built an automated script that spams random answers. Seems rather terrible to me to be fair.

probably_wrong 7y ago

I had to do some digging into crowdsourcing for my PhD thesis, and these numbers are well reported in the literature. Our group made the calculations to ensure that our task would pay at least minimum wage. As a result, we had to throttle our participants because they would hit our servers pretty hard. That's not to say that the experience was easy - we had to implement every possible sanity check, and even then we ended up throwing away half our data. For us it was still worth it, since twice the rate was still cheap and we knew not to trust the internet. But a less internet-savy researcher blindly trusting their data would have gotten some surprising results.

namanyayg 7y ago
What kind of sanity checks did you implement, or how did you know what data to throw?
- probably_wrong 7y ago
  We had an interactive task (hosted in our own server) and a questionnaire afterwards. Sanity checks off the top of my head: * Control words: our server gave them two (unique) control words (one for joining, one for winning), and we asked for them in the post-task questionnaire. Some of these words ended up being reused among several participants, even from participants that never even started the task. * IP checks: we had an experiment that you could "win" (and earn a bonus), but you only were allowed to play once. Some people restarted the task several times, so we only used the first attempt in a sequence (as reported by their IP and timestamp). * Data thrown away: we further removed data where we had more than one player per IP (to control for both multiple accounts per person and use of proxies), experiments that were way too fast, and experiments with unsupported browsers (which we explictly mentioned in the description). Regarding money, we were not allowed by the TOS to withold payment to anyone that filled the questionnaire, even if we knew they did it in bad faith. We therefore implemented the "winning" bonus, and also gave bonus to people who lost but really tried. I want to point out that a LARGE percentage of participants played honestly, and some of their data was thrown away only out of an abundance of caution. Once you keep the first bad apples out, they simply move to other, easily-exploitable tasks.

closetOperator 7y ago

Unless you have a clear strategy, MTurk work is a complete waste of time. True. I perused Mechanical Turk tasks some months ago, and was confronted by an array of plaintive, useless demands to perform what amounts to unproductive garbage picking for listless and disinterested college grad students running unimaginative projects without a clue as to whether the requested task is even possible. It would be a request like: Gather phone numbers from this queue of web pages. And they'd seem to have paid for a "database" of "leads" which was more than likely an excel spreadsheet of "hyperlinks" categorized according to a search query of keywords from the data provider. You get into the queue, and start pulling up each URL in series, and they're all these expired domains with parked registrar pages for GoDaddy and Tucows or whatever, Along with some Geocities, Angelfile, Tripod and AOL home pages thrown in. Quickly, you get a sense that some fool of data science masters program enrollee paid good money for a dusty, mouldering text file, didn't even look at it, and dropped it right the fuck into a template for a Mechanical Turk task. Now, there are three immediately obvious courses of action. One, abort and never again consider Mechanical Turk as a useful platform for operating an exchange of effort for rewards. Two, plead with the task owner by reporting feedback to them and ask them to stop for a moment and consider the flaws inherent to this framing of a human activity deemed worthy of compensation. Three, obey the letter of the law, and not the spirit, hold your nose, bellow the words "you asked for it!" and proceed to fill the task with the tech support numbers for all of the domain registrars, hoping that you'll not only get paid for grifting on the task owner, but also possibly inundate all these domain squatting registrars with robocalls trawling for psyche student surveys and questionnaires that will attempt to publish similarly terrible research papers designed with the intent to ostensibly "prove" a flawed hypothesis of human behavior with results that couldn't possibly be replicated because the hypothesis itself begs its own question. Valuing my time, I just logged out, and haven't looked back since. They need moderators to mechanical turk the quality of each task, because it benefits no one and wastes people's time, to even propose fruitless, unredeeming tasks. Unless things have changed since winter, from what I witnessed, there is perhaps zero review of tasks to assess whether a request fits the profile of anything even remotely possible or worth trying.

TrackerFF 7y ago

The problem with these gigs is that the tasks are extremely generalized. So while they attract more workers, it is always going to be a sellers market. The only way a worker can scale his income, is by increasing volume - and picking the right assignments. I've been doing side-hustles since I was a teen, and continue today, even though I have a well-paying professional job. My observations are that domain knowledge / specialized expertise always pays better. You're much better off learning some hobby, almost any hobby, and monetizing it. How do you monetize it? Do content writing. Start flipping on Ebay / Craigslist / etc. Perform / create if you can. etc. Now - I understand that these (MTurk) kinds of jobs are for desperate people, that want/need the money now - not 4 years down the road. Building expert knowledge will take time, and you will more likely than not get burnt a few times in the start - but the long-term payoff is much better. Last month, I made $800 in selling two musical instruments. Time invested in it all was around 2-3 hours, mostly on picking 'em up, and shipping them. Maybe not the answer for everyone, but it's extremely hard for me to imagine any other way now. I currently have three different hobbies which I enjoy very much (and have for years), and I'm lucky enough to have so much knowledge in that I also make a decent side-income off. Best of all? It doesn't feel like you're working - though you need to treat it like work, if you're gonna make money.

arvinsim 7y ago
Interesting. May I ask what hobbies are these?

s_Hogg 7y ago

Related to this, Stanford created something so you can use Mechanical Turk without being overly exploitative (ensures you pay at least US$15/hr): https://fairwork.stanford.edu/ https://fairwork.stanford.edu/

s_Hogg 7y ago
I particularly like this part of the FAQ: I have an even cleverer point suggesting that I might not be able to trust workers' self reports. Might my clever rebuttal be right? Yes. But you are using this tool because you want to help ensure fair treatment to workers, right? Fair treatment starts with trusting workers. They will make good faith estimates.
mruts 7y ago
I like the idea of having the market decide what is fair, instead of some Stanford website.

Mediterraneo10 7y ago

I signed up for Mechanical Turk soon after its creation. I transcribed podcasts and made about $10/hour, which was not bad money for a student, especially considering that these podcasts were on interesting subjects and it did not feel like hard work. After a few months I moved on to other things, and when I looked at MTurk again, I saw that it had quickly become a race to the bottom. No freelancing site can withstand those capitalist pressures for long.

ptero 7y ago

The person described is a "senior business analyst", reportedly making good money, but decides to work on MTurk, doing menial tasks, to help offset extra costs (diapers, etc.) of a baby that was born. This sounds insane to me. Almost anything else sounds better: reduce expenses, figure out if he can get more responsibilities and a raise, even get a low % credit to pay for those diapers for a year, etc. . Is this just me? This is an honest question.

londons_explore 7y ago
Some MTurk tasks are amenable to sub-contracting out or writing bots to mostly or fully automate. If he does that, I could see it being competitive on an hourly basis to a regular job.
- dalbasal 7y ago
  How open are Amazon to this? Ooh, the idea of mTurk is "human powered." OTOH, a market for tasks that can be do e via a combination of automations and humans could be powerful.
  - jwieczorek 7y ago
    Such a combination is what Unbabel does in translation. ML-driven automated translation followed by crowd-sourced human review and corrections. It's a neat idea and could be the future of translation (as opposed to full automation).
  - dahfizz 7y ago
    I would assume that Amazon already does this. If a task is trivial enough for an AI, they would do it themselves and get the posters money before the job gets forwarded to the Turks.
    - dalbasal 7y ago
      Sure, Amazon have all sorts of ml APIs. But a task platform where anyone can complete the tasks using ai, humans or a combination that f both could be substantial impressive. Say you need to look through 20k images and determine if they contain hipsters. There may not be an off-the-shelf hipster detector. Maybe you could make one, or use humans or a combination by training an NN.. etc.

Havoc 7y ago

Only way I can see this working is if you're in a CoL area that is low end 3rd world. Else pretty much anything else will be more profitable

rjf72 7y ago
I think many people do this not out of need but as a time waster that pays a bit. As the article mentions, 75% of 'Turkers' are American. The article's subject says, "I’d literally be sitting around bored half the day without it. Making money beats doing nothing." Of course he wouldn't be doing literally nothing. Instead he'd be just mindlessly wasting time browsing Facebook, checking out the latest clickbait, posting on forums, and engaging in the sort of activities that studies are increasingly showing are destroying our collective mental health and social cohesion. Instead, he's using that time to make an extra thousand a month.
- Havoc 7y ago
  >I think many people do this not out of need but as a time waster that pays a bit. As the article mentions, 75% of 'Turkers' are American. Been tempted myself. But then I crunch the numbers vs my actual job and go clean my apartment instead. That plus a very real fear of RSI from all the mouse clicking. I mean I already click at work all day then on hn/reddit in afternoons...

sschueller 7y ago

I wonder how many task on Mechanical Turk can actually be automated. By that I don't mean building some fancy AI but tasks that can easily be automated and have just been posted on Mechanical Turk.

kuu 7y ago
I guess automating is not always the cheapest option, and for "once in a life" experiment/test/dataset it can be sometimes cheaper to hire people than develop an algorithm. I guess XKCD said it before: https://xkcd.com/974/ https://xkcd.com/974/
- taneq 7y ago
  XKCD gave me a 503 "Backend unavailable" and I thought it was some advanced meta-joke for a second.
H8crilA 7y ago
Much of what's happening in such markets is collecting data to build ML systems.

Yuval_Halevi 7y ago

MTurk is the closest there is for a modern slave.

shawabawa3 7y ago
Slaves are the closest thing to a modern slave
systemtest 7y ago
The workers in Qatar beg to differ.

lordnacho 7y ago

Is there anything stopping people from just randomly filling in the surveys? It seems like the kind of thing where someone would have made a bot. How many people who are paying a couple of bucks for something like that are going to complain at you?

jonchang 7y ago
Survey instruments generally have a few ways to check for this, usually via attention checks which require you to answer in a specific way as stated in the question text (defeating your bot). Similarly a survey could be designed to ask the same or similar question multiple ways, and a survey response that lacks the expected correlation can also be rejected.

Phemist 7y ago

Fun anecdote about some of my own dabblings in online psych experiments and MTurk. 7 or 8 years ago, my then supervisor and I experimented with running psychological (behavioural) experiments on MTurk. We had created a method that would run in browsers in native javascript and were looking to validate it. Coming from a testing-1st-year-psych-students-in-the-basement-of-the-faculty-building kind of thing, we naively took what the going wage was for that (something like 8 euros/hour), and put an experiment online that would take around 20 min to complete, and paid 2 euros. My supervisor used his own credit card and set the spend limit to around 1000 euros; thinking that we'd never hit it. Boy how wrong were we. Apparently this 6 euro/hour wage was _much_ _much_ higher than the going rate, and we hit the spend limit in around 2 hours. Even though we had to throw out around 70% of the completions, we ended up with usable data from around 150 participants. We went in expecting to run the experiment for a week or 2, and get maybe 50 participants, but came out with 500 in a span of 2 hours. Safe to say, we celebrated a job well done that night over one or two drinks. People were commenting on how nice of a change of pace it was compared to the then-usual MTurk tasks and that they would have done it for free. Some even left their e-mail addresses should we run another online experiment. I've since been out of the field of online behavioural experimentation, but it seems to have taken off quite a bit.

bob_theslob646 7y ago
Holy smokes that is an amazing story. Do you have any more? I am always fascinated to hear how researchers design studies to prevent people from just spamming in any old answer.
- Phemist 7y ago
  I'm not sure what the state of the art is nowadays. Back in the day. The method at the time was so new, we just assumed that there would be no bots that would be able to complete the experiment successfully, or that they would stand out like a sore thumb upon analysing the data. Modern frameworks will eventually have to deal with this I'm sure. The structured way in which experiments are defined also make it easy to develop tools to automate this. Unfortunately I'm not into the field enough anymore to know how people are dealing with it. The biggest problem we had was with people with high latency connections. The effects we were looking for were measured in 10s of milliseconds. In order to tease out these effects, we had to be very particular about the timing of when certain stimuli were presented to the participant. High latency page reloads (which were unavoidable in the system we built our method on) would mess with this high-precision presentation requirement. We measured the latency, but did not pre-emptively exclude anyone based on their latency, hence the high % of people with unusable data in our initial validation experiment. For subsequent experiments I built a "loader" screen that would pretend to be loading the experiment. What it in fact was doing was refreshing the webpage several times (ofcourse, while progressing a progress bar) to measure the latency of the connection of the participant. High average latency + high variance latency connections were excluded. The tresholds were based on what we found in the early validation study. Surprisingly, after throwing out the high-latency data, there were no other exclusions necessary. It seemed that for our validation study, all participants were very attentive during the experiment. In "in-person" cases, researchers would add attention checks to their experiments with the logic that failing these attention checks by itself is no indication of "spamming", but seeing weird quirks in the data + failed attention checks would be. One that my supervisor was fond of was throwing in instruction screens, in the middle of runs of trials that required you to press a very specific button to continue. E.g., the experiment has you press 'A' and 'J' constantly, and to continue you have to press 'N'. Secretly though, A and J were also valid ways to continue with the experiment. The thought being that if you were hammering one of those buttons to quickly get through the experiment, you would also skip past the instruction screen very quickly with it.

blintz 7y ago

How do people feel about the research ethics of this? Many academic studies involve the use of MTurk participants, but it seems as though they are not being paid close to a living or even fair wage ($2 an hour is abysmal!). Should universities require that studies using MTurk pay some minimum pro-rated hourly wage? That is, if a Stanford HCI study (which seem to use MTurk often) wants workers to fill out a 15-minute survey, they would need to pay at least $8 an hour (so $2). I’ve been to thesis defenses at Stanford where researchers have explicitly stated that they chose to use MTurk because they can pay workers far less than locally recruited participants.

wsc981 7y ago
2 USD per hour can be decent depending the location. Where I live in Thailand people on average make 7 USD per day.
- dsr_ 7y ago
  Do the people in Thailand who make $7/day generally have reliable access to the Internet for 8 hours a day, with a cost for network access and smartphone under $8/day? And a ready source of loans to cover the initial purchase? And can they read English?
  - nl 7y ago
    Yes. You will probably get better English skills in Philippines or India, but the economics are similar.
  - wsc981 7y ago
    Yeah, internet in our remote village is quite excellent. I've worked from home with on mobile 4G internet, was good enough for video calling. And now have proper broadband in home of course. English skills aren't great for most people, though if people really wanted to could learn for free using apps like Duolingo (there's English lessons for Thai people). A lot of people have decent smartphones. Good 4G internet for a month cost me around 600 Baht per month (~15 USD). Sure could be expensive if people make 300 Baht a day, but if they could work through Mechanical Turk, they could quickly earn back the cost, since they would likely earn more compared to working on a farm. Thai people like to lend money a lot. They're the most indebted of SEA when looking at loans compared to their incomes [0]. The use these loans to buy new cars, houses and phones as well. --- [0]: https://www.chiangmaicitylife.com/citylife-articles/drowning-in-debt/ https://www.chiangmaicitylife.com/citylife-articles/drowning...

dgivney 7y ago

A somewhat interesting article which strongly turns into a puff piece of MT. I started scrolling to the bottom to find out who was paying for this content and there it was "In his book, Side Hustle From Home: How To Make Money Online With Amazon Mechanical Turk".. I wonder just how does he manage a new baby, a side hustle that fills every minute with productive money-making and writing a book!.. The author really should be writing a book about his exceptional time management skills instead.

mikehollinger 7y ago
Yeah I noticed as well. It seems like what this is really saying is that the Mechanical Turk marketplace is an interesting study in economics. Buyers are offering to buy pieces of information for too high of a price and a select few people have found out how to nab those contracts. I halfway expected to see a secondary market emerge where you “complete” a top expensive task like “translate this text” by bundling it with five other translation tasks and posting it for three times the price.
Traster 7y ago
This seems like an almost constant feature of ways to make money online - the people who are making the money are almost uniformly making money by selling tutorials on how to make money online. Want to make money online playing poker? Pay $800 for Upswing Poker's masterclass! Want to make money online trading stocks? Check out the ultimate Penny Stock Playbook for only $50! Want to make money online drop shipping? For only $1600 you can buy the ProfitableOnlineStore course and have all your questions answered! So it makes you wonder doesn't it: If people are so good at making millions making money online, why are they offering to sell you the secret rather than actually making the millions themselves. The answer is: It's easier to sell gullible people tutorials than it is to actually make a living exploiting arbitrage opportunities in one of the most efficient markets ever known to man.
- tpkj 7y ago
  Bingo. "Yo dawg, I heard you're thinking of a new way to make money...so I wrote a book about thinking of a new way to make money so you can think about a new way to make money while I make money from you thinking about my new way to think about making money."

RocketSyntax 7y ago

You could totally automate commonly requested tasks and then apply your algorithm to many other tasks to print $

mhh__ 7y ago

In my brief experience, bursts of actually quite a lot (My favourite being one where you got £1-£2 for "Does this sentence make sense" Yes/No) surrounded by almost nothing. I've seen some people on Reddit who are effectively gaming it well enough to actually make consistent money, but that's clearly the exception not the rule.

PaulHoule 7y ago

Every time I have looked at Turk lately the HITs I've seen have been awful. For instance there used to be a lot of HITs that involves transcribing printed receipts. I wouldn't mind doing this so much if the receipts weren't smudged and creased or otherwise illegible, but I think these people have an OCR that can handle the easy ones and you're left with a residuum of hard cases which sometimes can't be transcribed at all, or for which you'll probably make enough mistakes to get in trouble. I find that mentally fatiguing on top of the extra time.

anm89 7y ago

What would the argument be that some workers in the economy but not these people specifically deserve a minimum wage? I get that they are designated contractors but it seems pretty arbitrary. People would lose their minds if they found out Amazon was paying warehouse workers $2.50 an hour but couldn't they just arbitrarily assign prices per package and call them contractors as well?

sct202 7y ago

I tried mTurk after reading an article like this years ago, and it was the biggest waste of time ever. It was ungodly tedious for like pennies per task. I don't know how anyone could be doing so well at getting tasks that average out to $0.47 ($45k/95k tasks) like this guy is claiming; this is an absurd outlier.

neilv 7y ago

A few years ago, a friend of mine, in her 20s, worked full-time as a technician in a university-affiliated research lab, she managed to find a steal of a studio apartment (half the price of a bottom-end one-bedroom in the neighborhood), and she bought awful cheap bulk food... but money was still so tight that, in the evening, when she was too tired to do anything else, she did these online "gigs" that paid only $1-$2 an hour. Since she could write, she mostly did writing assignments (which I suspected were for SEO Web sites), in which they tell you a topic and how many words to write, you research and write the article, and you get paid a pittance. But $2 will buy steel-cut oats for the day. Her time should've been worth more than $1-$2/hour, and I don't like the idea of companies arguably exploiting desperate people this way. Though it's not just companies: university researchers sometimes use Mechanical Turk workers to process data, and as research subjects.

alexashka 7y ago
Every time I hear these stories, I don't know if they've done the math. Let's say I live in Toronto (an expensive city to live in) and make 10$/h after taxes working at Starbucks. 40h/week, 4.5 weeks/mo = 1800$ Bedroom in apartment = 700$ Metropass = 150$ Phone = 50$ Internet = 50$ Food = 350$ Total = 1300$ That's 500$ left for things like going out, clothes, etc. Nobody is exploiting your friends - it sounds like she complained to you, you took her words at face value and concluded that those other people are at fault.
- ABCLAW 7y ago
  I live in Toronto, those numbers are hilarious. You cannot live in this city on a yearly wage of 21600. Not only are your numbers wrong individually, but they're also woefully under representative because your fake individual isn't buying toiletries, isn't paying for dental, doesn't purchase clothing, doesn't have any medical consumables to purchase, only eats a meal per day or so, etc. I've done the math. I've lived here. I've financed years of study in the city on loans and been on Ontario Works at times at the start of my career. Your model is poorly constructed.
  - 7y ago

simonebrunozzi 7y ago

Working as a "turker" on Mechanical Turk, or as a Tasker on Taskrabbit, or as a Lyft/Uber driver, all have one thing in common: the switching cost to/from that particular job is almost zero. In my view, this is why you can get paid less than what you deserve, but that option is still attractive to you. In other words, if you have 34 minutes between things, one of the few paying things you can do is MTurk, even if it doesn't pay well.

manubeauty 7y ago

Have you ever searched for a way to create a side income? If so, then you want to check out this free training how to make 6 figure income in 2019 https://pst.cr/CVC1J https://pst.cr/CVC1J https://pst.cr/CVC1J https://pst.cr/CVC1J

manubeauty 7y ago