4 ms·
Former longtime Leapforce/RaterLabs preferred/senior rater here. There was always some exploitation going on, but it got way worse under Ruth Porat's tenure. S
by rater266 4y ago
Former longtime Leapforce/RaterLabs preferred/senior rater here. There was always some exploitation going on, but it got way worse under Ruth Porat's tenure.
Standard Leapforce US English raters were paid $13.50/hr. If you were lucky enough to get promoted to "Preferred Agent" (PA), you made $17.40/hr. The criteria for PA promotion was basically to be in the top 5% or so on their quality metrics, plus various subjective quality requirements. Promotion rounds were irregular and random.
Leapforce's US English pay rates never changed over time. There were no raises, other than the aforementioned rare and seemingly luck-based promotion opportunity. I was one of the fortunate ones.
Leapforce raters were misclassified as 1099 contractors. There were no benefits. Training time was unpaid. Originally, raters could work as many hours as they wanted, subject to work availability. Around the end of 2013, they added a 40 hour/week limit because they were afraid a court or regulator might rule the "contractors" were employees and force them to pay retroactive overtime.
If a state dared to go after Leapforce or Google for misclassification, they would leave that state and fire the workers there. No severance, of course. There were numerous states they did not hire in, including their home state of California.
Task availability was usually very good for US English raters, but there were definitely periods (often around holidays) where that was not the case. There was never any communication about projected task availability. We certainly would have appreciated a quick heads-up of "hey, many of our engineers will be on vacation between mid-December and mid-January, so expect task availability to be light. Tasks will be added daily at 8AM PST, but probably won't last more than two or three hours."
I understand other locales were not as fortunate as we were for task availability.
Work availability for a particular rater was not necessarily as consistent. If you scored poorly on a monthly quality review or random quality audit, you could be abruptly suspended, fired, or limited to an hour or two a day until the next monthly review. There would be no prior warning of this. There were automated tripwires (which we called the "bot") that would flag and suspend you if your ratings were considered outliers, or if your task comments were too similar (even when there was a batch of tasks that were very similar), or if you worked too fast, or if you worked too slowly, or a million other reasons. If you were flagged by the bot, you had no work until the Leapforce quality people got around to reviewing you. There was no compensation for missed hours even if it turned out you hadn't done anything wrong.
Even more insultingly, Google loved to use quality reviews to informally implement guideline changes without actually changing the guidelines. If you weren't good at guessing their future direction, you could easily have your hours cut for the next month.
Around 2017, Google realized they (not just Leapforce or Lionbridge) could be sued for this misclassification nonsense, so they forced Leapforce and Lionbridge to convert their misclassified contractors to employees. Leapforce morphed into RaterLabs in what appeared to be some kind of liability-dodging scam. As employees, they cut us to 26 hours/week so they wouldn't have to pay for health insurance. There were still no benefits other than the minimum required by state law, if you were lucky enough to be in a state that required any. Nominal pay rates were cut in what seemed like a random and inconsistent (not even location based) way; depending on the cut, you got a few cents raise after accounting for the self-employment tax you no longer had to pay (which RaterLabs was quick to point out), or a pay cut.
There was a rater revolt around that time, but nothing seemed to come of it.
Appen bought Leapforce and RaterLabs later in 2017. Leapforce founders Daren Jackson and Caine Lai made off with a pile of cash. His employees weren't so fortunate. Now people are making $10/hr. for a job that paid $13.50/hr. in 2009, which would be the equivalent of $18.38/hr. in today's dollars.
- vgeek 4y agoThank you for all of this information! I've roughly followed it as a marketer (YMYL ranking factors from the guidelines type stuff) and as an observer-- it is interesting that the shifts lined up with the Porat era. The two people I knew who did Leapforce also did human powered answers for ChaCha(?) where they were paid per answer versus an hourly rate, and then migrated to Leapforce at a lower date. They were both in US and I would remember them complaining on AIM over task availability-- sometimes they'd have work, others they wouldn't. They would take breaks from playing Counterstrike to see if new tasks were available and disappear if there was work available. With Leapforce, were there different rigid time allotments per task? E.g. 30 seconds to classify an image vs 2 minutes for ranking a webpage to be shown as a search result? If you were flagged as a low performer, was there any recourse or remediation, or just getting disabled and hoping for a review?
- rater266 4y agoThere were different time allotments per task, yes. In the Raterhub era, each task had an "Average Estimated Time" (AET) set by Google. A typical 5x5 side-by-side search result comparison task would have been 9 minutes, for example. I may be forgetting something but I don't think we ever saw the individual URL rating tasks (which I think I recall were 1.5 mins? maybe it was 2. It's been so long) after the Raterhub switch. AETs were usually reasonable until the Ruth Porat era. After Porat, they seemed to be getting pretty aggressive about reducing them as far as possible to the point of absurdity. We could release tasks for insufficient AET. I suspect they added automated tooling in the Porat era to "optimize" AET in their favor. With only a few exceptions I can remember, AETs only went down, not up. If you were flagged by the bot, there was nothing you could do except wait for Leapforce to review you. That could take anywhere from a few hours to several days depending on how busy they were. I wouldn't say there was any meaningful appeal route; I mean, you could email their quality team, and maybe you could plead for leniency, but I don't know anyone had any real success with that. (I don't have personal experience with that; as I said, I was a PA/senior, which was a quality-based designation, so it follows that I tended to be pretty good at the job! But I always felt bad for other raters in chat who had to deal with that, seeing their livelihoods put at risk in such an arbitrary and robotic manner. If someone isn't a good fit for the job, then yes, it's not going to work out, but I do believe they should be given feedback and guidance on how to improve and a fair chance to do so.)