12 ms·

'Rater' is paid $10 hourly to teach Google's algorithm

106 points by exz 4y ago

nyhwtt 4y ago
Better than working at Wendys!
- Crosseye_Jack 4y ago
  Depends on how you look at it, The wage fits in around the rough ave pay for a Wendys "crew memeber", however you are responsable for your own costs while working (computer, internet bill, electricity bill, heating) but on the flip side, its part-time, no comute, no long hours standing, no dress code.
Gravyness 4y ago
Definitely explains the quality of search as of late.
- TheGoddessInari 4y ago
  They (and everyone else) have been doing this forever, if not obvious
- amelius 4y ago
  Those raters must really love Pinterest ...
  - kappuchino 4y ago
    Acually ... no, but pinterest provides results that are relevant and satisfying to the general search (and the SQR - the Search Quality Rating, see link in discussion). Hence Pinterest stays in good standing even if most people like me (you, too?) hate it for its login/content walling tactics.
    - Avamander 4y ago
      > but pinterest provides results that are relevant and satisfying to the general search More like it matches some keywords like run-of-the-mill WordPress malware generates. I've seen plenty of both. Not to mention the similarity to the masses of biz/buzz/xyz spam websites.
Proven 4y ago
How much should they be paid? It's not like you need a PhD to confirm if a search result sucks. Results are supposed to be good enough for the average user, so you need common sense and some on-job training to do a good job. And having 10 common-sense guys churning through user feedback @$11/h is better than 2 geniuses @$55/h. As one comment says, it probably beats fliping burgers.
- closedloop129 4y ago
  Where is the limit? Why not 20 guys in a low-cost country @$5.50/h? They cannot rate all results anyway, so why not get the best possible evaluations? Shouldn't quality data beat quantity if you extrapolate from the evaluated samples to the entire index? Spammers target the lower part of the bell curve. If you let that part evaluate the quality of the results, do you think that they will recognize spam?
- echen 4y ago
  It can be suprisingly difficult. A lot of the difficulty is in understanding the user intent behind a search query (since you're usually rating other people's queries, not your own). For example, here are some queries in my history - would the average person understand what I'm looking for and what a good search result would be? "neovim worktrees" "django models through" "pandas econdb"
  - kappuchino 4y ago
    Good examples. Raters have - afaik- to assume multiple possible intentions on two word searches (and general searches on one). This results in better ratings for results (top 10 generally) that partially match different search intentions. so for "pandas databases" it would match the panda python libraries, but also databases literally of pandas ... and maybe a company that is called pandas and has databases. And this is the reason why we can never have "pure" results (in most cases) for a specific search, because the algorithm and the search rating is aiming for result-topic-diversity most of the time (if not all of the time).
- kappuchino 4y ago
rcktmrtn 4y ago
On a related note, am I right in understand that the "Google Experts" [1] do it for free? [1] https://developers.google.com/community/experts/ https://developers.google.com/community/experts/
- ocdtrekkie 4y ago
  Yes, Google has tricked a massive army of people into doing free work for one of the most profitable companies in the world. Things like "Google Local Guides" give people a gold star (and maybe a pair of branded socks) for doing unpaid labor. It's kinda hilarious, but also sad. I imagine there are Google executives in a room somewhere laughing their full heads off every time they think about how there are people who do free work for Google who think they are making the world a better place or something.
  - rurban 4y ago
    When they tried to lure me into a "Local Guide" I was offered a free flight, hotel and food at Menlo Park. Once, but still good enough for no work at all.
  - vba616 4y ago
    >laughing their full heads off every time they think about how there are people who do free work for Google who think they are making the world a better place or something. So when you go to a store that Google Maps says is open, and find it's closed, you refuse to correct the hours and go "MUHAHAHAHA...I refuse to help Google, take that, corporate scum!"
    - ocdtrekkie 4y ago
      Why would I bother checking Google Maps? But basically, yeah. If Google wants my expertise and data collection ability, they can pay me for it. As it stands now they just cold call businesses and ask them with an AI thing.
      - nine_k 4y ago
vgeek 4y ago
This is 100% a race to the bottom. Years ago the Leapforce (now Appen as mentioned in the article) and Lionsbridge "contractors" could average over $18/hr. They would only work on Google tasks and adhere to Google's policies. The work seemed inconsistent (people would have 5 hours of work some days, none the next) and about what you would expect for AI training. No worker protections, benefits, etc.-- all while doing work, albeit opaquely, for one of the richest companies of all time. Why can't companies at least try to take care of their workers?
- daniel-cussen 4y ago
  > all while doing work, albeit opaquely, for one of the richest companies of all time. It doesn't work that way. Contrariwise.
- Tenoke 4y ago
  I worked for Leapforce from the UK in 2015-16 and I definitely didn't reach 18$/hr and the availability of work wasnt too great. Probably not bad if you have another gig you can do in the mean time though.
- rater266 4y ago
  Former longtime Leapforce/RaterLabs preferred/senior rater here. There was always some exploitation going on, but it got way worse under Ruth Porat's tenure. Standard Leapforce US English raters were paid $13.50/hr. If you were lucky enough to get promoted to "Preferred Agent" (PA), you made $17.40/hr. The criteria for PA promotion was basically to be in the top 5% or so on their quality metrics, plus various subjective quality requirements. Promotion rounds were irregular and random. Leapforce's US English pay rates never changed over time. There were no raises, other than the aforementioned rare and seemingly luck-based promotion opportunity. I was one of the fortunate ones. Leapforce raters were misclassified as 1099 contractors. There were no benefits. Training time was unpaid. Originally, raters could work as many hours as they wanted, subject to work availability. Around the end of 2013, they added a 40 hour/week limit because they were afraid a court or regulator might rule the "contractors" were employees and force them to pay retroactive overtime. If a state dared to go after Leapforce or Google for misclassification, they would leave that state and fire the workers there. No severance, of course. There were numerous states they did not hire in, including their home state of California. Task availability was usually very good for US English raters, but there were definitely periods (often around holidays) where that was not the case. There was never any communication about projected task availability. We certainly would have appreciated a quick heads-up of "hey, many of our engineers will be on vacation between mid-December and mid-January, so expect task availability to be light. Tasks will be added daily at 8AM PST, but probably won't last more than two or three hours." I understand other locales were not as fortunate as we were for task availability. Work availability for a particular rater was not necessarily as consistent. If you scored poorly on a monthly quality review or random quality audit, you could be abruptly suspended, fired, or limited to an hour or two a day until the next monthly review. There would be no prior warning of this. There were automated tripwires (which we called the "bot") that would flag and suspend you if your ratings were considered outliers, or if your task comments were too similar (even when there was a batch of tasks that were very similar), or if you worked too fast, or if you worked too slowly, or a million other reasons. If you were flagged by the bot, you had no work until the Leapforce quality people got around to reviewing you. There was no compensation for missed hours even if it turned out you hadn't done anything wrong. Even more insultingly, Google loved to use quality reviews to informally implement guideline changes without actually changing the guidelines. If you weren't good at guessing their future direction, you could easily have your hours cut for the next month. Around 2017, Google realized they (not just Leapforce or Lionbridge) could be sued for this misclassification nonsense, so they forced Leapforce and Lionbridge to convert their misclassified contractors to employees. Leapforce morphed into RaterLabs in what appeared to be some kind of liability-dodging scam. As employees, they cut us to 26 hours/week so they wouldn't have to pay for health insurance. There were still no benefits other than the minimum required by state law, if you were lucky enough to be in a state that required any. Nominal pay rates were cut in what seemed like a random and inconsistent (not even location based) way; depending on the cut, you got a few cents raise after accounting for the self-employment tax you no longer had to pay (which RaterLabs was quick to point out), or a pay cut. There was a rater revolt around that time, but nothing seemed to come of it. Appen bought Leapforce and RaterLabs later in 2017. Leapforce founders Daren Jackson and Caine Lai made off with a pile of cash. His employees weren't so fortunate. Now people are making $10/hr. for a job that paid $13.50/hr. in 2009, which would be the equivalent of $18.38/hr. in today's dollars.
advisedwang 4y ago
I thought the management theory was to in-source your core-competencies and out-source everything else. Surely quality search results is one of Google's core competencies!
- ocdtrekkie 4y ago
  Google's core competency is ads. Any time you click a search result, it's a failure: Google wanted you to click the ad above it.
  - kappuchino 4y ago
    Totaly oversimplification. If google wanted you only to click on ads, it would be a type of yellow pages. What it is (as it was with newspapers and magazines a time ago): A place where you can place paid ads based on context. If you remove the context, it becomes worthless. So while everyone is complaining about search quality, its an uphill battle for any search company. Like PR tries to sneak in Ad-Material into reguluar journalism, companies try to manipulate search results with SEO ranging form keyword-stuffing to bait and switch search pages. (which are one reason why google needs humans and not ai to verify).
- kurupt213 4y ago
  If you use Google you know the search results have been getting worse and worse
akomtu 4y ago
There is your "AI".
Justin_K 4y ago
Another prime example of how ML and AI aren't mature. With all the data google collects, they have to rely on minimum wagers to "find" the best results.
- xnx 4y ago
  In the sense that ML and AI aren't yet AGI, yes, but this doesn't mean that Google's ML and AI aren't very useful and benefit from having another source of training and validation data.
- dfdz 4y ago
  This criticism seems unfair. There is nothing wrong with semi-supervised learning.
- kappuchino 4y ago
  Actually, ML/AI would not know by the rules of Google for "Quality" how to rate content, since content is ever changing and diverse as it gets. If you spend some time reading the guide (https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf https://static.googleusercontent.com/media/guidelines.raterh...) you will realize that its not about the "best" results but on classificaion / verificationin the YMYL (Your Money, Your Life) areas that should not be left to machines in the first place. Money is mainly low because there are so many people that can do the job and so many that are needed to redundantly rate the content.
civilized 4y ago
Dumb question - why can't I just upvote and downvote pages, and Google enhances its ranking for me based on that? Maybe that's why the search "X reddit" is so popular - it's got a bit of that social validation.
- arthurcolle 4y ago
  this used to exist but they got rid of it like 10 yrs ago IIRC
- lordnacho 4y ago
  Who do you reckon would cast the most votes? Correct, SEO bots.
  - jancsika 4y ago
    If Google can't use it's vast array of user data to tell the difference between SEO bots and non-bot users, what's even the point of it collecting all that fucking data in the first place?
    - deleted 4y ago
      [deleted]
    - bryanrasmussen 4y ago
      Good point, but I have my reasons for suspecting they can't tell the difference https://news.ycombinator.com/item?id=31296953 https://news.ycombinator.com/item?id=31296953
      - scoopertrooper 4y ago
        Maybe that's part of how they train it?
    - rednalexa 4y ago
      I would assume that if Google is paying 10$/hour for humans to rank content, someone else is paying 11$/hour to fool it with a human generating content, clicks, etc. Additionally, people writing bots learn just like Google learns, and start generating what appears to be legitimate interactions. Same algorithms that fit to critique between 'bot' and 'non-bot' class can be used to create bots that fall into the 'non-bot' category. It's a losing battle - but what we get is better than we would get if they collected less data.
hobo_mark 4y ago
Ha, so they still do it, that is how I paid rent for a few months back in 2012.
ggm 4y ago
Surprisingly badly paid. I wonder how strongly quality would track pay? Probably they act as a CAPTCHA mutually cross checking and so QA is by volume, and not coincidentally demands they work in isolation. Sad story.
- echen 4y ago
  Quality control is often indeed by volume. When I was at Google / Facebook, search evaluations would typically have 5 raters each. One of the difficulties, though, is that search evaluation can be quite subjective. And honestly, most of the search engine raters we used were low quality and did a bad job (so the aggregate answer was often nonsense).
  - kappuchino 4y ago
    This. In the early days of my SQR rating time, the rating was not sliced into time per rating. And you had to write a short blurb about your reasons, often using the acronyms so it read like "NR, LQ / SPAM". Later you would see the other SQR notes and would wonder if both of you had seen the same page.
  - rater266 4y ago
demadog 4y ago
When those reports come out that “The average Google employee makes $220k a year” or whatever, it’s totally misleading since it doesn’t account for all the outsourced labor at $10 an hour that is critical to tuning their algorithms.
- Nexxxeh 4y ago
  I did voice training tasks for Google through a 3rd party I can't remember the name of. I was recruited via /r/beermoney or something. I enjoyed it, and it was easy work that fit around my schedule. It also gave a feeling of improving the lives of users, which was cool.
  - daniel-cussen 4y ago
    It for sure can be good, win-win.
- kappuchino 4y ago
  Its better paid than Amazon Mechanical Turk afaik (not saying it should not be paid better). The tuning is done literally by thousands world wide and since every rating task is done by multiple quality raters (afaik side by side is done by at least 5), it seems to be a cost vs. scale factor.
  - demadog 4y ago
    Amazon Turk is more of a platform for small business to hire other workers for micro-tasks. Raterlabs is used only by Google to improve their core product.
- hn_throwaway_99 4y ago
  I don't think that's misleading at all. Why would I assume that non-Google employee compensation would go into the calculation for what the average Google employee makes? As someone else pointed out, this pays substantially more than AWS Mechanical Turk. To be honest, it's difficult for me to imagine an easier job than this one. I mean, most people who would do this job could easily now make more at McDonald's or Walmart or whatever. But I think jobs at McDonald's or Walmart are considerably harder, so I don't see why this job should pay the same.
kappuchino 4y ago
Ex Google Search quality Rater here, based in Germany. From 2004 to 2006 I worked on SQR and "special projects", with a much better pay (maybe because of german circumstances). I've been tracking the SQR-guide since then (public version here https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf https://static.googleusercontent.com/media/guidelines.raterh...), so I have still some current understanding. If you have questions, I'd be happy to try to answer them since my NDA has expired ;-).
- dmarchand90 4y ago
  I don't have any specific questions but would be really interested in hearing more!
  - kappuchino 4y ago
    Its an interesting boring job in its core. At least when I did it. You agree to see theoretically everything that people search for and try to rate it according to the linked manual. In my time - really long ago - britney spears was one of the most common searches (I remember it probably wrong, but the test I had to do was half rating questions on spears and tom hanks). But I also got to rate "donkey s*x" and things like "divorce help" which makes you think about the impact search results can have. The boring part was that it is kind of repetetive to rate and you seem to learn how simplistic people search most of the time (but then again, only common/frequent searches are tested, at least most of the time). And it was enlightning how search spammers adapted to each no measure to keep them out. I still know the names of the people who were infamous at that time for setting up fake, keyword stuffed "blogs" to get people to their shop.
- sfmike 4y ago
  What do you think about gpt3 and ai content on serps and how will it be rated and ranked?
  - kappuchino 4y ago
    Good question, disapointing answer: Only time will tell. I have a hunch however, from stylometry work. Its that for short texts "generated" text will be virtually indestinguishable and this will grow as ml/ai systems advance to longer and longer texts. This will probably lead to different techniques to rate content (like more emphasis on the Your life/Your money factor).
cpeterso 4y ago
The search quality rater guide says: “Your ratings will not directly affect how a particular webpage, website, or result appears in Google Search, nor will they cause specific webpages, websites, or results to move up or down on the search results page. Instead, your ratings will be used to measure how well search engine algorithms are performing for a broad range of searches.” Is this accurate? Or is the guide bending the truth to discourage malicious raters from trying to influence site rankings? Google only uses the rater data to evaluate its search algorithm and doesn’t use the rater data to teach the algorithm? Seems like a missed opportunity. https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf#page57 https://static.googleusercontent.com/media/guidelines.raterh...
- kappuchino 4y ago
  Mostly five raters indenpendently rate the same task and are randomly chosen from a pool. This should reduce SQR conspiracies to near zero. Also, afaik, google engineers use the SQR Feedback to enhance algorithms (example: spam detection) Second: In my active time - long ago - white text was still a thing to make keyword stuffing "invisible". As a result of the SQRs work (and maybe a lot of complaints about bad search results ;-D), the next revision of search ranking "killed" that method - at least for some time - for all pages and not only the rated ones (which were only a sample of all in existance).
- lupire 4y ago
  Google hires professionals, and surveys the Internet username (Search users and Analytics-integrated website users) to teach the algorithm.
- kentonv 4y ago
  The quote matches my own understanding of the system from when I worked on Google Search circa 2005. Human ratings were used to facilitate an automated test that could tell you if your new algorithm actually made search results better or not. That let search quality engineers iterate on ideas quickly. But the human ratings were not directly used to produce search results, since that wouldn't scale (only a relatively small number of queries actually had rated results). Obviously 2005 was a long time ago though and everything could have changed since then.
anm89 4y ago
I did this job right out of school for a company called lionbridge that contracted to Google. It was OK at first and then it started having me watch increasingly extreme porn. I think I made it about 3 months
geoduck14 4y ago
If there is a web site with a LOT of content, does Google get SQR for each page or do they get SQR for "a couple of pages" and apply that rating for the entire site? When you rate "accruacy" of information or "authorativeness" of a site, how do you know the site is "an authority"? are you guessing?
- kappuchino 4y ago
  SQR is used to test if google search results are "right" across most searches, not if a website is "right" in general. Rating of a site from the point of google relies on other data points like speed, support of https, links to the site (aka page rank) and a lot more. Have a look at the SQR Guide (PDF Link in my comment above) for examples how Google classifies information results. (Especially the YMYL - Your money or your life classification and Expertise, Authoritativeness, and Trustworthiness (E-A-T))
iamleppert 4y ago
Google absolutely employs more raters than software engineers. This is the big fraud of Google search. It’s not PageRank, which hasn’t been used for years. It’s exploited human labor. That’s the real lie behind Google.
- kappuchino 4y ago
  First sentence is true. Second is not, as is the third. I believe you don't know what you are talking about. A software engineer for example writes code that scales across thousands of systems and might be (human) language independet. A SQR tests samples of searches if they match google quality standards. As this cannot scale across all websites it is done with a subset of the web, like a one in a million ratio. It has to be done by 5 people to get an average so it is independent from one rating = one result. And it has to be done in context to countries / cultures.
jccalhoun 4y ago
"The raters Yahoo Finance spoke to work from home, but they have no clue who their direct boss is, or if they have one boss or many bosses. They don’t even know their boss’ (or bosses’) name." "Across the board, the raters Yahoo Finance spoke to worry about waking up to a pink-slip-email, and two said they know of instances where people haven’t received an email at all — the worker’s account was just deactivated. " This is some Kafka-esque shit. Unreal.
astrange 4y ago
Is there a way to sign up to tell them Gmail's spam detection doesn't work anymore? Seems like anyone can get through by putting "invoice" in the subject.
freeflight 4y ago
I'm pretty sure that's a lot more than the army of shadow content moderators in the Philippines are being paid [0] [0] https://webcache.googleusercontent.com/search?q=cache:3HkKSZrt2MwJ:https://www.mediapost.com/publications/article/327887/pbs-the-cleaners-uncovers-secret-world-of-socia.html+&cd=14&hl=de&ct=clnk&gl=de https://webcache.googleusercontent.com/search?q=cache:3HkKSZ...

106 points by exz 4y ago

nyhwtt 4y ago

Better than working at Wendys!

Crosseye_Jack 4y ago
Depends on how you look at it, The wage fits in around the rough ave pay for a Wendys "crew memeber", however you are responsable for your own costs while working (computer, internet bill, electricity bill, heating) but on the flip side, its part-time, no comute, no long hours standing, no dress code.

Gravyness 4y ago

Definitely explains the quality of search as of late.

TheGoddessInari 4y ago
They (and everyone else) have been doing this forever, if not obvious
amelius 4y ago
Those raters must really love Pinterest ...
- kappuchino 4y ago
  Acually ... no, but pinterest provides results that are relevant and satisfying to the general search (and the SQR - the Search Quality Rating, see link in discussion). Hence Pinterest stays in good standing even if most people like me (you, too?) hate it for its login/content walling tactics.
  - Avamander 4y ago
    > but pinterest provides results that are relevant and satisfying to the general search More like it matches some keywords like run-of-the-mill WordPress malware generates. I've seen plenty of both. Not to mention the similarity to the masses of biz/buzz/xyz spam websites.

Proven 4y ago

How much should they be paid? It's not like you need a PhD to confirm if a search result sucks. Results are supposed to be good enough for the average user, so you need common sense and some on-job training to do a good job. And having 10 common-sense guys churning through user feedback @$11/h is better than 2 geniuses @$55/h. As one comment says, it probably beats fliping burgers.

closedloop129 4y ago
Where is the limit? Why not 20 guys in a low-cost country @$5.50/h? They cannot rate all results anyway, so why not get the best possible evaluations? Shouldn't quality data beat quantity if you extrapolate from the evaluated samples to the entire index? Spammers target the lower part of the bell curve. If you let that part evaluate the quality of the results, do you think that they will recognize spam?
echen 4y ago
It can be suprisingly difficult. A lot of the difficulty is in understanding the user intent behind a search query (since you're usually rating other people's queries, not your own). For example, here are some queries in my history - would the average person understand what I'm looking for and what a good search result would be? "neovim worktrees" "django models through" "pandas econdb"
- kappuchino 4y ago
  Good examples. Raters have - afaik- to assume multiple possible intentions on two word searches (and general searches on one). This results in better ratings for results (top 10 generally) that partially match different search intentions. so for "pandas databases" it would match the panda python libraries, but also databases literally of pandas ... and maybe a company that is called pandas and has databases. And this is the reason why we can never have "pure" results (in most cases) for a specific search, because the algorithm and the search rating is aiming for result-topic-diversity most of the time (if not all of the time).
kappuchino 4y ago

rcktmrtn 4y ago

On a related note, am I right in understand that the "Google Experts" [1] do it for free? [1] https://developers.google.com/community/experts/ https://developers.google.com/community/experts/

ocdtrekkie 4y ago
Yes, Google has tricked a massive army of people into doing free work for one of the most profitable companies in the world. Things like "Google Local Guides" give people a gold star (and maybe a pair of branded socks) for doing unpaid labor. It's kinda hilarious, but also sad. I imagine there are Google executives in a room somewhere laughing their full heads off every time they think about how there are people who do free work for Google who think they are making the world a better place or something.
- rurban 4y ago
  When they tried to lure me into a "Local Guide" I was offered a free flight, hotel and food at Menlo Park. Once, but still good enough for no work at all.
- vba616 4y ago
  >laughing their full heads off every time they think about how there are people who do free work for Google who think they are making the world a better place or something. So when you go to a store that Google Maps says is open, and find it's closed, you refuse to correct the hours and go "MUHAHAHAHA...I refuse to help Google, take that, corporate scum!"
  - ocdtrekkie 4y ago
    Why would I bother checking Google Maps? But basically, yeah. If Google wants my expertise and data collection ability, they can pay me for it. As it stands now they just cold call businesses and ask them with an AI thing.
    - nine_k 4y ago

vgeek 4y ago

This is 100% a race to the bottom. Years ago the Leapforce (now Appen as mentioned in the article) and Lionsbridge "contractors" could average over $18/hr. They would only work on Google tasks and adhere to Google's policies. The work seemed inconsistent (people would have 5 hours of work some days, none the next) and about what you would expect for AI training. No worker protections, benefits, etc.-- all while doing work, albeit opaquely, for one of the richest companies of all time. Why can't companies at least try to take care of their workers?

daniel-cussen 4y ago
> all while doing work, albeit opaquely, for one of the richest companies of all time. It doesn't work that way. Contrariwise.
Tenoke 4y ago
I worked for Leapforce from the UK in 2015-16 and I definitely didn't reach 18$/hr and the availability of work wasnt too great. Probably not bad if you have another gig you can do in the mean time though.
rater266 4y ago
Former longtime Leapforce/RaterLabs preferred/senior rater here. There was always some exploitation going on, but it got way worse under Ruth Porat's tenure. Standard Leapforce US English raters were paid $13.50/hr. If you were lucky enough to get promoted to "Preferred Agent" (PA), you made $17.40/hr. The criteria for PA promotion was basically to be in the top 5% or so on their quality metrics, plus various subjective quality requirements. Promotion rounds were irregular and random. Leapforce's US English pay rates never changed over time. There were no raises, other than the aforementioned rare and seemingly luck-based promotion opportunity. I was one of the fortunate ones. Leapforce raters were misclassified as 1099 contractors. There were no benefits. Training time was unpaid. Originally, raters could work as many hours as they wanted, subject to work availability. Around the end of 2013, they added a 40 hour/week limit because they were afraid a court or regulator might rule the "contractors" were employees and force them to pay retroactive overtime. If a state dared to go after Leapforce or Google for misclassification, they would leave that state and fire the workers there. No severance, of course. There were numerous states they did not hire in, including their home state of California. Task availability was usually very good for US English raters, but there were definitely periods (often around holidays) where that was not the case. There was never any communication about projected task availability. We certainly would have appreciated a quick heads-up of "hey, many of our engineers will be on vacation between mid-December and mid-January, so expect task availability to be light. Tasks will be added daily at 8AM PST, but probably won't last more than two or three hours." I understand other locales were not as fortunate as we were for task availability. Work availability for a particular rater was not necessarily as consistent. If you scored poorly on a monthly quality review or random quality audit, you could be abruptly suspended, fired, or limited to an hour or two a day until the next monthly review. There would be no prior warning of this. There were automated tripwires (which we called the "bot") that would flag and suspend you if your ratings were considered outliers, or if your task comments were too similar (even when there was a batch of tasks that were very similar), or if you worked too fast, or if you worked too slowly, or a million other reasons. If you were flagged by the bot, you had no work until the Leapforce quality people got around to reviewing you. There was no compensation for missed hours even if it turned out you hadn't done anything wrong. Even more insultingly, Google loved to use quality reviews to informally implement guideline changes without actually changing the guidelines. If you weren't good at guessing their future direction, you could easily have your hours cut for the next month. Around 2017, Google realized they (not just Leapforce or Lionbridge) could be sued for this misclassification nonsense, so they forced Leapforce and Lionbridge to convert their misclassified contractors to employees. Leapforce morphed into RaterLabs in what appeared to be some kind of liability-dodging scam. As employees, they cut us to 26 hours/week so they wouldn't have to pay for health insurance. There were still no benefits other than the minimum required by state law, if you were lucky enough to be in a state that required any. Nominal pay rates were cut in what seemed like a random and inconsistent (not even location based) way; depending on the cut, you got a few cents raise after accounting for the self-employment tax you no longer had to pay (which RaterLabs was quick to point out), or a pay cut. There was a rater revolt around that time, but nothing seemed to come of it. Appen bought Leapforce and RaterLabs later in 2017. Leapforce founders Daren Jackson and Caine Lai made off with a pile of cash. His employees weren't so fortunate. Now people are making $10/hr. for a job that paid $13.50/hr. in 2009, which would be the equivalent of $18.38/hr. in today's dollars.

advisedwang 4y ago

I thought the management theory was to in-source your core-competencies and out-source everything else. Surely quality search results is one of Google's core competencies!

ocdtrekkie 4y ago
Google's core competency is ads. Any time you click a search result, it's a failure: Google wanted you to click the ad above it.
- kappuchino 4y ago
  Totaly oversimplification. If google wanted you only to click on ads, it would be a type of yellow pages. What it is (as it was with newspapers and magazines a time ago): A place where you can place paid ads based on context. If you remove the context, it becomes worthless. So while everyone is complaining about search quality, its an uphill battle for any search company. Like PR tries to sneak in Ad-Material into reguluar journalism, companies try to manipulate search results with SEO ranging form keyword-stuffing to bait and switch search pages. (which are one reason why google needs humans and not ai to verify).
kurupt213 4y ago
If you use Google you know the search results have been getting worse and worse

akomtu 4y ago

There is your "AI".

Justin_K 4y ago

Another prime example of how ML and AI aren't mature. With all the data google collects, they have to rely on minimum wagers to "find" the best results.

xnx 4y ago
In the sense that ML and AI aren't yet AGI, yes, but this doesn't mean that Google's ML and AI aren't very useful and benefit from having another source of training and validation data.
dfdz 4y ago
This criticism seems unfair. There is nothing wrong with semi-supervised learning.
kappuchino 4y ago
Actually, ML/AI would not know by the rules of Google for "Quality" how to rate content, since content is ever changing and diverse as it gets. If you spend some time reading the guide (https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf https://static.googleusercontent.com/media/guidelines.raterh...) you will realize that its not about the "best" results but on classificaion / verificationin the YMYL (Your Money, Your Life) areas that should not be left to machines in the first place. Money is mainly low because there are so many people that can do the job and so many that are needed to redundantly rate the content.

civilized 4y ago

Dumb question - why can't I just upvote and downvote pages, and Google enhances its ranking for me based on that? Maybe that's why the search "X reddit" is so popular - it's got a bit of that social validation.

arthurcolle 4y ago
this used to exist but they got rid of it like 10 yrs ago IIRC
lordnacho 4y ago
Who do you reckon would cast the most votes? Correct, SEO bots.
- jancsika 4y ago
  If Google can't use it's vast array of user data to tell the difference between SEO bots and non-bot users, what's even the point of it collecting all that fucking data in the first place?
  - deleted 4y ago
    [deleted]
  - bryanrasmussen 4y ago
    Good point, but I have my reasons for suspecting they can't tell the difference https://news.ycombinator.com/item?id=31296953 https://news.ycombinator.com/item?id=31296953
    - scoopertrooper 4y ago
      Maybe that's part of how they train it?
  - rednalexa 4y ago
    I would assume that if Google is paying 10$/hour for humans to rank content, someone else is paying 11$/hour to fool it with a human generating content, clicks, etc. Additionally, people writing bots learn just like Google learns, and start generating what appears to be legitimate interactions. Same algorithms that fit to critique between 'bot' and 'non-bot' class can be used to create bots that fall into the 'non-bot' category. It's a losing battle - but what we get is better than we would get if they collected less data.

hobo_mark 4y ago

Ha, so they still do it, that is how I paid rent for a few months back in 2012.

ggm 4y ago

Surprisingly badly paid. I wonder how strongly quality would track pay? Probably they act as a CAPTCHA mutually cross checking and so QA is by volume, and not coincidentally demands they work in isolation. Sad story.

echen 4y ago
Quality control is often indeed by volume. When I was at Google / Facebook, search evaluations would typically have 5 raters each. One of the difficulties, though, is that search evaluation can be quite subjective. And honestly, most of the search engine raters we used were low quality and did a bad job (so the aggregate answer was often nonsense).
- kappuchino 4y ago
  This. In the early days of my SQR rating time, the rating was not sliced into time per rating. And you had to write a short blurb about your reasons, often using the acronyms so it read like "NR, LQ / SPAM". Later you would see the other SQR notes and would wonder if both of you had seen the same page.
- rater266 4y ago

demadog 4y ago

When those reports come out that “The average Google employee makes $220k a year” or whatever, it’s totally misleading since it doesn’t account for all the outsourced labor at $10 an hour that is critical to tuning their algorithms.

Nexxxeh 4y ago
I did voice training tasks for Google through a 3rd party I can't remember the name of. I was recruited via /r/beermoney or something. I enjoyed it, and it was easy work that fit around my schedule. It also gave a feeling of improving the lives of users, which was cool.
- daniel-cussen 4y ago
  It for sure can be good, win-win.
kappuchino 4y ago
Its better paid than Amazon Mechanical Turk afaik (not saying it should not be paid better). The tuning is done literally by thousands world wide and since every rating task is done by multiple quality raters (afaik side by side is done by at least 5), it seems to be a cost vs. scale factor.
- demadog 4y ago
  Amazon Turk is more of a platform for small business to hire other workers for micro-tasks. Raterlabs is used only by Google to improve their core product.
hn_throwaway_99 4y ago
I don't think that's misleading at all. Why would I assume that non-Google employee compensation would go into the calculation for what the average Google employee makes? As someone else pointed out, this pays substantially more than AWS Mechanical Turk. To be honest, it's difficult for me to imagine an easier job than this one. I mean, most people who would do this job could easily now make more at McDonald's or Walmart or whatever. But I think jobs at McDonald's or Walmart are considerably harder, so I don't see why this job should pay the same.

kappuchino 4y ago

Ex Google Search quality Rater here, based in Germany. From 2004 to 2006 I worked on SQR and "special projects", with a much better pay (maybe because of german circumstances). I've been tracking the SQR-guide since then (public version here https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf https://static.googleusercontent.com/media/guidelines.raterh...), so I have still some current understanding. If you have questions, I'd be happy to try to answer them since my NDA has expired ;-).

dmarchand90 4y ago
I don't have any specific questions but would be really interested in hearing more!
- kappuchino 4y ago
  Its an interesting boring job in its core. At least when I did it. You agree to see theoretically everything that people search for and try to rate it according to the linked manual. In my time - really long ago - britney spears was one of the most common searches (I remember it probably wrong, but the test I had to do was half rating questions on spears and tom hanks). But I also got to rate "donkey s*x" and things like "divorce help" which makes you think about the impact search results can have. The boring part was that it is kind of repetetive to rate and you seem to learn how simplistic people search most of the time (but then again, only common/frequent searches are tested, at least most of the time). And it was enlightning how search spammers adapted to each no measure to keep them out. I still know the names of the people who were infamous at that time for setting up fake, keyword stuffed "blogs" to get people to their shop.
sfmike 4y ago
What do you think about gpt3 and ai content on serps and how will it be rated and ranked?
- kappuchino 4y ago
  Good question, disapointing answer: Only time will tell. I have a hunch however, from stylometry work. Its that for short texts "generated" text will be virtually indestinguishable and this will grow as ml/ai systems advance to longer and longer texts. This will probably lead to different techniques to rate content (like more emphasis on the Your life/Your money factor).

cpeterso 4y ago

The search quality rater guide says: “Your ratings will not directly affect how a particular webpage, website, or result appears in Google Search, nor will they cause specific webpages, websites, or results to move up or down on the search results page. Instead, your ratings will be used to measure how well search engine algorithms are performing for a broad range of searches.” Is this accurate? Or is the guide bending the truth to discourage malicious raters from trying to influence site rankings? Google only uses the rater data to evaluate its search algorithm and doesn’t use the rater data to teach the algorithm? Seems like a missed opportunity. https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf#page57 https://static.googleusercontent.com/media/guidelines.raterh...

kappuchino 4y ago
Mostly five raters indenpendently rate the same task and are randomly chosen from a pool. This should reduce SQR conspiracies to near zero. Also, afaik, google engineers use the SQR Feedback to enhance algorithms (example: spam detection) Second: In my active time - long ago - white text was still a thing to make keyword stuffing "invisible". As a result of the SQRs work (and maybe a lot of complaints about bad search results ;-D), the next revision of search ranking "killed" that method - at least for some time - for all pages and not only the rated ones (which were only a sample of all in existance).
lupire 4y ago
Google hires professionals, and surveys the Internet username (Search users and Analytics-integrated website users) to teach the algorithm.
kentonv 4y ago
The quote matches my own understanding of the system from when I worked on Google Search circa 2005. Human ratings were used to facilitate an automated test that could tell you if your new algorithm actually made search results better or not. That let search quality engineers iterate on ideas quickly. But the human ratings were not directly used to produce search results, since that wouldn't scale (only a relatively small number of queries actually had rated results). Obviously 2005 was a long time ago though and everything could have changed since then.

anm89 4y ago

I did this job right out of school for a company called lionbridge that contracted to Google. It was OK at first and then it started having me watch increasingly extreme porn. I think I made it about 3 months

geoduck14 4y ago

If there is a web site with a LOT of content, does Google get SQR for each page or do they get SQR for "a couple of pages" and apply that rating for the entire site? When you rate "accruacy" of information or "authorativeness" of a site, how do you know the site is "an authority"? are you guessing?

kappuchino 4y ago
SQR is used to test if google search results are "right" across most searches, not if a website is "right" in general. Rating of a site from the point of google relies on other data points like speed, support of https, links to the site (aka page rank) and a lot more. Have a look at the SQR Guide (PDF Link in my comment above) for examples how Google classifies information results. (Especially the YMYL - Your money or your life classification and Expertise, Authoritativeness, and Trustworthiness (E-A-T))

iamleppert 4y ago

Google absolutely employs more raters than software engineers. This is the big fraud of Google search. It’s not PageRank, which hasn’t been used for years. It’s exploited human labor. That’s the real lie behind Google.

kappuchino 4y ago
First sentence is true. Second is not, as is the third. I believe you don't know what you are talking about. A software engineer for example writes code that scales across thousands of systems and might be (human) language independet. A SQR tests samples of searches if they match google quality standards. As this cannot scale across all websites it is done with a subset of the web, like a one in a million ratio. It has to be done by 5 people to get an average so it is independent from one rating = one result. And it has to be done in context to countries / cultures.

jccalhoun 4y ago

"The raters Yahoo Finance spoke to work from home, but they have no clue who their direct boss is, or if they have one boss or many bosses. They don’t even know their boss’ (or bosses’) name." "Across the board, the raters Yahoo Finance spoke to worry about waking up to a pink-slip-email, and two said they know of instances where people haven’t received an email at all — the worker’s account was just deactivated. " This is some Kafka-esque shit. Unreal.

astrange 4y ago

Is there a way to sign up to tell them Gmail's spam detection doesn't work anymore? Seems like anyone can get through by putting "invoice" in the subject.

freeflight 4y ago

I'm pretty sure that's a lot more than the army of shadow content moderators in the Philippines are being paid [0] [0] https://webcache.googleusercontent.com/search?q=cache:3HkKSZrt2MwJ:https://www.mediapost.com/publications/article/327887/pbs-the-cleaners-uncovers-secret-world-of-socia.html+&cd=14&hl=de&ct=clnk&gl=de https://webcache.googleusercontent.com/search?q=cache:3HkKSZ...