6 ms·

How much should they be paid? It's not like you need a PhD to confirm if a search result sucks. Results are supposed to be good enough for the average user, so

by Proven 4y ago

How much should they be paid? It's not like you need a PhD to confirm if a search result sucks. Results are supposed to be good enough for the average user, so you need common sense and some on-job training to do a good job. And having 10 common-sense guys churning through user feedback @$11/h is better than 2 geniuses @$55/h. As one comment says, it probably beats fliping burgers.

closedloop129 4y ago
Where is the limit? Why not 20 guys in a low-cost country @$5.50/h? They cannot rate all results anyway, so why not get the best possible evaluations? Shouldn't quality data beat quantity if you extrapolate from the evaluated samples to the entire index? Spammers target the lower part of the bell curve. If you let that part evaluate the quality of the results, do you think that they will recognize spam?
echen 4y ago
It can be suprisingly difficult. A lot of the difficulty is in understanding the user intent behind a search query (since you're usually rating other people's queries, not your own). For example, here are some queries in my history - would the average person understand what I'm looking for and what a good search result would be? "neovim worktrees" "django models through" "pandas econdb"
- kappuchino 4y ago
  Good examples. Raters have - afaik- to assume multiple possible intentions on two word searches (and general searches on one). This results in better ratings for results (top 10 generally) that partially match different search intentions. so for "pandas databases" it would match the panda python libraries, but also databases literally of pandas ... and maybe a company that is called pandas and has databases. And this is the reason why we can never have "pure" results (in most cases) for a specific search, because the algorithm and the search rating is aiming for result-topic-diversity most of the time (if not all of the time).
kappuchino 4y ago
I think everyone in SQR is based on the national average/context. In germany - even 15 years ago - SQRs were paid quite okay (around 16€/hour back then) including health insurance and social security pay.
deleted 4y ago
[deleted]

by Proven 4y ago

closedloop129 4y ago

Where is the limit? Why not 20 guys in a low-cost country @$5.50/h? They cannot rate all results anyway, so why not get the best possible evaluations? Shouldn't quality data beat quantity if you extrapolate from the evaluated samples to the entire index? Spammers target the lower part of the bell curve. If you let that part evaluate the quality of the results, do you think that they will recognize spam?

echen 4y ago

It can be suprisingly difficult. A lot of the difficulty is in understanding the user intent behind a search query (since you're usually rating other people's queries, not your own). For example, here are some queries in my history - would the average person understand what I'm looking for and what a good search result would be? "neovim worktrees" "django models through" "pandas econdb"

kappuchino 4y ago
Good examples. Raters have - afaik- to assume multiple possible intentions on two word searches (and general searches on one). This results in better ratings for results (top 10 generally) that partially match different search intentions. so for "pandas databases" it would match the panda python libraries, but also databases literally of pandas ... and maybe a company that is called pandas and has databases. And this is the reason why we can never have "pure" results (in most cases) for a specific search, because the algorithm and the search rating is aiming for result-topic-diversity most of the time (if not all of the time).

kappuchino 4y ago

I think everyone in SQR is based on the national average/context. In germany - even 15 years ago - SQRs were paid quite okay (around 16€/hour back then) including health insurance and social security pay.

deleted 4y ago

[deleted]