7 ms·

Presumably, many Googlers who are directly involved in designing and implementing content moderation are active readers of Hacker News, as is evidenced by the f

Presumably, many Googlers who are directly involved in designing and implementing content moderation are active readers of Hacker News, as is evidenced by the fact that having a story hit the front page is an effective way to solve problems like these. But what I never see is any of those Googlers taking the time to explain why they feel these seemingly-broken approaches are the only ways to solve the problem of moderation at scale. To me, it seems insulting to one's intelligence to receive an automated message that your channel's been banned, be given the option to appeal, and to receive an equally automated message rnd() minutes later that your appeal has been denied. But what do I know? I've never been tasked with solving such an engineering problem. I know FAANGers here may feel put upon by what appears to be a hostile environment against their companies, but that's at least in part because you've never said exactly which technical and practical challenges lead to this state of affairs. If it's not against your company's NDAs, I challenge you all to try; whether the community agrees with you or not, I think we would all learn something useful from a productive discussion about the problem.

wombatpm 5y ago
I understand the need for automation, and can imagine that the spammers, grifters and crooks would overwhelm any manual review system. I'm suprised that Google has not implemented pay for support. If the issue is really important, can I just pay $50 and speak to a real person? Just because manual processes don't scale, doesn't meam they shouldn't exist.
lotsofpulp 5y ago
> But what I never see is any of those Googlers taking the time to explain why they feel these seemingly-broken approaches are the only ways to solve the problem of moderation at scale. They are not broken approaches. They are optimizing for net income. Higher net income for the company means higher compensation for individuals at the company, since they are being paid with equity. Spending more money on labor means less net income for the company means lower compensation for individuals.
throw10920 5y ago
> But what I never see is any of those Googlers taking the time to explain why they feel these seemingly-broken approaches are the only ways to solve the problem of moderation at scale. From my experience with a very large bureaucracy, I believe the answer to this is "the worker bees don't believe this; the requirement to implement automated moderation [in this way] is imposed by managers and/or executives" - and speaking up can get you retaliated against.
- sombremesa 5y ago
  Yup. Here’s a good video for what goes on behind the scenes at Amazon’s gaming org for example. https://youtu.be/aggk93Nazuk https://youtu.be/aggk93Nazuk I can verify this is true since I used to work there.
  - dredmorbius 5y ago
    That's apparently based on a Bloomberg article for those who'd prefer not sitting through a 40-minute re-reading of it: https://www.bloomberg.com/news/features/2021-01-29/amazon-game-studios-struggles-to-find-a-hit https://www.bloomberg.com/news/features/2021-01-29/amazon-ga...
- hackitup7 5y ago
  It's probably not a singular insane executive who just hates good user experiences, either. It's probably a web of incentives that emerge to produce this poor experience: - CEO has a goal of getting support ticket resolution times to X - Some exec is tasked with flagging content, and allowing very little unapproved content through (creates lawsuits) - Some exec has a budget of Y for resolving support issues - Some other execc else wants to get a higher level technical win on their resume Then in total you get someone who builds an automatic flagging system. This creates support tickets. You need to drive them down, but you only have Y budget to do so (so you can't hire an army of high-quality reps). You hire a smaller amount of reps on a budget. They do a bad job from being undergunned; someone builds an internal product that will magically help them resolve issues faster but it basically just helps them resolve issues really fast by identifying the obviously innocent and denying appeal on most cases. At the end of the day an understaffed team closes a huge # of tickets without prejudice. But at no point did anyone say "F those users." This is actually worse than some tone deaf exec who hates users, because you can't solve it by replacing someone bad.
maxwellb 5y ago
> If it's not against your company's NDAs This ^ is nontrivial (maybe impossible) to determine without consulting an attorney.
- KMag 5y ago
  Technically, you'll need to go to court to get a canonical answer. Without a court ruling, it's just expert opinions.
whimsicalism 5y ago
Honestly, what service do you get for free that is also rolled out to billions of people and has better customer service? Hiring individual people to address your issues is going to be hard to scale in a way that allows a free service to be rolled out to billions. I don't work at Google, do work at another company that I have no doubt has the same issue, but not discussing in the context of that company either.
- luma 5y ago
  OP did note that he happens to be a YT premium user (for music), so in this situation it isn't a "free" service, they are interacting with a paying customer.
KennyBlanken 5y ago
From knowing a lot of Googlers: - They are extremely self-assured of their competence and the effectiveness/completeness of anything they design. - Google has nearly two-decade-culture of drive-by product development. People work on something, it becomes a product, the laser pointer of Shiny dances nearby, and they're off to work on something else and the product matures at a glacial rate, if at all. They also don't care. Money printing machine go brrrrrrrrrr to the tune of $10BN/year.
- vernacular 5y ago
  Yes, bless daddy Ads for bankrolling our negligent product practices. Can be awful for users, but hey, the company lives on.
jyounker 5y ago
There is a really simple reason for this: Discussing basically anything with the outside world is violation a of your employment agreement.
aznicht 5y ago
I wonder what do you expect to hear? I don't have anything to do with youtube or its moderation approaches. But I had some experience with automatic content moderation. Imaging a system contains of tens (hundreds) of signal sources. Signal might be as complex as ML model resolution or as simple as static list of stopwords. And everything in between. With sufficiently large rate of new content to moderate (and I guess YT has a lot), you don't deal with specific moderation decisions. For you they are just points in your metrics. So you can't treat all blocks as probably false-positive, you react only to drastic change in trends. In such system the only way to solve single false-positive - human intervention. So you have some operators who process appeals. But human operators are not cheap, so first you try to increase their productivity - for example by preprocessing data for them. So they (for example) don't need to watch the video, they get transcript, probably part of it, that triggered the signal. So it might be possible that these 15 minutes were enough for human operator to make a decision based on some substring of transcript, which without context was enough for them to mark author as fraud. Or you can go even further - you can skip operator if content got enough bad signals. My point here is that the problem is not entirely technical. You can't make ideal model without false-positive. You just try to optimise it by using discovered cases like this as input to bake better one. So the only thing to blame here - not enough human operators. Thats why you probably don't get any answers here from google engineers. The root cause either not technical, or too technical to comment it outside.

wombatpm 5y ago

I understand the need for automation, and can imagine that the spammers, grifters and crooks would overwhelm any manual review system. I'm suprised that Google has not implemented pay for support. If the issue is really important, can I just pay $50 and speak to a real person? Just because manual processes don't scale, doesn't meam they shouldn't exist.

lotsofpulp 5y ago

> But what I never see is any of those Googlers taking the time to explain why they feel these seemingly-broken approaches are the only ways to solve the problem of moderation at scale. They are not broken approaches. They are optimizing for net income. Higher net income for the company means higher compensation for individuals at the company, since they are being paid with equity. Spending more money on labor means less net income for the company means lower compensation for individuals.

throw10920 5y ago

> But what I never see is any of those Googlers taking the time to explain why they feel these seemingly-broken approaches are the only ways to solve the problem of moderation at scale. From my experience with a very large bureaucracy, I believe the answer to this is "the worker bees don't believe this; the requirement to implement automated moderation [in this way] is imposed by managers and/or executives" - and speaking up can get you retaliated against.

sombremesa 5y ago
Yup. Here’s a good video for what goes on behind the scenes at Amazon’s gaming org for example. https://youtu.be/aggk93Nazuk https://youtu.be/aggk93Nazuk I can verify this is true since I used to work there.
- dredmorbius 5y ago
  That's apparently based on a Bloomberg article for those who'd prefer not sitting through a 40-minute re-reading of it: https://www.bloomberg.com/news/features/2021-01-29/amazon-game-studios-struggles-to-find-a-hit https://www.bloomberg.com/news/features/2021-01-29/amazon-ga...
hackitup7 5y ago
It's probably not a singular insane executive who just hates good user experiences, either. It's probably a web of incentives that emerge to produce this poor experience: - CEO has a goal of getting support ticket resolution times to X - Some exec is tasked with flagging content, and allowing very little unapproved content through (creates lawsuits) - Some exec has a budget of Y for resolving support issues - Some other execc else wants to get a higher level technical win on their resume Then in total you get someone who builds an automatic flagging system. This creates support tickets. You need to drive them down, but you only have Y budget to do so (so you can't hire an army of high-quality reps). You hire a smaller amount of reps on a budget. They do a bad job from being undergunned; someone builds an internal product that will magically help them resolve issues faster but it basically just helps them resolve issues really fast by identifying the obviously innocent and denying appeal on most cases. At the end of the day an understaffed team closes a huge # of tickets without prejudice. But at no point did anyone say "F those users." This is actually worse than some tone deaf exec who hates users, because you can't solve it by replacing someone bad.

maxwellb 5y ago

> If it's not against your company's NDAs This ^ is nontrivial (maybe impossible) to determine without consulting an attorney.

KMag 5y ago
Technically, you'll need to go to court to get a canonical answer. Without a court ruling, it's just expert opinions.

whimsicalism 5y ago

Honestly, what service do you get for free that is also rolled out to billions of people and has better customer service? Hiring individual people to address your issues is going to be hard to scale in a way that allows a free service to be rolled out to billions. I don't work at Google, do work at another company that I have no doubt has the same issue, but not discussing in the context of that company either.

luma 5y ago
OP did note that he happens to be a YT premium user (for music), so in this situation it isn't a "free" service, they are interacting with a paying customer.

KennyBlanken 5y ago

From knowing a lot of Googlers: - They are extremely self-assured of their competence and the effectiveness/completeness of anything they design. - Google has nearly two-decade-culture of drive-by product development. People work on something, it becomes a product, the laser pointer of Shiny dances nearby, and they're off to work on something else and the product matures at a glacial rate, if at all. They also don't care. Money printing machine go brrrrrrrrrr to the tune of $10BN/year.

vernacular 5y ago
Yes, bless daddy Ads for bankrolling our negligent product practices. Can be awful for users, but hey, the company lives on.

jyounker 5y ago

There is a really simple reason for this: Discussing basically anything with the outside world is violation a of your employment agreement.

aznicht 5y ago

I wonder what do you expect to hear? I don't have anything to do with youtube or its moderation approaches. But I had some experience with automatic content moderation. Imaging a system contains of tens (hundreds) of signal sources. Signal might be as complex as ML model resolution or as simple as static list of stopwords. And everything in between. With sufficiently large rate of new content to moderate (and I guess YT has a lot), you don't deal with specific moderation decisions. For you they are just points in your metrics. So you can't treat all blocks as probably false-positive, you react only to drastic change in trends. In such system the only way to solve single false-positive - human intervention. So you have some operators who process appeals. But human operators are not cheap, so first you try to increase their productivity - for example by preprocessing data for them. So they (for example) don't need to watch the video, they get transcript, probably part of it, that triggered the signal. So it might be possible that these 15 minutes were enough for human operator to make a decision based on some substring of transcript, which without context was enough for them to mark author as fraud. Or you can go even further - you can skip operator if content got enough bad signals. My point here is that the problem is not entirely technical. You can't make ideal model without false-positive. You just try to optimise it by using discovered cases like this as input to bake better one. So the only thing to blame here - not enough human operators. Thats why you probably don't get any answers here from google engineers. The root cause either not technical, or too technical to comment it outside.