7 ms·

Disclaimer: Former Technical Solutions Engineer for GCP, aka Support for Customers. Also Former Engineer on YouTube Caching. To get it out of the way, I do not

Disclaimer: Former Technical Solutions Engineer for GCP, aka Support for Customers. Also Former Engineer on YouTube Caching. To get it out of the way, I do not agree that it should've taken a journalist to get involved to have this situation solved. However, I'd like to prompt Hacker News with how would you handle receiving support requests from a product that has >2.7B users. Almost all of which are non-directly revenue generating, across hundreds of different languages, in every conceivable location in the world. It's an extremely hard problem to solve, but I don't think anyone has got it right. I'll be playing devil's advocate in the comments. Keep me busy for my flights.

gooosle 2y ago
You start by not banning people for nothing.
- qmarchi 2y ago
  Except for that you risk breaking laws in several jurisdictions if you don't. And you can't fall back to "manual" enforcement think of the massive capital of human resources you'd have to invest in to make it keep up with the upload rate of YouTube.
  - gooosle 2y ago
    Give me a single example of laws in any jurisdiction that require this user to be banned. There was absolutely no reason to ban in this case and you know it. If there was one video where rules were unintentionally broken, and the user has no history of that before, you remove the video, not completely ban the user.
  - prmoustache 2y ago
    If it is not sustainable, the service shouldn't be run.
  - deleted 2y ago
    [deleted]
  - CamperBob2 2y ago
    Except for that you risk breaking laws in several jurisdictions if you don't. Why isn't the market addressing this? Why doesn't someone start a video service in the US and for the US -- one that follows US law, and not Pakistan's or Iran's or India's or Germany's or anyone else's? Oh, right, because then they can't sell ads in those places, or use them as tax shelters. Silly question, I guess. By forcing compliance with oppressive regimes (and with pearl-clutching sponsors in less-oppressive ones), advertising will eventually end everything good about the Internet. If you work in that business, you're the problem. Consider a different career.
CamperBob2 2y ago
It's a social problem, not a technical one. Facilitate a hierarchy of trust among users who more-or-less volunteer to moderate at the lower trust levels and engage on unjustly-banned users' behalf with YouTube staff at higher trust levels. Better yet, pay them. It is, after all, work.
- andrewinardeer 2y ago
  Why pay? Reddit, a multi-billion dollar company has perfected the art of exploiting unpaid volunteer work. So much so that when said workers rebel against the administration they get booted from their position. Moderators are easily replaced as there is always someone willing to toe the administration's line.
tossandthrow 2y ago
It is easy: Allocate the resources needed - hire more customer reps. This would cut into margins, but maybe it is not possible to run hyper scale companies only managed by a couple of engineers. And maybe we should not accept that profit seeking people want to do that anyways.
- lolinder 2y ago
  > This would cut into margins, but maybe it is not possible to run hyper scale companies only managed by a couple of engineers. It wouldn't cut into margins, it would make YouTube wildly unprofitable, with no viable path to monetization that would ever pay for the support burden. I realize that some on HN—it sounds like you included—are perfectly happy to argue that if a company can't provide human customer support to every one of their users then that company shouldn't exist, but most of YouTube's users would fervently disagree.
  - tossandthrow 2y ago
    Clearly, as the parent commentor wrote, they are not able to solve it technically. There is a segment on HN, it sounds like you are included, who believes that is is OK to entirely defer to algorithmic governance without any legal oversigt.
  - taormina 2y ago
    That is absolutely hilarious. There is definitely a middle ground between today’s “spend $0 on live human support” and “spend X% of YT revenue on the support a platform like YT demands”. There exists an X% where they are plenty profitable. It’s a video hosting platform subsidized by the largest ad network in the world.
    - lolinder 2y ago
      They don't spend $0 on live customer support, they spend X%. At their scale there will always be high profile stories like this unless and until they spend enough to support every single user who runs into problems with their automated systems. Spending that much is completely cost prohibitive, so we should never expect to reach a point where we stop seeing stories like this.
ryandrake 2y ago
In this case, an entire channel was shut down (with no opportunity for appeal) on account of a single instance of someone zoom-bombing a live stream with pornography. Presumably the decision was entirely automated and there was no human in the loop. It doesn't matter how many users you have. This "solution" seems like swatting a fly with a nuclear weapon. Why not just take down the offending video until the user takes corrective action? YouTube can clearly identify the offending video out of the non-offending ones, so that's not a technical problem. And it can be done entirely with automation, so it wouldn't need humans. Further, they obviously can tell that the user does not have a history or track record of this kind of activity. Why do these tech companies always go straight to the "no recourse ban hammer"?
- sbarre 2y ago
  Yeah it feels like there should be an escalating series of consequences for infractions.. It should have been pretty straightforward to establish a pattern (or lack of) around whether this was an intentionally abusive channel or a first-time offence. But of course this costs money and takes time to build.
  - some_random 2y ago
    They sorta have this already for copyright strikes, but I guess porn was judged to be a more impactful offense?
    - Cthulhu_ 2y ago
      Yes; they cannot take any risks with that, laws around showing porn to people / minors (accidentally or not) are stricter than copyright violations. I'm not a legal expert and honestly I'm pulling this out of my ass, but, porn is a legal matter, but copyright violations is a civil matter. There's also the advertisers and payment processors to consider, most of them will have nothing to do with porn. This triggered some major pruning of various websites too, Tumblr and Reddit on the one side, but huge chunks of Pornhub and co as well for any porn that didn't come with the right paperwork, including identification and consent forms from the participants.
levkk 2y ago
Reddit works just fine. Where there is a will, there is a way. I do believe (without proof) that incentives for Google are misaligned here.
- sbarre 2y ago
  So while Reddit is huge, I don't think it's even close to the same scale as Google. And problems at Google's scale are hard to compare to smaller scenarios. But I do agree with your 2nd point about misaligned incentives though. I don't think "how do we ensure that every user can get fair support" was ever on any product roadmap for these free global-scale products.. Or more accurately, the "users" in this case are the advertisers, not the uploaders.
- some_random 2y ago
  Reddit is definitely a different beast, it's easier to admin and has unpaid community moderators who pick up pretty much all the grunt work
- qingcharles 2y ago
  This isn't correct. I've just been discussing the shadowbanning/appeal system with Huffman because it doesn't work right now.
fakedang 2y ago
Well for starters, cut the issues at source? The article mentions that the channel was taken down because a hacker in the live Zoom meeting (being streamcast into YouTube) played porn. YouTube could have simply blocked that single YT video while retaining the rest of the channel. If multiple instances of users hacking Zoom meetings came to light, Google could simply block Zoom from streamcasting videos into YouTube until they fixed their shit.
vundercind 2y ago
> However, I'd like to prompt Hacker News with how would you handle receiving support requests from a product that has >2.7B users. Almost all of which are non-directly revenue generating, across hundreds of different languages, in every conceivable location in the world. One way: dollars. Another way: don't provide services you can't support. The only way it's actually going to happen, and when it does we'll shockingly find it was always possible and companies just didn't feel like doing it: regulation (which will just result in a mix of the first two options)
- sbarre 2y ago
  > don't provide services you can't support. Also playing devil's advocate here: Define "support". I'm sure Google will tell you that they support their billions of users just fine, relatively speaking, and that the percentage of people who fall through the cracks is an acceptable margin (to them, obviously not to the users themselves). To your point about "not providing the service", do you believe that the trade-off that would happen if Google, for example, stopped offering free tier Youtube uploads, would be worth it for providing better support to paying users? Would the incredibly massive reduction in uploaded content be worth it? Or do we have to live with these kinds of gaps in order to get the rest?
  - vundercind 2y ago
    I don't think the vacuum left by Google exiting free video hosting would last long. There are lots of potential solutions to the problem, that aren't all very-expensive centralized services, but YouTube existing takes up all the oxygen in the room.
    - sbarre 2y ago
      Wouldn't a new player in the space run into the same challenges though? I don't think this is a Google-specific problem (although they are particularly bad at it).
      - vundercind 2y ago
criddell 2y ago
Do what Microsoft does (or at least used to do). Charge a fee for support and if the problem is due to a bug or poor online help, refund the fee. The risk is of bad incentives when providing support becomes profitable...
some_random 2y ago
I agree completely that it's a really hard problem, but this isn't an unusual case in a non-english language. Accounts are going to be hacked all the time, and all the evidence should be easily available to verify the claim that an attacker uploaded the porn, not the original account holder. There are plenty of other platforms with monstrous support requirements and while none do it perfectly or maybe even well, the popular perception of Google is that they have a policy from on high to not even try. This is across all Google products by the way, ReLogic (developers of Terraria) were locked out of their gmail and canceled their Stadia port over it [1]. I was trying to find a specific story related to GCP, but all I keep running into is people complaining about support [2] [3] [4] These are especially bad because it's affecting customers who are contributing to a very high margin part of the business, and in some cases customers have a line item for the support. Obviously bad experiences get talked about orders of magnitude more than good ones, but Google really is well known for this. [1] https://www.techspot.com/news/88563-re-logic-cancels-terraria-stadia-after-losing-access.html https://www.techspot.com/news/88563-re-logic-cancels-terrari... [2] https://www.reddit.com/r/googlecloud/comments/m3hi63/whats_googles_problem_with_customer_service/ https://www.reddit.com/r/googlecloud/comments/m3hi63/whats_g... [3] https://www.reddit.com/r/googlecloud/comments/1ey0rx8/gcp_support_so_bad/ https://www.reddit.com/r/googlecloud/comments/1ey0rx8/gcp_su... [4] https://www.reddit.com/r/googlecloud/comments/owt679/how_does_gclouds_support_compare_to_awss/ https://www.reddit.com/r/googlecloud/comments/owt679/how_doe...
ksynwa 2y ago
Seems like you want all the benefits of having monopolised video upload but none of the downsides.
papageek 2y ago
All the services are revenue generating. Advertisers pay for eyeballs.
prmoustache 2y ago
Do you have actual numbers on how many accounts are banned and how many appeal processes are triggered every day ? worldwide and by countries?
oidar 2y ago
>> However, I'd like to prompt Hacker News with how would you handle receiving support requests from a product that has >2.7B users. Almost all of which are non-directly revenue generating, across hundreds of different languages, in every conceivable location in the world. 2.7B users is a lot, but how many of those are established content creators (like say - more than 10k subscribers) that are banned on a daily basis? How many people would it take to review those cases?
deleted 2y ago
[deleted]
Brian_K_White 2y ago
Less than 2.7bn:1 user:staff ratio. Do you have any other impossible conundrums I can clear up before coffee?
mikequinlan 2y ago
Is there any reason that governments should allow that business model to exist? Why can't the government require a reasonable level of support, and if that restricts how big a service can get then that is just fine.

gooosle 2y ago

You start by not banning people for nothing.

qmarchi 2y ago
Except for that you risk breaking laws in several jurisdictions if you don't. And you can't fall back to "manual" enforcement think of the massive capital of human resources you'd have to invest in to make it keep up with the upload rate of YouTube.
- gooosle 2y ago
  Give me a single example of laws in any jurisdiction that require this user to be banned. There was absolutely no reason to ban in this case and you know it. If there was one video where rules were unintentionally broken, and the user has no history of that before, you remove the video, not completely ban the user.
- prmoustache 2y ago
  If it is not sustainable, the service shouldn't be run.
- deleted 2y ago
  [deleted]
- CamperBob2 2y ago
  Except for that you risk breaking laws in several jurisdictions if you don't. Why isn't the market addressing this? Why doesn't someone start a video service in the US and for the US -- one that follows US law, and not Pakistan's or Iran's or India's or Germany's or anyone else's? Oh, right, because then they can't sell ads in those places, or use them as tax shelters. Silly question, I guess. By forcing compliance with oppressive regimes (and with pearl-clutching sponsors in less-oppressive ones), advertising will eventually end everything good about the Internet. If you work in that business, you're the problem. Consider a different career.

CamperBob2 2y ago

It's a social problem, not a technical one. Facilitate a hierarchy of trust among users who more-or-less volunteer to moderate at the lower trust levels and engage on unjustly-banned users' behalf with YouTube staff at higher trust levels. Better yet, pay them. It is, after all, work.

andrewinardeer 2y ago
Why pay? Reddit, a multi-billion dollar company has perfected the art of exploiting unpaid volunteer work. So much so that when said workers rebel against the administration they get booted from their position. Moderators are easily replaced as there is always someone willing to toe the administration's line.

tossandthrow 2y ago

It is easy: Allocate the resources needed - hire more customer reps. This would cut into margins, but maybe it is not possible to run hyper scale companies only managed by a couple of engineers. And maybe we should not accept that profit seeking people want to do that anyways.

lolinder 2y ago
> This would cut into margins, but maybe it is not possible to run hyper scale companies only managed by a couple of engineers. It wouldn't cut into margins, it would make YouTube wildly unprofitable, with no viable path to monetization that would ever pay for the support burden. I realize that some on HN—it sounds like you included—are perfectly happy to argue that if a company can't provide human customer support to every one of their users then that company shouldn't exist, but most of YouTube's users would fervently disagree.
- tossandthrow 2y ago
  Clearly, as the parent commentor wrote, they are not able to solve it technically. There is a segment on HN, it sounds like you are included, who believes that is is OK to entirely defer to algorithmic governance without any legal oversigt.
- taormina 2y ago
  That is absolutely hilarious. There is definitely a middle ground between today’s “spend $0 on live human support” and “spend X% of YT revenue on the support a platform like YT demands”. There exists an X% where they are plenty profitable. It’s a video hosting platform subsidized by the largest ad network in the world.
  - lolinder 2y ago
    They don't spend $0 on live customer support, they spend X%. At their scale there will always be high profile stories like this unless and until they spend enough to support every single user who runs into problems with their automated systems. Spending that much is completely cost prohibitive, so we should never expect to reach a point where we stop seeing stories like this.

ryandrake 2y ago

In this case, an entire channel was shut down (with no opportunity for appeal) on account of a single instance of someone zoom-bombing a live stream with pornography. Presumably the decision was entirely automated and there was no human in the loop. It doesn't matter how many users you have. This "solution" seems like swatting a fly with a nuclear weapon. Why not just take down the offending video until the user takes corrective action? YouTube can clearly identify the offending video out of the non-offending ones, so that's not a technical problem. And it can be done entirely with automation, so it wouldn't need humans. Further, they obviously can tell that the user does not have a history or track record of this kind of activity. Why do these tech companies always go straight to the "no recourse ban hammer"?

sbarre 2y ago
Yeah it feels like there should be an escalating series of consequences for infractions.. It should have been pretty straightforward to establish a pattern (or lack of) around whether this was an intentionally abusive channel or a first-time offence. But of course this costs money and takes time to build.
- some_random 2y ago
  They sorta have this already for copyright strikes, but I guess porn was judged to be a more impactful offense?
  - Cthulhu_ 2y ago
    Yes; they cannot take any risks with that, laws around showing porn to people / minors (accidentally or not) are stricter than copyright violations. I'm not a legal expert and honestly I'm pulling this out of my ass, but, porn is a legal matter, but copyright violations is a civil matter. There's also the advertisers and payment processors to consider, most of them will have nothing to do with porn. This triggered some major pruning of various websites too, Tumblr and Reddit on the one side, but huge chunks of Pornhub and co as well for any porn that didn't come with the right paperwork, including identification and consent forms from the participants.

levkk 2y ago

Reddit works just fine. Where there is a will, there is a way. I do believe (without proof) that incentives for Google are misaligned here.

sbarre 2y ago
So while Reddit is huge, I don't think it's even close to the same scale as Google. And problems at Google's scale are hard to compare to smaller scenarios. But I do agree with your 2nd point about misaligned incentives though. I don't think "how do we ensure that every user can get fair support" was ever on any product roadmap for these free global-scale products.. Or more accurately, the "users" in this case are the advertisers, not the uploaders.
some_random 2y ago
Reddit is definitely a different beast, it's easier to admin and has unpaid community moderators who pick up pretty much all the grunt work
qingcharles 2y ago
This isn't correct. I've just been discussing the shadowbanning/appeal system with Huffman because it doesn't work right now.

fakedang 2y ago

Well for starters, cut the issues at source? The article mentions that the channel was taken down because a hacker in the live Zoom meeting (being streamcast into YouTube) played porn. YouTube could have simply blocked that single YT video while retaining the rest of the channel. If multiple instances of users hacking Zoom meetings came to light, Google could simply block Zoom from streamcasting videos into YouTube until they fixed their shit.

vundercind 2y ago

> However, I'd like to prompt Hacker News with how would you handle receiving support requests from a product that has >2.7B users. Almost all of which are non-directly revenue generating, across hundreds of different languages, in every conceivable location in the world. One way: dollars. Another way: don't provide services you can't support. The only way it's actually going to happen, and when it does we'll shockingly find it was always possible and companies just didn't feel like doing it: regulation (which will just result in a mix of the first two options)

sbarre 2y ago
> don't provide services you can't support. Also playing devil's advocate here: Define "support". I'm sure Google will tell you that they support their billions of users just fine, relatively speaking, and that the percentage of people who fall through the cracks is an acceptable margin (to them, obviously not to the users themselves). To your point about "not providing the service", do you believe that the trade-off that would happen if Google, for example, stopped offering free tier Youtube uploads, would be worth it for providing better support to paying users? Would the incredibly massive reduction in uploaded content be worth it? Or do we have to live with these kinds of gaps in order to get the rest?
- vundercind 2y ago
  I don't think the vacuum left by Google exiting free video hosting would last long. There are lots of potential solutions to the problem, that aren't all very-expensive centralized services, but YouTube existing takes up all the oxygen in the room.
  - sbarre 2y ago
    Wouldn't a new player in the space run into the same challenges though? I don't think this is a Google-specific problem (although they are particularly bad at it).
    - vundercind 2y ago

criddell 2y ago

Do what Microsoft does (or at least used to do). Charge a fee for support and if the problem is due to a bug or poor online help, refund the fee. The risk is of bad incentives when providing support becomes profitable...

some_random 2y ago

I agree completely that it's a really hard problem, but this isn't an unusual case in a non-english language. Accounts are going to be hacked all the time, and all the evidence should be easily available to verify the claim that an attacker uploaded the porn, not the original account holder. There are plenty of other platforms with monstrous support requirements and while none do it perfectly or maybe even well, the popular perception of Google is that they have a policy from on high to not even try. This is across all Google products by the way, ReLogic (developers of Terraria) were locked out of their gmail and canceled their Stadia port over it [1]. I was trying to find a specific story related to GCP, but all I keep running into is people complaining about support [2] [3] [4] These are especially bad because it's affecting customers who are contributing to a very high margin part of the business, and in some cases customers have a line item for the support. Obviously bad experiences get talked about orders of magnitude more than good ones, but Google really is well known for this. [1] https://www.techspot.com/news/88563-re-logic-cancels-terraria-stadia-after-losing-access.html https://www.techspot.com/news/88563-re-logic-cancels-terrari... [2] https://www.reddit.com/r/googlecloud/comments/m3hi63/whats_googles_problem_with_customer_service/ https://www.reddit.com/r/googlecloud/comments/m3hi63/whats_g... [3] https://www.reddit.com/r/googlecloud/comments/1ey0rx8/gcp_support_so_bad/ https://www.reddit.com/r/googlecloud/comments/1ey0rx8/gcp_su... [4] https://www.reddit.com/r/googlecloud/comments/owt679/how_does_gclouds_support_compare_to_awss/ https://www.reddit.com/r/googlecloud/comments/owt679/how_doe...

ksynwa 2y ago

Seems like you want all the benefits of having monopolised video upload but none of the downsides.

papageek 2y ago

All the services are revenue generating. Advertisers pay for eyeballs.

prmoustache 2y ago

Do you have actual numbers on how many accounts are banned and how many appeal processes are triggered every day ? worldwide and by countries?

oidar 2y ago

>> However, I'd like to prompt Hacker News with how would you handle receiving support requests from a product that has >2.7B users. Almost all of which are non-directly revenue generating, across hundreds of different languages, in every conceivable location in the world. 2.7B users is a lot, but how many of those are established content creators (like say - more than 10k subscribers) that are banned on a daily basis? How many people would it take to review those cases?

deleted 2y ago

[deleted]

Brian_K_White 2y ago

Less than 2.7bn:1 user:staff ratio. Do you have any other impossible conundrums I can clear up before coffee?

mikequinlan 2y ago

Is there any reason that governments should allow that business model to exist? Why can't the government require a reasonable level of support, and if that restricts how big a service can get then that is just fine.