4 ms·

Because when you're dealing with a billion users, a one-in-a-million mistake still screws a thousand users, and everyone realizes that getting the rate of mista

Because when you're dealing with a billion users, a one-in-a-million mistake still screws a thousand users, and everyone realizes that getting the rate of mistakes anywhere near that low is impossible. At the same time, you're dealing with actual bad actors that genuinely need to be banned, and there is a lot of this abuse. Mistakes will happen. You can't stop banning users. The bad actors will file appeals (in some cases including public escalations - see the various cases where someone complained about being unfairly banned from a game for cheating, and after a bit of a shitstorm, the company posted the evidence of him cheating). The appeal processes often work, sometimes don't - the public shitstorms are often cases where multiple things went wrong. There is no easy solution, and at the scale these companies operate it's obvious to everyone involved that "just having a highly skilled human review every case" is completely impractical (not just "too expensive for the company to want to pay for it"). Because for each genuine user affected you have many abusers. The pay-for-support proposals have several issues (PR impact from "screwing customers then extorting them to pay", stolen credit cards, engineering required to make it happen). Abuse teams are (understandably) rather tight-lipped, and also tend to insist that telling the user what they did wrong would enable abusers to dodge the protections - this is something I don't understand (the abuser presumably knows what they did, while a user wrongfully accused does not), but it seems to be consistently said by abuse teams from many different companies. All this combined makes it very hard to push for improvement, because there is no clear path towards a solution. You can ask a question at an all-hand generally raising the problem, but you'll get the usual "our abuse teams are working very hard on this, it's a hard problem" non-answer. At the same time, yes, genuine users are absolutely getting screwed, and for the individual affected user, the consequences can be pretty severe.

jsnell 2y ago
(This reply is probably too late, I missed it while scanning this thread in the morning.) > Abuse teams are (understandably) rather tight-lipped, and also tend to insist that telling the user what they did wrong would enable abusers to dodge the protections - this is something I don't understand (the abuser presumably knows what they did, while a user wrongfully accused does not), but it seems to be consistently said by abuse teams from many different companies. The abuser will have done a lot of things. They don't know which one(s) got them caught. The more information they're revealed, the faster they can iterate to avoid getting caught next time. But also, the real user doesn't actually benefit that much from knowing what got them banned. What are they going to do? Go back in time and not do it? Now, a thing the user will greatly benefit from is knowing exactly what (if anything) they need to do to get their account back. That's where the focus on informing users should be. But since the abusers will also be told the same information, this needs to be something that legitimate users will find easy, but abusers or account hijackers will find hard to do at scale. So it can't just be "write a sob story about how you need the account back because it has photos of your dead grandmother". The abusers will be more competent at that than real users. Typically it has to be spending a limited resource, or at least proving you have access to some limited resource and rate-limiting recovery actions by that resource. Some examples: non-VOIP phone numbers, proof of real world identity, social vouching by accounts in good standing.
johnnyanmac 2y ago
>it's obvious to everyone involved that "just having a highly skilled human review every case" is completely impractical (not just "too expensive for the company to want to pay for it"). It's not obvious to me. These are trillion dollar companies and it's not like the appeals process doesn't take weeks to begin with (too long, but I digress). On top of all that you gotta keep in mind that some influencers are literally making the company money. To have them take weeks over a false negative is unaccetptable. So yeah, get a proper review team, make sure they know the actual message that got them banned instead of needing to scour an account, give them a proper minute to review and decide.

jsnell 2y ago

(This reply is probably too late, I missed it while scanning this thread in the morning.) > Abuse teams are (understandably) rather tight-lipped, and also tend to insist that telling the user what they did wrong would enable abusers to dodge the protections - this is something I don't understand (the abuser presumably knows what they did, while a user wrongfully accused does not), but it seems to be consistently said by abuse teams from many different companies. The abuser will have done a lot of things. They don't know which one(s) got them caught. The more information they're revealed, the faster they can iterate to avoid getting caught next time. But also, the real user doesn't actually benefit that much from knowing what got them banned. What are they going to do? Go back in time and not do it? Now, a thing the user will greatly benefit from is knowing exactly what (if anything) they need to do to get their account back. That's where the focus on informing users should be. But since the abusers will also be told the same information, this needs to be something that legitimate users will find easy, but abusers or account hijackers will find hard to do at scale. So it can't just be "write a sob story about how you need the account back because it has photos of your dead grandmother". The abusers will be more competent at that than real users. Typically it has to be spending a limited resource, or at least proving you have access to some limited resource and rate-limiting recovery actions by that resource. Some examples: non-VOIP phone numbers, proof of real world identity, social vouching by accounts in good standing.

johnnyanmac 2y ago

>it's obvious to everyone involved that "just having a highly skilled human review every case" is completely impractical (not just "too expensive for the company to want to pay for it"). It's not obvious to me. These are trillion dollar companies and it's not like the appeals process doesn't take weeks to begin with (too long, but I digress). On top of all that you gotta keep in mind that some influencers are literally making the company money. To have them take weeks over a false negative is unaccetptable. So yeah, get a proper review team, make sure they know the actual message that got them banned instead of needing to scour an account, give them a proper minute to review and decide.