7 ms·

Examples like this show the time is ripe for alternative social media platforms to replace the established networks. Dealing with bad actors in public forums i

Examples like this show the time is ripe for alternative social media platforms to replace the established networks. Dealing with bad actors in public forums is not as hard as tech companies would want you to believe. This is evident, because we do it today. Its our system of courts. Take any platform, and implement a simple e-court system with juries, being random users from the network. The injured party can effectively e-sue , and the pool of randomly users must all agree that abuse took place. This takes care of " No more frivolous takedowns. This also takes care of the pesky catch-all ToS. Platform no longer needs to define abuse. Instead, we take approach of Justice Steward: "I know it when I see it" Juries participating get rewards (badges, recognition, opportunities as judges / advocates, etc) The reason companies do not implement such a system is because it takes power away from them (management, staff) and gives it to the users. That's precisely why such a system would be a winner.

deleted 5y ago
[deleted]
jsnell 5y ago
I can't imagine what could go wrong with showing a random pool of your users suspected CSAM, beheadings, animal cruelty, and so on. Just out of curiosity, what do you plan to do in the unlikely case that you can't get people to do this grueling work for free? Just leave all of that content on the site? It does mean that you won't be able to act on anything except the grossest of abuse. Given the hyper-partisan divide in America, you'd never get 10 people to agree on any case with even a whiff of politics. People will not vote on "is this abusive", they'll vote on "do I agree with this". But, hey it's not all bad. Want some viewers for your video? Here's a growth hacking tip: Just have a friend file an abuse complaint against it, and you'll get it sent into the jury rotation pool.
- IG_Semmelweiss 5y ago
  1) RE: what could go wrong with showing suspected CSAM to users. You are not describing something that doesn't happen in real life. Jury pools are introduced to a summary of the case to precisely ask jurors to recuse themselves if the content of said case is too X. Ejury adjudication would make this incredibly simple via flagging system. "X user has stated this is animal cruelty, and the algo concurrs". So you know what you are dealing with before anyone hits play. 2) RE: No one wants the work. Real-life Jury pools consistently have 100+ candidates for a final 12. E-juries are much better because they can actually scale and can cycle much faster. The world has 7B people. With the right incentives, you can find 10 or so people to take a case on. Plus, you are forgetting that there are also real penalties for losing a case that can act to fend off the undesirables. For example, you could make it so that losing a case means the platform discloses your IP. Makes all your private postings/messages public. Open up the logs of use. Take your pick. There's a million ways to punish degenerates which now do not happen because algos decide and they have such a high false positive rate, to punish would be doubling down on a failed strategy. 3) RE: No agreement except for gross abuse. I have to disagree. 98%+ of people would be in agreement that death, CSAM, animal cruelty etc are offenses that should be dealt with. You already have consensus there. Anything else is in the spectrum of spam / hate / etc. Yet, if it is not beyond reasonable doubt, then its probably not gross. 4) Growth hacking: You seem to forget there are penalties for false accusations in real life. This is no different. If your friend wants to lose access to his own account, he can go right ahead and report a video for the 10 extra views....
  - jsnell 5y ago
    I don't think you quite appreciate what a cesspool the Internet can be, especially when you plan to run a service that's going to be basically unmoderated. The amount of material to review is going to absolutely overwhelm the few volunteers. So what are you doing when there are not enough reviewers? Is your platform going to block that content until it is reviewed, or show it until it is reviewed? Now, whichever option you choose, take a minute to think about how it would be exploited. As for your idea on "real penalties"... Somebody posts abusive material, and as a penalty the platform would distribute more of their material? Harsh. Oh, I sure hope they don't close my throwaway account created from a VPN IP. How could I ever replace it with a new one? People doing this work professionally get psychological counseling for it. You simply could not subject normal users to it. Not morally, not legally, and not if you want to actually run a business. I don't know what kind of support the judicial systems give juries who have to view that kind of material. But at least in those cases there are real consequences if the defendant is found guilty. Not so on the Internet.
- swiley 5y ago
  The problem is that all of the content is siloed. When the services are run by small groups of like-minded people (or people self-hosting) this tends to be less of a problem.
xvector 5y ago
Take that one shotgun suicide viral video incident on TikTok. Do you really think that should be shown to "a jury of random users" to decide whether it is safe to view? Now imagine periodically reviewing videos like this being your 'duty'. In addition to scarring your users for life, they'd be leaving in droves, and you'd be inundated in lawsuits.
- IG_Semmelweiss 5y ago
  1) Tiktok - People actually curate this content already. The only difference is they are staff, and being paid. So maybetake the same compensation, and open it up at scale, so its an improved version of what is already happening for curation. Again, there are people in India paid to do this today. Its not like its not happening you just need to randomize it and spread out responsibility across a wider body. 2) This already happens in real life. I am sure there are mothers serving in jury pools about the murder of a toddler by a parent. The difference is that there are a number of things one could do to manage user preferences in such cases. Also, content would not be as pervasive as implied if penalties are effective at making borderline actors self-moderate, and permanently removing bad actors with real consequences (IP banning, public logs, etc).
  - deleted 5y ago
    [deleted]
  - xvector 5y ago
    I’m sorry, you can’t seriously expect a platform to willingly show users potentially horrific content. That’s just disconnected from reality entirely.
KingOfCoders 5y ago
SaaS rule on the internet "Every payment service becomes like PayPal. Every social network becomes Facebook. Every search engine becomes Google." Most people at these company are fine people. There are forces (revenue, survival instanct, compliance, network effect, competition, being a monopol is nice, ...) that drive these companies. PayPal, Google, Facebook are the end state of the current forces in the market. You do need to remove one or several of those (e.g. revenue driven) to end up in a different end state.
thebigman433 5y ago
This would definitely end up with most users not wanting to sift through whatever case they get "assigned", and the platform effectively being moderated by a big group of power users. You can already see this happening with Reddit. A handful of people control the subs that make up for like 40% of the content on the site. Also this completely breaks down when you start talking about having a jury agree on what is/isnt hate speech or something similar. You are never going to get an agreement on that.
- IG_Semmelweiss 5y ago
  1) Real juries perform exactly as you describe. The vast majority of people want to get out. This is not different 2) That's exactly the point. If a group of random people can't agree that some post is hate speech, then its not hate speech beyond reasonable doubt.
  - vkou 5y ago
    > 2) That's exactly the point. If a group of random people can't agree that some post is hate speech, then its not hate speech beyond reasonable doubt. Real juries have a selection process that it supposed to weed out biased individuals. Real juries have judges that steer them towards a particular way of viewing evidence. Real juries are forbidden from doing their own research, and can only passively listen to lawyers make arguments.
  - thebigman433 5y ago
    > 1) Real juries perform exactly as you describe. The vast majority of people want to get out. This is not different Its a requirement in real life. If you try making it a requirement to use your platform, I dont see why any users would stay. I think users will overwhelmingly prefer to have the content curated for them, so they dont have to look at child porn and beheadings every week for the opportunity to use social media. > 2) That's exactly the point. If a group of random people can't agree that some post is hate speech, then its not hate speech beyond reasonable doubt. This sounds like an incredibly quick way to make your platform just a collection of the worst parts of the internet. A place with a TOS determined by the users where you need an overwhelming majority[1] to convict anything isnt going to last long with people who just wanna share some pictures with friends. [1] Your post says unanimous but Ill give it the benefit of the doubt and assume thats not literal, since getting a unanimous decision is essentially impossible on the internet.
nradov 5y ago
No thanks, I don't want my content judged by random people from Pakistan or Venezuela. That seems worse than the current moderation system.
curryst 5y ago
I like the idea, but it would need some refinement. > Juries participating get rewards (badges, recognition, opportunities as judges / advocates, etc) There are also external rewards that need to be considered. For example, being a juror because you want to silence someone. I wouldn't put it past 4chan to try something like that. > The injured party can effectively e-sue Without any kind of cost or risk, this will get abused. Remember that it takes 1 person to make a report, and 12 people to form the jury. It doesn't take much to flood the "court", and then there's effectively no moderation. > The reason companies do not implement such a system is because it takes power away from them (management, staff) and gives it to the users. I don't think it's directly about power. It's largely about advertising money. They need to make sure advertisers are okay with the ads being run next to the content. Advertisers will not care that a jury voted to keep a racist tirade.
foxhill 5y ago
this approach is used by the game, Counter-Strike: Global Offensive. the system is called Overwatch (not to be confused with the game of the same name..). how it works: once a player has received enough reports for suspicious behaviour (i.e., cheating), or bad behaviour (griefing, throwing, etc.), a replay of the match is submitted to tens of other players. these players review the "demo", decide on the likelihood of the "suspect" being guilty of those offenses. if all reviewers come to a unanimous decision, then the suspect receives a ban. the system is thoroughly abused[1] by hackers/cheaters who are able to spam reports for players they don't like, and by controlling hundreds, if not thousands, of bot accounts, can reach unanimous verdicts that no normal person would decide, inflicting bans on innocent players. [1] - https://www.youtube.com/watch?v=N0OiOiqmi-c https://www.youtube.com/watch?v=N0OiOiqmi-c
- esyir 5y ago
  So, I think the base concept is fine, but requires oversight. It's an augmentation of anticheat, not a replacement. Hidden reliability scores should aid the classification of "judges". Misclassify often? Get downprioritized. As CS is a paid game, the typical issue of mass account creation to game this should be at least slightly mitigated. Anticheat has to be defense in depth.
  - foxhill 5y ago
    well, therein lies the issue: this approach, one of scoring jurors (i think that’s a more appropriate term :), already happens. almost exactly as you describe. defense in depth, indeed: Overwatch is one of three systems, the other two being VAC, and Trust. the former being more traditional cheat detection (which, empirically, is not effective at all), and the latter being an observation of a user’s interaction on the entire steam platform, amongst other things, to determine a “trust factor”. anecdotally, myself and many others can tell you that this is also ineffective. for what it’s worth, 2020 and by the looks of it, 2021, will go down in CSGO’s history as the years that the cheaters won.
    - esyir 5y ago
yosito 5y ago
> Examples like this show the time is ripe for alternative social media platforms to replace the established networks. AFAICT social media replaced "the established networks" a decade ago.
mycologos 5y ago
This idea seems to ignore two things: 1) jury duty is maybe 1 day per 1-2 years in America for most people, and it's still widely disliked. In my experience, it's the most reliable place you can go and find adults acting like children who want to get out of school early. I'm not saying that's unreasonable, but I am saying that most people don't like expending their time on it. 2) A possible response to point 1) is that the frictions involved in virtual jury duty are much lower than those of real-world jury duty. But, this cuts both ways. One reason our legal system works is that people can't just automatically spam it with garbage claims (well, they can, but it requires spending time and resources). This defense doesn't apply when the whole thing is online. Heuristics for automatic defense of the jury system then run into the same problems of automatic moderating. Overall, I agree that it's unfortunate that automated moderating makes so many high-profile screw-ups. But I'm also confused by how many people are sure that it's because tech companies are ignoring obvious solutions.

deleted 5y ago

[deleted]

jsnell 5y ago

I can't imagine what could go wrong with showing a random pool of your users suspected CSAM, beheadings, animal cruelty, and so on. Just out of curiosity, what do you plan to do in the unlikely case that you can't get people to do this grueling work for free? Just leave all of that content on the site? It does mean that you won't be able to act on anything except the grossest of abuse. Given the hyper-partisan divide in America, you'd never get 10 people to agree on any case with even a whiff of politics. People will not vote on "is this abusive", they'll vote on "do I agree with this". But, hey it's not all bad. Want some viewers for your video? Here's a growth hacking tip: Just have a friend file an abuse complaint against it, and you'll get it sent into the jury rotation pool.

IG_Semmelweiss 5y ago
1) RE: what could go wrong with showing suspected CSAM to users. You are not describing something that doesn't happen in real life. Jury pools are introduced to a summary of the case to precisely ask jurors to recuse themselves if the content of said case is too X. Ejury adjudication would make this incredibly simple via flagging system. "X user has stated this is animal cruelty, and the algo concurrs". So you know what you are dealing with before anyone hits play. 2) RE: No one wants the work. Real-life Jury pools consistently have 100+ candidates for a final 12. E-juries are much better because they can actually scale and can cycle much faster. The world has 7B people. With the right incentives, you can find 10 or so people to take a case on. Plus, you are forgetting that there are also real penalties for losing a case that can act to fend off the undesirables. For example, you could make it so that losing a case means the platform discloses your IP. Makes all your private postings/messages public. Open up the logs of use. Take your pick. There's a million ways to punish degenerates which now do not happen because algos decide and they have such a high false positive rate, to punish would be doubling down on a failed strategy. 3) RE: No agreement except for gross abuse. I have to disagree. 98%+ of people would be in agreement that death, CSAM, animal cruelty etc are offenses that should be dealt with. You already have consensus there. Anything else is in the spectrum of spam / hate / etc. Yet, if it is not beyond reasonable doubt, then its probably not gross. 4) Growth hacking: You seem to forget there are penalties for false accusations in real life. This is no different. If your friend wants to lose access to his own account, he can go right ahead and report a video for the 10 extra views....
- jsnell 5y ago
  I don't think you quite appreciate what a cesspool the Internet can be, especially when you plan to run a service that's going to be basically unmoderated. The amount of material to review is going to absolutely overwhelm the few volunteers. So what are you doing when there are not enough reviewers? Is your platform going to block that content until it is reviewed, or show it until it is reviewed? Now, whichever option you choose, take a minute to think about how it would be exploited. As for your idea on "real penalties"... Somebody posts abusive material, and as a penalty the platform would distribute more of their material? Harsh. Oh, I sure hope they don't close my throwaway account created from a VPN IP. How could I ever replace it with a new one? People doing this work professionally get psychological counseling for it. You simply could not subject normal users to it. Not morally, not legally, and not if you want to actually run a business. I don't know what kind of support the judicial systems give juries who have to view that kind of material. But at least in those cases there are real consequences if the defendant is found guilty. Not so on the Internet.
swiley 5y ago
The problem is that all of the content is siloed. When the services are run by small groups of like-minded people (or people self-hosting) this tends to be less of a problem.

xvector 5y ago

Take that one shotgun suicide viral video incident on TikTok. Do you really think that should be shown to "a jury of random users" to decide whether it is safe to view? Now imagine periodically reviewing videos like this being your 'duty'. In addition to scarring your users for life, they'd be leaving in droves, and you'd be inundated in lawsuits.

IG_Semmelweiss 5y ago
1) Tiktok - People actually curate this content already. The only difference is they are staff, and being paid. So maybetake the same compensation, and open it up at scale, so its an improved version of what is already happening for curation. Again, there are people in India paid to do this today. Its not like its not happening you just need to randomize it and spread out responsibility across a wider body. 2) This already happens in real life. I am sure there are mothers serving in jury pools about the murder of a toddler by a parent. The difference is that there are a number of things one could do to manage user preferences in such cases. Also, content would not be as pervasive as implied if penalties are effective at making borderline actors self-moderate, and permanently removing bad actors with real consequences (IP banning, public logs, etc).
- deleted 5y ago
  [deleted]
- xvector 5y ago
  I’m sorry, you can’t seriously expect a platform to willingly show users potentially horrific content. That’s just disconnected from reality entirely.

KingOfCoders 5y ago

SaaS rule on the internet "Every payment service becomes like PayPal. Every social network becomes Facebook. Every search engine becomes Google." Most people at these company are fine people. There are forces (revenue, survival instanct, compliance, network effect, competition, being a monopol is nice, ...) that drive these companies. PayPal, Google, Facebook are the end state of the current forces in the market. You do need to remove one or several of those (e.g. revenue driven) to end up in a different end state.

thebigman433 5y ago

This would definitely end up with most users not wanting to sift through whatever case they get "assigned", and the platform effectively being moderated by a big group of power users. You can already see this happening with Reddit. A handful of people control the subs that make up for like 40% of the content on the site. Also this completely breaks down when you start talking about having a jury agree on what is/isnt hate speech or something similar. You are never going to get an agreement on that.

IG_Semmelweiss 5y ago
1) Real juries perform exactly as you describe. The vast majority of people want to get out. This is not different 2) That's exactly the point. If a group of random people can't agree that some post is hate speech, then its not hate speech beyond reasonable doubt.
- vkou 5y ago
  > 2) That's exactly the point. If a group of random people can't agree that some post is hate speech, then its not hate speech beyond reasonable doubt. Real juries have a selection process that it supposed to weed out biased individuals. Real juries have judges that steer them towards a particular way of viewing evidence. Real juries are forbidden from doing their own research, and can only passively listen to lawyers make arguments.
- thebigman433 5y ago
  > 1) Real juries perform exactly as you describe. The vast majority of people want to get out. This is not different Its a requirement in real life. If you try making it a requirement to use your platform, I dont see why any users would stay. I think users will overwhelmingly prefer to have the content curated for them, so they dont have to look at child porn and beheadings every week for the opportunity to use social media. > 2) That's exactly the point. If a group of random people can't agree that some post is hate speech, then its not hate speech beyond reasonable doubt. This sounds like an incredibly quick way to make your platform just a collection of the worst parts of the internet. A place with a TOS determined by the users where you need an overwhelming majority[1] to convict anything isnt going to last long with people who just wanna share some pictures with friends. [1] Your post says unanimous but Ill give it the benefit of the doubt and assume thats not literal, since getting a unanimous decision is essentially impossible on the internet.

nradov 5y ago

No thanks, I don't want my content judged by random people from Pakistan or Venezuela. That seems worse than the current moderation system.

curryst 5y ago

I like the idea, but it would need some refinement. > Juries participating get rewards (badges, recognition, opportunities as judges / advocates, etc) There are also external rewards that need to be considered. For example, being a juror because you want to silence someone. I wouldn't put it past 4chan to try something like that. > The injured party can effectively e-sue Without any kind of cost or risk, this will get abused. Remember that it takes 1 person to make a report, and 12 people to form the jury. It doesn't take much to flood the "court", and then there's effectively no moderation. > The reason companies do not implement such a system is because it takes power away from them (management, staff) and gives it to the users. I don't think it's directly about power. It's largely about advertising money. They need to make sure advertisers are okay with the ads being run next to the content. Advertisers will not care that a jury voted to keep a racist tirade.

foxhill 5y ago

this approach is used by the game, Counter-Strike: Global Offensive. the system is called Overwatch (not to be confused with the game of the same name..). how it works: once a player has received enough reports for suspicious behaviour (i.e., cheating), or bad behaviour (griefing, throwing, etc.), a replay of the match is submitted to tens of other players. these players review the "demo", decide on the likelihood of the "suspect" being guilty of those offenses. if all reviewers come to a unanimous decision, then the suspect receives a ban. the system is thoroughly abused[1] by hackers/cheaters who are able to spam reports for players they don't like, and by controlling hundreds, if not thousands, of bot accounts, can reach unanimous verdicts that no normal person would decide, inflicting bans on innocent players. [1] - https://www.youtube.com/watch?v=N0OiOiqmi-c https://www.youtube.com/watch?v=N0OiOiqmi-c

esyir 5y ago
So, I think the base concept is fine, but requires oversight. It's an augmentation of anticheat, not a replacement. Hidden reliability scores should aid the classification of "judges". Misclassify often? Get downprioritized. As CS is a paid game, the typical issue of mass account creation to game this should be at least slightly mitigated. Anticheat has to be defense in depth.
- foxhill 5y ago
  well, therein lies the issue: this approach, one of scoring jurors (i think that’s a more appropriate term :), already happens. almost exactly as you describe. defense in depth, indeed: Overwatch is one of three systems, the other two being VAC, and Trust. the former being more traditional cheat detection (which, empirically, is not effective at all), and the latter being an observation of a user’s interaction on the entire steam platform, amongst other things, to determine a “trust factor”. anecdotally, myself and many others can tell you that this is also ineffective. for what it’s worth, 2020 and by the looks of it, 2021, will go down in CSGO’s history as the years that the cheaters won.
  - esyir 5y ago

yosito 5y ago

> Examples like this show the time is ripe for alternative social media platforms to replace the established networks. AFAICT social media replaced "the established networks" a decade ago.

mycologos 5y ago

This idea seems to ignore two things: 1) jury duty is maybe 1 day per 1-2 years in America for most people, and it's still widely disliked. In my experience, it's the most reliable place you can go and find adults acting like children who want to get out of school early. I'm not saying that's unreasonable, but I am saying that most people don't like expending their time on it. 2) A possible response to point 1) is that the frictions involved in virtual jury duty are much lower than those of real-world jury duty. But, this cuts both ways. One reason our legal system works is that people can't just automatically spam it with garbage claims (well, they can, but it requires spending time and resources). This defense doesn't apply when the whole thing is online. Heuristics for automatic defense of the jury system then run into the same problems of automatic moderating. Overall, I agree that it's unfortunate that automated moderating makes so many high-profile screw-ups. But I'm also confused by how many people are sure that it's because tech companies are ignoring obvious solutions.