3 ms·

I can't imagine what could go wrong with showing a random pool of your users suspected CSAM, beheadings, animal cruelty, and so on. Just out of curiosity, what

by jsnell 5y ago

I can't imagine what could go wrong with showing a random pool of your users suspected CSAM, beheadings, animal cruelty, and so on. Just out of curiosity, what do you plan to do in the unlikely case that you can't get people to do this grueling work for free? Just leave all of that content on the site? It does mean that you won't be able to act on anything except the grossest of abuse. Given the hyper-partisan divide in America, you'd never get 10 people to agree on any case with even a whiff of politics. People will not vote on "is this abusive", they'll vote on "do I agree with this". But, hey it's not all bad. Want some viewers for your video? Here's a growth hacking tip: Just have a friend file an abuse complaint against it, and you'll get it sent into the jury rotation pool.

IG_Semmelweiss 5y ago
1) RE: what could go wrong with showing suspected CSAM to users. You are not describing something that doesn't happen in real life. Jury pools are introduced to a summary of the case to precisely ask jurors to recuse themselves if the content of said case is too X. Ejury adjudication would make this incredibly simple via flagging system. "X user has stated this is animal cruelty, and the algo concurrs". So you know what you are dealing with before anyone hits play. 2) RE: No one wants the work. Real-life Jury pools consistently have 100+ candidates for a final 12. E-juries are much better because they can actually scale and can cycle much faster. The world has 7B people. With the right incentives, you can find 10 or so people to take a case on. Plus, you are forgetting that there are also real penalties for losing a case that can act to fend off the undesirables. For example, you could make it so that losing a case means the platform discloses your IP. Makes all your private postings/messages public. Open up the logs of use. Take your pick. There's a million ways to punish degenerates which now do not happen because algos decide and they have such a high false positive rate, to punish would be doubling down on a failed strategy. 3) RE: No agreement except for gross abuse. I have to disagree. 98%+ of people would be in agreement that death, CSAM, animal cruelty etc are offenses that should be dealt with. You already have consensus there. Anything else is in the spectrum of spam / hate / etc. Yet, if it is not beyond reasonable doubt, then its probably not gross. 4) Growth hacking: You seem to forget there are penalties for false accusations in real life. This is no different. If your friend wants to lose access to his own account, he can go right ahead and report a video for the 10 extra views....
- jsnell 5y ago
  I don't think you quite appreciate what a cesspool the Internet can be, especially when you plan to run a service that's going to be basically unmoderated. The amount of material to review is going to absolutely overwhelm the few volunteers. So what are you doing when there are not enough reviewers? Is your platform going to block that content until it is reviewed, or show it until it is reviewed? Now, whichever option you choose, take a minute to think about how it would be exploited. As for your idea on "real penalties"... Somebody posts abusive material, and as a penalty the platform would distribute more of their material? Harsh. Oh, I sure hope they don't close my throwaway account created from a VPN IP. How could I ever replace it with a new one? People doing this work professionally get psychological counseling for it. You simply could not subject normal users to it. Not morally, not legally, and not if you want to actually run a business. I don't know what kind of support the judicial systems give juries who have to view that kind of material. But at least in those cases there are real consequences if the defendant is found guilty. Not so on the Internet.
swiley 5y ago
The problem is that all of the content is siloed. When the services are run by small groups of like-minded people (or people self-hosting) this tends to be less of a problem.

by jsnell 5y ago

IG_Semmelweiss 5y ago

1) RE: what could go wrong with showing suspected CSAM to users. You are not describing something that doesn't happen in real life. Jury pools are introduced to a summary of the case to precisely ask jurors to recuse themselves if the content of said case is too X. Ejury adjudication would make this incredibly simple via flagging system. "X user has stated this is animal cruelty, and the algo concurrs". So you know what you are dealing with before anyone hits play. 2) RE: No one wants the work. Real-life Jury pools consistently have 100+ candidates for a final 12. E-juries are much better because they can actually scale and can cycle much faster. The world has 7B people. With the right incentives, you can find 10 or so people to take a case on. Plus, you are forgetting that there are also real penalties for losing a case that can act to fend off the undesirables. For example, you could make it so that losing a case means the platform discloses your IP. Makes all your private postings/messages public. Open up the logs of use. Take your pick. There's a million ways to punish degenerates which now do not happen because algos decide and they have such a high false positive rate, to punish would be doubling down on a failed strategy. 3) RE: No agreement except for gross abuse. I have to disagree. 98%+ of people would be in agreement that death, CSAM, animal cruelty etc are offenses that should be dealt with. You already have consensus there. Anything else is in the spectrum of spam / hate / etc. Yet, if it is not beyond reasonable doubt, then its probably not gross. 4) Growth hacking: You seem to forget there are penalties for false accusations in real life. This is no different. If your friend wants to lose access to his own account, he can go right ahead and report a video for the 10 extra views....

jsnell 5y ago
I don't think you quite appreciate what a cesspool the Internet can be, especially when you plan to run a service that's going to be basically unmoderated. The amount of material to review is going to absolutely overwhelm the few volunteers. So what are you doing when there are not enough reviewers? Is your platform going to block that content until it is reviewed, or show it until it is reviewed? Now, whichever option you choose, take a minute to think about how it would be exploited. As for your idea on "real penalties"... Somebody posts abusive material, and as a penalty the platform would distribute more of their material? Harsh. Oh, I sure hope they don't close my throwaway account created from a VPN IP. How could I ever replace it with a new one? People doing this work professionally get psychological counseling for it. You simply could not subject normal users to it. Not morally, not legally, and not if you want to actually run a business. I don't know what kind of support the judicial systems give juries who have to view that kind of material. But at least in those cases there are real consequences if the defendant is found guilty. Not so on the Internet.

swiley 5y ago

The problem is that all of the content is siloed. When the services are run by small groups of like-minded people (or people self-hosting) this tends to be less of a problem.