3 ms·

I'm very excited to see this discussed! Even without pressure from government entities, I would certainly want some sort of way to prevent something like child

I'm very excited to see this discussed! Even without pressure from government entities, I would certainly want some sort of way to prevent something like child abuse content from being uploaded to my server, so I'd love for a system like this to work well. 1. Would the system discussed have anyway to diminish "cancel culture" like attacks? For instance, if a user were to say something egregious like "Empire Strikes Back is bad", what is preventing especially enthusiastic fans from labeling them with having uploaded child abuse content. Presumably that would be a good way to get that user banned from most popular servers. Is there a way to "appeal" this is the moderator or server host is a bad actor? Is having a good enough reputation before an incident like that enough to not be blocked everywhere? 2. Is there a way we could test this outside of Matrix? It seems useful to social platforms, particularly other decentralized ones. If it could be done in a platform independent way, it could be a nice open source collaboration (granted, how people interact on different platforms might be enough to make that impossible to generalize effectively). I also ask, because this sort of thing is really dependent on how people interact, so "play testing" it might be beneficial to getting a more realistic design going early on. Even doing something low-tech might find some issues that should be addressed. Is there a room discussing this in particular? I'd be interested in participating.

Arathorn 6y ago
> 1. Would the system discussed have anyway to diminish "cancel culture" like attacks? I would differentiate between "cancel culture", where folks gang up and react disproportionately to a misdeed - versus a smear campaign where people try to weaponise the moderation system by spreading malicious reputation (e.g. your ESB example). The idea is that both could be solved by having an open ecosystem of relative reputation lists. If a trusted authority (e.g. iwf.org.uk) were to publish a vetted list of hashes of known bad content that you definitely don't want on your server, then you'd obviously expect them to review and verify entries to that list. If they got sloppy and let randoms try to smear each other by submitting ESB-haters to the list, then there would be a scandal and people would stop trusting that reputation list. Alternatively, if ESB-fans started framing ESB-haters with child abuse imagery, then obviously that's a problem for the cops rather than the moderation list verifiers. Alternatively, if some random vigilante decided to start publishing up a reputation feed of people they happen to think are are child abusers, and then started to stuff it with other randoms they happen not to like (whether that's ESB-haters), then the hope is that armed with appropriate visibility on the filters which they are applying, people would spot that this is a disreputable source of reputation asap and run a mile. For instance, you might have accidentally subscribed to the #child-abuse-vigilante reputation feed, and your Matrix client might say "btw, 98% of the rooms in this community have been filtered out by the #child-abuse-vigilante feed data". You might click on the link to check the names of the rooms which have disappeared, and if they turn out actually to have names like "ESB was overrated" then you might choose to dig further and then ring the alarm that #child-abuse-vigilante is not to be trusted, and unsub. You could also publish a reputation feed for reputation feeds (no, really). > 2. Is there a way we could test this outside of Matrix? Absolutely. There is nothing Matrix specific here, or even decentralisation-specific. We're just trying to prove it in Matrix because we have a relatively large community, containing a fairly representative mix of different actors, but we're not so big as to make it hard to experiment - and because this is frankly an existential threat to the long-term success of the project :) The room for discussing this is #matrix-reputation:matrix.org (although public convo is a bit sporadic; this should change as we spin up more experiments in the coming weeks/months however).
- optimuspaul 6y ago
  > Alternatively, if ESB-fans started framing ESB-haters with child abuse imagery, then obviously that's a problem for the cops rather than the moderation list verifiers. Also a problem with ESB fans. Note: I came here because I assumed this was about the ancestor simulation we are all living in, I'm a bit disappointed.
- mirdaki 6y ago
  Thank you for the response! > I would differentiate between "cancel culture", where folks gang up and react disproportionately to a misdeed - versus a smear campaign where people try to weaponise the moderation system by spreading malicious reputation (e.g. your ESB example). Good distinction! > The room for discussing this is #matrix-reputation:matrix.org (although public convo is a bit sporadic; this should change as we spin up more experiments in the coming weeks/months however). Great, I've joined!
- jimmygrapes 6y ago
  Just a drunken musing here... what you describe as (implied to be) unlikely is real in the form of the Anti-Defamation League for other platforms/social behaviors, who have listed individual numbers and letters as "hate symbols" due to obscure uses by extremists. If all it takes to become a hate symbol is for a small segment of the internet to say "what should we make the ADL freak out about today?" how would that sort of "anti-brigading" work? If I see the letters HH together I assume Hulk Hogan, but others (such as the ADL or SLPC) see Heil Hitler. Not sure how this would be defended against, if any group became the de-facto standard arbiter of what constitutes bad.
grey_earthling 6y ago
> For instance, if a user were to say something egregious like "Empire Strikes Back is bad", what is preventing especially enthusiastic fans from labeling them with having uploaded child abuse content. Presumably that would be a good way to get that user banned from most popular servers. Suppose @alice:example.com says “Empire Strikes Back is bad”, and @bob:example.org labels Alice with “uploaded child abuse”. The label attached to @alice:example.com is not “uploaded child abuse” but “uploaded child abuse, according to @bob:example.org”. Perhaps @charlie:example.net discovers this and labels Bob with “false moderator”. Maybe @drew:example.horse labels Bob with “false moderator” and Charlie with “true moderator”. Each server can decide whether they trust Bob, Charlie and/or Drew. Anyone who cares enough can check whether Bob's assertion about Alice was correct, and so they can learn whether to trust Charlie and Drew. This is similar to how blocking works on [scuttlebutt](https://ssb.nz https://ssb.nz), which is subjective by design — if I can see that a few people I trust have blocked a certain identity, and they've left a message saying why, I can choose to trust them and block that identity myself.

Arathorn 6y ago

> 1. Would the system discussed have anyway to diminish "cancel culture" like attacks? I would differentiate between "cancel culture", where folks gang up and react disproportionately to a misdeed - versus a smear campaign where people try to weaponise the moderation system by spreading malicious reputation (e.g. your ESB example). The idea is that both could be solved by having an open ecosystem of relative reputation lists. If a trusted authority (e.g. iwf.org.uk) were to publish a vetted list of hashes of known bad content that you definitely don't want on your server, then you'd obviously expect them to review and verify entries to that list. If they got sloppy and let randoms try to smear each other by submitting ESB-haters to the list, then there would be a scandal and people would stop trusting that reputation list. Alternatively, if ESB-fans started framing ESB-haters with child abuse imagery, then obviously that's a problem for the cops rather than the moderation list verifiers. Alternatively, if some random vigilante decided to start publishing up a reputation feed of people they happen to think are are child abusers, and then started to stuff it with other randoms they happen not to like (whether that's ESB-haters), then the hope is that armed with appropriate visibility on the filters which they are applying, people would spot that this is a disreputable source of reputation asap and run a mile. For instance, you might have accidentally subscribed to the #child-abuse-vigilante reputation feed, and your Matrix client might say "btw, 98% of the rooms in this community have been filtered out by the #child-abuse-vigilante feed data". You might click on the link to check the names of the rooms which have disappeared, and if they turn out actually to have names like "ESB was overrated" then you might choose to dig further and then ring the alarm that #child-abuse-vigilante is not to be trusted, and unsub. You could also publish a reputation feed for reputation feeds (no, really). > 2. Is there a way we could test this outside of Matrix? Absolutely. There is nothing Matrix specific here, or even decentralisation-specific. We're just trying to prove it in Matrix because we have a relatively large community, containing a fairly representative mix of different actors, but we're not so big as to make it hard to experiment - and because this is frankly an existential threat to the long-term success of the project :) The room for discussing this is #matrix-reputation:matrix.org (although public convo is a bit sporadic; this should change as we spin up more experiments in the coming weeks/months however).

optimuspaul 6y ago
> Alternatively, if ESB-fans started framing ESB-haters with child abuse imagery, then obviously that's a problem for the cops rather than the moderation list verifiers. Also a problem with ESB fans. Note: I came here because I assumed this was about the ancestor simulation we are all living in, I'm a bit disappointed.
mirdaki 6y ago
Thank you for the response! > I would differentiate between "cancel culture", where folks gang up and react disproportionately to a misdeed - versus a smear campaign where people try to weaponise the moderation system by spreading malicious reputation (e.g. your ESB example). Good distinction! > The room for discussing this is #matrix-reputation:matrix.org (although public convo is a bit sporadic; this should change as we spin up more experiments in the coming weeks/months however). Great, I've joined!
jimmygrapes 6y ago
Just a drunken musing here... what you describe as (implied to be) unlikely is real in the form of the Anti-Defamation League for other platforms/social behaviors, who have listed individual numbers and letters as "hate symbols" due to obscure uses by extremists. If all it takes to become a hate symbol is for a small segment of the internet to say "what should we make the ADL freak out about today?" how would that sort of "anti-brigading" work? If I see the letters HH together I assume Hulk Hogan, but others (such as the ADL or SLPC) see Heil Hitler. Not sure how this would be defended against, if any group became the de-facto standard arbiter of what constitutes bad.

grey_earthling 6y ago

> For instance, if a user were to say something egregious like "Empire Strikes Back is bad", what is preventing especially enthusiastic fans from labeling them with having uploaded child abuse content. Presumably that would be a good way to get that user banned from most popular servers. Suppose @alice:example.com says “Empire Strikes Back is bad”, and @bob:example.org labels Alice with “uploaded child abuse”. The label attached to @alice:example.com is not “uploaded child abuse” but “uploaded child abuse, according to @bob:example.org”. Perhaps @charlie:example.net discovers this and labels Bob with “false moderator”. Maybe @drew:example.horse labels Bob with “false moderator” and Charlie with “true moderator”. Each server can decide whether they trust Bob, Charlie and/or Drew. Anyone who cares enough can check whether Bob's assertion about Alice was correct, and so they can learn whether to trust Charlie and Drew. This is similar to how blocking works on [scuttlebutt](https://ssb.nz https://ssb.nz), which is subjective by design — if I can see that a few people I trust have blocked a certain identity, and they've left a message saying why, I can choose to trust them and block that identity myself.