5 ms·

I've had to give this some thought for other reasons, and after a couple decades solving analogous problems to moderation in security, I agree with yishan about

I've had to give this some thought for other reasons, and after a couple decades solving analogous problems to moderation in security, I agree with yishan about signal to noise over the specific content, but what I have effectively spent a career studying and detecting with data is a single factor: malice. It's something every person is capable of, and it takes a lot of exercise and practice with higher values to reach for something else when your expectations are challenged, and often it's an active choice to recognize the urge and act differently. If there were a rule or razor I would make on a forum or platform, it's that all content has to pass the bar of being without malice. It's not "assume good intent," it's recognizing that there are ways of having very difficult opinions without malice, and one can have conventional views that are malicious, and unconventional ones that are not. If you have ever dealt with a prosecutor or been on the wrong side of a legal dispute, these are people fundamentally actuated by malice, and the similar prosecution of ideas and opinions (and ultimately people) is what wrecks a forum. It's not about being polite or civil, avoiding conflict, or even avoiding mockery and some very funny and unexpected smackdowns either. It's a quality that in being universally capable of it, I think we're also able to know it when we see it. "Hate," is a weak substitute because it is so vague we can apply it to anything, but malice is ancient and essential. Of course someone malicious can just redefine malice the way they have done other things and use it as an accusation because words have no meaning other than as a means in struggle, but really, you can see when someone is actuated by it. I think there is a point where a person decides, consciously or not, that they will relate to the world around them with malice, and the first casulty of that is an alignment to honesty and truth. What makes it useful is that you can address malice directly and restore an equillibrium in the discourse, whereas accusations of hate and others are irrevocable judgments. I'd wonder if given it's applicability, this may be the tool.

dang 4y ago
For me, malice relates to intent. Intent isn't observable. When person X makes a claim about Y's intent, they're almost always filling in invisible gaps using their imagination. You can't moderate on that basis. We have to go by effects, not intent (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20effects%20intent&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...). It took me a long time to learn that if I tell a user "you were being $FOO" where $FOO relates to their intent, they can simply say "no I wasn't" and no one can prove otherwise, making the moderation position a weak one. Mostly they will deny it sincerely because they never had such an intent, at least not consciously. If you do that as a mod, you've just given them a reason to feel entirely in the right, and if you proceed to moderate them anyway, they will feel treated unjustly. This is a way to generate bad blood, make enemies, and lose the high ground. The reverse strategy is better: describe the effects of someone's posts and explain why they are bad. When inevitably they respond with "but my intent was $BAR", the answer is "I believe you [what else can you say about something only that person could know?], but nonetheless the effects were $BAZ and we have to moderate based on effects. Intent doesn't communicate itself—the burden is on the commenter to disambiguate it." (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=burden%20disambiguate%20by%3Adang&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...) Often when people get moderated in this way, they respond by writing the comment they originally had in their head, as a defense of what they actually meant. It's often astonishing what a gap there is between the two. Then you can respond that if they had posted that in the first place, it would have been fine, and that while they know what they have in their head while posting, the rest of us have no access to that—it needs to be spelled out explicitly. Being able to tell someone "if you had posted that in the first place, it would have been fine" is a strong moderation position, because it takes off the table the idea "you're only moderating me because you dislike my opinions", which is otherwise ubiquitous.
- Eliezer 4y ago
  This exchange ought to be a post in its own right. It seems to me that malice, hate, Warp contamination, whatever you want to call it, is very much a large part of the modern problem; and also it's a true and deep statement that you should moderate based on effects and not tell anyone what their inner intentions were, because you aren't sure of those and most users won't know them either.
- motohagiography 4y ago
  This aspect of people writing what they meant again after being challenged and it being different - I'd assert that when there is malice (or another intent) present, they double down or use other tactics toward a specific end other than improving the forum or relationship they are contributing to. When there is none, you get that different or broader answer, which is really often worth it. However, yes it is intent, as you identify. I have heard the view that intent is not observable, and I agree with the link examples that the effect of a comment is the best available heuristic. It is also consistent with a lot of other necessary and altruistic principles to say it's not knowable. On detecting malice from data, however, the security business is predicated on detecting intent from network data, so while it's not perfect, there are precedents for (more-) structured data. I might refine it to say that intent is not passively observable in a reliable way, as if you interrogate the source, we get revealed intent. On the intent taking place in the imagination of the observer, that's a deep question. I think I have reasonably been called out on some of my views being the artifacts of the logic of underlying ideas that may not have been apparent to me. I've also challenged authors with the same criticism, where I think there are ideas that are sincere, and ones that are artifacts of exogenous intent and the logic of other ideas, and that there is a way of telling the difference by interrogating the idea (via the person.) I even agree with the principle of not assuming malice, but professionally, my job has been to assess it from indirect structured data (a hawkish, is this an attack?) - whereas I interpret the moderator role as assessing intent directly by its effects, but from unstructured data (is this comment/person causing harm?). Malice is the example I used because I think it has persisted in roughly its same meaning since the earliest writing, and if that subset of effectively 'evil' intent only existed in the imaginations of its observers, there's a continuity of imagination and false consciousness about their relationship to the world that would be pretty radical. I think it's right to not assume malice, but fatal to deny it. Perhaps there is a more concrete path to take than my conflating it with the problem of evil, even if on these discussions of global platform rules, it seems like a useful source of prior art?
danans 4y ago
> "Hate," is a weak substitute because it is so vague we can apply it to anything That is a big stretch. Hate can't be applied to many things, including disagreements like this comment. But it can be pretty clearly applied to statements that, if carried out in life, would deny another person or peoples' human rights. Another is denigration or mocking someone on the basis of things that can't or shouldn't have to change about themselves, like their race or religion. There is a pretty bright line there. Malice (per the conventional meaning of something bad intended, but not necessarily revealed or acted out) is a much lower bar that includes outright hate speech. > but really, you can see when someone is actuated by it. How can you identify this systematically (vs it being just your opinion), but not identify hate speech?
- Manuel_D 4y ago
  Hate absolutely can, and is, applied to disagreements: Plenty of people consider disagreement around allowing natal males in women's sport is hateful. Plenty of people consider opposition to the abolishment of police is hateful. Plenty of people immigration enforcement hateful. I could go on...
  - bombcar 4y ago
    Any disagreement is classified as hate now; the word is empty and worthless. We cannot regulate the internal forum, only the actions we perceive.
  - danans 4y ago
    > Plenty of people consider disagreement around allowing natal males in women's sport is hateful. Plenty of people consider opposition to the abolishment of police is hateful. Plenty of people immigration enforcement hateful. Those things aren't deemed hate speech, but they might be disagreed with and downvoted on some forums (i.e. HN), and championed on others (i.e. Parler) but that has nothing to do with them being hate speech. They are just unpopular opinions in some places, and I can understand how it might bother you if those are your beliefs and you get downvoted. Actual hate speech based on your examples is: promoting violence/harassment against non-cisgender people, promoting violence/harassment by police, and promoting violence/harassment by immigration authorities against migrants. Promoting violence and harassment is a fundamentally different type of speech than disagreeing with the prevailing local opinion on a controversial subject that has many shades of gray (that your examples intentionally lack).
nostrebored 4y ago
I find most forums that advocate against behavior they view as malicious wind up becoming hugboxes as people skirt this arbitrary boundary. I will never, never come back to platforms or groups after I get this feeling. Hugbox environments wind up having a loose relationship with the truth and a strong emphasis on emotional well-being. Setting your moderation boundaries determines the values of your platform. I’d much rather talk to someone who wants to hurt my feelings than someone who is detached from reality or saying what they think.
creeble 4y ago
I think it's two things: the power of malice, and popularity measurement. Malice and fame. Social networks are devices for measuring popularity; if you took the up/down arrows off, no one would be interested in playing. And we have proven once again that nothing gets up arrows like being mean. HN has the unusual property that you can't (readily) see others' score, just your own. That doesn't really make it any less about fame, but maybe it helps. When advertising can fund these devices to scale to billions, it's tough to be optimistic about how it reflects human nature.
INTPenis 4y ago
I don't think it's as simple as malice either because the people who get worked up about online debate often believe they are doing the right thing, they believe they are trying to save someone else from thinking the wrong thing. There's no malice there.

dang 4y ago

For me, malice relates to intent. Intent isn't observable. When person X makes a claim about Y's intent, they're almost always filling in invisible gaps using their imagination. You can't moderate on that basis. We have to go by effects, not intent (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20effects%20intent&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...). It took me a long time to learn that if I tell a user "you were being $FOO" where $FOO relates to their intent, they can simply say "no I wasn't" and no one can prove otherwise, making the moderation position a weak one. Mostly they will deny it sincerely because they never had such an intent, at least not consciously. If you do that as a mod, you've just given them a reason to feel entirely in the right, and if you proceed to moderate them anyway, they will feel treated unjustly. This is a way to generate bad blood, make enemies, and lose the high ground. The reverse strategy is better: describe the effects of someone's posts and explain why they are bad. When inevitably they respond with "but my intent was $BAR", the answer is "I believe you [what else can you say about something only that person could know?], but nonetheless the effects were $BAZ and we have to moderate based on effects. Intent doesn't communicate itself—the burden is on the commenter to disambiguate it." (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=burden%20disambiguate%20by%3Adang&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...) Often when people get moderated in this way, they respond by writing the comment they originally had in their head, as a defense of what they actually meant. It's often astonishing what a gap there is between the two. Then you can respond that if they had posted that in the first place, it would have been fine, and that while they know what they have in their head while posting, the rest of us have no access to that—it needs to be spelled out explicitly. Being able to tell someone "if you had posted that in the first place, it would have been fine" is a strong moderation position, because it takes off the table the idea "you're only moderating me because you dislike my opinions", which is otherwise ubiquitous.

Eliezer 4y ago
This exchange ought to be a post in its own right. It seems to me that malice, hate, Warp contamination, whatever you want to call it, is very much a large part of the modern problem; and also it's a true and deep statement that you should moderate based on effects and not tell anyone what their inner intentions were, because you aren't sure of those and most users won't know them either.
motohagiography 4y ago
This aspect of people writing what they meant again after being challenged and it being different - I'd assert that when there is malice (or another intent) present, they double down or use other tactics toward a specific end other than improving the forum or relationship they are contributing to. When there is none, you get that different or broader answer, which is really often worth it. However, yes it is intent, as you identify. I have heard the view that intent is not observable, and I agree with the link examples that the effect of a comment is the best available heuristic. It is also consistent with a lot of other necessary and altruistic principles to say it's not knowable. On detecting malice from data, however, the security business is predicated on detecting intent from network data, so while it's not perfect, there are precedents for (more-) structured data. I might refine it to say that intent is not passively observable in a reliable way, as if you interrogate the source, we get revealed intent. On the intent taking place in the imagination of the observer, that's a deep question. I think I have reasonably been called out on some of my views being the artifacts of the logic of underlying ideas that may not have been apparent to me. I've also challenged authors with the same criticism, where I think there are ideas that are sincere, and ones that are artifacts of exogenous intent and the logic of other ideas, and that there is a way of telling the difference by interrogating the idea (via the person.) I even agree with the principle of not assuming malice, but professionally, my job has been to assess it from indirect structured data (a hawkish, is this an attack?) - whereas I interpret the moderator role as assessing intent directly by its effects, but from unstructured data (is this comment/person causing harm?). Malice is the example I used because I think it has persisted in roughly its same meaning since the earliest writing, and if that subset of effectively 'evil' intent only existed in the imaginations of its observers, there's a continuity of imagination and false consciousness about their relationship to the world that would be pretty radical. I think it's right to not assume malice, but fatal to deny it. Perhaps there is a more concrete path to take than my conflating it with the problem of evil, even if on these discussions of global platform rules, it seems like a useful source of prior art?

danans 4y ago

> "Hate," is a weak substitute because it is so vague we can apply it to anything That is a big stretch. Hate can't be applied to many things, including disagreements like this comment. But it can be pretty clearly applied to statements that, if carried out in life, would deny another person or peoples' human rights. Another is denigration or mocking someone on the basis of things that can't or shouldn't have to change about themselves, like their race or religion. There is a pretty bright line there. Malice (per the conventional meaning of something bad intended, but not necessarily revealed or acted out) is a much lower bar that includes outright hate speech. > but really, you can see when someone is actuated by it. How can you identify this systematically (vs it being just your opinion), but not identify hate speech?

Manuel_D 4y ago
Hate absolutely can, and is, applied to disagreements: Plenty of people consider disagreement around allowing natal males in women's sport is hateful. Plenty of people consider opposition to the abolishment of police is hateful. Plenty of people immigration enforcement hateful. I could go on...
- bombcar 4y ago
  Any disagreement is classified as hate now; the word is empty and worthless. We cannot regulate the internal forum, only the actions we perceive.
- danans 4y ago
  > Plenty of people consider disagreement around allowing natal males in women's sport is hateful. Plenty of people consider opposition to the abolishment of police is hateful. Plenty of people immigration enforcement hateful. Those things aren't deemed hate speech, but they might be disagreed with and downvoted on some forums (i.e. HN), and championed on others (i.e. Parler) but that has nothing to do with them being hate speech. They are just unpopular opinions in some places, and I can understand how it might bother you if those are your beliefs and you get downvoted. Actual hate speech based on your examples is: promoting violence/harassment against non-cisgender people, promoting violence/harassment by police, and promoting violence/harassment by immigration authorities against migrants. Promoting violence and harassment is a fundamentally different type of speech than disagreeing with the prevailing local opinion on a controversial subject that has many shades of gray (that your examples intentionally lack).

nostrebored 4y ago

I find most forums that advocate against behavior they view as malicious wind up becoming hugboxes as people skirt this arbitrary boundary. I will never, never come back to platforms or groups after I get this feeling. Hugbox environments wind up having a loose relationship with the truth and a strong emphasis on emotional well-being. Setting your moderation boundaries determines the values of your platform. I’d much rather talk to someone who wants to hurt my feelings than someone who is detached from reality or saying what they think.

creeble 4y ago

I think it's two things: the power of malice, and popularity measurement. Malice and fame. Social networks are devices for measuring popularity; if you took the up/down arrows off, no one would be interested in playing. And we have proven once again that nothing gets up arrows like being mean. HN has the unusual property that you can't (readily) see others' score, just your own. That doesn't really make it any less about fame, but maybe it helps. When advertising can fund these devices to scale to billions, it's tough to be optimistic about how it reflects human nature.

INTPenis 4y ago

I don't think it's as simple as malice either because the people who get worked up about online debate often believe they are doing the right thing, they believe they are trying to save someone else from thinking the wrong thing. There's no malice there.