6 ms·
Content moderation at scale is impossible to do well (2019)
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- dang 6y agoI always look at these things from a parochial perspective, i.e. how does it apply to HN. His first two points ("moderation is likely to end up pissing off those who are moderated" and "content moderation is always going to rely on judgment calls") are true of all moderation and certainly HN. They have nothing to do with scale though. His third point is more interesting: at scale, even 99.9% accuracy is going to leave hundreds of thousands of mistakes a day. No one will care about the 99.9% you got right, only the things you missed. Reporters will call public attention to these. That's a good point and not widely understood. But he stops there. All that shows is that content moderation is impossible to do perfectly, which we already know because of points 1 and 2. More interesting would be the part he didn't write: given that a large number of misses is inevitable and reporters will call attention to them, what follows next? Probably that the public image will be that moderation sucks. That's rough, but it doesn't necessarily follow that content moderation is impossible to do well.
- tqi 6y agoI think the way scale comes in to play for the first two points comes down to who is being moderated / doing the moderation. At scale, the users being moderated are more capable of becoming a politically or economically effective coalition. And at scale, the consistency of judgement becomes harder to maintain. Now you have mistakes not just because judgement was incorrectly applied, but also because the judgement itself was incorrectly implemented.
- dang 6y agoI agree with you that, at scale, issues with moderation can become politically influential, and that's a huge difference from moderation at smaller scales. Lucky for us, says parochial me. Consistency is probably a red herring. Moderation will never be perceived as consistent, at any scale. Different mods will always make different calls, and even the same mod will make different-seeming calls because humans are complicated.
- 082349872349872 6y agoConsistency being a red herring is behind the discussions over "instant" replay vs ref call in sports. If players can trust a ref to be not perfect, but fair (that is, screwing up equally for both teams), the game will flow much more than if it is continually stopped so everyone can argue and check the replay.
- dang 6y agoI think the analogy between mods and refs is a good one (https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=by%3Adang%20refs&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...). I don't follow sports that do instant replays but I'd expect that people don't agree about them anyway. The best way to signal fairness is to find a thread like a butterfly option where you can scold one of each side. Readers are alert to that and they love it. Everyone feels reassured that things are fair. That would be analogous to a ref giving a penalty to each side. It's not always possible, of course, and when the scolding is one-sided people will automatically assume that you secretly favor the other team.
- halfFact 6y agoHN is like the Apple AppStore, the rules are loose and apply to people who are find themselves against Apple. Any other programming fourms with a different set of mods?
- dang 6y agoIf I understand you correctly, I think you may have missed the OP's points 1 and 2. All content moderation sucks in this way.
- pbhjpbhj 6y agoSurely the 'scale' issue's answer is for a community to moderate itself. That way, the community should be happy. That's what Slashdot did (does?). All users take part in moderation [if they wish], and take part in meta-moderation. Using a statistical consensus, and correction of outliers, but doing so through the userbase such that as the community grows so too does the base of moderators. Making it statistical means that no one moderator can have undue influence (other than site owners). HN does some 2nd order stuff, flags/vouches from trusted users i think are weighted (?) but doesn't go to the next level of then decreasing the weight of votes for those whose votes were meta-moderated as 'wrong' (ie contrary to community consensus). In theory that should handle brigading, for example, because you don't know what you'll moderate before you say yes, and you can't choose to moderate - the site chooses you. Slashdot also had several dimensions to it's moderation and allowed users to choose to consume content contrary to the community norm - eg you can choose to see more humour, even if the community in general doesn't like to view humour. Still the best system, on paper (I have no idea if they have to constantly fix things!?!) that I've seen.
- blast 6y agoSlashdot was never close to "scale". OP is talking about FB numbers.
- jandrese 6y agoThe Slashdot system won't work so well for Facebook because on Slashdot everything is public. On Facebook the meta-moderators would be either seeing only posts from their friends (defeating the purpose), or posts from total strangers that they might not want leaking out. It would break the encapsulation of your friends list. One thing I do like about Slashdot is that they limit the total amount of score a post can get and how much each person can moderate at once. This makes it much much harder to brigade content, which is the usual problem with community based moderation.
- jasode 6y ago>Surely the 'scale' issue's answer is for a community to moderate itself. That way, the community should be happy. Self-moderation is basically what Stackoverflow is and a lot of users are not happy with the outcome. Most of the votes that close questions as "duplicates" and "not constructive" are done by fellow users. I tried to explain this previously[0] but unhappy Stackoverflow users still insist there's an "us vs them". And consider that Stackoverflow's userbase is a much smaller scale than Facebook. [0] https://news.ycombinator.com/item?id=23977165 https://news.ycombinator.com/item?id=23977165
- deleted 6y ago[deleted]
- klyrs 6y ago> His third point is more interesting: at scale, even 99.9% accuracy is going to leave hundreds of thousands of mistakes a day. No one will care about the 99.9% you got right, only the things you missed. Reporters will call public attention to these. This is indeed the crux. The .1% failures are quite inevitable. They represent the most valuable training data, and that deserves attention. The breakdown that I see is that it becomes a popularity contest: if you have enough friends, followers, or you can bend the ear of somebody who does, then and only then do these cases receive human attention. In a sense, this is outsourcing the human-level moderation to the public, a level of bad PR determines the escalation. This will fail to capture a large portion of the false positives, fail to feed back into the training set, and the number of failures will continue to scale with the system. The goal should be to eradicate these mistakes, and I can't see how that can be done without investing in human metamoderation -- even if they catch 1% of the .1%, that could provide an opportunity to continuously improve the system. If your profits scale, then so must your investments, or as you say, public confidence will collapse.
- searchableguy 6y agoYou described the problem well. How would you convince someone to increase budget for moderation without bad PR?
- klyrs 6y agoRepeated bad PR stories erodes public confidence, which can result in regulations? My reasons include altruism and a long term view, which don't always play well with business types.
- deleted 6y ago[deleted]
- jimnotgym 6y agoNot impossible at all. Divide the subject matter into thousands of specialist areas, and let each area be moderated by it's own community moderators/editors. It only becomes a problem when you try and publish every comment on every subject through a single organisation.
- pbhjpbhj 6y agoDoesn't that reinforce extremes and encourage enclaves: which is why Reddit end up closing whole communities (the good and bad in those communities). You're just putting off moderation of extreme content that doesn't confirm to a community consensus at all by hiding it in a corner and letting it grow without control.
- jimnotgym 6y agoAre the readers of 'Nature' or 'The Economist' extreme? Quality curation and editorial control are possible at those levels. I can write a letter, or indeed an article or paper for either, but it only gets accepted if it is good enough.
- searchableguy 6y ago> Doesn't that reinforce extremes and encourage enclaves: which is why Reddit end up closing whole communities (the good and bad in those communities). Nope. It works well on discord. I can think of few reasons why: 1. The reliance on advertising model pushes them to retain users on their platform for longer and increase engagement in unnatural ways. You can see with how they push you to use the app or throw big subs everywhere. 2. The front-page has limited slots and there is no way to divide the community into smaller parts in itself. On discord, you can do this with channels and roles. 3. There is no voting system and realtime communication is more human like. You can get feedback naturally which you won't on reddit.
- djoldman 6y agoI've always been curious about "content moderation." Theoretically, if you could perfectly allow only content that was Legal in the jurisdiction, wouldn't every website allow everything Legal and be able to just point at the laws and say, "if you don't like it, change the law"? If so, is the crux of the problem that 1. we don't know what is legal, 2. We can't perfectly allow it, and or 3. Something else. If 3, is it that websites don't really fear legal repercussions, but fear bad PR, thus alienating users/revenue? If so, doesn't it follow that the level of content moderation will track the projected cost of losing the projected users/revenue?
- krapp 6y agoThe crux of the problem is that limiting moderation only to allowing content which is illegal and banning what is illegal still allows content many people may find objectionable, spam, trolling, racist and hateful speech and off topic content. Not every online community wants to be a 4chan or Voat, which is what that sort of hands off moderation inevitably leads to.
- searchableguy 6y agoWhat if you just give each person a way to censor the content they don't want to see? Something like show dead posts on hn. Although, why host content that isn't profitable?
- cptroot 6y agoThat only really works if it can hide harrassment from you automatically
- u801e 6y ago> What if you just gave each person a way to censor the content they don't want to see? > > Something like show dead posts on hn. Or a killfile in a news client. While usenet was to some extent full of troll posts and spam, with the right filters, the signal to noise ratio was very high.
- 6y ago
- new_guy 6y agoModeration, even at scale, is a decades old solved problem, and not even a particularly hard one. The reason places like FB etc say it's 'impossible' so they don't open themselves up to legal liability.
- pbhjpbhj 6y agoPerhaps you could expand on that with details of a solution you have in mind, references to a moderation system using that solution would be useful too.
- auganov 6y agoObviously right on the numbers but I have to disagree with the notion that reporters and people are just making a lot of noise over the 0.1% of inevitable "bad calls". Usually only notable popular accounts get coverage over their censorship (or lack of). The number of such accounts is a few orders of magnitude smaller than the general userbase. Any network would want to protect these accounts from arbitrary bans and they probably do. Most big stories about social media censorship often include comments from company reps who stand by their decision. Stories ending in "Oops we made a mistake" tend to be much less common as far as I can tell. As such I'd say people are actually pretty good at identifying the cases that aren't just a "bad call" but a contentious matter.
- ameliaquining 6y agoGiven the "impossibility theorem" headline, I was hoping for at least a sketch of a formalizable argument. I wasn't really expecting one, and maybe this is a silly thing to be annoyed about, but I think it's better to be clear about what exactly you have backing up a particular argument.
- mountainboy 6y agomy approach: let everyone rate content on multiple relevant criteria. Eg, quality, grammar, agreement, sexuality, violence, hate, etc. Make some default filters that choose societally acceptable settings for each filter, and hides new content that is unrated. Enable individuals to tweak the filters to any setting they wish, for a totally custom view, including viewing unrated content. Enable individuals to share their filters with others. Voila! A system that allows any content to be posted, but shows only "safe" content by default and allows individuals to bypass the filters if they wish.
- deleted 6y ago[deleted]
- lordCarbonFiber 6y agoExcept, in this system you're relying on users who choose to view unrated content to provide the initial ratings. Even places like 4chan have moderation and most people won't be willing to engage with content at that level to positively rate memes or more general discussion. Add onto that that untrained users are going to be largely garbage at actually classifying your content; this is somethign social media struggles with and they have trained moderators. What's the difference between hate and ironic self ribbing, what's the difference between sexuality, art, and medicine (see tumblr's hilarious attempt at banning "femme presenting nipples"), what kind of content can be legitimately traumatizing or otherwise dangerous (calls to violence; deadly misinformation; etc). The closest thing to that model is something like the fediverse where you're choosing to federate with instances based on their moderation rules and your users choose to stay on your instance based on yours. Even that has problems with chains of trust and discovery though. At scale you can't assume good faith anymore
- aspaceman 6y agoThis sounds like granular trigger warnings, which people have voiced opposition to for reasons
- jorvi 6y agoAt least for voting, the fix is simple and I don’t understand why both Reddit and HN (with the default settings) don’t do it: make voting invisible. That is, you sort the comments by the vote value they have, as is done now, but you don’t display the value. Reddit displays the value directly, and HN by gradually greying out comments. The problem with doing that is that it leads to bandwagoning. If I’d had to hazard a guess, a solid 50% of downvotes are people blindly ramming the downvote button because that comment already has a negative vote value.
- chr1 6y agoCentralized moderation at scale is impossible to do for the same reason that centralized planning of economy is impossible to do: which is that goals, knowledge and resources are not known to the centralized authority that is supposed to handle the task. But it is possible to give everyone tools to modereate for themeselves https://adecentralizedworld.com/2020/06/a-trust-and-moderation-system-for-the-decentralized-web/ https://adecentralizedworld.com/2020/06/a-trust-and-moderati...
- ameliaquining 6y agoMasnick responds to this in the original post: > There is the argument (that I regularly advocate) that pushing out the moderation to the ends of the network (i.e., giving more controls to the end users) is better, but that also has some complications in that it puts the burden on end users, and they have neither the time nor inclination to continually tweak their own settings.
- xg15 6y ago> The first obvious issue is the creation of echo chambers or online cults, where instead of choosing truth and seeking the best information, people trust those who confirm their existing beliefs. This can be somewhat mitigated with the ability to set a trust filter level to a low level, even in the negatives, to see posts and opinions one may not usually be exposed to. This again is up to the end user if they wish to expose themselves to differing opinions or not. It is up to individuals to determine what’s best for them. Of course, one would hope they make good choices, but it is not society’s job to save people from themselves. It’s far better to have this freedom and the power to use or misuse it, than to allow companies to build the echo chamber behind closed doors. That's a nice dogma - which breaks down as soon as one of the cults/echo chambers starts harrassing or doxxing or otherwise hurting someone they declared an enemy. This whole concept is based on an assumption that if someone bothers you online, this is always a personal issue that can be resolved by you pretending they didn't exist. By now we have enough experience with online communities to know that things don't work that way.
- erichocean 6y agoCan GPT-3 be re-purposed to do automatic content moderation?
- sova 6y agoThis title is sarcastic right? It's doable just not well because there is no way to create a totalitarian panopticon without having your state first be a totalitarian panoply of optics and cons.
- sova 6y agoScore Voting notably defies Arrow's Paradox. Score Voting (= Range Voting) is where candidates are given a score from, say zero to 99, and an average is taken. Effectively the same style as in figure skating. Yes, a once-every-four-years global aesthetic athletic event is the only place to get voting right.* *what do you mean by "right?" Well, there is something called Bayesian regret and you can minimize it by having Score Voting (=Range Voting) or by having the magical harry potter wizard hat choose the perfect candidate. Other voting systems have more Bayesian Regret and the current First-Past-the-Post voting style has quite a lot, actually. https://rangevoting.org/BayRegsFig.html https://rangevoting.org/BayRegsFig.html
- mehrdadn 6y agoRelated: https://politics.stackexchange.com/questions/14015/does-gibbard-satterthwaite-theorem-apply-to-all-voting-systems/14245#14245 https://politics.stackexchange.com/questions/14015/does-gibb...
- ulucs 6y agoAny voting system that incorporates mean scores will in no way be immune to strategic deviations. (imagine a tie between your second ranked candidate and some other one, you simply inflate your score of your second ranked candidate by one) Also isn't achieving the lowest regret trivial, as the other voting systems never report anything cardinally?
- sova 6y agoIn regards to your first statement, that's absolutely the point of Score Voting, to have a clear leaderboard with 1st, 2nd, 3rd and so forth. With regards to your second statement, "because we cannot access data means the data never existed?" Bayesian regret exists in all elections with all candidates, just because it's not "reported" doesn't mean it's not statistically there. The point is that Score Voting is a superset of all voting systems, and everything else is a blurrying of the detail that is available only in Score Voting. Bayesian Regret is just used to demonstrate that it will yield the highest rated candidate overall, which is logical given how it works. Nevertheless, examine the science for yourself and I believe you will soon find that all voting systems are some sort of blind or ignorant form of Score Voting. Means or averages are convenient, but you could simply take the greatest total score as well.