6 ms·

How Automated Content Moderation Works (Even When It Doesn't)

45 points by atg_abhishek 3y ago

verisimi 3y ago
[flagged]
- deleted 3y ago
  [deleted]
- Sohcahtoa82 3y ago
  > There is also a parallel legal reality - that people have the right to free speech. Here are go again. Are we going to have to have the same argument that's already been on HN a million times about whether or not "free speech" means compelling a private company to let you use their platform for speech?
- gizmo686 3y ago
  In the US, the legal reality of free speech applies only to actions the government might take to restrict it. Regardless of what is good policy, the legal reality at the moment is that private companies have no obligation to provide free speech to their users. This is being challenged by Texas and Florida, which have passed laws to require a level of free speech on the platforms. The platforms argue that their editorial control on what they host is itself protected speech (in the same way that magazines deciding what articles to accept is). The states, for their part, argue that the media companies are more analogous to common carriers, which can be subject to non-discrimination laws. The case was just heard by the supreme court, so we should get a decision at some point.
- rowyourboat 3y ago
  > There is also a parallel legal reality - that people have the right to free speech. The right to free speech only means you can't be prosecuted for what you say. It does not compel any private entity to distribute everything that you want to say.
  - dragonwriter 3y ago
    > The right to free speech only means you can't be prosecuted for what you say. It is a much broader protection against government retaliation than just "cannot be prosecuted", but it is a protection against the government, and it includes protection against being forced to endorse and relay others' speech. Compelled speech is the opposite of free speech.
  - gosub100 3y ago
    I don't think they should be able to claim "safe harbor" protection (or am I getting that confused with common carrier?) and then get to use censorship however they like. The spirit of safe harbor is that "we're not responsible for what random people post on here". It doesn't seem logically consistent to claim this legal protection but then curtail the content to anyone's personal sensitivities.
    - gizmo686 3y ago
      Section 230(c)(2)(a) is pretty explicit in allowing moderation. Just because they are not legally liable for what users post, Congress did not want to prevent platforms from going beyond their minimal legal requirements if they so chose. The alternative us a law that says "It is unreasonable for us to expect you to perfectly police your users; but we will hold you liable for imperfectly doing so".
      - gosub100 3y ago
        Just curious: do you believe in net neutrality? (I think that's the cause of my conflation with "common carrier"). But if censorship is a valid reason to deny service, why can't charter or AT&T say "we dont like what you're sending over our wires, so we're blocking or throttling you" ? Is it merely the fact that encryption blocks the carrier from knowing what the line is used for? Or said another way, should AT&T have the right to terminate a user's service if they were certain a user was posting, say, white supremacy?
- bitwize 3y ago
  Obligatory xkcd: https://xkcd.com/1357/ https://xkcd.com/1357/
  - TheMode 3y ago
    There's a difference between your show being cancelled, and nobody coming to listen.
    - futuretaint 3y ago
      exactly. like when Yoel Roth (former safety lead at twitter) worked w/ FBI to ban conservative accounts, critical of Hunter Biden. No one cared, and Yoel and the FBI intuitively understood that.
- Etheryte 3y ago
  Hate speech is most definitely not an idea that's made up by corporations on the spot. It's an idea that's codified into law in many countries worldwide. In most of the world, free speech isn't a right with no strings attached whatsoever.
Sohcahtoa82 3y ago
I feel that moderating a platform as massive as Facebook is essentially impossible. Automating exact matches for known bad content is easy enough. But when trying to automate without using exact matches, it becomes impossible when nuance and context can make a huge difference in the meaning of a word or sentence. Automation can get it wrong, and when it does, users want the ability to force a human to review it and make a decision, but bad actors will abuse and overwhelm the system.
- CM30 3y ago
  Moderation is one of those things that basically cannot scale (properly) without human level AI or an absolute TON of resources. Any generalised system lacks too much context to do a good job, and any specialised one is too expensive/resource heavy. You can somewhat get away with it on something like Reddit where communities moderate themselves, but that brings its own problems for both the platform and the communities on it.
  - kmeisthax 3y ago
    Moderation absolutely can scale, platforms just don't want to pay for it. For two reasons: - Moderation is a 'cost center', which is MBAspeak for "thing that doesn't provide immediate returns disproportional to investment". For context, so is engineering (us). So instead of paying a reasonable amount to hire moderators, Facebook and other platforms will spend as little as possible and barely do anything. This mentality tends to be enforced early on in the growth phase where users are being added way faster than you can afford to add moderators, but remains even after sustainable revenues have been discovered and you have plenty of money to hire people with. - Certain types of against-the-rules posts provide a benefit to the platform hosting them. Copyright infringement is an obvious example, but that has liability associated to it, so platforms will at least pretend to care. More subtle would be things like outrage bait and political misinformation. You can hook people for life with that shit. Why would you pay money to hire people to punish your best posters? That last one dovetails with certain calls for "free speech" online. The thing is, while all the content people want removed is harmful to users, some of it is actually beneficial to the platform. Any institutional support for freedom of speech by social media companies is motivated not by a high-minded support for liberal values, but by the fact that it's an excuse to cut moderation budgets and publish more lurid garbage.

45 points by atg_abhishek 3y ago

verisimi 3y ago

[flagged]

deleted 3y ago
[deleted]
Sohcahtoa82 3y ago
> There is also a parallel legal reality - that people have the right to free speech. Here are go again. Are we going to have to have the same argument that's already been on HN a million times about whether or not "free speech" means compelling a private company to let you use their platform for speech?
gizmo686 3y ago
In the US, the legal reality of free speech applies only to actions the government might take to restrict it. Regardless of what is good policy, the legal reality at the moment is that private companies have no obligation to provide free speech to their users. This is being challenged by Texas and Florida, which have passed laws to require a level of free speech on the platforms. The platforms argue that their editorial control on what they host is itself protected speech (in the same way that magazines deciding what articles to accept is). The states, for their part, argue that the media companies are more analogous to common carriers, which can be subject to non-discrimination laws. The case was just heard by the supreme court, so we should get a decision at some point.
rowyourboat 3y ago
> There is also a parallel legal reality - that people have the right to free speech. The right to free speech only means you can't be prosecuted for what you say. It does not compel any private entity to distribute everything that you want to say.
- dragonwriter 3y ago
  > The right to free speech only means you can't be prosecuted for what you say. It is a much broader protection against government retaliation than just "cannot be prosecuted", but it is a protection against the government, and it includes protection against being forced to endorse and relay others' speech. Compelled speech is the opposite of free speech.
- gosub100 3y ago
  I don't think they should be able to claim "safe harbor" protection (or am I getting that confused with common carrier?) and then get to use censorship however they like. The spirit of safe harbor is that "we're not responsible for what random people post on here". It doesn't seem logically consistent to claim this legal protection but then curtail the content to anyone's personal sensitivities.
  - gizmo686 3y ago
    Section 230(c)(2)(a) is pretty explicit in allowing moderation. Just because they are not legally liable for what users post, Congress did not want to prevent platforms from going beyond their minimal legal requirements if they so chose. The alternative us a law that says "It is unreasonable for us to expect you to perfectly police your users; but we will hold you liable for imperfectly doing so".
    - gosub100 3y ago
      Just curious: do you believe in net neutrality? (I think that's the cause of my conflation with "common carrier"). But if censorship is a valid reason to deny service, why can't charter or AT&T say "we dont like what you're sending over our wires, so we're blocking or throttling you" ? Is it merely the fact that encryption blocks the carrier from knowing what the line is used for? Or said another way, should AT&T have the right to terminate a user's service if they were certain a user was posting, say, white supremacy?
bitwize 3y ago
Obligatory xkcd: https://xkcd.com/1357/ https://xkcd.com/1357/
- TheMode 3y ago
  There's a difference between your show being cancelled, and nobody coming to listen.
  - futuretaint 3y ago
    exactly. like when Yoel Roth (former safety lead at twitter) worked w/ FBI to ban conservative accounts, critical of Hunter Biden. No one cared, and Yoel and the FBI intuitively understood that.
Etheryte 3y ago
Hate speech is most definitely not an idea that's made up by corporations on the spot. It's an idea that's codified into law in many countries worldwide. In most of the world, free speech isn't a right with no strings attached whatsoever.

Sohcahtoa82 3y ago

I feel that moderating a platform as massive as Facebook is essentially impossible. Automating exact matches for known bad content is easy enough. But when trying to automate without using exact matches, it becomes impossible when nuance and context can make a huge difference in the meaning of a word or sentence. Automation can get it wrong, and when it does, users want the ability to force a human to review it and make a decision, but bad actors will abuse and overwhelm the system.

CM30 3y ago
Moderation is one of those things that basically cannot scale (properly) without human level AI or an absolute TON of resources. Any generalised system lacks too much context to do a good job, and any specialised one is too expensive/resource heavy. You can somewhat get away with it on something like Reddit where communities moderate themselves, but that brings its own problems for both the platform and the communities on it.
- kmeisthax 3y ago
  Moderation absolutely can scale, platforms just don't want to pay for it. For two reasons: - Moderation is a 'cost center', which is MBAspeak for "thing that doesn't provide immediate returns disproportional to investment". For context, so is engineering (us). So instead of paying a reasonable amount to hire moderators, Facebook and other platforms will spend as little as possible and barely do anything. This mentality tends to be enforced early on in the growth phase where users are being added way faster than you can afford to add moderators, but remains even after sustainable revenues have been discovered and you have plenty of money to hire people with. - Certain types of against-the-rules posts provide a benefit to the platform hosting them. Copyright infringement is an obvious example, but that has liability associated to it, so platforms will at least pretend to care. More subtle would be things like outrage bait and political misinformation. You can hook people for life with that shit. Why would you pay money to hire people to punish your best posters? That last one dovetails with certain calls for "free speech" online. The thing is, while all the content people want removed is harmful to users, some of it is actually beneficial to the platform. Any institutional support for freedom of speech by social media companies is motivated not by a high-minded support for liberal values, but by the fact that it's an excuse to cut moderation budgets and publish more lurid garbage.