4 ms·

Couldn't violence (both towards animals and humans) be detected in an automated fashion? Especially screams in the audio should be fairly easy to find. Then

Couldn't violence (both towards animals and humans) be detected in an automated fashion? Especially screams in the audio should be fairly easy to find. Then block those videos by default, with a manual appeals process that sends it to a moderator, combined with a big warning that submitting videos against the TOS will get you suspended. Of course this could lead to people submitting videos without audio, but this will always be a cat and mouse game. Or is FB under legal duty to review potentially criminal content?

mschuster91 7y ago
> Shouldn't violence (both towards animals and humans) be farely easy to detect in an automated fashion? No. Already filters fuck up way too much and restrict perfectly legal content.
- the_duke 7y ago
  That's why I mentioned an appeals process for requesting human review. As long as this exists and actually leads to human review, I'd much rather wait a while for my videos to be posted if it means easing the burden on 15.000 people having to watch highly damaging content.
  - Konnstann 7y ago
    Why wouldn't the people who upload violent/pornographic content not always appeal their videos? It's not like they're accidentally posting banned content. This would lead to nothing more than a slight delay before it gets banned by a moderator, solving nothing.
    - mschuster91 7y ago
      > It's not like they're accidentally posting banned content. They simply hope it ain't going to be flagged by another Facebook user or their AI.
- onion2k 7y ago
  Already filters fuck up way too much and restrict perfectly legal content. Would users actually have a problem with that though? People have the freedom to share whatever they want, but that freedom doesn't necessarily have to include sharing it on Facebook. Facebook could decide to tell their users "We're filtering out the bad stuff much more aggressively now and unfortunately that might block your content by mistake." I think a lot of users would be fine with that.
  - marvin 7y ago
    Such moderation has caused a significant degree of political grumbling in the recent past. E.g. where Facebook has blocked women's nipples but not men's, and also photos containing nudity that are politically important. E.g. the photo of Vietnam refugees fleeing a napalm attack, as well as examples of art that are politically important or relevant in an ongoing discussion, +++. The difference between bona fide political censorship and decency/legality/prevent-minors-from-seeing-supposedly-horrible-things/stuff-we-just-don't-like-and-assign-a-political-label censorship is very much a matter of judgement.
lemcoe9 7y ago
"should be fa[i]rly easy to find" Nope. It's just not that easy. Why do you think the biggest companies on the planet struggle so mightily to reliably and un-controversially figure out what content to block.
- mulmen 7y ago
  Because they optimize for profit. Moderation is a cost center.
  - cloakandswagger 7y ago
    Right, so wouldn't they be highly motivated to create models for detecting it automatically? Even if it cost millions of dollars to develop, they would both save money on human labor and boost PR by avoiding articles like this haranguing them about "torturous" working conditions. The fact is that the state of the art for ML is simply not to the point where you could build a model for reliably identifying a category as broad and ambiguous as "violence". I would hope HN users, even if they aren't directly familiar with ML, would appreciate that it isn't trivial.
    - mulmen 7y ago
      I'm not suggesting that moderation is easy to automate. I'm saying Facebook is an advertising company that cares about content moderation exactly as far as it impacts their advertising business and no more. They have no concern for the moderators or the content, they just want to solve this problem (as it relates to profit) as cheaply as possible. At this point in time they believe the best way to do this is by utilizing low-cost labor. Facebook could easily devote more resources to caring for these moderators but they don't because doing so has no further positive impact on the bottom line.
      - cloakandswagger 7y ago
        The cheapest route to moderation is automation, and I would be stunned if there isn't a thick layer of it applied before this content reaches human moderators. My point is that ML is not a magic wand and we're still in its early days. Facebook would love to have a model that accurately identifies offensive content, but that level of AI does not exist today, and no amount of money thrown at it will instantly advance the state of the art to that point.
jsty 7y ago
> Especially screams should be farely easy to detect I guess it depends how reliably it could determine between ordinary screaming (especially with children who scream at just about anything that makes them happy or unhappy), and 'terrible things are happening' screaming. Even most humans would probably struggle to tell the difference just based on a short audio clip of a scream, which is what your algo would have to work with.
- SketchySeaBeast 7y ago
  I think we've all sat in our house trying to figure out if the next door neighbour's kids are being murdered or not. Rollercoaster videos would be banned, as well as haunted houses, neither of which are terribly offensive.
  - logfromblammo 7y ago
    I can barely tell the difference between the parrot call that means "Ohshithawkhawkhawkhawk! Take cover!" and the one that means "Hawk! Just kidding; pay attention to me now." I'm not certain a machine learning system could classify the emotional context of screams across the entire range of animal voices. And what are you going to train it on, exactly? That's a nightmarish corpus to be tagged, right there.
dexen 7y ago
>Couldn't violence (both towards animals and humans) be detected in an automated fashion? Even if it could (and parallel posters argue it cannot), there is a meta-problem: "Who is allowed to post violent content?". Violence is contextual, and also legality of posting it is contextual. Same goes for other objectionable materials (nudity, extreme ideologies etc.). Point in case, if all users were subject to the same criteria, the big names in media would quickly get banned. And/or would enter war path with Facebook, accusing it of heavy-handed censorship against the freedom of the press. Same for reportage of popular events with any degree of nudity. A news piece about a Femen protest, a Pride event, a World Naked Bike Ride, maybe even the No Pants Subway Ride would cause a ban. Likewise, anybody documenting & discussing historical events, and anybody documenting & discussing present-day uprising, revolutions, civil wars, persecutions, etc., would quickly get banned, essentially sweeping a lot of abuse under the rug. Probably even discussion & documentation of domestic violence would have this problem. As a lighter aside, cartoonish violence (video games & movies) could also easily fall prey to automated violence take-downs. All in all, Facebook really really wants to give certain users (mostly press & historians) broad, sweeping exceptions to the general rules.
- lallysingh 7y ago
  Train with signals about the sender?
  - the_duke 7y ago
    Also my first thought, but thinking it through, this can easily lead to (potentially valid) criticism of intrusive profiling and censorship.
    - raxxorrax 7y ago
      And that would be bad because...?
  - dexen 7y ago
    That's my point: it would enforce an automated two-tier system. If you are "ingroup", you would be allowed to post virtually anything. If you are a regular citizen, good luck.
jdietrich 7y ago
It's a whole bunch more complicated than that - I'd highly recommend the Radiolab episode Post No Evil and the documentary The Cleaners for the full story. Facebook doesn't have a blanket policy of removing violent content, for wholly legitimate reasons. Many users in Mexico very strongly want videos of cartel murders to stay on the site, because they feel that they are important reportage of something that the mainstream media is unwilling or unable to report. Many users in Syria very strongly want videos of IS murders to remain on the site, because they want the world to see the suffering that IS has wrought on their country. If Facebook just delete all violent content, they're robbing the victims of violence of their voice - it doesn't feel like Facebook are protecting users of their platform, it feels like Facebook are complicit in covering up the crimes of drug cartels and terrorists. It's not enough to identify that a video is violent - they need to know context. Is this particular video in this particular context a cry for help or a celebration of evil? Will removing this video silence the propaganda of a murderer, or will it help to conceal their crimes? Does this video show the death of an unidentifiable person in an ungoverned warzone, or is it vital evidence that urgently needs to be forwarded to the relevant authorities? For now, only well-trained human beings are capable of making that call. This sort of dilemma points to the fundamental impossibility of moderating Facebook to the satisfaction of everyone. Some users complain about pornography, so Facebook make a policy to take down the porn. Other users complain that these policies are sexist and protest outside Facebook's offices to free the nipple. Facebook end up with several pages of rules that allow their army of moderators to consistently distinguish between pornographic and empowering nipples, but nobody's really happy with the outcome. Facebook are doing a really crappy job of looking after their moderators, but that issue is fixable if we apply enough pressure. There are many other problems with moderating Facebook that are far less tractable and have far wider consequences.

mschuster91 7y ago

> Shouldn't violence (both towards animals and humans) be farely easy to detect in an automated fashion? No. Already filters fuck up way too much and restrict perfectly legal content.

the_duke 7y ago
That's why I mentioned an appeals process for requesting human review. As long as this exists and actually leads to human review, I'd much rather wait a while for my videos to be posted if it means easing the burden on 15.000 people having to watch highly damaging content.
- Konnstann 7y ago
  Why wouldn't the people who upload violent/pornographic content not always appeal their videos? It's not like they're accidentally posting banned content. This would lead to nothing more than a slight delay before it gets banned by a moderator, solving nothing.
  - mschuster91 7y ago
    > It's not like they're accidentally posting banned content. They simply hope it ain't going to be flagged by another Facebook user or their AI.
onion2k 7y ago
Already filters fuck up way too much and restrict perfectly legal content. Would users actually have a problem with that though? People have the freedom to share whatever they want, but that freedom doesn't necessarily have to include sharing it on Facebook. Facebook could decide to tell their users "We're filtering out the bad stuff much more aggressively now and unfortunately that might block your content by mistake." I think a lot of users would be fine with that.
- marvin 7y ago
  Such moderation has caused a significant degree of political grumbling in the recent past. E.g. where Facebook has blocked women's nipples but not men's, and also photos containing nudity that are politically important. E.g. the photo of Vietnam refugees fleeing a napalm attack, as well as examples of art that are politically important or relevant in an ongoing discussion, +++. The difference between bona fide political censorship and decency/legality/prevent-minors-from-seeing-supposedly-horrible-things/stuff-we-just-don't-like-and-assign-a-political-label censorship is very much a matter of judgement.

lemcoe9 7y ago

"should be fa[i]rly easy to find" Nope. It's just not that easy. Why do you think the biggest companies on the planet struggle so mightily to reliably and un-controversially figure out what content to block.

mulmen 7y ago
Because they optimize for profit. Moderation is a cost center.
- cloakandswagger 7y ago
  Right, so wouldn't they be highly motivated to create models for detecting it automatically? Even if it cost millions of dollars to develop, they would both save money on human labor and boost PR by avoiding articles like this haranguing them about "torturous" working conditions. The fact is that the state of the art for ML is simply not to the point where you could build a model for reliably identifying a category as broad and ambiguous as "violence". I would hope HN users, even if they aren't directly familiar with ML, would appreciate that it isn't trivial.
  - mulmen 7y ago
    I'm not suggesting that moderation is easy to automate. I'm saying Facebook is an advertising company that cares about content moderation exactly as far as it impacts their advertising business and no more. They have no concern for the moderators or the content, they just want to solve this problem (as it relates to profit) as cheaply as possible. At this point in time they believe the best way to do this is by utilizing low-cost labor. Facebook could easily devote more resources to caring for these moderators but they don't because doing so has no further positive impact on the bottom line.
    - cloakandswagger 7y ago
      The cheapest route to moderation is automation, and I would be stunned if there isn't a thick layer of it applied before this content reaches human moderators. My point is that ML is not a magic wand and we're still in its early days. Facebook would love to have a model that accurately identifies offensive content, but that level of AI does not exist today, and no amount of money thrown at it will instantly advance the state of the art to that point.

jsty 7y ago

> Especially screams should be farely easy to detect I guess it depends how reliably it could determine between ordinary screaming (especially with children who scream at just about anything that makes them happy or unhappy), and 'terrible things are happening' screaming. Even most humans would probably struggle to tell the difference just based on a short audio clip of a scream, which is what your algo would have to work with.

SketchySeaBeast 7y ago
I think we've all sat in our house trying to figure out if the next door neighbour's kids are being murdered or not. Rollercoaster videos would be banned, as well as haunted houses, neither of which are terribly offensive.
- logfromblammo 7y ago
  I can barely tell the difference between the parrot call that means "Ohshithawkhawkhawkhawk! Take cover!" and the one that means "Hawk! Just kidding; pay attention to me now." I'm not certain a machine learning system could classify the emotional context of screams across the entire range of animal voices. And what are you going to train it on, exactly? That's a nightmarish corpus to be tagged, right there.

dexen 7y ago

>Couldn't violence (both towards animals and humans) be detected in an automated fashion? Even if it could (and parallel posters argue it cannot), there is a meta-problem: "Who is allowed to post violent content?". Violence is contextual, and also legality of posting it is contextual. Same goes for other objectionable materials (nudity, extreme ideologies etc.). Point in case, if all users were subject to the same criteria, the big names in media would quickly get banned. And/or would enter war path with Facebook, accusing it of heavy-handed censorship against the freedom of the press. Same for reportage of popular events with any degree of nudity. A news piece about a Femen protest, a Pride event, a World Naked Bike Ride, maybe even the No Pants Subway Ride would cause a ban. Likewise, anybody documenting & discussing historical events, and anybody documenting & discussing present-day uprising, revolutions, civil wars, persecutions, etc., would quickly get banned, essentially sweeping a lot of abuse under the rug. Probably even discussion & documentation of domestic violence would have this problem. As a lighter aside, cartoonish violence (video games & movies) could also easily fall prey to automated violence take-downs. All in all, Facebook really really wants to give certain users (mostly press & historians) broad, sweeping exceptions to the general rules.

lallysingh 7y ago
Train with signals about the sender?
- the_duke 7y ago
  Also my first thought, but thinking it through, this can easily lead to (potentially valid) criticism of intrusive profiling and censorship.
  - raxxorrax 7y ago
    And that would be bad because...?
- dexen 7y ago
  That's my point: it would enforce an automated two-tier system. If you are "ingroup", you would be allowed to post virtually anything. If you are a regular citizen, good luck.

jdietrich 7y ago

It's a whole bunch more complicated than that - I'd highly recommend the Radiolab episode Post No Evil and the documentary The Cleaners for the full story. Facebook doesn't have a blanket policy of removing violent content, for wholly legitimate reasons. Many users in Mexico very strongly want videos of cartel murders to stay on the site, because they feel that they are important reportage of something that the mainstream media is unwilling or unable to report. Many users in Syria very strongly want videos of IS murders to remain on the site, because they want the world to see the suffering that IS has wrought on their country. If Facebook just delete all violent content, they're robbing the victims of violence of their voice - it doesn't feel like Facebook are protecting users of their platform, it feels like Facebook are complicit in covering up the crimes of drug cartels and terrorists. It's not enough to identify that a video is violent - they need to know context. Is this particular video in this particular context a cry for help or a celebration of evil? Will removing this video silence the propaganda of a murderer, or will it help to conceal their crimes? Does this video show the death of an unidentifiable person in an ungoverned warzone, or is it vital evidence that urgently needs to be forwarded to the relevant authorities? For now, only well-trained human beings are capable of making that call. This sort of dilemma points to the fundamental impossibility of moderating Facebook to the satisfaction of everyone. Some users complain about pornography, so Facebook make a policy to take down the porn. Other users complain that these policies are sexist and protest outside Facebook's offices to free the nipple. Facebook end up with several pages of rules that allow their army of moderators to consistently distinguish between pornographic and empowering nipples, but nobody's really happy with the outcome. Facebook are doing a really crappy job of looking after their moderators, but that issue is fixable if we apply enough pressure. There are many other problems with moderating Facebook that are far less tractable and have far wider consequences.