8 ms·

Somebody from Facebook recently told me they developed the best system to flag radical/violent content in the world, I pointed out they first developed the best

by nr2x 6y ago

Somebody from Facebook recently told me they developed the best system to flag radical/violent content in the world, I pointed out they first developed the best system to spread it.

umvi 6y ago
Well they didn't develop the platform specifically for spreading radical/violent content. They build a generic platform for spreading thoughts among friends, and racist/violent people used the generic platform to spread their thoughts.
- tolbish 6y ago
  Not quite. They tweaked the way it was run to keep people addicted.
  - 2pEXgD0fZ5cF 6y ago
    That is the thing, many argue as if Facebook is a flat, neutral forum, and leave out the whole big algorithmic blackbox it entails. In reality there are no chronological timelines and the regular user is pushed towards the most rage inducing (or as Facebook employees probably euphemize it: "most engaging") content and group recommendations Facebook can apply to an individual. Facebook was very much an actor in this, not some mere communication infrastructure. Facebook wasn't "warped" by malicious actors, it fanned the flames and by doing so nurtured their hatred and multiplied their reach, all to get the users as addicted as possible to this endless loop. (The same goes for Twitter and Google/Youtube too, of course)
- nr2x 6y ago
  Philip Morris didn’t intentionally develop a system to spread lung cancer, and yet they did.
  - redisman 6y ago
    It's not a bad metaphor since PM and FB both did develop their addiction systems on purpose. Cigarettes are not just dried tobacco leaf, there was a ton of R&D to identify and add chemicals that make them extremely addicting. Same with FB, they have optimized for outrage culture etc. - whatever brings the most "engagement".
- riscy 6y ago
  The key thing is that they built a reward system into sharing each of those thoughts, which enables an addiction for acceptance from users on the platform. Facebook’s engagement systems / algorithms feed on these addictions. As the article discusses, this causes users to radicalize their posts to garner more attention.
mrweasel 6y ago
What makes Facebook worse, in my mind, is that people don't seem to care that it's their real name associated with terrible radical, violent and angry comment. At least on something like 4Chan you'd be anonymous, but nope, people will write the most grotesque comments, while everyone can see their real name, where they live and who they work for. Some people don't seem to care, and that's pretty scary.
- JumpCrisscross 6y ago
  > people don't seem to care that it's their real name associated with terrible radical, violent and angry comment The hypothesis of real name policies was that people would be civil online if their real-world identities grounded them. In reality, the tail wagged the dog.
  - nr2x 6y ago
    Very astute observation.
  - coffeefirst 6y ago
    Yeah, it may have worked in some contexts, but people also read the room. And when the algorithm is setting the room to "1 tick more extreme than you" they gradually dial up. I've watched this process turn friends who had pretty mild political leanings to "burn it all down" radicals or transform into right wing talk radio hosts. It's not good.
  - dragontamer 6y ago
    Yeah, but anyone who knew a thing about Usenet back in the day knows that was false. I mean, it was a theory that was popular in the 00s, mostly because a ton of people at that time didn't have experience with Usenet. But a lot of vile language was used in the clear, signed with the real names of many individuals. Anonymity doesn't seem to change much. Humans will be humans, whether or not they're anonymous.
  - brlewis 6y ago
    Was that the hypothesis, or just the marketing? Obviously real names make it easier to integrate marketing data from disparate sources.
tomp 6y ago
Well, with #BlackLivesMatter and #KillAllMen, I kind-of doubt the former assertion...
sct202 6y ago
I've had fb friends report spammed into being temp or permabanned very easily (personal feuds, stalkers), so I would question how they're even measuring quality at this point.
ezzzzz 6y ago
Although I am concerned about the 1A and big tech being the arbiters of it, it seems like such a system has yet to go to production (probably never will). I understand that there are technical, financial, and legal complexities that are much more nuanced than 'FB doesn't care, as long as they can profit from potentially dangerous content' and 'FB just wants to suppress any speech which doesn't align with their neo-liberal ideologies'. What I find troubling is that, a system to find/flag radical/violent content likely exists within the auspices of the NSA. Despite the massive infringement of privacy that these systems create, it seems that they still can't prevent people from being 'radicalized' by content from the platforms that is designed to do exactly that. Furthermore, they couldn't (or didn't) do anything to actually prevent people from storming the Capital. This begs the question, does this massive surveillance state need to exist if it does not do the thing it is purported to do (find and suppress violence, prevent terrorist attacks, etc). If the owners of the platforms have any conscience, unfortunately, it seems that the responsibility will be up to them. It's murky territory for sure, and you have to strike a balance between far reaching censorship (i.e. infringing of 1A) and allowing disinformation, hate speech, and calls to violence to spread like wild-fire (i.e. how we got into this mess in the first place). I hate to say it, but the older I get, the more something like 'the great firewall' of China seems to make more sense.
- agumonkey 6y ago
  I don't think the NSA was engineered with internet / social networks scale though. But they were already failing in the 9/11 days soo..
  - ezzzzz 6y ago
    I mean, PRISM was a post 9/11 thing and yeah, Snowden's leaks directly noted Facebook, YouTube, etc. There is really no telling what exactly is out there, but we know the NSA has massive Data Centers and a massive budget. Basically, even the most personal private information about literally any citizen is just a FISA Warrant away. Things posted in a public forum are a cakewalk. Maybe they aren't doing anything novel with this data, whatever, this is beside the point. The point is, we can't really count on the State to regulate this stuff, even though, you know, creating and enforcing laws is literally the reason the state exists... IDK, I hesitantly have to side with big-tech on this, but this is a problem that the State either can't fix, or will not fix. The internet is a wonderful thing, but it is also very dangerous. Having unfettered access to information, be it true or false, can wield some horrible side effects when such a large swath of the population lacks basic BS detection, critical-thinking and like, base level understanding of civics. Do we need a Vanguard? I wouldn't trust one coming from either side of the aisle in the current climate. Seems like 'censorship' from the platforms is the only viable solution I can see, but that can take a pretty dark turn. At least if it's a private company, you can always leave, create your own thing or find something else...for now.
sydd 6y ago
Not trying to defend Facebook, but they did not do it on purpose. They developed a really addicting social network where you do as many interactions as possible. When Facebook started it was a different world: Almost no data privacy laws, internet security in its infancy (e.g. in 2008 HTTPS was a rare new thing, not cross-origin security etc.) and people were spamming Farmville notifications to each other. It just turned out that this structure is an excellent ground for lots of bad things.
- deleted 6y ago
  [deleted]
- nr2x 6y ago
  That's not true: the EU had lots of national privacy laws under the privacy directive (GDPR was partially intended to unify/simplify that), and the US had sectoral laws such as COPPA and HIPAA, the former of which very much applies to Facebook.
marcod 6y ago
Wasn't there also a system developed that was really good at flagging radical content and proceeded to flag all the conservative posts? :p

by nr2x 6y ago

Somebody from Facebook recently told me they developed the best system to flag radical/violent content in the world, I pointed out they first developed the best system to spread it.

umvi 6y ago

Well they didn't develop the platform specifically for spreading radical/violent content. They build a generic platform for spreading thoughts among friends, and racist/violent people used the generic platform to spread their thoughts.

tolbish 6y ago
Not quite. They tweaked the way it was run to keep people addicted.
- 2pEXgD0fZ5cF 6y ago
  That is the thing, many argue as if Facebook is a flat, neutral forum, and leave out the whole big algorithmic blackbox it entails. In reality there are no chronological timelines and the regular user is pushed towards the most rage inducing (or as Facebook employees probably euphemize it: "most engaging") content and group recommendations Facebook can apply to an individual. Facebook was very much an actor in this, not some mere communication infrastructure. Facebook wasn't "warped" by malicious actors, it fanned the flames and by doing so nurtured their hatred and multiplied their reach, all to get the users as addicted as possible to this endless loop. (The same goes for Twitter and Google/Youtube too, of course)
nr2x 6y ago
Philip Morris didn’t intentionally develop a system to spread lung cancer, and yet they did.
- redisman 6y ago
  It's not a bad metaphor since PM and FB both did develop their addiction systems on purpose. Cigarettes are not just dried tobacco leaf, there was a ton of R&D to identify and add chemicals that make them extremely addicting. Same with FB, they have optimized for outrage culture etc. - whatever brings the most "engagement".
riscy 6y ago
The key thing is that they built a reward system into sharing each of those thoughts, which enables an addiction for acceptance from users on the platform. Facebook’s engagement systems / algorithms feed on these addictions. As the article discusses, this causes users to radicalize their posts to garner more attention.

mrweasel 6y ago

What makes Facebook worse, in my mind, is that people don't seem to care that it's their real name associated with terrible radical, violent and angry comment. At least on something like 4Chan you'd be anonymous, but nope, people will write the most grotesque comments, while everyone can see their real name, where they live and who they work for. Some people don't seem to care, and that's pretty scary.

JumpCrisscross 6y ago
> people don't seem to care that it's their real name associated with terrible radical, violent and angry comment The hypothesis of real name policies was that people would be civil online if their real-world identities grounded them. In reality, the tail wagged the dog.
- nr2x 6y ago
  Very astute observation.
- coffeefirst 6y ago
  Yeah, it may have worked in some contexts, but people also read the room. And when the algorithm is setting the room to "1 tick more extreme than you" they gradually dial up. I've watched this process turn friends who had pretty mild political leanings to "burn it all down" radicals or transform into right wing talk radio hosts. It's not good.
- dragontamer 6y ago
  Yeah, but anyone who knew a thing about Usenet back in the day knows that was false. I mean, it was a theory that was popular in the 00s, mostly because a ton of people at that time didn't have experience with Usenet. But a lot of vile language was used in the clear, signed with the real names of many individuals. Anonymity doesn't seem to change much. Humans will be humans, whether or not they're anonymous.
- brlewis 6y ago
  Was that the hypothesis, or just the marketing? Obviously real names make it easier to integrate marketing data from disparate sources.

tomp 6y ago

Well, with #BlackLivesMatter and #KillAllMen, I kind-of doubt the former assertion...

sct202 6y ago

I've had fb friends report spammed into being temp or permabanned very easily (personal feuds, stalkers), so I would question how they're even measuring quality at this point.

ezzzzz 6y ago

Although I am concerned about the 1A and big tech being the arbiters of it, it seems like such a system has yet to go to production (probably never will). I understand that there are technical, financial, and legal complexities that are much more nuanced than 'FB doesn't care, as long as they can profit from potentially dangerous content' and 'FB just wants to suppress any speech which doesn't align with their neo-liberal ideologies'. What I find troubling is that, a system to find/flag radical/violent content likely exists within the auspices of the NSA. Despite the massive infringement of privacy that these systems create, it seems that they still can't prevent people from being 'radicalized' by content from the platforms that is designed to do exactly that. Furthermore, they couldn't (or didn't) do anything to actually prevent people from storming the Capital. This begs the question, does this massive surveillance state need to exist if it does not do the thing it is purported to do (find and suppress violence, prevent terrorist attacks, etc). If the owners of the platforms have any conscience, unfortunately, it seems that the responsibility will be up to them. It's murky territory for sure, and you have to strike a balance between far reaching censorship (i.e. infringing of 1A) and allowing disinformation, hate speech, and calls to violence to spread like wild-fire (i.e. how we got into this mess in the first place). I hate to say it, but the older I get, the more something like 'the great firewall' of China seems to make more sense.

agumonkey 6y ago
I don't think the NSA was engineered with internet / social networks scale though. But they were already failing in the 9/11 days soo..
- ezzzzz 6y ago
  I mean, PRISM was a post 9/11 thing and yeah, Snowden's leaks directly noted Facebook, YouTube, etc. There is really no telling what exactly is out there, but we know the NSA has massive Data Centers and a massive budget. Basically, even the most personal private information about literally any citizen is just a FISA Warrant away. Things posted in a public forum are a cakewalk. Maybe they aren't doing anything novel with this data, whatever, this is beside the point. The point is, we can't really count on the State to regulate this stuff, even though, you know, creating and enforcing laws is literally the reason the state exists... IDK, I hesitantly have to side with big-tech on this, but this is a problem that the State either can't fix, or will not fix. The internet is a wonderful thing, but it is also very dangerous. Having unfettered access to information, be it true or false, can wield some horrible side effects when such a large swath of the population lacks basic BS detection, critical-thinking and like, base level understanding of civics. Do we need a Vanguard? I wouldn't trust one coming from either side of the aisle in the current climate. Seems like 'censorship' from the platforms is the only viable solution I can see, but that can take a pretty dark turn. At least if it's a private company, you can always leave, create your own thing or find something else...for now.

sydd 6y ago

Not trying to defend Facebook, but they did not do it on purpose. They developed a really addicting social network where you do as many interactions as possible. When Facebook started it was a different world: Almost no data privacy laws, internet security in its infancy (e.g. in 2008 HTTPS was a rare new thing, not cross-origin security etc.) and people were spamming Farmville notifications to each other. It just turned out that this structure is an excellent ground for lots of bad things.

deleted 6y ago
[deleted]
nr2x 6y ago
That's not true: the EU had lots of national privacy laws under the privacy directive (GDPR was partially intended to unify/simplify that), and the US had sectoral laws such as COPPA and HIPAA, the former of which very much applies to Facebook.

marcod 6y ago

Wasn't there also a system developed that was really good at flagging radical content and proceeded to flag all the conservative posts? :p