8 ms·

How Facebook got addicted to spreading misinformation

128 points by 7d7n 6y ago

ilamont 6y ago
The algorithms that underpin Facebook’s business weren’t created to filter out what was false or inflammatory; they were designed to make people share and engage with as much content as possible by showing them things they were most likely to be outraged or titillated by. I find it interesting that this article does not even mention some of the other platforms that have also grown off this strategy, and have contributed to the societal problems named in the article. I joke with my wife that you can innocently watch a YouTube press conference featuring our moderate Republican governor discussing COVID policy one day, and before you know it you're being recommended videos by rabid anti-vaxxers and QAnon deep state conspiracy theorists. It's not just a Facebook problem or an "AI" problem, it's a platform problem which requires wider solutions than the broken internal approaches described in this article.
- TheJoYo 6y ago
  Platforms that stick to chronological order are growing without the influence of this strategy. Mastodon has been doing very well by simply offering the Email of social media.
  - Nextgrid 6y ago
    I agree that chronological order and more generally not overriding user's preferences to maximize engagement would solve this problem. I disagree that Mastodon (or other federated alternatives) are a good example of this because these projects don't have a business model to support, so obviously they don't have to care about maximizing engagement. I'm not saying maximizing engagement is a good idea, or that engagement-based business models are a good idea, I'm just saying that comparing a platform with an incentive to make money to one that does not doesn't answer anything.
    - TheJoYo 6y ago
      everything feeds the basilisk.
  - throwawayboise 6y ago
    Outside of HN I have never heard or seen anyone talk or write about Mastodon. For the general pubic, it doesn't exist.
- ggggtez 6y ago
  I think it's worth pointing out that Facebook isn't the only platform, though I don't think every article needs to cover all social media equally. This article does a deeper dive on one particular company's issues (and the company that's widely acknowledged to have known about the issue the longest, and done the least to fix it). There are other articles that tackle this topic for YouTube [1]. But as far as I know, we're still waiting on any expose that shows that YouTube had done extensive research into the issue and opted to do nothing to continue growth. That dubious honor seems to be reserved for Facebook at least for now. [1] https://www.usatoday.com/story/tech/2021/02/12/youtube-channel-recommendation-white-supremacist-extremist-videos/6712787002/ https://www.usatoday.com/story/tech/2021/02/12/youtube-chann... > [In 2017,] YouTube limited recommendations on those videos and disabled features such as commenting and sharing. But it didn’t remove them. The company said the crackdown reduced views of supremacist videos by 80%.
contemporary343 6y ago
"In 2017, Chris Cox, Facebook’s longtime chief product officer, formed a new task force to understand whether maximizing user engagement on Facebook was contributing to political polarization. It found that there was indeed a correlation, and that reducing polarization would mean taking a hit on engagement. In a mid-2018 document reviewed by the Journal, the task force proposed several potential fixes, such as tweaking the recommendation algorithms to suggest a more diverse range of groups for people to join. But it acknowledged that some of the ideas were “antigrowth.” Most of the proposals didn’t move forward, and the task force disbanded. Since then, other employees have corroborated these findings. A former Facebook AI researcher who joined in 2018 says he and his team conducted “study after study” confirming the same basic idea: models that maximize engagement increase polarization. They could easily track how strongly users agreed or disagreed on different issues, what content they liked to engage with, and how their stances changed as a result. Regardless of the issue, the models learned to feed users increasingly extreme viewpoints. “Over time they measurably become more polarized,” he says." Facebook has repeatedly shown that it will maximize its bottom line and growth (in particular, as measured by engagement metrics) above everything else. Fair enough, they're a for-profit corporation! I just can't really take seriously their claims of 'self-regulation'. They simply can never be trusted to self-regulate because it's just not in their interest, and they've never shown meaningful actions in this regard. They will only respond to the legal stick - and it is most certainly coming.
- contemporary343 6y ago
  Another key quote that explains the fundamental misalignment of incentives at the employee level - you're rewarded for increasing engagment metrics. So why do something that might decrease it? "But anything that reduced engagement, even for reasons such as not exacerbating someone’s depression, led to a lot of hemming and hawing among leadership. With their performance reviews and salaries tied to the successful completion of projects, employees quickly learned to drop those that received pushback and continue working on those dictated from the top down."
  - cosmodisk 6y ago
    It's funny how probably less than 50 people in one private company can have such a great impact on hundreds of millions of people worldwide.
ctocoder 6y ago
When the 1st Facebook API was released, I helped build the largest application called SuperWall. The way to increase virality and activity on the wall, was to allow a copy paste virus to perpetuate throughout the network. The DAU with this virus that "Zuckerberg is going to delete your account if you do not log in" kept DAU and virial growth at peak levels. Facebook could do this too and get the same results. More DAU more money.
adjkant 6y ago
Despite the title, this is very much more than fluff and would highly recommend reading it in full.
syamilmj 6y ago
You could go on Facebook with no opinion on anything and leave an extremist.
- barbazoo 6y ago
  The same way you could be on Facebook and not being radicalized at all. At the end of the day it's people who chose to be misinformed, who chose to not question, not look elsewhere, not to be skeptic about how content is making them feel. Facebook just shamelessly takes advantage of people's shortcomings, for profit.
  - aidenn0 6y ago
    I mean, if I walk up to random people at a bar and tell them that their mother is a whore, anybody who punches me does so of their own free will, but I'm not entirely free of culpability for the violence.
  - eigenket 6y ago
    I would agree with the statement: > At the end of the day it's people who are more easily misinformed, who do not question, do not look elsewhere, are not skeptical about how content is making them feel What is your reasoning for declaring that this is a matter of "choice"?
    - barbazoo 6y ago
      I feel that it's a choice because it's under our control what we do. We chose to believe, we chose to hate, to judge. Believing everything FB puts in front of you in their feed is just lazy. We have to do better than that.
  - temp8964 6y ago
    It's not fair to just blame Facebook though. Twitter, reddit, CNN, FoxNews, all have the same kind of behavior. It's also not fair to just blame the media. How about the politicians? Any major political leader dare to elaborate a balanced view at all? It seems to be deeply rooted in partisan politics.
coldcode 6y ago
When your founder and CEO makes growth the only thing that matters, everything else is not important. If selling people lies makes more money, sell more lies, etc. If Facebook charged people $1 a month per account, and deleted all the ads, the data selling, the lie promotions, etc, 95% of their company could be laid off. They would also make less money for sure, and that's something Zuckerberg could never support.
mooneater 6y ago
Killer article here by Karen Hao, hard hitting ending. Wonder what fb thought they were getting into here with MTR!
sodality2 6y ago
♪We know it's not right♪ ♪We know it's not funny♪ ♪But we'll stop beating this dead horse♪ ♪when it stops spitting out money♪
- kaycebasques 6y ago
  Tangentially related, this quote is from Bo Burnham's song about the formulaic-ness of pop music: https://www.youtube.com/watch?v=nsCdP-Q_HXI https://www.youtube.com/watch?v=nsCdP-Q_HXI Edit: actually regarding formulaic-ness I'm thinking about Country Song. Repeat Stuff is about how pop songs just repeat the same lines now, as you might guess from the title https://youtu.be/y7im5LT09a0 https://youtu.be/y7im5LT09a0
  - sodality2 6y ago
    Yep! His songs are really funny. Very sad he sort of retired but I'm happy he's working on what he enjoys, with the movie he directed.
strangeloops85 6y ago
One take-away I had from reading this excellent and detailed article is the fundamental tension in Facebook's approach to moderating content: They rely on a hammer to catch what is objectionable or misinformation, but optimize for engagement maximally with everything else. However the hammer misses plenty of newly emerging inflammatory content (for example, anti-Rohingya content would look very different from anything prior), and this content gets maximally amplified because the overall newsfeed algorithms optimize for engagement. An alternate approach would be to sacrifice engagement overall (perhaps only a little!) to reduce the very real and negative consequences of undesirable content that slips past the detection algorithms (which it always will). I suspect some of the fixes that were proposed, but never implemented, effectively did this. But they were shot down because, well, the top-line engagement numbers would dip.
thitcanh 6y ago
Has anyone ever looked at history and figured out that humans are pretty damn awful? Facebook is just a catalyzer just like Twitter is. Every platform has their own disgusting self-centered bubbles and that’s not because of the platform. The bar to get banned from any online platform is pretty high. Just look at how long it took for the most prominent Twitter offender to get rejected. You can ban all public forums you want, then people will just continue sharing BS on WhatsApp without you even noticing. The problem isn’t Facebook but people. Governments should punish people, not outsource the policing to companies.
- sneak 6y ago
  > Governments should punish people, not outsource the policing to companies. Governments shouldn't be in the business of deciding what is truth and what is "misinformation", and they certainly shouldn't be punishing people based on the outcomes of such determinations. Otherwise, we'd have to ban the church and jail all the priests.
  - shkkmo 6y ago
    Governments are already in the business of deciding many truths. It is called the court system. There are people who are deliberately spreading misinformation to make money. I think that we absolutely should be investigating and prosecuting such people. We simply lack the policing resources to effectively combat online fraud, online bullying, and online extortion. I think we need a fundamental restructuring of policing on our society to make this feasible.
    - sneak 6y ago
      > There are people who are deliberately spreading misinformation to make money. I think that we absolutely should be investigating and prosecuting such people. The US constitution prevents bans on televangelists. In the US, to achieve your goal you'll have to start off with a constitutional amendment repealing the first amendment.
      - 6y ago
sneak 6y ago
Why does everyone seem to be convinced that the spread of misinformation is somehow suddenly a problem? It has been widespread for centuries in our societies and with the exception of a few cases (say, Galileo, crusades, et c) it hasn't really consistently caused any major issues. This feels very much to me like a "something must be done!"
- deleted 6y ago
  [deleted]
- helloworld11 6y ago
  The mainstream media remains furious about Trump having won unexpectedly right under their noses in 2016 and couple with this their general disdain for social media platforms or anything at all that steals their power to impose their notion of authoritative opinion to get the current rage over the largely invented demon of misinformation. To make "misinformation" into an even bigger boogeyman, we have a large progressive, left, woke-oriented subset of the population that largely gets pandered to by the main media outlets increasingly polarizing towards absolute intolerance of anything against their cherished dogmas and being more than happy to have a convenient label for even very moderate divergent arguments. It's noteworthy that so much of the supposed issues around the misinformation debate focus entirely on painting all right-oriented opinion as part of the extremes, while visibly ignoring how much polarization in the other direction also exists towards its own extremes.
andred14 6y ago
This year I was suspended multiple times from Facebook for posting real science and facts that challenge the current narrative. This should alarm and encourage you to seek other platforms.
andyxor 6y ago
A better question is how MSM got addicted to spreading misinformation. 'Hate speech' these days means 'speech I hate and want silenced'. Freedom of speech protections exist for this exact reason. Why should a social media app engage in political censorship and selecting right vs. wrong opinion?
ldbooth 6y ago
amazing how the salary changes the employee's perspective. it has been said elsewhere: engagement = addiction dial up the addiction, dial up the profit. "Regulators! mount up"
throwitaway1235 6y ago
Stop trying to police what people think. It's disgusting.
threesmegiste 6y ago
Again facebook news. Tricky as always. There is only one evil in big tech. Others are angel. Dont say they all same. These kind of frequent news will guide you. And you start to say “ X is also evil i know but not much like facebook”. And that what they want us to think. On safari you dont need to run faster than lion. Just dont be the last among runners. When somebody talk about privacy, sentences start with Facebook and the rest is unimportant. Basic media manupilation and psychology.
blendo 6y ago
“I think what happens to most people who work at Facebook—and definitely has been my story—is that there’s no boundary between Facebook and me,” he says. “It’s extremely personal.”

128 points by 7d7n 6y ago

The algorithms that underpin Facebook’s business weren’t created to filter out what was false or inflammatory; they were designed to make people share and engage with as much content as possible by showing them things they were most likely to be outraged or titillated by. I find it interesting that this article does not even mention some of the other platforms that have also grown off this strategy, and have contributed to the societal problems named in the article. I joke with my wife that you can innocently watch a YouTube press conference featuring our moderate Republican governor discussing COVID policy one day, and before you know it you're being recommended videos by rabid anti-vaxxers and QAnon deep state conspiracy theorists. It's not just a Facebook problem or an "AI" problem, it's a platform problem which requires wider solutions than the broken internal approaches described in this article.

TheJoYo 6y ago
Platforms that stick to chronological order are growing without the influence of this strategy. Mastodon has been doing very well by simply offering the Email of social media.
- Nextgrid 6y ago
  I agree that chronological order and more generally not overriding user's preferences to maximize engagement would solve this problem. I disagree that Mastodon (or other federated alternatives) are a good example of this because these projects don't have a business model to support, so obviously they don't have to care about maximizing engagement. I'm not saying maximizing engagement is a good idea, or that engagement-based business models are a good idea, I'm just saying that comparing a platform with an incentive to make money to one that does not doesn't answer anything.
  - TheJoYo 6y ago
    everything feeds the basilisk.
- throwawayboise 6y ago
  Outside of HN I have never heard or seen anyone talk or write about Mastodon. For the general pubic, it doesn't exist.
ggggtez 6y ago
I think it's worth pointing out that Facebook isn't the only platform, though I don't think every article needs to cover all social media equally. This article does a deeper dive on one particular company's issues (and the company that's widely acknowledged to have known about the issue the longest, and done the least to fix it). There are other articles that tackle this topic for YouTube [1]. But as far as I know, we're still waiting on any expose that shows that YouTube had done extensive research into the issue and opted to do nothing to continue growth. That dubious honor seems to be reserved for Facebook at least for now. [1] https://www.usatoday.com/story/tech/2021/02/12/youtube-channel-recommendation-white-supremacist-extremist-videos/6712787002/ https://www.usatoday.com/story/tech/2021/02/12/youtube-chann... > [In 2017,] YouTube limited recommendations on those videos and disabled features such as commenting and sharing. But it didn’t remove them. The company said the crackdown reduced views of supremacist videos by 80%.

contemporary343 6y ago

"In 2017, Chris Cox, Facebook’s longtime chief product officer, formed a new task force to understand whether maximizing user engagement on Facebook was contributing to political polarization. It found that there was indeed a correlation, and that reducing polarization would mean taking a hit on engagement. In a mid-2018 document reviewed by the Journal, the task force proposed several potential fixes, such as tweaking the recommendation algorithms to suggest a more diverse range of groups for people to join. But it acknowledged that some of the ideas were “antigrowth.” Most of the proposals didn’t move forward, and the task force disbanded. Since then, other employees have corroborated these findings. A former Facebook AI researcher who joined in 2018 says he and his team conducted “study after study” confirming the same basic idea: models that maximize engagement increase polarization. They could easily track how strongly users agreed or disagreed on different issues, what content they liked to engage with, and how their stances changed as a result. Regardless of the issue, the models learned to feed users increasingly extreme viewpoints. “Over time they measurably become more polarized,” he says." Facebook has repeatedly shown that it will maximize its bottom line and growth (in particular, as measured by engagement metrics) above everything else. Fair enough, they're a for-profit corporation! I just can't really take seriously their claims of 'self-regulation'. They simply can never be trusted to self-regulate because it's just not in their interest, and they've never shown meaningful actions in this regard. They will only respond to the legal stick - and it is most certainly coming.

contemporary343 6y ago
Another key quote that explains the fundamental misalignment of incentives at the employee level - you're rewarded for increasing engagment metrics. So why do something that might decrease it? "But anything that reduced engagement, even for reasons such as not exacerbating someone’s depression, led to a lot of hemming and hawing among leadership. With their performance reviews and salaries tied to the successful completion of projects, employees quickly learned to drop those that received pushback and continue working on those dictated from the top down."
- cosmodisk 6y ago
  It's funny how probably less than 50 people in one private company can have such a great impact on hundreds of millions of people worldwide.

ctocoder 6y ago

When the 1st Facebook API was released, I helped build the largest application called SuperWall. The way to increase virality and activity on the wall, was to allow a copy paste virus to perpetuate throughout the network. The DAU with this virus that "Zuckerberg is going to delete your account if you do not log in" kept DAU and virial growth at peak levels. Facebook could do this too and get the same results. More DAU more money.

adjkant 6y ago

Despite the title, this is very much more than fluff and would highly recommend reading it in full.

syamilmj 6y ago

You could go on Facebook with no opinion on anything and leave an extremist.

barbazoo 6y ago
The same way you could be on Facebook and not being radicalized at all. At the end of the day it's people who chose to be misinformed, who chose to not question, not look elsewhere, not to be skeptic about how content is making them feel. Facebook just shamelessly takes advantage of people's shortcomings, for profit.
- aidenn0 6y ago
  I mean, if I walk up to random people at a bar and tell them that their mother is a whore, anybody who punches me does so of their own free will, but I'm not entirely free of culpability for the violence.
- eigenket 6y ago
  I would agree with the statement: > At the end of the day it's people who are more easily misinformed, who do not question, do not look elsewhere, are not skeptical about how content is making them feel What is your reasoning for declaring that this is a matter of "choice"?
  - barbazoo 6y ago
    I feel that it's a choice because it's under our control what we do. We chose to believe, we chose to hate, to judge. Believing everything FB puts in front of you in their feed is just lazy. We have to do better than that.
- temp8964 6y ago
  It's not fair to just blame Facebook though. Twitter, reddit, CNN, FoxNews, all have the same kind of behavior. It's also not fair to just blame the media. How about the politicians? Any major political leader dare to elaborate a balanced view at all? It seems to be deeply rooted in partisan politics.

coldcode 6y ago

When your founder and CEO makes growth the only thing that matters, everything else is not important. If selling people lies makes more money, sell more lies, etc. If Facebook charged people $1 a month per account, and deleted all the ads, the data selling, the lie promotions, etc, 95% of their company could be laid off. They would also make less money for sure, and that's something Zuckerberg could never support.

mooneater 6y ago

Killer article here by Karen Hao, hard hitting ending. Wonder what fb thought they were getting into here with MTR!

sodality2 6y ago

♪We know it's not right♪ ♪We know it's not funny♪ ♪But we'll stop beating this dead horse♪ ♪when it stops spitting out money♪

kaycebasques 6y ago
Tangentially related, this quote is from Bo Burnham's song about the formulaic-ness of pop music: https://www.youtube.com/watch?v=nsCdP-Q_HXI https://www.youtube.com/watch?v=nsCdP-Q_HXI Edit: actually regarding formulaic-ness I'm thinking about Country Song. Repeat Stuff is about how pop songs just repeat the same lines now, as you might guess from the title https://youtu.be/y7im5LT09a0 https://youtu.be/y7im5LT09a0
- sodality2 6y ago
  Yep! His songs are really funny. Very sad he sort of retired but I'm happy he's working on what he enjoys, with the movie he directed.

strangeloops85 6y ago

One take-away I had from reading this excellent and detailed article is the fundamental tension in Facebook's approach to moderating content: They rely on a hammer to catch what is objectionable or misinformation, but optimize for engagement maximally with everything else. However the hammer misses plenty of newly emerging inflammatory content (for example, anti-Rohingya content would look very different from anything prior), and this content gets maximally amplified because the overall newsfeed algorithms optimize for engagement. An alternate approach would be to sacrifice engagement overall (perhaps only a little!) to reduce the very real and negative consequences of undesirable content that slips past the detection algorithms (which it always will). I suspect some of the fixes that were proposed, but never implemented, effectively did this. But they were shot down because, well, the top-line engagement numbers would dip.

thitcanh 6y ago

Has anyone ever looked at history and figured out that humans are pretty damn awful? Facebook is just a catalyzer just like Twitter is. Every platform has their own disgusting self-centered bubbles and that’s not because of the platform. The bar to get banned from any online platform is pretty high. Just look at how long it took for the most prominent Twitter offender to get rejected. You can ban all public forums you want, then people will just continue sharing BS on WhatsApp without you even noticing. The problem isn’t Facebook but people. Governments should punish people, not outsource the policing to companies.

sneak 6y ago
> Governments should punish people, not outsource the policing to companies. Governments shouldn't be in the business of deciding what is truth and what is "misinformation", and they certainly shouldn't be punishing people based on the outcomes of such determinations. Otherwise, we'd have to ban the church and jail all the priests.
- shkkmo 6y ago
  Governments are already in the business of deciding many truths. It is called the court system. There are people who are deliberately spreading misinformation to make money. I think that we absolutely should be investigating and prosecuting such people. We simply lack the policing resources to effectively combat online fraud, online bullying, and online extortion. I think we need a fundamental restructuring of policing on our society to make this feasible.
  - sneak 6y ago
    > There are people who are deliberately spreading misinformation to make money. I think that we absolutely should be investigating and prosecuting such people. The US constitution prevents bans on televangelists. In the US, to achieve your goal you'll have to start off with a constitutional amendment repealing the first amendment.
    - 6y ago

sneak 6y ago

Why does everyone seem to be convinced that the spread of misinformation is somehow suddenly a problem? It has been widespread for centuries in our societies and with the exception of a few cases (say, Galileo, crusades, et c) it hasn't really consistently caused any major issues. This feels very much to me like a "something must be done!"

deleted 6y ago
[deleted]
helloworld11 6y ago
The mainstream media remains furious about Trump having won unexpectedly right under their noses in 2016 and couple with this their general disdain for social media platforms or anything at all that steals their power to impose their notion of authoritative opinion to get the current rage over the largely invented demon of misinformation. To make "misinformation" into an even bigger boogeyman, we have a large progressive, left, woke-oriented subset of the population that largely gets pandered to by the main media outlets increasingly polarizing towards absolute intolerance of anything against their cherished dogmas and being more than happy to have a convenient label for even very moderate divergent arguments. It's noteworthy that so much of the supposed issues around the misinformation debate focus entirely on painting all right-oriented opinion as part of the extremes, while visibly ignoring how much polarization in the other direction also exists towards its own extremes.

andred14 6y ago

This year I was suspended multiple times from Facebook for posting real science and facts that challenge the current narrative. This should alarm and encourage you to seek other platforms.

andyxor 6y ago

A better question is how MSM got addicted to spreading misinformation. 'Hate speech' these days means 'speech I hate and want silenced'. Freedom of speech protections exist for this exact reason. Why should a social media app engage in political censorship and selecting right vs. wrong opinion?

ldbooth 6y ago

amazing how the salary changes the employee's perspective. it has been said elsewhere: engagement = addiction dial up the addiction, dial up the profit. "Regulators! mount up"

throwitaway1235 6y ago

Stop trying to police what people think. It's disgusting.

threesmegiste 6y ago

Again facebook news. Tricky as always. There is only one evil in big tech. Others are angel. Dont say they all same. These kind of frequent news will guide you. And you start to say “ X is also evil i know but not much like facebook”. And that what they want us to think. On safari you dont need to run faster than lion. Just dont be the last among runners. When somebody talk about privacy, sentences start with Facebook and the rest is unimportant. Basic media manupilation and psychology.

blendo 6y ago

“I think what happens to most people who work at Facebook—and definitely has been my story—is that there’s no boundary between Facebook and me,” he says. “It’s extremely personal.”