4 ms·

This is a very hard problem that YouTube and Facebook made for themselves by becoming the world’s largest advertising platforms. They depend on engagement for a

This is a very hard problem that YouTube and Facebook made for themselves by becoming the world’s largest advertising platforms. They depend on engagement for ad revenue, they designed world-class algorithms to promote this engagement, and it turned out that extremist content happens to be very engaging. And so the problem is, building an algorithm that blindly promotes whatever keeps users on the site, for all its complexity, is a far more tractable problem compared to building a system that can avoid promoting content that promotes violence. In the meantime, they throw armies of people at the problem, to moderate content and respond to user reports, but it’s a losing battle. They had the technology to create a monster, but don’t have the technology to stop it.

rcoveson 7y ago
> They had the technology to create a monster, but don’t have the technology to stop it. At first I was going to argue that they could stop it and that they did have the "technology" to do so, but the more I thought about it the more I concluded that in the system we have built, they really cannot. Profitable ideas are unstoppable until they are proved less profitable than some other idea, or regulated out of existence. Fiduciary duty enshrines this into law. Sufficient competition ensures that it is the only winning strategy. Obligatory SSC: https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/
- astrange 7y ago
  Facebook doesn't have fiduciary duties to anyone, unless they're registered financial advisors.
- deogeo 7y ago
  > Fiduciary duty enshrines this into law. A common misconception: https://www.nytimes.com/roomfordebate/2015/04/16/what-are-corporations-obligations-to-shareholders/corporations-dont-have-to-maximize-profits https://www.nytimes.com/roomfordebate/2015/04/16/what-are-co...
kragen 7y ago
Suppose you have two video-sharing platforms which use basically the same recommendation algorithms (etc.) to keep people engaged and clicking on videos, and they're equally effective — except that one of them censors recommendations based on some political criteria, for example, being "extremist", such as whoever the modern equivalent of Martin Luther King is. Let's call the censored platform "JedgarTube" and the other one "MLKTube", for lack of better terms. There are a couple of possibilities: 1. Those recommendations were actually less effective at keeping people engaged than whatever recommendations replace them on JedgarTube. In that case, MLKTube needs to copy the decision in order not to lose users gradually to JedgarTube. In fact, whichever platform blocks those recommendations first will experience improved user growth and engagement. Essentially the recommendations were just a bug of a primitive recommendation algorithm. 2. Those recommendations were actually more effective at keeping people engaged than whatever recommendations replace them on JedgarTube. In that case, JedgarTube will gradually lose users to MLKTube, again, assuming the platforms are otherwise equal. Of course, the truth is that any particular censorship decision could fall into either #1 or #2. The #1 censorship decisions will be copied by MLKTube, if they aren't too incompetent or principled, while the #2 decisions will gradually accumulate into a competitive disadvantage for eyeballs at JedgarTube. That's why the media platforms and government censors are trying to set up a global censorship system — whichever platform steps up first to be JedgarTube will lose viewers to whoever's censorship implementation is a step or two behind. Note that none of this logic depends on normative judgments such as "extremist content is bad", "extremist content is good", "people should have freedom of speech", "people shouldn't have freedom of speech", "platforms shouldn't manipulate people with algorithmic recommendations", "algorithmic recommendations are good for people", or anything like that. It's purely reasoning about objective causes and effects about people's behavior, although I've chosen my terms to weaken the evident bias against "extremism" in this discussion so readers can reason about these causes and effects instead of being thrown around by their emotional biases.
- stcredzero 7y ago
  That's why the media platforms and government censors are trying to set up a global censorship system — whichever platform steps up first to be JedgarTube will lose viewers to whoever's censorship implementation is a step or two behind. The market isn't going to do what we want, so we're going to implement it using centralized power instead. I didn't elect these big companies to censor the public discourse, and it's highly disturbing that they're working hand in hand with the government to do this. Note that none of this logic depends on normative judgments The "logic" depends on the normative judgement that what would result from a free market would be bad. There have been too few big players doing too much meddling and manipulation of the markets contained within their walled gardens to know whether that would be the case or not.
bsder 7y ago
> They had the technology to create a monster, but don’t have the technology to stop it. They don't have the incentive to stop it. All you have to do is make posting have an actual price, no matter how nominal, and most of this goes away. Unfortunately for their profits, it will wipe out 90+% of their user base. And it will kill virality cold (you won't forward something to your whole address book if it costs you a dollar).
- ocdtrekkie 7y ago
  The other incentive option is to charge the platform for the violation. Aka, remove Section 230. If the platform was to be charged for posting dangerous content, the platform would magically be able to largely prevent it overnight.
  - qball 7y ago
    >Aka, remove Section 230. If the platform was to be charged for posting dangerous content, the platform would magically be able to largely prevent it overnight. That's not actually what Section 230 does. It prevents the platform from being charged if while moderating it makes a mistake. Removing Section 230 would do more harm than good if you consider censorship good- because then the only option to escape legal liability would be to allow everything. Remember that Section 230 was part of the Communications Decency Act and was designed to ensure that websites could remove pornographic and objectionable content without being responsible (of course, it's also the only part of that law that was found constitutional, and for good reason). By taking the only legal immunity for half-baked censorship away (i.e. private company gets to moderate what's posted but won't suffer consequences for getting it wrong) you've closed the door even harder on it- US law won't actually let you censor consequence-free otherwise, which is the whole reason S.230 was added in the first place.
  - wbl 7y ago
    Charge them with what? In the US there is nothing illegal about being a Nazi. There is nothing illegal in selling Mein Kampf etc.
    - 7y ago
skybrian 7y ago
Except that advertisers are not interested in sponsoring extremist content, so you could argue the incentives go in the other direction.
- kragen 7y ago
  YouTube permits a lot of demonetized videos they don't dare to sell advertising on, or (previously) that YouTubers wouldn't give permission to sell advertising on, because the alternative of erasing the videos would send the YouTubers and their viewers to DailyMotion or Vimeo. As long as the demonetized viewers occasionally watch a non-demonetized video, or the sometimes-demonetized YouTubers occasionally make non-demonetized videos, YouTube makes more money that way.
  - skybrian 7y ago
    That's a weaker incentive, though. And it comes with a risk of occasional scandals that scare away advertisers, at least temporarily. I don't think incentives are enough to predict what YouTube will do. The future is uncertain, there are a lot of policies that could plausibly make money, and the decision-makers are people who read the news and have opinions about what's right, not paperclip maximizers. Politics matters as well as economics.
    - kragen 7y ago
      True!

rcoveson 7y ago

> They had the technology to create a monster, but don’t have the technology to stop it. At first I was going to argue that they could stop it and that they did have the "technology" to do so, but the more I thought about it the more I concluded that in the system we have built, they really cannot. Profitable ideas are unstoppable until they are proved less profitable than some other idea, or regulated out of existence. Fiduciary duty enshrines this into law. Sufficient competition ensures that it is the only winning strategy. Obligatory SSC: https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/

astrange 7y ago
Facebook doesn't have fiduciary duties to anyone, unless they're registered financial advisors.
deogeo 7y ago
> Fiduciary duty enshrines this into law. A common misconception: https://www.nytimes.com/roomfordebate/2015/04/16/what-are-corporations-obligations-to-shareholders/corporations-dont-have-to-maximize-profits https://www.nytimes.com/roomfordebate/2015/04/16/what-are-co...

kragen 7y ago

Suppose you have two video-sharing platforms which use basically the same recommendation algorithms (etc.) to keep people engaged and clicking on videos, and they're equally effective — except that one of them censors recommendations based on some political criteria, for example, being "extremist", such as whoever the modern equivalent of Martin Luther King is. Let's call the censored platform "JedgarTube" and the other one "MLKTube", for lack of better terms. There are a couple of possibilities: 1. Those recommendations were actually less effective at keeping people engaged than whatever recommendations replace them on JedgarTube. In that case, MLKTube needs to copy the decision in order not to lose users gradually to JedgarTube. In fact, whichever platform blocks those recommendations first will experience improved user growth and engagement. Essentially the recommendations were just a bug of a primitive recommendation algorithm. 2. Those recommendations were actually more effective at keeping people engaged than whatever recommendations replace them on JedgarTube. In that case, JedgarTube will gradually lose users to MLKTube, again, assuming the platforms are otherwise equal. Of course, the truth is that any particular censorship decision could fall into either #1 or #2. The #1 censorship decisions will be copied by MLKTube, if they aren't too incompetent or principled, while the #2 decisions will gradually accumulate into a competitive disadvantage for eyeballs at JedgarTube. That's why the media platforms and government censors are trying to set up a global censorship system — whichever platform steps up first to be JedgarTube will lose viewers to whoever's censorship implementation is a step or two behind. Note that none of this logic depends on normative judgments such as "extremist content is bad", "extremist content is good", "people should have freedom of speech", "people shouldn't have freedom of speech", "platforms shouldn't manipulate people with algorithmic recommendations", "algorithmic recommendations are good for people", or anything like that. It's purely reasoning about objective causes and effects about people's behavior, although I've chosen my terms to weaken the evident bias against "extremism" in this discussion so readers can reason about these causes and effects instead of being thrown around by their emotional biases.

stcredzero 7y ago
That's why the media platforms and government censors are trying to set up a global censorship system — whichever platform steps up first to be JedgarTube will lose viewers to whoever's censorship implementation is a step or two behind. The market isn't going to do what we want, so we're going to implement it using centralized power instead. I didn't elect these big companies to censor the public discourse, and it's highly disturbing that they're working hand in hand with the government to do this. Note that none of this logic depends on normative judgments The "logic" depends on the normative judgement that what would result from a free market would be bad. There have been too few big players doing too much meddling and manipulation of the markets contained within their walled gardens to know whether that would be the case or not.

bsder 7y ago

> They had the technology to create a monster, but don’t have the technology to stop it. They don't have the incentive to stop it. All you have to do is make posting have an actual price, no matter how nominal, and most of this goes away. Unfortunately for their profits, it will wipe out 90+% of their user base. And it will kill virality cold (you won't forward something to your whole address book if it costs you a dollar).

ocdtrekkie 7y ago
The other incentive option is to charge the platform for the violation. Aka, remove Section 230. If the platform was to be charged for posting dangerous content, the platform would magically be able to largely prevent it overnight.
- qball 7y ago
  >Aka, remove Section 230. If the platform was to be charged for posting dangerous content, the platform would magically be able to largely prevent it overnight. That's not actually what Section 230 does. It prevents the platform from being charged if while moderating it makes a mistake. Removing Section 230 would do more harm than good if you consider censorship good- because then the only option to escape legal liability would be to allow everything. Remember that Section 230 was part of the Communications Decency Act and was designed to ensure that websites could remove pornographic and objectionable content without being responsible (of course, it's also the only part of that law that was found constitutional, and for good reason). By taking the only legal immunity for half-baked censorship away (i.e. private company gets to moderate what's posted but won't suffer consequences for getting it wrong) you've closed the door even harder on it- US law won't actually let you censor consequence-free otherwise, which is the whole reason S.230 was added in the first place.
- wbl 7y ago
  Charge them with what? In the US there is nothing illegal about being a Nazi. There is nothing illegal in selling Mein Kampf etc.
  - 7y ago

skybrian 7y ago

Except that advertisers are not interested in sponsoring extremist content, so you could argue the incentives go in the other direction.

kragen 7y ago
YouTube permits a lot of demonetized videos they don't dare to sell advertising on, or (previously) that YouTubers wouldn't give permission to sell advertising on, because the alternative of erasing the videos would send the YouTubers and their viewers to DailyMotion or Vimeo. As long as the demonetized viewers occasionally watch a non-demonetized video, or the sometimes-demonetized YouTubers occasionally make non-demonetized videos, YouTube makes more money that way.
- skybrian 7y ago
  That's a weaker incentive, though. And it comes with a risk of occasional scandals that scare away advertisers, at least temporarily. I don't think incentives are enough to predict what YouTube will do. The future is uncertain, there are a lot of policies that could plausibly make money, and the decision-makers are people who read the news and have opinions about what's right, not paperclip maximizers. Politics matters as well as economics.
  - kragen 7y ago
    True!