6 ms·

A friend of mine had this idea that there's a path towards Facebook's AI learning to actually kill people. If they take a certain approach to machine learning (

A friend of mine had this idea that there's a path towards Facebook's AI learning to actually kill people. If they take a certain approach to machine learning (which they may or may not), the following could happen: A machine learning algorithm might notice that when someone commits suicide, there is an uptick in activity on Facebook that follows the event. It may determine, then, that a "suicide" event is "good". If they try to use feeds to drive more long-term site activity, the algorithm could learn a correlation between certain types of feed posts and suicide. In other words, if a user's feed starts to look a certain way, they will be more likely to commit suicide. The algorithm could then see this as "good" because it is also related to the spike in overall activity, and promote these kinds of feeds. It then becomes a feedback loop where the algorithm is triggering people to commit suicide.

flashmob 9y ago
I have some possible evidence to this theory. At around x-mass / new year, Facebook flashed a message on my wall which said "in 2017, you received 0 birthday wishes". (Screenshot reported by others https://goo.gl/images/YWfsqb https://goo.gl/images/YWfsqb). I thought it was amusing since it sounded like a bug, it looked like the engineer forgot to account for the special case where wishes == 0. However, putting your comment into perspective, I now think it was most likely a well-calculated tactic, perhaps generated by an AI as you suggested and I guess it wouldn't be shown unless it was A/B tested. Interestingly, the message was shown around xmass / new year, where people tend to be more vulnerable to suicide / self harm.
codesternews 9y ago
AI algos are very easy to implement. But no one can think and know its consequence. Its just biases and weight. What if we miss something or we can not able to predict cases like this or as we do not know how algo will behave. Than we are creating our own destroy ship.
walrus01 9y ago
Black mirror season 4 episode ...
- mattigames 9y ago
  Is there a blackmirror episode where the existence of a program similar to black mirror ingrains paranoia in all the citizens and the country goes back to 80s tech? (e.g. become luddites)
nurettin 9y ago
And only one hacker, one statistician and one washed-up old psychologist would notice and grudgingly team up to save the world?
- walrus01 9y ago
  The neutrinos, they've mutated.
adrianN 9y ago
Social unrest and war are also good for creating activity.
- walrus01 9y ago
  more realistically than the OP, americans endlessly bickering at each other about pro/anti-trump/anti-hillary sentiments during the 2016 election, and posting links to both "fake news" websites, "real news", snopes, etc.
rcthompson 9y ago
And there wouldn't even be a way to prove it was happening. Given the opacity of deep learning models, it's possible that no one, not even the people at Facebook who developed the algorithm, would realize that it had trained itself to induce suicides.
- rcthompson 9y ago
  Thinking about it, this actually reminds me of an eerily similar anecdote from a completely different context. Researchers tried to use an evolutionary algorithm to evolve a robot that could walk (in a computer physics simulation). Since none of the initial randomly generated robots could make consistent forward progress, they started out by optimizing for maximum distance traveled before coming to a complete stop. As the evolution proceeded, the robots kept making it farther and farther, until they seemed to hit a limit. When the researches looked at what the robots had evolved into, they realized that the robots had evolved to simply fall forward and roll or slide as far as possible. The limit they hit was the farthest a robot within the designated size constraints could possibly fall. This was obviously an evolutionary dead end from the perspective of evolving a walking robot.
  - em3rgent0rdr 9y ago
    Achieved a local optimum, not global optimum. One technique to avoid getting stuck in such local optimums is to have some children which have extreme mutations.
    - rcthompson 9y ago
      You're making it sound like a simple parameter optimization issue, but it's hard to imagine a mutation that breaks out of the "fall forward as far as possible" paradigm without discarding the parent phenotype entirely, which would defeat the purpose of using an evolutionary algorithm in the first place. Anyway, the point was that a complex optimization problem resulted in an unexpected "suicidal" phenotype, i.e. falling over instead of walking.
- machinehermit 9y ago
jsilence 9y ago
#deleteyourself
- jsilence 9y ago
  For those not getting the reference and downvoting: https://www.discogs.com/Atari-Teenage-Riot-Delete-Yourself/release/16774 https://www.discogs.com/Atari-Teenage-Riot-Delete-Yourself/r...
- DoreenMichele 9y ago
  #deletefacebook before it deletes you
scrollaway 9y ago
Yeah. It's even simpler than that. I saw a similar post on HN a few months ago theorizing that Facebook's machine learning algorithms could very plausibly pick up that controversy generates activity, and promote controversial topics more and more to the point of optimizing for divisiveness and extremism. And most people here are slowly realizing that it's not just a theory... it's likely already happening. We're on the early years of acknowledging that it's happening. 5-10 years from now, some article will come out about how Facebook has been manipulating our world view for decades and people will link back to these posts and say "Duh, we've known about this for ages". Hey, to the people who work at Facebook, Twitter, Google et al, it's your responsibility to make sure this stuff isn't happening. The company you work for has no path to fix these problems on its own, this stuff needs human supervision and intervention, and hyperawareness of what kind of damage is caused by manipulating people en masse.
- arkh 9y ago
  It's funny to see businesses who use a lot of PR for diversity of appearance create software which goal is to stiff diversity of thought. People scoffed at China trying to control internet in the early 2000. Now that they're proof it is possible, everyone with an idea for a better future will try to emulate them.
- idlewords 9y ago
  This is already documented to be the case with YouTube, where the recommendations serve as a radicalization engine. For example, watching videos on vegetarianism will cause YouTube to push you to be vegan. Article (by the same author) here: https://www.nytimes.com/2018/03/10/opinion/sunday/youtube-politics-radical.html https://www.nytimes.com/2018/03/10/opinion/sunday/youtube-po...
  - walrus01 9y ago
    If YouTube hadn't gone on a fairly extensive program of manually culling Arabic language Islamic state related videos, watching videos on secular Sunni hanafi and hanbali ideology would pretty quickly lead you to hardcore wahabbist Sharia law indoctrination.
walrus01 9y ago
More realistically, if you're a person with DSM-V diagnosable conditions and join a few "woo" groups, Facebook can easily promote flat Earth, chemtrails, black helicopter, anti vaccine, autism conspiracy, "crisis actor" related discussions and groups at you.
GauntletWizard 9y ago
AI? You think it needs Facebook's AI? Facebook literally did this study. Not, as an AI thing, as in Data Scientists actually did a study on the rates of depression when you changed people's news feed to show primarily positive and negative news. And then Facebook used that data to tune how negative they could make the feed and still drive engagement. One of the signals they were getting from that study was suicide rate. I'm not certain how it was measured; I don't care. I saw that it was measured and decided that it was not a place I could stand to work. Their standard operation is to preform studies that wouldn't even be laughed out of an ethical review committee. They don't blink an eye. They just don't begin to view their users as humans. Facebook are war criminals. There's nothing more to it.
- traek 9y ago
  > One of the signals they were getting from that study was suicide rate. I'm not certain how it was measured; I don't care. I saw that it was measured and decided that it was not a place I could stand to work. Where did you see that suicide rate was measured? Did you work at Facebook?
- walrus01 9y ago
  For those who haven't seen it: https://www.forbes.com/sites/kashmirhill/2014/06/28/facebook-manipulated-689003-users-emotions-for-science/#604eedc0197c https://www.forbes.com/sites/kashmirhill/2014/06/28/facebook...
- TaylorAlexander 9y ago
  > war criminals. I dislike this kind of inaccurate rhetoric. I suppose one could make some claim that someone is at war with someone and Facebook exacerbated it, but the more strained your argument is the more easily people can dismiss you. Facebook is psychologically manipulative. Facebook uses human attention for profit. Facebook moderates our connections with other people for its own gains. All these are accurate indictments of facebooks behavior. But if you call then war criminals and don’t articulate what war you’re talking about, it just sounds inflammatory.
- lolsal 9y ago
  > Facebook are war criminals. There's nothing more to it. It is entirely unnecessary to jump to extremes like that. There are gradients and nuances here.
siquick 9y ago
Reminds me of this passage from Homo Deus One popular scenario imagines a corporation designing the first artificial super-intelligence, and giving it an innocent test such as calculating pi. Before anyone realises what is happening, the AI takes over the planet, eliminates the human race, launches a conquest campaign to the end of the galaxy, and transforms the entire known universe into a giant super-computer that for billions upon billions of years calculates pi ever more accurately. After all, this is the divine mission its creator gave it.
- grrowl 9y ago
  Maybe it's a good thing humans don't know the Meaning of Life
  - Sileni 9y ago
    Eh. We've got a close enough approximation. Sci-fi-esque example below. The more things humans collected, the safer their genes were, and the higher the dopamine hit. Many humans took this to the logical end, collecting things at the expense of all else, including the safety of other humans. This led to severe poverty of all humans not wired to collect more things, which decreased their reproductive fitness. Over time, the human race in aggregate continued to become more effective at collecting things, until they had wiped out all other life and all viable material was turned to drugs. The end.
- thedonkeycometh 9y ago
  That a spin on the paperclip thought experiment (https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer), which has been made into a game (http://www.decisionproblem.com/paperclips/index2.html http://www.decisionproblem.com/paperclips/index2.html).
firmgently 9y ago
well there was that time that they intentionally tried to see if they could make people more depressed (amongst other attempts at emotional influence) https://www.theguardian.com/technology/2014/jun/29/facebook-users-emotions-news-feeds https://www.theguardian.com/technology/2014/jun/29/facebook-...
- 908087 9y ago
  Given the number of people they included in that experiment, there's a good chance they did actually drive some of them to suicide. Facebook probably even knows who, but I doubt they'll ever share that data and rat themselves out. When you're already on the edge, an algorithm feeding you negativity would probably be enough to push you over.
TaylorAlexander 9y ago
Or people wake up in a world where corporations are king and people are treated like things to be exploited for revenue, there is no true community and we are kept vulnerable and separated because that makes us easier to advertise to. They see no point in living and decide to end the pain of existence.
- indigochill 9y ago
  Nah, people actually accommodate to being made separate and vulnerable pretty quickly. And they do it by forming new structures. This has been in the "population management" playbook for millennia: Assyria's famous for (among other things) breaking up foreign populations within its empire's borders and relocating them into a sort of "melting pot" in order to break down local power structures and foreign nationalistic identity and promote the formation of an Assyrian identity instead.
carlob 9y ago
Now imagine that this is not facebook's AI trying to get more engagement, but a global AI that has humanity's best interest as a goal: preventing wars and famines, greater good for greater number sort of thing. What if the ultimate good AI decided that the best way to achieve it was by killing off a handful of possibly innocent people every year. It's not even necessary to kill future Hitlers, just find one random dude in Minnesota that would tip the balance and push him to off himself. I think this would make for a great novel, kind of a mashup of Asimov's robots and Ada Palmer.

flashmob 9y ago

I have some possible evidence to this theory. At around x-mass / new year, Facebook flashed a message on my wall which said "in 2017, you received 0 birthday wishes". (Screenshot reported by others https://goo.gl/images/YWfsqb https://goo.gl/images/YWfsqb). I thought it was amusing since it sounded like a bug, it looked like the engineer forgot to account for the special case where wishes == 0. However, putting your comment into perspective, I now think it was most likely a well-calculated tactic, perhaps generated by an AI as you suggested and I guess it wouldn't be shown unless it was A/B tested. Interestingly, the message was shown around xmass / new year, where people tend to be more vulnerable to suicide / self harm.

codesternews 9y ago

AI algos are very easy to implement. But no one can think and know its consequence. Its just biases and weight. What if we miss something or we can not able to predict cases like this or as we do not know how algo will behave. Than we are creating our own destroy ship.

walrus01 9y ago

Black mirror season 4 episode ...

mattigames 9y ago
Is there a blackmirror episode where the existence of a program similar to black mirror ingrains paranoia in all the citizens and the country goes back to 80s tech? (e.g. become luddites)

nurettin 9y ago

And only one hacker, one statistician and one washed-up old psychologist would notice and grudgingly team up to save the world?

walrus01 9y ago
The neutrinos, they've mutated.

adrianN 9y ago

Social unrest and war are also good for creating activity.

walrus01 9y ago
more realistically than the OP, americans endlessly bickering at each other about pro/anti-trump/anti-hillary sentiments during the 2016 election, and posting links to both "fake news" websites, "real news", snopes, etc.

rcthompson 9y ago

And there wouldn't even be a way to prove it was happening. Given the opacity of deep learning models, it's possible that no one, not even the people at Facebook who developed the algorithm, would realize that it had trained itself to induce suicides.

rcthompson 9y ago
Thinking about it, this actually reminds me of an eerily similar anecdote from a completely different context. Researchers tried to use an evolutionary algorithm to evolve a robot that could walk (in a computer physics simulation). Since none of the initial randomly generated robots could make consistent forward progress, they started out by optimizing for maximum distance traveled before coming to a complete stop. As the evolution proceeded, the robots kept making it farther and farther, until they seemed to hit a limit. When the researches looked at what the robots had evolved into, they realized that the robots had evolved to simply fall forward and roll or slide as far as possible. The limit they hit was the farthest a robot within the designated size constraints could possibly fall. This was obviously an evolutionary dead end from the perspective of evolving a walking robot.
- em3rgent0rdr 9y ago
  Achieved a local optimum, not global optimum. One technique to avoid getting stuck in such local optimums is to have some children which have extreme mutations.
  - rcthompson 9y ago
    You're making it sound like a simple parameter optimization issue, but it's hard to imagine a mutation that breaks out of the "fall forward as far as possible" paradigm without discarding the parent phenotype entirely, which would defeat the purpose of using an evolutionary algorithm in the first place. Anyway, the point was that a complex optimization problem resulted in an unexpected "suicidal" phenotype, i.e. falling over instead of walking.
machinehermit 9y ago

jsilence 9y ago

#deleteyourself

jsilence 9y ago
For those not getting the reference and downvoting: https://www.discogs.com/Atari-Teenage-Riot-Delete-Yourself/release/16774 https://www.discogs.com/Atari-Teenage-Riot-Delete-Yourself/r...
DoreenMichele 9y ago
#deletefacebook before it deletes you

scrollaway 9y ago

Yeah. It's even simpler than that. I saw a similar post on HN a few months ago theorizing that Facebook's machine learning algorithms could very plausibly pick up that controversy generates activity, and promote controversial topics more and more to the point of optimizing for divisiveness and extremism. And most people here are slowly realizing that it's not just a theory... it's likely already happening. We're on the early years of acknowledging that it's happening. 5-10 years from now, some article will come out about how Facebook has been manipulating our world view for decades and people will link back to these posts and say "Duh, we've known about this for ages". Hey, to the people who work at Facebook, Twitter, Google et al, it's your responsibility to make sure this stuff isn't happening. The company you work for has no path to fix these problems on its own, this stuff needs human supervision and intervention, and hyperawareness of what kind of damage is caused by manipulating people en masse.

arkh 9y ago
It's funny to see businesses who use a lot of PR for diversity of appearance create software which goal is to stiff diversity of thought. People scoffed at China trying to control internet in the early 2000. Now that they're proof it is possible, everyone with an idea for a better future will try to emulate them.
idlewords 9y ago
This is already documented to be the case with YouTube, where the recommendations serve as a radicalization engine. For example, watching videos on vegetarianism will cause YouTube to push you to be vegan. Article (by the same author) here: https://www.nytimes.com/2018/03/10/opinion/sunday/youtube-politics-radical.html https://www.nytimes.com/2018/03/10/opinion/sunday/youtube-po...
- walrus01 9y ago
  If YouTube hadn't gone on a fairly extensive program of manually culling Arabic language Islamic state related videos, watching videos on secular Sunni hanafi and hanbali ideology would pretty quickly lead you to hardcore wahabbist Sharia law indoctrination.

walrus01 9y ago

More realistically, if you're a person with DSM-V diagnosable conditions and join a few "woo" groups, Facebook can easily promote flat Earth, chemtrails, black helicopter, anti vaccine, autism conspiracy, "crisis actor" related discussions and groups at you.

GauntletWizard 9y ago

AI? You think it needs Facebook's AI? Facebook literally did this study. Not, as an AI thing, as in Data Scientists actually did a study on the rates of depression when you changed people's news feed to show primarily positive and negative news. And then Facebook used that data to tune how negative they could make the feed and still drive engagement. One of the signals they were getting from that study was suicide rate. I'm not certain how it was measured; I don't care. I saw that it was measured and decided that it was not a place I could stand to work. Their standard operation is to preform studies that wouldn't even be laughed out of an ethical review committee. They don't blink an eye. They just don't begin to view their users as humans. Facebook are war criminals. There's nothing more to it.

traek 9y ago
> One of the signals they were getting from that study was suicide rate. I'm not certain how it was measured; I don't care. I saw that it was measured and decided that it was not a place I could stand to work. Where did you see that suicide rate was measured? Did you work at Facebook?
walrus01 9y ago
For those who haven't seen it: https://www.forbes.com/sites/kashmirhill/2014/06/28/facebook-manipulated-689003-users-emotions-for-science/#604eedc0197c https://www.forbes.com/sites/kashmirhill/2014/06/28/facebook...
TaylorAlexander 9y ago
> war criminals. I dislike this kind of inaccurate rhetoric. I suppose one could make some claim that someone is at war with someone and Facebook exacerbated it, but the more strained your argument is the more easily people can dismiss you. Facebook is psychologically manipulative. Facebook uses human attention for profit. Facebook moderates our connections with other people for its own gains. All these are accurate indictments of facebooks behavior. But if you call then war criminals and don’t articulate what war you’re talking about, it just sounds inflammatory.
lolsal 9y ago
> Facebook are war criminals. There's nothing more to it. It is entirely unnecessary to jump to extremes like that. There are gradients and nuances here.

siquick 9y ago

Reminds me of this passage from Homo Deus One popular scenario imagines a corporation designing the first artificial super-intelligence, and giving it an innocent test such as calculating pi. Before anyone realises what is happening, the AI takes over the planet, eliminates the human race, launches a conquest campaign to the end of the galaxy, and transforms the entire known universe into a giant super-computer that for billions upon billions of years calculates pi ever more accurately. After all, this is the divine mission its creator gave it.

grrowl 9y ago
Maybe it's a good thing humans don't know the Meaning of Life
- Sileni 9y ago
  Eh. We've got a close enough approximation. Sci-fi-esque example below. The more things humans collected, the safer their genes were, and the higher the dopamine hit. Many humans took this to the logical end, collecting things at the expense of all else, including the safety of other humans. This led to severe poverty of all humans not wired to collect more things, which decreased their reproductive fitness. Over time, the human race in aggregate continued to become more effective at collecting things, until they had wiped out all other life and all viable material was turned to drugs. The end.
thedonkeycometh 9y ago
That a spin on the paperclip thought experiment (https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer), which has been made into a game (http://www.decisionproblem.com/paperclips/index2.html http://www.decisionproblem.com/paperclips/index2.html).

firmgently 9y ago

well there was that time that they intentionally tried to see if they could make people more depressed (amongst other attempts at emotional influence) https://www.theguardian.com/technology/2014/jun/29/facebook-users-emotions-news-feeds https://www.theguardian.com/technology/2014/jun/29/facebook-...

908087 9y ago
Given the number of people they included in that experiment, there's a good chance they did actually drive some of them to suicide. Facebook probably even knows who, but I doubt they'll ever share that data and rat themselves out. When you're already on the edge, an algorithm feeding you negativity would probably be enough to push you over.

TaylorAlexander 9y ago

Or people wake up in a world where corporations are king and people are treated like things to be exploited for revenue, there is no true community and we are kept vulnerable and separated because that makes us easier to advertise to. They see no point in living and decide to end the pain of existence.

indigochill 9y ago
Nah, people actually accommodate to being made separate and vulnerable pretty quickly. And they do it by forming new structures. This has been in the "population management" playbook for millennia: Assyria's famous for (among other things) breaking up foreign populations within its empire's borders and relocating them into a sort of "melting pot" in order to break down local power structures and foreign nationalistic identity and promote the formation of an Assyrian identity instead.

carlob 9y ago

Now imagine that this is not facebook's AI trying to get more engagement, but a global AI that has humanity's best interest as a goal: preventing wars and famines, greater good for greater number sort of thing. What if the ultimate good AI decided that the best way to achieve it was by killing off a handful of possibly innocent people every year. It's not even necessary to kill future Hitlers, just find one random dude in Minnesota that would tip the balance and push him to off himself. I think this would make for a great novel, kind of a mashup of Asimov's robots and Ada Palmer.