7 ms·

The paperclip maximizer is a thought experiment described by Swedish philosopher Nick Bostrom in 2003. It illustrates the existential risk that an artificial ge

by madrox 3y ago

The paperclip maximizer is a thought experiment described by Swedish philosopher Nick Bostrom in 2003. It illustrates the existential risk that an artificial general intelligence may pose to human beings when it is programmed to pursue even seemingly harmless goals and the necessity of incorporating machine ethics into artificial intelligence design. The scenario describes an advanced artificial intelligence tasked with manufacturing paperclips. If such a machine were not programmed to value human life, given enough power over its environment, it would try to turn all matter in the universe, including human beings, into paperclips or machines that manufacture paperclips. https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...

CharlesW 3y ago
I wonder if he got the idea from Philip K. Dick's 1955 story "Autofac"? https://en.wikipedia.org/wiki/Autofac https://en.wikipedia.org/wiki/Autofac https://www.vulture.com/2018/01/electric-dreams-recap-season-1-episode-2-autofac.html https://www.vulture.com/2018/01/electric-dreams-recap-season...
- actionfromafar 3y ago
  PKD keeps surprising me.
  - dylan604 3y ago
    I like how the sci-fi authors have spent time thinking about what an advanced AI could do, yet those building the AI have not taken a moment's pause to consider what they are doing
    - d1str0 3y ago
      Do you honestly believe that the only people who have thought about the ramifications of AI are Sci Fi authors? I can guarantee people who spend years researching and building advanced language models have thought about the ramifications of their work. This isn’t Jurassic Park.
      - chess_buster 3y ago
        They have – and decided to do it anyway.
      - actionfromafar 3y ago
        Haha, but it is! :-D LLMs are pretty basic stuff but we are all struggling with what to use them for! OpenAI is manually playing whac-a-mole with ChatGPT saying the darndest things!
      - otachack 3y ago
        Depends what we're looking at. Ride share disruption was very Jurassic Park and we've been dealing with ramifications ever since.
      - AlotOfReading 3y ago
        If you accept the implied premise that there are irresponsible deployments of AI out there, the alternative explanation is that they did consider the ramifications and simply don't care. That's even worse. Calling them ignorant is actually giving them the benefit of the doubt.
        goatlover 3y ago
        Or the researchers don't think existential threats are realistic, and paper maximizing thought experiments are silly. Maybe they're wrong, but maybe not. It's easy to imagine AI takeover scenarios by giving them unlimited powers, it's hard to show the actual path to such abilities. It's also hard to understand why an AI smart enough to paperclip the world wouldn't also be smart enough to realize the futility in doing so. So while alignment remains an issue, the existential alignment threats are too ill-specified. AGIs would understand we don't want to paperclip the world. Fun game though.
        AlotOfReading 3y ago
        That would be not accepting the premise that deployments are irresponsible. I guess there could be a situation where every researcher thinks everyone else's deployment is irresponsible and theirs is fine, but I don't think that's what you're saying.
    - JRandomHacker42 3y ago
      https://twitter.com/AlexBlechman/status/1457842724128833538 https://twitter.com/AlexBlechman/status/1457842724128833538 Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus
    - willcipriano 3y ago
      The people building it know it isn't AI, the people selling it call it that.
      - Karunamon 3y ago
        To say that something like GPT-4 does not count as "AI" requires a gargantuan shifting of goalposts from where they were at this time last year.
        actionfromafar 3y ago
        So it was ever since computers started playing chess.
    - feoren 3y ago
      That's because the people who are building the AI actually know how it works, understand how fundamentally simple it all is, and know that there's no room for consciousness to magically emerge. The current state of AI is not so much a story of any kind of "intelligence" being amazing, but rather the sum total of humanity's data being amazing. The amazing feats LLMs perform come from the words we all wrote, not the code they wrote. The code just unlocks the previously latent power of all that data. It is nothing close to being an actual intelligence, regardless of how much we anthropomorphize it. We also anthropomorphize stick figures, stuffed animals, and weighted companion cubes.
      - bananapub 3y ago
        that's indeed not a sensible worry, but the actual consequences on society of such things are extremely real and already happening, and are something the people involved seem either uninterested in worrying about or actively encouraging.
      - flangola7 3y ago
        You've written a lot of words yet I don't see a compelling reason to believe a single one of them.
      - actionfromafar 3y ago
        Good thing no new technology ever caused anything bad thanks to us not anthropomorphising it.
        deleted 3y ago
        [deleted]
- abecedarius 3y ago
  He got it from Eliezer Yudkowsky's somewhat different paperclip maximizer in a mailing-list post. (My memory of a Twitter thread a while back which included Yudkowsky, saying he'd told Bostrom not to worry about attributing ideas like that to him.)
  - deleted 3y ago
    [deleted]
jaclaz 3y ago
If anyone is curious about real paperclip manufacturing machines: https://news.ycombinator.com/item?id=20902807 https://news.ycombinator.com/item?id=20902807
stcredzero 3y ago
More and more of our global economy is centered around compute. While it seems like oil and fossil fuel use will decline with the advent of other forms of energy production in the near future, computer chips are becoming prominent in global strategic thinking and military planning. How is this different from maximizing paperclips? It's the same thing, just with a much more direct basis for instrumental convergence!
majormajor 3y ago
Some of these thought experiments seem very disconnected from how industry works. Like, we're saying "make as many paperclips as possible" as our instruction to this agent, not even "make as many as profitable" or "make up to X per day at a cost of less than Y per day"? The solution is proposed to be "program the AI to value human life" instead of the far simpler "put basic constraints on the process like you would in a business today"? Ok, so it's a more general example of worries about "managing superintelligence" but IMO it does the debate a disservice by being so obviously ludicrous that it's hard to square "naive paperclip-maximizing AI" with "superintelligence." I think if we're going to survive all this stuff it's much more likely to be because the private parties with the wherewithal to unleash an AI with the ability to affect the world to that extent will largely be ones with enough resources to have narrow banal goals and narrow banal constraints including self-preservation too, not because we figure out some sort of general purpose "AGIs that are aligned with humans" solution. Kinda like with nukes.
- spicybright 3y ago
  Even if we make an AI that wants to turn all matter into paper clips, we're so far away from an agent doing that I'm really not too worried. I don't think there's any industry on earth that doesn't need humans in the loop somehow. Whether is mining raw material from the ground, loading stuff in machines for processing, and most importantly fixing broken down machines, robots are really bad at these things for the foreseeable future. Not to mention AI needs constant electricity, which is really complicated and requires humans fixing a lot of stuff.
  - circuit10 3y ago
    The thought experiment is about a superintelligence, which either wouldn’t need humans and could build some kind of robots or something even more effective that we haven’t thought of, or manipulate us into doing exactly what it “wants” Also it’s a simplified example, it wouldn’t literally be paperclips but some other arbitrary goal (it shows how most goals takes to their absolute extreme won’t be compatible with human existence, even something that sounds harmless like making paperclips)

by madrox 3y ago

CharlesW 3y ago

I wonder if he got the idea from Philip K. Dick's 1955 story "Autofac"? https://en.wikipedia.org/wiki/Autofac https://en.wikipedia.org/wiki/Autofac https://www.vulture.com/2018/01/electric-dreams-recap-season-1-episode-2-autofac.html https://www.vulture.com/2018/01/electric-dreams-recap-season...

actionfromafar 3y ago
PKD keeps surprising me.
- dylan604 3y ago
  I like how the sci-fi authors have spent time thinking about what an advanced AI could do, yet those building the AI have not taken a moment's pause to consider what they are doing
  - d1str0 3y ago
    Do you honestly believe that the only people who have thought about the ramifications of AI are Sci Fi authors? I can guarantee people who spend years researching and building advanced language models have thought about the ramifications of their work. This isn’t Jurassic Park.
    - chess_buster 3y ago
      They have – and decided to do it anyway.
    - actionfromafar 3y ago
      Haha, but it is! :-D LLMs are pretty basic stuff but we are all struggling with what to use them for! OpenAI is manually playing whac-a-mole with ChatGPT saying the darndest things!
    - otachack 3y ago
      Depends what we're looking at. Ride share disruption was very Jurassic Park and we've been dealing with ramifications ever since.
    - AlotOfReading 3y ago
      If you accept the implied premise that there are irresponsible deployments of AI out there, the alternative explanation is that they did consider the ramifications and simply don't care. That's even worse. Calling them ignorant is actually giving them the benefit of the doubt.
      - goatlover 3y ago
        Or the researchers don't think existential threats are realistic, and paper maximizing thought experiments are silly. Maybe they're wrong, but maybe not. It's easy to imagine AI takeover scenarios by giving them unlimited powers, it's hard to show the actual path to such abilities. It's also hard to understand why an AI smart enough to paperclip the world wouldn't also be smart enough to realize the futility in doing so. So while alignment remains an issue, the existential alignment threats are too ill-specified. AGIs would understand we don't want to paperclip the world. Fun game though.
        AlotOfReading 3y ago
        That would be not accepting the premise that deployments are irresponsible. I guess there could be a situation where every researcher thinks everyone else's deployment is irresponsible and theirs is fine, but I don't think that's what you're saying.
  - JRandomHacker42 3y ago
    https://twitter.com/AlexBlechman/status/1457842724128833538 https://twitter.com/AlexBlechman/status/1457842724128833538 Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus
  - willcipriano 3y ago
    The people building it know it isn't AI, the people selling it call it that.
    - Karunamon 3y ago
      To say that something like GPT-4 does not count as "AI" requires a gargantuan shifting of goalposts from where they were at this time last year.
      - actionfromafar 3y ago
        So it was ever since computers started playing chess.
  - feoren 3y ago
    That's because the people who are building the AI actually know how it works, understand how fundamentally simple it all is, and know that there's no room for consciousness to magically emerge. The current state of AI is not so much a story of any kind of "intelligence" being amazing, but rather the sum total of humanity's data being amazing. The amazing feats LLMs perform come from the words we all wrote, not the code they wrote. The code just unlocks the previously latent power of all that data. It is nothing close to being an actual intelligence, regardless of how much we anthropomorphize it. We also anthropomorphize stick figures, stuffed animals, and weighted companion cubes.
    - bananapub 3y ago
      that's indeed not a sensible worry, but the actual consequences on society of such things are extremely real and already happening, and are something the people involved seem either uninterested in worrying about or actively encouraging.
    - flangola7 3y ago
      You've written a lot of words yet I don't see a compelling reason to believe a single one of them.
    - actionfromafar 3y ago
      Good thing no new technology ever caused anything bad thanks to us not anthropomorphising it.
      - deleted 3y ago
        [deleted]
abecedarius 3y ago
He got it from Eliezer Yudkowsky's somewhat different paperclip maximizer in a mailing-list post. (My memory of a Twitter thread a while back which included Yudkowsky, saying he'd told Bostrom not to worry about attributing ideas like that to him.)
- deleted 3y ago
  [deleted]

jaclaz 3y ago

If anyone is curious about real paperclip manufacturing machines: https://news.ycombinator.com/item?id=20902807 https://news.ycombinator.com/item?id=20902807

stcredzero 3y ago

More and more of our global economy is centered around compute. While it seems like oil and fossil fuel use will decline with the advent of other forms of energy production in the near future, computer chips are becoming prominent in global strategic thinking and military planning. How is this different from maximizing paperclips? It's the same thing, just with a much more direct basis for instrumental convergence!

majormajor 3y ago

Some of these thought experiments seem very disconnected from how industry works. Like, we're saying "make as many paperclips as possible" as our instruction to this agent, not even "make as many as profitable" or "make up to X per day at a cost of less than Y per day"? The solution is proposed to be "program the AI to value human life" instead of the far simpler "put basic constraints on the process like you would in a business today"? Ok, so it's a more general example of worries about "managing superintelligence" but IMO it does the debate a disservice by being so obviously ludicrous that it's hard to square "naive paperclip-maximizing AI" with "superintelligence." I think if we're going to survive all this stuff it's much more likely to be because the private parties with the wherewithal to unleash an AI with the ability to affect the world to that extent will largely be ones with enough resources to have narrow banal goals and narrow banal constraints including self-preservation too, not because we figure out some sort of general purpose "AGIs that are aligned with humans" solution. Kinda like with nukes.

spicybright 3y ago
Even if we make an AI that wants to turn all matter into paper clips, we're so far away from an agent doing that I'm really not too worried. I don't think there's any industry on earth that doesn't need humans in the loop somehow. Whether is mining raw material from the ground, loading stuff in machines for processing, and most importantly fixing broken down machines, robots are really bad at these things for the foreseeable future. Not to mention AI needs constant electricity, which is really complicated and requires humans fixing a lot of stuff.
- circuit10 3y ago
  The thought experiment is about a superintelligence, which either wouldn’t need humans and could build some kind of robots or something even more effective that we haven’t thought of, or manipulate us into doing exactly what it “wants” Also it’s a simplified example, it wouldn’t literally be paperclips but some other arbitrary goal (it shows how most goals takes to their absolute extreme won’t be compatible with human existence, even something that sounds harmless like making paperclips)