5 ms·

>Observe that in these scenarios the AIs are evil by default, just like a plant on an alien planet would probably be poisonous by default. I believe this is a

>Observe that in these scenarios the AIs are evil by default, just like a plant on an alien planet would probably be poisonous by default. I believe this is a core misunderstanding. Bostrom never says that a superintelligent AI is evil by default. Bostrom argues the AI will be orthogonal, it's goals will be underspecified in such a way that leads it to destroy humanity. The paperclip optimizer AI doesn't want to kill people, it just doesn't notice them, the same way you don't notice ants you drive over in your daily commute. AIs with goals orthogonal to our own will attack humanity in the same way humanity attacks the rainforests, piecemeal, as-needed, and without remorse or care for what was there before. It won't be evil, it will be uncaring, and blind.

monk_e_boy 10y ago
An uncaring blind AI wouldn't be very interesting, I would assume whoever had turned it on and was observing it would just turn it off and tweek the algorithm. But I don't think this is even possible as when the AI was in child stage and was learning, it would learn from people, so it would become like people. Or at least understand them. At some point it has to know less than a human and will learn at a rate that humans can measure. As we measure we can make decisions about it. I don't agree with the assumption that a factory making paperclips will make the transition to a super intelligent AI in a short time frame. I think it will take years. And along that route (of years of learning) we'll have time to talk to it and decide if it should be kept switched on.
- lmm 10y ago
  > I don't agree with the assumption that a factory making paperclips will make the transition to a super intelligent AI in a short time frame. Why? Because factories run by humans take a long time to transition? It might take weeks or months to make a process tweak in a human-run factory, but deploying new code happens in seconds or faster.
  - monk_e_boy 10y ago
    Yes because the deplyment of the code is the hard stage. rolls eyes How long would it take to write -- scientists are going through this stage right now and it's taking about 50 years so far. You think a paperclip factory is going to be quicker? Nope. And any smart AI that is being used to design a better AI will take years and will be front page news (assuming non-military) just like any other tech company (funding, profit, getting good hires) This assumption that the factory is smart in secret and buys more servers in secret and creates code somewhere in secret is just so wrong. It will be dog or dolphin smart at some point and will ask for things, it'll communicate with us.
    - lmm 10y ago
      > How long would it take to write -- scientists are going through this stage right now and it's taking about 50 years so far. You think a paperclip factory is going to be quicker? Nope. And any smart AI that is being used to design a better AI will take years and will be front page news (assuming non-military) just like any other tech company (funding, profit, getting good hires) It may take years or decades, but the point is that once it reaches the point that it can make itself smarter, it'd go from "barely smarter than a human" to "inconceivably smart" in a matter of milliseconds. > This assumption that the factory is smart in secret and buys more servers in secret and creates code somewhere in secret is just so wrong. It will be dog or dolphin smart at some point and will ask for things, it'll communicate with us. Dogs and dolphins don't communicate with us very effectively. I think it's very plausible that the AI could get to beyond-human-level smart without ever spontaneously deciding to talk to us. Are you sure there's a region where it's smart enough to talk but not smart enough to hide how smart it is?
      - monk_e_boy 10y ago
        > it'd go from "barely smarter than a human" to "inconceivably smart" in a matter of milliseconds Incorrect. It will be interesting to come back to this thread in 50 years time and see who is right.
  - mcguire 10y ago
    Therefore, to paraphrase Cato, AI research must be destroyed.
fjdlwlv 10y ago
The emergent system of human society already does this (read The Wealth of Nations) and you gave some examples yourself,so why (aside from amusement) should we care about dangerous Superintelligence when we already have so many out of control processes heading towards destroying our habitats, societies, and selves?

monk_e_boy 10y ago

An uncaring blind AI wouldn't be very interesting, I would assume whoever had turned it on and was observing it would just turn it off and tweek the algorithm. But I don't think this is even possible as when the AI was in child stage and was learning, it would learn from people, so it would become like people. Or at least understand them. At some point it has to know less than a human and will learn at a rate that humans can measure. As we measure we can make decisions about it. I don't agree with the assumption that a factory making paperclips will make the transition to a super intelligent AI in a short time frame. I think it will take years. And along that route (of years of learning) we'll have time to talk to it and decide if it should be kept switched on.

lmm 10y ago
> I don't agree with the assumption that a factory making paperclips will make the transition to a super intelligent AI in a short time frame. Why? Because factories run by humans take a long time to transition? It might take weeks or months to make a process tweak in a human-run factory, but deploying new code happens in seconds or faster.
- monk_e_boy 10y ago
  Yes because the deplyment of the code is the hard stage. rolls eyes How long would it take to write -- scientists are going through this stage right now and it's taking about 50 years so far. You think a paperclip factory is going to be quicker? Nope. And any smart AI that is being used to design a better AI will take years and will be front page news (assuming non-military) just like any other tech company (funding, profit, getting good hires) This assumption that the factory is smart in secret and buys more servers in secret and creates code somewhere in secret is just so wrong. It will be dog or dolphin smart at some point and will ask for things, it'll communicate with us.
  - lmm 10y ago
    > How long would it take to write -- scientists are going through this stage right now and it's taking about 50 years so far. You think a paperclip factory is going to be quicker? Nope. And any smart AI that is being used to design a better AI will take years and will be front page news (assuming non-military) just like any other tech company (funding, profit, getting good hires) It may take years or decades, but the point is that once it reaches the point that it can make itself smarter, it'd go from "barely smarter than a human" to "inconceivably smart" in a matter of milliseconds. > This assumption that the factory is smart in secret and buys more servers in secret and creates code somewhere in secret is just so wrong. It will be dog or dolphin smart at some point and will ask for things, it'll communicate with us. Dogs and dolphins don't communicate with us very effectively. I think it's very plausible that the AI could get to beyond-human-level smart without ever spontaneously deciding to talk to us. Are you sure there's a region where it's smart enough to talk but not smart enough to hide how smart it is?
    - monk_e_boy 10y ago
      > it'd go from "barely smarter than a human" to "inconceivably smart" in a matter of milliseconds Incorrect. It will be interesting to come back to this thread in 50 years time and see who is right.
- mcguire 10y ago
  Therefore, to paraphrase Cato, AI research must be destroyed.

fjdlwlv 10y ago

The emergent system of human society already does this (read The Wealth of Nations) and you gave some examples yourself,so why (aside from amusement) should we care about dangerous Superintelligence when we already have so many out of control processes heading towards destroying our habitats, societies, and selves?