4 ms·

> after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later. That's physically impossible. AI stil

> after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later. That's physically impossible. AI still needs to improve incrementally and this will take time and resources. We can consider such AI to act like a virus - it will need a host to grow - in this case computer resources. It will be limited by the CPU, network speeds, storage and power. If it grows too fast it will starve the host of these resources which in turn will slow down it's progress. Even if the AI starts on a supercomputer in some university somewhere and decides that it needs more power, most likely it will start by "infecting" other computers on the network. These computers will not be nearly powerful enough to add much to the current processing power of the supercomputer. Such invasion is likely to get noticed. Each infected node will not be another super intelligent being, but due to hardware limitations it will most likely be a software that's limited in scope and designed to propagate and control the resources, linking them with the main host. We can safely assume that restarting, or reformatting the drives will restore the node to it's non-infected state thus slowing down the "invasion". Why we can assume that? It will take AI long time to learn everything about every architecture and operating system and apply that knowledge. The AI that's hosted on a supercomputer will not be able to suddenly start rewriting its software, take over nearby nodes, learn everything about our world etc. >You're assuming that AI will grow with us and necessarily reach the same value system we have (one we don't really comprehend ourselves, mind you). No I'm not. Only thing I can assume is that if the AI is smart enough to figure out how "convert" humans to paper clips, it will be smart enough to understand that it's pointless. It may decide to use it's super intelligence to take over the world, and I can't argue with that.. But if it continues to make paper clips even if they are no longer necessary then by definition this is not AI, just a stupid replicator.

TeMPOraL 12y ago
You're assuming quite a lot about how such event would develop. Keep in mind that a sufficiently smart AI wouldn't have to infect stuff, it could just ask. Think about IQ 200 sociopath that has Internet plugged into its head. It would be extremely difficult to outsmart it. > it will be smart enough to understand that it's pointless. Pointless for humans, not for an AI. You're projecting a lot of human values and thinking on AI. See my comment [0]. [0] - https://news.ycombinator.com/item?id=8387771 https://news.ycombinator.com/item?id=8387771
- kordless 12y ago
  His assumptions are logically sound given our universe's rules. There is no such thing as a free lunch here. The AI will need to pay for its way, just like each of us do. If it succeeds into making us into paper clips, it will have to be equally as savvy in marketing and selling those paper clips to someone. Given the shipping charges to the nearest habitable planet are very high, their market likely will have to exist here. I'm sure as shit not going to buy a paper clip made out of you!
  - TeMPOraL 12y ago
    You're being sarcastic, right? Since when our universe is governed by capitalist economy?
    - kordless 12y ago
      I'm quite serious. Our universe is governed by laws of conservation and efficiencies. For AI to exist, it needs power. Power must come from somewhere in the universe and, at least here on earth, it has been shown that there is competition for those resources. Whether this alludes to business or not is irrelevant; it's a fact of our reality that we are all competing against each other for resources.
      - TeMPOraL 12y ago
        But competition here has absolutely nothing to do with free-market economy that we have today. > The AI will need to pay for its way, just like each of us do. If it succeeds into making us into paper clips, it will have to be equally as savvy in marketing and selling those paper clips to someone. Given the shipping charges to the nearest habitable planet are very high, their market likely will have to exist here. I'm sure as shit not going to buy a paper clip made out of you! One, AI will not necessarily have to pay for anything. It may, or it could convince us to donate resources to it, or it could take them by force. Two, why would you ever think that it has to market paper clips for anyone (assuming anyone is left)? It optimizes for paper clips, so it makes paper clips. Marketing is a way humans bullshit each other into donating resources so that they don't have to fight for them. Three, what shipping costs, what buying? Again, AI wants to maximize amount of paper clips in the universe, period. It doesn't want to sell them (unless it leads to more paper clips). The only "shipping cost" it cares about is how much energy it takes to reach another planet, not some arbitrary dollar value people attach to movement of goods. My problem with your answer is that it lifts market economy from something completely irrelevant to the level of physical laws. The only time an AI would care about it would be if - and only as long as - interacting with humans was a best strategy for that AI to achieve its goals - which is likely only at the very beginning, when it needs to gather enough power to subdue us.

TeMPOraL 12y ago

You're assuming quite a lot about how such event would develop. Keep in mind that a sufficiently smart AI wouldn't have to infect stuff, it could just ask. Think about IQ 200 sociopath that has Internet plugged into its head. It would be extremely difficult to outsmart it. > it will be smart enough to understand that it's pointless. Pointless for humans, not for an AI. You're projecting a lot of human values and thinking on AI. See my comment [0]. [0] - https://news.ycombinator.com/item?id=8387771 https://news.ycombinator.com/item?id=8387771

kordless 12y ago
His assumptions are logically sound given our universe's rules. There is no such thing as a free lunch here. The AI will need to pay for its way, just like each of us do. If it succeeds into making us into paper clips, it will have to be equally as savvy in marketing and selling those paper clips to someone. Given the shipping charges to the nearest habitable planet are very high, their market likely will have to exist here. I'm sure as shit not going to buy a paper clip made out of you!
- TeMPOraL 12y ago
  You're being sarcastic, right? Since when our universe is governed by capitalist economy?
  - kordless 12y ago
    I'm quite serious. Our universe is governed by laws of conservation and efficiencies. For AI to exist, it needs power. Power must come from somewhere in the universe and, at least here on earth, it has been shown that there is competition for those resources. Whether this alludes to business or not is irrelevant; it's a fact of our reality that we are all competing against each other for resources.
    - TeMPOraL 12y ago
      But competition here has absolutely nothing to do with free-market economy that we have today. > The AI will need to pay for its way, just like each of us do. If it succeeds into making us into paper clips, it will have to be equally as savvy in marketing and selling those paper clips to someone. Given the shipping charges to the nearest habitable planet are very high, their market likely will have to exist here. I'm sure as shit not going to buy a paper clip made out of you! One, AI will not necessarily have to pay for anything. It may, or it could convince us to donate resources to it, or it could take them by force. Two, why would you ever think that it has to market paper clips for anyone (assuming anyone is left)? It optimizes for paper clips, so it makes paper clips. Marketing is a way humans bullshit each other into donating resources so that they don't have to fight for them. Three, what shipping costs, what buying? Again, AI wants to maximize amount of paper clips in the universe, period. It doesn't want to sell them (unless it leads to more paper clips). The only "shipping cost" it cares about is how much energy it takes to reach another planet, not some arbitrary dollar value people attach to movement of goods. My problem with your answer is that it lifts market economy from something completely irrelevant to the level of physical laws. The only time an AI would care about it would be if - and only as long as - interacting with humans was a best strategy for that AI to achieve its goals - which is likely only at the very beginning, when it needs to gather enough power to subdue us.