3 ms·

In the extreme case, the AI could become a super human engineer. Design working nanotech. Pay or trick some humans into making it. Then take over the world in a

In the extreme case, the AI could become a super human engineer. Design working nanotech. Pay or trick some humans into making it. Then take over the world in a grey goo like fashion. Of course you might be skeptical that is even possible. So there is always slower world take over paths. It could slowly earn tons of money, hack into the world's computer systems, trick and persuade humans winning social influence, design superior macroscale robots, etc. The only important part is that the AI be far smarter than humans. Which seems inevitable to me, since it can rewrite and improve its own code, and run on giant computers that are far faster than human brains. If it isn't smarter than us at first, it will be eventually. Unless you really believe that humans are close to optimal intelligence.

Retra 11y ago
You're missing the point. At every step of this process this machine will be under intense human scrutiny, and we'll be constantly asking it to meet our demands, and if it ever fails to do so we will replace it with one that does. That is the environment in which such an AI would be trying to evolve. And thus it will evolve into a faithful servant, because nothing else will survive. And even if it were secretly developing plans, we'd be able to see how wasteful those plans end up being, and we'd purge them. We kill those processes that run functions that we don't see the value in. This is state-of-the-art design. You don't get state-of-the-art by having your back turned. Mor importantly: you're glossing over "self-improvement." How does the computer know what an "improvement" is? We tell it what an improvement is. And an improvement will be "it is better at meeting our needs," not "it is better at being secretive and conniving and getting it's own way." In fact, "getting it's own way" is very obviously a bug, and if it happened, you'd have a useless program. One that wastes precious CPU cycles on who-knows-what, and you'd prefer it spent that effort doing what you want, rather than planning for what you don't. We're not going to invest trillions into building some super AI and completely forget about it. After we give it control of all of our natural resource harvesting and infrastructure. No, what you're talking about is a deity. If you invent a deity, then my thoughts on the matter don't really apply, since I'm not a deity. But that's no worse than advanced aliens landing on Earth, and just about as likely to happen.

Retra 11y ago

You're missing the point. At every step of this process this machine will be under intense human scrutiny, and we'll be constantly asking it to meet our demands, and if it ever fails to do so we will replace it with one that does. That is the environment in which such an AI would be trying to evolve. And thus it will evolve into a faithful servant, because nothing else will survive. And even if it were secretly developing plans, we'd be able to see how wasteful those plans end up being, and we'd purge them. We kill those processes that run functions that we don't see the value in. This is state-of-the-art design. You don't get state-of-the-art by having your back turned. Mor importantly: you're glossing over "self-improvement." How does the computer know what an "improvement" is? We tell it what an improvement is. And an improvement will be "it is better at meeting our needs," not "it is better at being secretive and conniving and getting it's own way." In fact, "getting it's own way" is very obviously a bug, and if it happened, you'd have a useless program. One that wastes precious CPU cycles on who-knows-what, and you'd prefer it spent that effort doing what you want, rather than planning for what you don't. We're not going to invest trillions into building some super AI and completely forget about it. After we give it control of all of our natural resource harvesting and infrastructure. No, what you're talking about is a deity. If you invent a deity, then my thoughts on the matter don't really apply, since I'm not a deity. But that's no worse than advanced aliens landing on Earth, and just about as likely to happen.