3 ms·
> I certainly hope the set of people with access to nuclear weapons would not be the same people building AI, or we'd indeed be quite screwed. Well, in fact th
by ars 11y ago
> I certainly hope the set of people with access to nuclear weapons would not be the same people building AI, or we'd indeed be quite screwed.
Well, in fact they are, (or would be) the exact same people. But despite that we won't be "screwed".
Or did you think that if strong AI showed up the Governments of the whole world would not rush to have their own?
> what runs on that substrate can emulate anything it likes.
At what speed though? It's not clear that an emulation in a computer would be anywhere near as fast as the actual thing (i.e. a human), so it might not actually be possible to upload people digitally. (If nothing else, the speed of light limits how fast you can run a CPU.) It's also not so obvious that it's even possible to copy a brain, never mind running one.
All your assumptions are predicated on the ability to emulate a human in software, if that's not possible then none of what you wrote about strong AI will happen, instead people will continue to be people and AI will just be another thing to ignore just like we ignore all the marvels we have now.
> The major assumption I'm making is that we can successfully build strong AI (AI capable of self-improvement past human), which seems plausible but not a given.
An even larger assumption is that it's even possible to do that. Never mind can we. Can an intelligence program something smarter than itself? I don't think so. I don't think the AI can, I don't think humans can.
So far the only known method of making a smarter version of something is by random permutation and picking the best one. It's not an accident that all AI used today is by that method. I'm not aware of any that are programmed from first principles.
- JoshTriplett 11y ago> Or did you think that if strong AI showed up the Governments of the whole world would not rush to have their own? Governments can't "rush" on the same scale an AI can, and a strong AI with an appropriate value function wouldn't permit the existence of any other strong AI that doesn't share its value function (since such an AI would fall somewhere between "counterproductive" and "potential existential threat"), nor would "of their own" have any meaning unless that value function was hopelessly broken. There shouldn't be a "second" strong AI, only a first. > It's not clear that an emulation in a computer would be anywhere near as fast as the actual thing Humans operate relatively slowly on the scale of computers; we're just massively parallel and have a fundamentally different architecture. That's not at all insurmountable even for human-built technology, let alone technology-built technology. > All your assumptions are predicated on the ability to emulate a human in software Several of my assumptions are, true, though I can imagine other possibilities that would have similar effect. > An even larger assumption is that it's even possible to do that. That's not a larger assumption, that's the same assumption I stated. > Can an intelligence program something smarter than itself? I don't think so. On what basis? > I don't think humans can. Humans don't have to build a system smarter than ourselves, just build one as smart as ourselves and run it a lot faster with access to a lot more data.