3 ms·
When it comes to building smarter-than-human AI, "try it and see" is never the right answer. Disagree emphatically. In fact it's the only way to do it because
by AndrewKemendo 10y ago
When it comes to building smarter-than-human AI, "try it and see" is never the right answer.
Disagree emphatically. In fact it's the only way to do it because there is no way to know certainly that a superhuman-AGI will ensure the longevity of humanity. I go so far as to argue that it's not even necessary because there is no long term longevity for humanity anyway.
There is this implicit assumption that humans are, should and will always be the apex entity - and I think that is misguided.
If you instead view superhuman-AGI as our rightful offspring, something that we can't understand and is better than us, then all of the existential dread around it goes away.
Dying elderly often express "comfort" in dying when they see that their offspring are reproducing and are smarter than they were. We should see Superhuman-AGI the same way except towards all of humanity.
- JoshTriplett 10y ago1) Coping mechanisms around death aside, there's no "comfort" in building a "successor" in the form of a bot that tiles the universe with paperclips, or with neural networks that minimally satisfy some notion of "interesting" while taking up as few atoms as possible to maximize the number of them, or many other utter failure modes (which far outnumber successful outcomes). We're not talking about "smart alien-like intelligence that just doesn't care about humans", we're talking about the equivalent of an industrial accident but on a species-wide scale. 2) It's reasonable to think about how our values might change in the presence of superintelligence; we certainly shouldn't assume that our present values should forever dictate how everything works. That's different than allowing a view that sentient beings who exist today might have no value. > In fact it's the only way to do it because there is no way to know certainly that a superhuman-AGI will ensure the longevity of humanity. There's no way to know certainly; there are ways to know that the outcome has higher expected value than not having it, given the vast set of problems it can solve and the massive negative values associated with those problems.
- AndrewKemendo 10y agoI am crucially aware of all of the "failure" scenarios and I find few of them plausible - even respecting that they are simple thought experiments. What every author Bostrom, Eliezer et al. seem to miss is that there will be a practical mechanism for a digital AGI taking physical control of systems out of the hands of humans. Eg. they would need to control the resources around mining or recovering the metal, then building production plants etc... So we either incrementally cede power to them, in which case in theory the humans previously controlling the systems are doing so "rationally" and thus see the AGI as better. Or the system outsmarts the humans controlling the systems in which case it is demonstrating that it is smarter. There is a tautology here that seems to be ignored: If we create a superhuman-AGI then by default it's goals will be more universally optimal than ours. They may or may not be aligned. However the definition of term is based on the fact that it is "better" in outcome than all manner of humans. So if we create one and it decides to maximize paperclips, then that means maximizing paperclips is more optimal goal universally than whatever humans could coordinate as a goal on our own. If we create a subhuman-AGI then we will be able to overcome it's goals by virtue of the fact that we are still superior. I'll go back to a very old example. An ant can't determine if building the Large-Hadron Collider is an optimal global goal - it's inscrutable to the Ant. All it knows is that it's house and all it's friends were destroyed. If it is the case that an AGI can in fact take the physical control reigns from humans, then by definition it is smarter and will make a more optimal long term goal than we could - to the point that we probably wouldn't understand what it's doing. I think the true concern is that we will make something that is superhuman-powerful without being superhuman-intelligent. Like the doomsday machine in Dr. Strangelove, but to me that is an altogether different question.
- dorgo 10y agoI understand what an "optimal solution" might be. But what is an optimal goal?
- AndrewKemendo 10y agoGreat question. More than likely everyone has a different answer, but each person, group, nation etc... has some implicit goals. I would argue the current human goal is "reproduce successfully" as that is what is baked into our genes - I doubt that is actually optimal though. The failed idea of a coherent extrapolated volition (CEV) that came up years ago was (roughly) the idea of using revealed preferences to understand what Humanity's goal is. This would give us a benchmark for what an AGI's goal should be. So if you want to be able to measure the capability of an AGI in comparison to human system, you need to understand the set of goals in humanity and then compare them to the AGI outcomes. The concept of goal direction in AGI is hugely contentious - but make no mistake there needs to be a goal if it is going to actually function at superhuman levels.
- parenthephobia 10y agoHow can a goal be optimal? What does that even mean? Your argument seems to imply that if an AGI tricks us into giving it the ability to destroy us, that's basically okay because its goals are "better" than human goals. Speaking as a human, I don't consider goals that are compatible with the destruction of humanity to be "better" than goals which are aligned with human interests.
- AndrewKemendo 10y agoYour argument seems to imply that if an AGI tricks us into giving it the ability to destroy us, that's basically okay because its goals are "better" than human goals. Yea that's about right. I don't consider goals that are compatible with the destruction of humanity to be "better" than goals which are aligned with human interests. Well of course you wouldn't, neither you nor I would possibly understand what a superhuman-AGI does or thinks. I don't think people realize that actually creating a superhuman-AGI is effectively creating a God in all the forms that people interpret it now.