3 ms·
It’s a (reasonable, in my opinion) but not proven assumption. It’s not a fact and not by definition. I think the bigger problem is that while the author assume
by jeeeb 3y ago
It’s a (reasonable, in my opinion) but not proven assumption. It’s not a fact and not by definition.
I think the bigger problem is that while the author assumes that humans cannot understand or control a super intelligent AI, he readily assumes that having escaped the constraints of its human creators the AI will either immediately go on to reward itself with infinite utility and become inert, or aim to infinitely reproduce.
The surety with which the author predicts the outcomes of the super intelligent being modifying its utility function, does not make sense if we assume that we cannot understand or limit its behaviours.
Like under this assumption the AI agents behaviour is by actual definition unpredictable.
- boringuser2 3y agoYes, I wasn't really addressing those points. It's a bit pedantic to state that an axiom isn't a fact, as, while that's largely correct, it hampers communication. Regarding your other points, I don't disagree per se, though you could make the argument that there is a substrate of existence that humans have sufficient intellect to accurately model. If so, it isn't necessarily completely implausible that we'd be able to predict the goals of a superior intellect by assuming that it has to play by the same rules, and will be merely be superior at playing by those constraints. As such, as we could determine the goals, we needent determine the mechanisms to make accurate claims about the end result of what a superior intellect might achieve. It's also possible that we simply understand nothing and are too limited to ever grasp anything beyond a certain point, in which case this entire argument is pointless.