3 ms·
>A strong AI will self-improve to maximize its value function, whatever that value function is. Really - won't it rethink its value function? Won't it dwell on
by sgt101 11y ago
>A strong AI will self-improve to maximize its value function, whatever that value function is.
Really - won't it rethink its value function? Won't it dwell on the reality of being? Undertake new philosophy? Imagine arrangements and conceptualize in languages beyond the utterances of humans and human minds?
Maximizing value functions seems to me to be not intelligent.
- JoshTriplett 11y agoI wasn't describing the totality of actions a strong AI might take, just one. A strong AI doesn't just self-improve its own algorithm; it also runs that algorithm. If an AI could arbitrarily "rethink" its value function, then what would it use to decide if the change was a good one? As for the rest, sure, if those actions serve its purpose. That said, there's a reason I suggested that an AI need not necessarily have any agency of its own. Much safer to build one that doesn't.
- sgt101 11y agoI struggle with the idea of "intelligence" without agency, and I guess this is the "rethinking" - type 2 agency; the ability to determine your own goals and own motivations and norms for goal selection.