2 ms·
What the thought experiment tries to get at is that it is hard to predict how a super intelligence will evolve from a set of simple objectives. No, it's pretty
by joe_the_user 11y ago
What the thought experiment tries to get at is that it is hard to predict how a super intelligence will evolve from a set of simple objectives.
No, it's pretty easy. I predict a super intelligence wouldn't evolve at all from a set of simple objectives.
(that may seem a little snarky but the original question has the implied assumption that super inteilligence could evolve from only a set of simple objectives and I think that assumption is implausible, is only accepted because it's made implicitly, etc)
But I guess that's a fundamental disagreement. I think it's relatively "obvious" that creating an intelligent system would involve the intentional crafting as well as inputs of particular immediate goals. If intelligence could from just whatever system gets complex and has feedback loops, then I'd agree we're in trouble and need to go around smash all the high-end thermostats in a fashion akin to a b-grade horror movie. But clearly I think that's a dubious "if".
- sawwit 11y agoI made that assumption explicitly, actually. To make it worse, it seems plausible that these kinds of dangerous AIs will be the most simple ones to build, since they won't require a lot of goal engineering. Intelligence is a superset of feedback loops. It is concerned with cases in which feedback loops are not sufficient to optimize the agents objectives, and attempts to reach them using learning and prediction. The complexity in behavior comes from interacting with a complex environment and having many model parameters; not (necessarily) from the initial goals. (As an intuition pump, have a look at GoogleMind's atari reinforcement learner. The essential parts of the code fit onto a single page [1].) [1] https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner