3 ms·

How does this relate to concepts like AIXI and Solomonoff induction?

chriswarbo 9y ago
Solomonoff induction is an (uncomputable) method which takes a sequence of inputs and predicts the subsequent inputs. If that sequence comes from some sensor, like a camera, then it can be used to predict what that sensor will detect in the future (and hence, indirectly, what the future state of the world will be). Solomonoff induction is completely passive, it doesn't say anything how to choose an action to take. AIXI applies Solomonoff induction to a reinforcement learning (RL) setting: the sequence is split into three parts: "observations" (passive input, e.g. from a camera), "actions" (which are under the agent's control) and "rewards" (which are numbers). AIXI uses Solomonoff induction to calculate what the total future rewards will be, if the sequence so far were followed by action A; or by action B; etc. and then performs whichever of those actions gave the largest predicted reward. This does tell us which action to take (at least, computable approximations do), but it relies on there being a source of reward; all sorts of "AI safety" research (e.g. intelligence.org ) is based around what such a reward should look like, and ways that an AI might achieve high reward whilst subverting our intentions. This 'causal entropic force' is a sort of implicit reward: the system is rewarded when it is able to efficiently reach other states; so it ends up 'putting itself in a good position', whatever that might mean in a particular situation. It hand-waves away a few key points: it needs a good predictor (e.g. Solomonoff induction, or something computable), and it also seems to need a world model which tells it what the "states" are. Solomonoff and AIXI don't need to be given a model: they build their own implicitly. They do need their input to be hooked up, e.g. to take pixels from a camera or whatever, but that's a known property of the implementation (e.g. the hardware available on a robot), whereas there's usually a bunch of ways we could model the world, with no "obvious" right answer, and that can directly affect how the system behaves.

Solomonoff induction is an (uncomputable) method which takes a sequence of inputs and predicts the subsequent inputs. If that sequence comes from some sensor, like a camera, then it can be used to predict what that sensor will detect in the future (and hence, indirectly, what the future state of the world will be). Solomonoff induction is completely passive, it doesn't say anything how to choose an action to take. AIXI applies Solomonoff induction to a reinforcement learning (RL) setting: the sequence is split into three parts: "observations" (passive input, e.g. from a camera), "actions" (which are under the agent's control) and "rewards" (which are numbers). AIXI uses Solomonoff induction to calculate what the total future rewards will be, if the sequence so far were followed by action A; or by action B; etc. and then performs whichever of those actions gave the largest predicted reward. This does tell us which action to take (at least, computable approximations do), but it relies on there being a source of reward; all sorts of "AI safety" research (e.g. intelligence.org ) is based around what such a reward should look like, and ways that an AI might achieve high reward whilst subverting our intentions. This 'causal entropic force' is a sort of implicit reward: the system is rewarded when it is able to efficiently reach other states; so it ends up 'putting itself in a good position', whatever that might mean in a particular situation. It hand-waves away a few key points: it needs a good predictor (e.g. Solomonoff induction, or something computable), and it also seems to need a world model which tells it what the "states" are. Solomonoff and AIXI don't need to be given a model: they build their own implicitly. They do need their input to be hooked up, e.g. to take pixels from a camera or whatever, but that's a known property of the implementation (e.g. the hardware available on a robot), whereas there's usually a bunch of ways we could model the world, with no "obvious" right answer, and that can directly affect how the system behaves.