4 ms·

The origins of the approach of a perfect world model equals AI, at least how I encountered it, are from Marcus Hutter. H proved that perfect compression (=predi

The origins of the approach of a perfect world model equals AI, at least how I encountered it, are from Marcus Hutter. H proved that perfect compression (=prediction of what will happen next) is the ideal way to drive an agent in reinforcement learning[1] (the choice of action being the likeliest high-reward action). So a perfect world model is enough to win at reinforcement learning. Can you show that if you maximized reward in some RL problem, it means you necessarily built a perfect world-model? [1] A Theory of Universal Artificial Intelligence based on Algorithmic Complexity, 2000 - https://arxiv.org/pdf/cs/0004001.pdf https://arxiv.org/pdf/cs/0004001.pdf Side-note: since building a perfect model is uncomputable, this is all a theoretical discussion. The paper also discusses time-bounded computation and has some interesting things to say about optimality in this case.

abeppu 5y ago
> So a perfect world model is enough to win at reinforcement learning. Can you show that if you maximized reward in some RL problem, it means you necessarily built a perfect world-model? No? Because maximizing reward for a specific problem may mean avoiding some states entirely, so your model has no need of understanding transitions out of those states, only how to avoid landing in them. E.g. if you have a medical application about interventions to improve outcomes for patients with disease X, it's unnecessary to refine the part of your model which would predict how fast a patient would die after you administer a toxic dose of A followed by a toxic dose of B. Your model only need to know that administering a toxic dose of A always leads to lower value states than some other action. I think a "perfect" world model is required by a "universal" AI in the sense that the range of problems it can handle must be solved by optimal policies which together "cover" all state transitions (in some universe of states).

abeppu 5y ago

> So a perfect world model is enough to win at reinforcement learning. Can you show that if you maximized reward in some RL problem, it means you necessarily built a perfect world-model? No? Because maximizing reward for a specific problem may mean avoiding some states entirely, so your model has no need of understanding transitions out of those states, only how to avoid landing in them. E.g. if you have a medical application about interventions to improve outcomes for patients with disease X, it's unnecessary to refine the part of your model which would predict how fast a patient would die after you administer a toxic dose of A followed by a toxic dose of B. Your model only need to know that administering a toxic dose of A always leads to lower value states than some other action. I think a "perfect" world model is required by a "universal" AI in the sense that the range of problems it can handle must be solved by optimal policies which together "cover" all state transitions (in some universe of states).