4 ms·

> The solution would be to improve embodiment for neural nets and to equip RL agents with internal world simulators (a world model) they could use to plan ahead

> The solution would be to improve embodiment for neural nets and to equip RL agents with internal world simulators (a world model) they could use to plan ahead. That’s one possible solution, but it only works to the extent that your simulator accurately reflects reality. Any hard coded simulation is therefore doomed to fail badly, and any adaptable “simulation” isn’t really a simulation at all — just more neural architecture. > So we need simulation both outside and inside agents. We don’t even need a simulation outside of agents, if you just use physical robot bodies :) Humans do just fine learning without extra simulation environments — the real world is enough. Of course, simulated environments are incredibly useful, but not in the ways you’re claiming. Mostly it’s because reinforcement learning is still so slow to learn that without a simulated environment it would take forever to test anything. Also, they allow repeatable, testable results that allow learning effectiveness to be measured scientifically, etc. Also I know you can formulate RL such that you use a simulator as part of the learning algorithm, but IMO that is cheating at worse, and a misguided rewsearch path at best. > Neural nets by themselves are not even the complete answer. Only to the extent that human brains are also not the complete answer to intelligence (i.e. maybe there’s a possible brain architecutre much better than ours), but I don’t think that’s what you’re saying. If you meant the current wave of artificial neurons are not the complete answer, then of course I agree. But I don’t think that’s what you’re saying either. Show me the part of our brain that you think contains a perfect simulation of reality and I’ll concede. Now certainly, our brain can do a kind of “neural simulations” to understand and reason about consequences of actions, but that “simulation” (if we can even call it such) is learned.

visarga 9y ago
RL agents (and humans) don't actually need perfect internal simulation of the environment. Instead we need ability to quickly recover from deviations, when the model diverges from reality, like self driving cars. We are using the world itself as a "simulator". The problem with AI agents doing the same is that it's expensive. We'd need cheap, fast, indestructible robots and environments to do that. Humans benefited form millions of years of evolution, and previous species took billions of years to evolve up to our species. That's a lot of "high resolution environment time" which would be impossible to recreate in silico as of now. Then there's the problem of sample complexity - humans need less examples than AI agents to learn, and that's because we have prior knowledge about the world encoded in our genetic code by evolution, while RL agents don't. We have to bake in some pretty clever prior knowledge in RL agents if we want them to be efficient. So AI's can make due with imperfect internal simulation, but they need perfect external simulation of the same complexity with the real world in order to become smart like us.

visarga 9y ago

RL agents (and humans) don't actually need perfect internal simulation of the environment. Instead we need ability to quickly recover from deviations, when the model diverges from reality, like self driving cars. We are using the world itself as a "simulator". The problem with AI agents doing the same is that it's expensive. We'd need cheap, fast, indestructible robots and environments to do that. Humans benefited form millions of years of evolution, and previous species took billions of years to evolve up to our species. That's a lot of "high resolution environment time" which would be impossible to recreate in silico as of now. Then there's the problem of sample complexity - humans need less examples than AI agents to learn, and that's because we have prior knowledge about the world encoded in our genetic code by evolution, while RL agents don't. We have to bake in some pretty clever prior knowledge in RL agents if we want them to be efficient. So AI's can make due with imperfect internal simulation, but they need perfect external simulation of the same complexity with the real world in order to become smart like us.