4 ms·
Humans, as opposed to deep learning, have embodiment. We can move about, push and prod, formulate ideas and test them in the world. A deep net can't do any of t
by visarga 9y ago
Humans, as opposed to deep learning, have embodiment. We can move about, push and prod, formulate ideas and test them in the world. A deep net can't do any of that in the supervised learning setting. The only way to do that is inside an RL agent. The problem is that any of our RL agents so far need to run inside a simulated environment, which is orders of magnitude less complex than reality. So they can't learn because they can't explore like us.
The solution would be to improve embodiment for neural nets and to equip RL agents with internal world simulators (a world model) they could use to plan ahead. So we need simulation both outside and inside agents. Neural nets by themselves are not even the complete answer. But what is missing is not necessarily a new algorithm or data representation, it's the whole world-agent complex.
Not to mention that a human alone is not much use - we need society and culture to unlock our potential. Before we knew the cause, we believed disease was caused by gods, and it took many deaths to unlock the mystery. We're not perfect either, we just sit on top of the previous generations. Another advantage we have - we have a builtin reward system that guides learning, which was created by evolution. We have to create this reward system for RL agents from scratch.
In some special cases like board games, the board is a perfect simulation in itself (happens to be trivial, just observe the rules, play against a replica of yourself). In that case RL agents can reach superhuman intelligence, but that is mostly on account of having a perfect playground to test ideas in.
In the future simulation and RL will form the next step in AI. The current limiting factor block is not the net, but the simulator. I think everyone here has noticed the blooming of many game environments used for training RL agents from DeepMind, OpenAI, Atari, StarCraft, Dota2, GTA, MuJoCo and others. It's a race to build the playground for the future intelligences.
Latest paper from DeepMind?
> IMPALA: Scalable Distributed DeepRL in DMLab-30. DMLab-30 is a collection of new levels designed using our open source RL environment DeepMind Lab. These environments enable any DeepRL researcher to test systems on a large spectrum of interesting tasks either individually or in a multi-task setting.
Before we build an AI, we need to build a world for that AI to be in.
- bitL 9y ago> run inside a simulated environment It's pretty common to use DRL in real-world robotics e.g. in self-driving cars or robotic motion planing, so you don't need to simulate the real thing. The obvious issue with RL is that it needs even more iterations/episodes/epochs than a typical DL, making real-world development either impractical (time) or too costly.
- bryananderson 9y agoEmbodiment is part of the picture, but I also am not sure that we have yet developed structures that are capable of learning in the way that humans do, no matter how rich the environment. Rocks, flatworms, and the Hoover Dam all have embodiment in the same complex world that we do, but none of them will ever coherently debate philosophy, because they have no structures capable of learning to do so. I’m not convinced that our RL agents do either.
- visarga 9y agoThat's easy. Flatworms can't because we took the top spot and hogged resources. Give them a world where they are the most advanced species, and a few billion years, and they can debate philosophy too. What would happen if a flatworm came out of the sea and tried to take up land from us? Hoover Dam is not embodied because it is not an agent. An agent has good an bad, life and death. The Dam doesn't give a damn about any of it.
- electrograv 9y ago> The solution would be to improve embodiment for neural nets and to equip RL agents with internal world simulators (a world model) they could use to plan ahead. That’s one possible solution, but it only works to the extent that your simulator accurately reflects reality. Any hard coded simulation is therefore doomed to fail badly, and any adaptable “simulation” isn’t really a simulation at all — just more neural architecture. > So we need simulation both outside and inside agents. We don’t even need a simulation outside of agents, if you just use physical robot bodies :) Humans do just fine learning without extra simulation environments — the real world is enough. Of course, simulated environments are incredibly useful, but not in the ways you’re claiming. Mostly it’s because reinforcement learning is still so slow to learn that without a simulated environment it would take forever to test anything. Also, they allow repeatable, testable results that allow learning effectiveness to be measured scientifically, etc. Also I know you can formulate RL such that you use a simulator as part of the learning algorithm, but IMO that is cheating at worse, and a misguided rewsearch path at best. > Neural nets by themselves are not even the complete answer. Only to the extent that human brains are also not the complete answer to intelligence (i.e. maybe there’s a possible brain architecutre much better than ours), but I don’t think that’s what you’re saying. If you meant the current wave of artificial neurons are not the complete answer, then of course I agree. But I don’t think that’s what you’re saying either. Show me the part of our brain that you think contains a perfect simulation of reality and I’ll concede. Now certainly, our brain can do a kind of “neural simulations” to understand and reason about consequences of actions, but that “simulation” (if we can even call it such) is learned.
- visarga 9y agoRL agents (and humans) don't actually need perfect internal simulation of the environment. Instead we need ability to quickly recover from deviations, when the model diverges from reality, like self driving cars. We are using the world itself as a "simulator". The problem with AI agents doing the same is that it's expensive. We'd need cheap, fast, indestructible robots and environments to do that. Humans benefited form millions of years of evolution, and previous species took billions of years to evolve up to our species. That's a lot of "high resolution environment time" which would be impossible to recreate in silico as of now. Then there's the problem of sample complexity - humans need less examples than AI agents to learn, and that's because we have prior knowledge about the world encoded in our genetic code by evolution, while RL agents don't. We have to bake in some pretty clever prior knowledge in RL agents if we want them to be efficient. So AI's can make due with imperfect internal simulation, but they need perfect external simulation of the same complexity with the real world in order to become smart like us.
- toisanji 9y agoagreed, this project is trying to do just that: https://github.com/jtoy/sensenet https://github.com/jtoy/sensenet
- yourapostasy 9y agoTo my untrained mind it is unexpected on the one hand to observe complete sensory deprivation for extended periods is considered highly cruel to sapients, yet here we are on the other hand, building what amounts to sensory deprivation tanks with tightly-controlled inputs, and expecting to somehow build sapient minds out of those modest beginnings. Even what we consider as "only" sentient-but-not-sapient phyla within the Animalia kingdom appear to wither on increasing degrees of sensory deprivation, perhaps until flatworm-scale sentience [1]. [1] https://www.sciencedirect.com/science/article/pii/0003347265900916 https://www.sciencedirect.com/science/article/pii/0003347265...