3 ms·

I assure you 99% of the problem of any RL project is the simulator. Generally you can't let an RL algorithm control anything real from the start, so you have to

I assure you 99% of the problem of any RL project is the simulator. Generally you can't let an RL algorithm control anything real from the start, so you have to implement a reasonably reliable simulator for whatever you want done. This is the big challenge in practice.

ronsor 5y ago
Correct me if I'm wrong, but wouldn't that mean the entire world would have to be simulated? Or at least some subset of society?
- ggggtez 5y ago
  The human brain does have a simulator. It's well known. How do you know where to move your hand to catch a ball? Or what is happening when you blink? Your brain is constantly simulating a few milliseconds ahead.
  - candiodari 5y ago
    That may be true, but that's not to simulate. "Dreaming" is used to "simulate", and it's only a hack, in humans probably not even explicitly programmed. The hack is to have the simulator itself ... also be a learned system, and not to simulate the world, because you don't act on the real world, only a tiny part of it that you can measure and actually get into your brain (which is a simplified version of what you see, or a "latent variable"). There's no need to simulate anything that doesn't affect your reasoning. The information flows in reality with an intelligent actor is like this. World (say, a tree falls) -> input representation (e.g. eyes) -> simplified version ("latent" version) -> intelligent actor -> muscles -> affects world. Now what everybody thinks of as a simulator is something that simulates the whole thing. But if you insert one more link (output of reasoning agent at time T -> simplified representation at time T+1) you can then run a "simulation": random simplified version ("what if your car became a tree and fell ?") -> intelligent actor -> next input for latent representation ("then what happens ?") For safety reasons, it is probably prudent to disconnect the muscles in this state. You know, so you don't knock out your mother when you dream about boxing. And as you say, this "predict the future" network is probably useful by itself in dealing with the world. So you can catch tennis balls and the like. https://arxiv.org/abs/1803.10122 https://arxiv.org/abs/1803.10122 Or, if you like: https://www.youtube.com/watch?v=dPsXxLyqpfs https://www.youtube.com/watch?v=dPsXxLyqpfs

ronsor 5y ago

Correct me if I'm wrong, but wouldn't that mean the entire world would have to be simulated? Or at least some subset of society?

ggggtez 5y ago
The human brain does have a simulator. It's well known. How do you know where to move your hand to catch a ball? Or what is happening when you blink? Your brain is constantly simulating a few milliseconds ahead.
- candiodari 5y ago
  That may be true, but that's not to simulate. "Dreaming" is used to "simulate", and it's only a hack, in humans probably not even explicitly programmed. The hack is to have the simulator itself ... also be a learned system, and not to simulate the world, because you don't act on the real world, only a tiny part of it that you can measure and actually get into your brain (which is a simplified version of what you see, or a "latent variable"). There's no need to simulate anything that doesn't affect your reasoning. The information flows in reality with an intelligent actor is like this. World (say, a tree falls) -> input representation (e.g. eyes) -> simplified version ("latent" version) -> intelligent actor -> muscles -> affects world. Now what everybody thinks of as a simulator is something that simulates the whole thing. But if you insert one more link (output of reasoning agent at time T -> simplified representation at time T+1) you can then run a "simulation": random simplified version ("what if your car became a tree and fell ?") -> intelligent actor -> next input for latent representation ("then what happens ?") For safety reasons, it is probably prudent to disconnect the muscles in this state. You know, so you don't knock out your mother when you dream about boxing. And as you say, this "predict the future" network is probably useful by itself in dealing with the world. So you can catch tennis balls and the like. https://arxiv.org/abs/1803.10122 https://arxiv.org/abs/1803.10122 Or, if you like: https://www.youtube.com/watch?v=dPsXxLyqpfs https://www.youtube.com/watch?v=dPsXxLyqpfs