4 ms·

Okay, TL;DR: "Causal Entropic Forcing" is something like an AI's utility function, where the agent attempts to maximize future possibilities. Since this is mea

Okay, TL;DR: "Causal Entropic Forcing" is something like an AI's utility function, where the agent attempts to maximize future possibilities. Since this is meaningless (all possible futures are possible), what you actually want to do is make it as easy as possible to get to those futures - aka, their entropic adjacency, hence the name, causal entropic forcing. However, CEF requires that the agent can actually predict possible future states of the system, which comes with some serious issues. In the original paper, this is covered by access to perfect simulators, but those aren't available in real-world situations. This post discusses how to (possibly) use recurrent neural networks to make such predictions; how to do so effectively, and with consideration of the NN's confidence in it's predictions. It's pretty cool!

highd 9y ago
In that case, aren't the big benchmark gains being claimed mostly the result of changing the problem to allow a perfect simulator in the system? The original author is claiming benchmark results either with a perfect simulator or with a pre-trained neural network mimicking a simulator. It seems like a massive change to the original problem. Otherwise the utility function is very similar to Q learning, just optimizing for future "flexibility" instead of future "reward". Basically we should be considering the current posted results versus Q learning with an equally accurate pre-trained forward simulator, which I don't think anyone has done.
- RangerScience 9y ago
  > gains being claimed AFAIK, the claims being gained are that they didn't have to supply the system with any goals, and it "figured out" the basic tests - included tool use and cooperation. > with an equally accurate pre-trained forward simulator AFAIK, that's exactly what the OP is discussing: how would this system perform when you replace the perfect simulator with an RNN trained to predict?
  - highd 9y ago
    I'm referring to posts like this: https://entropicai.blogspot.fr/2017/06/openai-first-record.html https://entropicai.blogspot.fr/2017/06/openai-first-record.h...
    - RangerScience 9y ago
      I'm not familiar with these works. Reading... [edit] Okay, I don't really understand what they're doing, so, my guess is that they have a component that predicts future states of the game, and they use something inspired by fractals to determine which future states to sample. Then, they're either using the score as the metric for the "value" of that future state (in which case it's not CEF), or they're ignoring the score and measuring something corresponding to entropy (or future-possibilies-remaining since it's pacman), and then they are using CEF. It's like Data playing that weird chess game; the CEF aspect doesn't help you play any better, but it gives you a different win condition that turns out to "win" better than directly trying to win. If, that is, my bad understanding is in any way accurate.
felippee 9y ago
Yes, this is very cool stuff and I've been advocating for it in previous posts. There is indeed the problem of prediction. Prediction seems to be an omnipresent paradigm in many aspects of perception, even in early vision. This, contrary to beating deep learning models to death, may hold the key to artificial intelligence. I have my own little model which I propose for prediction, I call the Predictive Vision Model, more info here: http://blog.piekniewski.info http://blog.piekniewski.info I'm looking for more people who see the potential of this.
clickok 9y ago
There's a lot details that need to be filled in to get a working algorithm from this idea, though. Like how to properly explore enough of the state space that you can estimate the ensuing entropy, and if it's possible to learn the utility and its variance in a sample-efficient manner using an RNN. It might be better to start off with an environment with unknown dynamics but an exact representation before going full nonlinear function approximation. Nonetheless it's an interesting post; I like the idea of coming up with abstract "goals" that can be applied to any environment (without having to construct a reward function) that yield complex behavior. Even if it doesn't do precisely what you want it's useful for exploration and perhaps a good stepping stone towards the desired behavior. On a related note, I believe you can learn to predict the entropy of a Markov process using reinforcement learning, so it might be possible to extend it towards control. I wrote up the basic idea: http://rl.ai/posts/generalized-returns-entropy.html http://rl.ai/posts/generalized-returns-entropy.html which argues that if you had some sort of state transition model you could construct a reward function from it, and then learning the value of a state is also the "expected entropy" starting from that state. The state transition model can itself be learned, so no simulator is required. The reason I say "I believe" is that this is the product of original research, that is, procrastinating on my thesis. So there's a risk that I've made an error somewhere or missed prior work.

highd 9y ago

In that case, aren't the big benchmark gains being claimed mostly the result of changing the problem to allow a perfect simulator in the system? The original author is claiming benchmark results either with a perfect simulator or with a pre-trained neural network mimicking a simulator. It seems like a massive change to the original problem. Otherwise the utility function is very similar to Q learning, just optimizing for future "flexibility" instead of future "reward". Basically we should be considering the current posted results versus Q learning with an equally accurate pre-trained forward simulator, which I don't think anyone has done.

RangerScience 9y ago
> gains being claimed AFAIK, the claims being gained are that they didn't have to supply the system with any goals, and it "figured out" the basic tests - included tool use and cooperation. > with an equally accurate pre-trained forward simulator AFAIK, that's exactly what the OP is discussing: how would this system perform when you replace the perfect simulator with an RNN trained to predict?
- highd 9y ago
  I'm referring to posts like this: https://entropicai.blogspot.fr/2017/06/openai-first-record.html https://entropicai.blogspot.fr/2017/06/openai-first-record.h...
  - RangerScience 9y ago
    I'm not familiar with these works. Reading... [edit] Okay, I don't really understand what they're doing, so, my guess is that they have a component that predicts future states of the game, and they use something inspired by fractals to determine which future states to sample. Then, they're either using the score as the metric for the "value" of that future state (in which case it's not CEF), or they're ignoring the score and measuring something corresponding to entropy (or future-possibilies-remaining since it's pacman), and then they are using CEF. It's like Data playing that weird chess game; the CEF aspect doesn't help you play any better, but it gives you a different win condition that turns out to "win" better than directly trying to win. If, that is, my bad understanding is in any way accurate.

felippee 9y ago

Yes, this is very cool stuff and I've been advocating for it in previous posts. There is indeed the problem of prediction. Prediction seems to be an omnipresent paradigm in many aspects of perception, even in early vision. This, contrary to beating deep learning models to death, may hold the key to artificial intelligence. I have my own little model which I propose for prediction, I call the Predictive Vision Model, more info here: http://blog.piekniewski.info http://blog.piekniewski.info I'm looking for more people who see the potential of this.

clickok 9y ago

There's a lot details that need to be filled in to get a working algorithm from this idea, though. Like how to properly explore enough of the state space that you can estimate the ensuing entropy, and if it's possible to learn the utility and its variance in a sample-efficient manner using an RNN. It might be better to start off with an environment with unknown dynamics but an exact representation before going full nonlinear function approximation. Nonetheless it's an interesting post; I like the idea of coming up with abstract "goals" that can be applied to any environment (without having to construct a reward function) that yield complex behavior. Even if it doesn't do precisely what you want it's useful for exploration and perhaps a good stepping stone towards the desired behavior. On a related note, I believe you can learn to predict the entropy of a Markov process using reinforcement learning, so it might be possible to extend it towards control. I wrote up the basic idea: http://rl.ai/posts/generalized-returns-entropy.html http://rl.ai/posts/generalized-returns-entropy.html which argues that if you had some sort of state transition model you could construct a reward function from it, and then learning the value of a state is also the "expected entropy" starting from that state. The state transition model can itself be learned, so no simulator is required. The reason I say "I believe" is that this is the product of original research, that is, procrastinating on my thesis. So there's a risk that I've made an error somewhere or missed prior work.