4 ms·
By no means will I be ever able to grasp Friston's theory, but the free energy minimisation vaguely reminds me of Curiosity-driven reinforcement learning. Can a
by snrji 8y ago
By no means will I be ever able to grasp Friston's theory, but the free energy minimisation vaguely reminds me of Curiosity-driven reinforcement learning. Can anyone with more understanding than me confirm or deny this apparent resemblance?
- Tenoke 8y agoThat has been my go-to comparison for a while as well, and it sounds like what they refer to in this article when they talk about using active inference-based AI vs classical reinforcement learning.
- snrji 8y agoYes, although looking at what other people have commented, "our" intuition is quite basic. I will try to read more about FEP.
- atschantz 8y agoThere is a large overlap, for instance, the popular VIME exploration algorithm [1] uses part of the free energy objective function. However, free energy isn't a theory of curiosity per se, its posed as description of self-organisation. It just so happens that you can express the free energy functional in terms of epistemic (curious) and instrumental (reward) components. [1] https://arxiv.org/abs/1605.09674 https://arxiv.org/abs/1605.09674
- snrji 8y agoI see, thanks. Is any AI research center/company trying to model this idea, instead of applying RL?
- snrji 8y agoExcuse me, another followup question (can't edit on mobile): can you ELI5 how do exploitation and exploration "emerge" naturally instead of the tradeoff being explicitly coded as in RL?
- paraschopra 8y agoThere are similarities. The difference in two approaches are: - FEP is Bayesian in nature, while there's usually no notion of uncertainties in curiosity driven RL - In FEP, there's no explicit weighting of explore/exploit tradeoff. It automatically emerges from equations - FEP, since it's Bayesian, allows for more complex reasoning (like counterfactuals) - Curiosity driven RL is scalable while FEP is not feasible for anything other than simple models
- snrji 8y agoMany thanks for your answer.
- snrji 8y agoExcuse me, another followup question (can't edit on mobile): can you ELI5 how do exploitation and exploration "emerge" naturally instead of the tradeoff being explicitly coded as in RL?
- paraschopra 8y agoIn forumations of FEP, there are two terms: cost and ambiguity. Minimisation of this combined term happens in a Bayesian optimal way. So you don't have to explicitly code weights for exploration and exploitation. Although what you do have to code is prior preferences, and since it is a distribution, you implicitly code the range of those preferences. But once you do that the FEP, algorithm figures out when to collect more data to build a better model and when to use the existing model to get near the prior preferences.
- snrji 8y agoI see. Much more elegant than explicitly coding the trade-off, actually :)
- atschantz 8y agoAs a general answer, the theory suggests that organisms maximize a quantity known as model evidence, which is just a way of saying 'how much evidence does some data provide for my model of the world?' There are two complementary ways to maximize this - change your model or change your world. If we now grant that actions also maximize model evidence, then actions can either be conducted to sample data that make the model a better fit of the data (exploration), or they can be conducted to sample observations that are consistent with the current model (exploitation).