3 ms·

Link to the actual paper. https://pathak22.github.io/large-scale-curiosity/ https://pathak22.github.io/large-scale-curiosity/ I was curious how they define or

Link to the actual paper. https://pathak22.github.io/large-scale-curiosity/ https://pathak22.github.io/large-scale-curiosity/ I was curious how they define or reward curiosity, it says it right here: Reinforcement learning algorithms rely on carefully engineering environment rewards that are extrinsic to the agent. However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent. Curiosity is a type of intrinsic reward function which uses prediction error as reward signal. So, the prediction error is the reward, nice.

gunshai 3y ago
This seems highly suseptible to what a human might consider irrelevant randomness. Given random images just shuffling indefinitely a curious individual will just give up and say, even though I can't predict the next thing it doesn't pertain to the domain of curiosity.
- dimatura 3y ago
  Right, that's why the "noisy TV problem" is a "problem" - it makes the simple model of curiosity used in the research less effective, because it causes agents to get stuck watching TV instead of exploring and advancing in the game, which is what we'd want them to do. (Though this does seem somewhat reminiscent of certain human behaviors...). One possible solution is to equip agents with a more expressive predictive model, capable of discerning "interesting" randomness from "uninteresting" randomness (for some TBD definition of "interesting", of course I'm handwaving here - I'm sure in the 5 past years there's been progress on this front).
  - jhardy54 3y ago
    First thing that popped into my head: “interesting randomness” is when you can’t predict future frames, but [within some interval] you gain that ability. Static on the TV is random but uninteresting, whereas morse code is “random” at first, but after enough exposure can be understood and predicted.

gunshai 3y ago

This seems highly suseptible to what a human might consider irrelevant randomness. Given random images just shuffling indefinitely a curious individual will just give up and say, even though I can't predict the next thing it doesn't pertain to the domain of curiosity.

dimatura 3y ago
Right, that's why the "noisy TV problem" is a "problem" - it makes the simple model of curiosity used in the research less effective, because it causes agents to get stuck watching TV instead of exploring and advancing in the game, which is what we'd want them to do. (Though this does seem somewhat reminiscent of certain human behaviors...). One possible solution is to equip agents with a more expressive predictive model, capable of discerning "interesting" randomness from "uninteresting" randomness (for some TBD definition of "interesting", of course I'm handwaving here - I'm sure in the 5 past years there's been progress on this front).
- jhardy54 3y ago
  First thing that popped into my head: “interesting randomness” is when you can’t predict future frames, but [within some interval] you gain that ability. Static on the TV is random but uninteresting, whereas morse code is “random” at first, but after enough exposure can be understood and predicted.