4 ms·
You don't have to repeat 4 billion years of evolution, an RL agent lives inside a strange universe where the basic axioms happen to be exactly aligned with what
by toxik 2y ago
You don't have to repeat 4 billion years of evolution, an RL agent lives inside a strange universe where the basic axioms happen to be exactly aligned with what you can do in that universe.
Its actions are not muscular, they are literal gameplay actions. It is orders of magnitude easier to learn that the same action should be performed until completion, than that the finger should be pressed against a surface while the hand is stabilized with respect to the cursor on a screen.
One of the most interesting (and pathological) things about humans is that we learn what is rewarding. Not how to get a reward, but actually we train ourselves to be rewarded by doing difficult/novel/funny/etc things. Notably this is communicated largely by being social, i.e., we feel reward for doing something difficult because other people are impressed by that.
In Castaway, Hanks' only companion is a mute, deflated ball, but nonetheless he must keep that relationship alive---to keep himself alive. The climax of the movie is when Hanks returns home and people are so impressed, his efforts are validated.
Contrast that to RL, there is no intrinsic motivation. The agents do not play, or meaningfully explore, really. The extent of its exploration is a nervous tic that makes it press the wrong button with probability ε. The reason it cannot hold down buttons is because it explores by having Parkinson's disease, by accident, not because it thought it might find out something useful/novel/funny/etc. In fact, it can't even have a definition of those words, because they are defined in the space between beings.
- orbifold 2y agoPersonally I am almost certain that the current framing of RL and its relationship to animal behavior is deeply misguided. It proves close to impossible to train animals using this paradigm (not for a lack of trying), i.e. animals such as mice only make any progress when water deprived and under conditions that exploit their natural instincts. Nevertheless they are capable of far more complex natural behaviors. There is a non-zero chance that RL as an explanation of animal behavior is just plain wrong or not applicable.
- nomel 2y agoI naively believe that the lack of performance is one of connectivity. Animal brains don't use directed graphs, probably for the very reason that latching states, like holding a button, become unreasonable. Our brains probably use small network graphs [1][2]. [1] definition: https://en.wikipedia.org/wiki/Small-world_network https://en.wikipedia.org/wiki/Small-world_network [2] evidence for our brains: https://www.semanticscholar.org/paper/Small-world-directed-networks-in-the-human-brain%3A-Liao-Ding/dafff1dba09b935f92bd6c5184d131c37b5339bd?p2df https://www.semanticscholar.org/paper/Small-world-directed-n...
- wegfawefgawefg 2y agoGo read the Intrisic Curiosity Module papers, 1 and 2.
- exe34 2y agoAre you referring to this one? https://arxiv.org/pdf/1905.10071 https://arxiv.org/pdf/1905.10071
- wegfawefgawefg 2y agono not that one. the first icm paper: https://pathak22.github.io/noreward-rl/ https://pathak22.github.io/noreward-rl/ and the followup which address the noise impredictability problem. there are more after that which i believe fail the black pill and miss the point of ml, asicifying the architecture with human priors. But the broader point is to show that rl is not just discovering solutions by chance in random actions. Nature starts with priors, and curiosity is one of the universal policy bootstrapping techniques. (others might be imitation, next state prediction, total nearby replication count) There is also a paper that deployed ICM on a physical robot and it just played with a ball because it was the only source of novel stimuli, and inadvertantly learned how to operate its arms. There was no other reward in the environment except for curiosity. It is amazing, and slightly creepy. I think the ICM will be rediscovered later in ML tech.
- red75prime 2y ago> The agents do not play, or meaningfully explore, really As others already pointed it's not an intrinsic limitation of RL agents. > In fact, it can't even have a definition of those words, because they are defined in the space between beings. In fact an agent doesn't need to know definitions to act. A bacterium don't know what it means to reproduce, but reproduces anyway.