3 ms·

I think a key problem is our understanding of the quality of an ML system is tied to a task. Our mechanism of training is tied to a loss, or some optimization p

by abeppu 4y ago

I think a key problem is our understanding of the quality of an ML system is tied to a task. Our mechanism of training is tied to a loss, or some optimization problem. The design, training, and evaluation of these systems is dependent on an externally provided definition of "correct". But this seems structurally different from how we or even less intelligent animals operate. DALL-E may make "better" art than most humans -- but it does so in response to a human-provided prompt, according to a system trained on human produced or selected images, improving on an externally-provided loss. Whereas a human artist, even if mediocre, is directed by their own interests and judges according to their own aesthetics. Even if some of their outputs are sometimes comparable, they're not really engaged in the same activity. Methodologically, how do we create agents that aren't just good at several tasks, but make up their own tasks, "play", develop changing preferences for different activities (I think this is more than just "exploration"), etc? Even a dog sometimes wants to play with a toy, sometimes wants to run and chase, sometimes wants to be warm inside. We don't "score" how well it plays with a toy, but we take its desire to play as a signs of greater intelligence than, e.g. a pet iguana which doesn't seem to have such a desire. Further, how do we create agents that can learn without ever seriously failing? RL systems have many episodes, some of which can end very badly (e.g. your simulated runner falls off the world) and they get to learn from this. We die exactly once, and we don't get to learn from it. Note, learning from others in a social context may be part of it, but non-social animals also can learn to avoid many kinds of serious harm without first experiencing it. I don't mean to overly discount the current methods -- they're achieving amazing results. But I think even an optimist should be open to the possibility / opportunity that perhaps the current techniques will get us 80% of the way there, but that there are still some important tricks to be discovered.

phreeza 4y ago
> Methodologically, how do we create agents that aren't just good at several tasks, but make up their own tasks, "play", develop changing preferences for different activities (I think this is more than just "exploration"), etc? Even a dog sometimes wants to play with a toy, sometimes wants to run and chase, sometimes wants to be warm inside. We don't "score" how well it plays with a toy, but we take its desire to play as a signs of greater intelligence than, e.g. a pet iguana which doesn't seem to have such a desire. This doesn't sound like it would be so hard to do if you have an agent or ensemble of agents that can already do it. What you probably really want is this behavior to somehow emerge from simple ground rules, which is probably a lot harder.
sinenomine 4y ago
> Methodologically, how do we create agents that aren't just good at several tasks, but make up their own tasks It's a good question, it has been asked a few times, and there are some answers[1][2] already, with the most general being to endow the agent with intrinsic motivation defined as an information-theoretic objective to maximize some definition of surprise. Then the agent in question will develop a general curious exploration policy, if trained long enough. > Further, how do we create agents that can learn without ever seriously failing? Another good question. One of the good enough answers here is that you should design a sequence of value functions[3] for your agent, in such a way, as to enforce some invariants over its future, possibly open-ended, lifetime. For this specific concern you should ensure that your agent develops some approximation of fear, leading to aversion of catastrophic failure regions in its state space. It's pretty self-evident that we develop such a fear in the young age ourselves, and where we don't, evolution gives us a hand and makes us preemptively fear heights, or snakes, even before we ever see one. The other answer being, of course, to prove[4] a mathematical theorem around some hard definition of safe exploration in reinforcement learning. 1. https://people.idsia.ch/~juergen/interest.html https://people.idsia.ch/~juergen/interest.html 2. https://www.deepmind.com/publications/is-curiosity-all-you-need-on-the-utility-of-emergent-behaviours-from-curious-exploration https://www.deepmind.com/publications/is-curiosity-all-you-n... 3. https://www.frontiersin.org/articles/10.3389/fncom.2016.00094/full https://www.frontiersin.org/articles/10.3389/fncom.2016.0009... 4. https://arxiv.org/abs/2006.03357 https://arxiv.org/abs/2006.03357

by abeppu 4y ago

phreeza 4y ago

> Methodologically, how do we create agents that aren't just good at several tasks, but make up their own tasks, "play", develop changing preferences for different activities (I think this is more than just "exploration"), etc? Even a dog sometimes wants to play with a toy, sometimes wants to run and chase, sometimes wants to be warm inside. We don't "score" how well it plays with a toy, but we take its desire to play as a signs of greater intelligence than, e.g. a pet iguana which doesn't seem to have such a desire. This doesn't sound like it would be so hard to do if you have an agent or ensemble of agents that can already do it. What you probably really want is this behavior to somehow emerge from simple ground rules, which is probably a lot harder.

sinenomine 4y ago

> Methodologically, how do we create agents that aren't just good at several tasks, but make up their own tasks It's a good question, it has been asked a few times, and there are some answers[1][2] already, with the most general being to endow the agent with intrinsic motivation defined as an information-theoretic objective to maximize some definition of surprise. Then the agent in question will develop a general curious exploration policy, if trained long enough. > Further, how do we create agents that can learn without ever seriously failing? Another good question. One of the good enough answers here is that you should design a sequence of value functions[3] for your agent, in such a way, as to enforce some invariants over its future, possibly open-ended, lifetime. For this specific concern you should ensure that your agent develops some approximation of fear, leading to aversion of catastrophic failure regions in its state space. It's pretty self-evident that we develop such a fear in the young age ourselves, and where we don't, evolution gives us a hand and makes us preemptively fear heights, or snakes, even before we ever see one. The other answer being, of course, to prove[4] a mathematical theorem around some hard definition of safe exploration in reinforcement learning. 1. https://people.idsia.ch/~juergen/interest.html https://people.idsia.ch/~juergen/interest.html 2. https://www.deepmind.com/publications/is-curiosity-all-you-need-on-the-utility-of-emergent-behaviours-from-curious-exploration https://www.deepmind.com/publications/is-curiosity-all-you-n... 3. https://www.frontiersin.org/articles/10.3389/fncom.2016.00094/full https://www.frontiersin.org/articles/10.3389/fncom.2016.0009... 4. https://arxiv.org/abs/2006.03357 https://arxiv.org/abs/2006.03357