4 ms·

> leads to the least certain outcome What's the loss function for this? How do you cross-validate this loss? The only generally applicable methods I can think

> leads to the least certain outcome What's the loss function for this? How do you cross-validate this loss? The only generally applicable methods I can think of (essentially some Bayesian-type updating, which then uses Monte-Carlo-type methods for evaluating this loss and "uncertainty") is just so much more expensive than anything we do at the moment that, at the current computing level, we wouldn't even be able to do it in most cases. The problem is, an AI model will tell you an "uncertainty" but these "uncertainties" will often not reflect the true underlying probability that it is wrong. This is, of course, all a bit silly since it's hard knowing what you don't know, and a proper update to this number is also not well-defined unless you already have a bunch of samples about potential future outcomes (via, like I mentioned before, a Monte-Carlo type approach. This would multiply the cost of something as simple as an evaluation by many times.)

klmadfejno 6y ago
In my mental model you don't need a loss function for this. It's just, given these potential short term goals predict the future state of the game 1 minute from now. Then choose the short term goal that maximizes uncertainty for the future. The idea being that you have a game playing bot that learns to do "things" in the game world, and a game state encoding bot that learns to describe the game world. I'm not suggesting you train the bot TO do the least certain idea, I'm saying you train the bot BY doing the least certain idea; and update your understanding of the world to be more certain about that choice in the future. The measure of uncertainty could be as simple as a multi armed bandit where the reward is % of accurate guesses following that path. The pseudo adversarial aspect of this being that both models will learn to agree on how choices translate to gameplay changes. The bot will not learn to beat the game necessarily, but a Style GANN kind of approach would probably be just fine to saying, "Please get to a state that looks like this" with a few manual samples. Perhaps all of this to say, it's better to learn how to play the game, and specify that, in this particular run, you'd like the bot to try and reach the game end credits.
- johnmoberg 6y ago
  Your last paragraph reminds me of Hindsight Experience Replay[1] where they use goal-conditioned policies that learn how to reach essentially any state. The idea is that, even if you don't reach your true goal, you've learned how to reach some other goal, so you can successively learn to navigate the environment until the end goal is easy. It's a really nice paper, highly recommended! [1] https://arxiv.org/abs/1707.01495 https://arxiv.org/abs/1707.01495

klmadfejno 6y ago

In my mental model you don't need a loss function for this. It's just, given these potential short term goals predict the future state of the game 1 minute from now. Then choose the short term goal that maximizes uncertainty for the future. The idea being that you have a game playing bot that learns to do "things" in the game world, and a game state encoding bot that learns to describe the game world. I'm not suggesting you train the bot TO do the least certain idea, I'm saying you train the bot BY doing the least certain idea; and update your understanding of the world to be more certain about that choice in the future. The measure of uncertainty could be as simple as a multi armed bandit where the reward is % of accurate guesses following that path. The pseudo adversarial aspect of this being that both models will learn to agree on how choices translate to gameplay changes. The bot will not learn to beat the game necessarily, but a Style GANN kind of approach would probably be just fine to saying, "Please get to a state that looks like this" with a few manual samples. Perhaps all of this to say, it's better to learn how to play the game, and specify that, in this particular run, you'd like the bot to try and reach the game end credits.

johnmoberg 6y ago
Your last paragraph reminds me of Hindsight Experience Replay[1] where they use goal-conditioned policies that learn how to reach essentially any state. The idea is that, even if you don't reach your true goal, you've learned how to reach some other goal, so you can successively learn to navigate the environment until the end goal is easy. It's a really nice paper, highly recommended! [1] https://arxiv.org/abs/1707.01495 https://arxiv.org/abs/1707.01495