5 ms·

I have a bone to pick with these "win probability" charts. <rant> If it was truly a "win probability" chart, that means it's a forecast. Except it's a shit fo

I have a bone to pick with these "win probability" charts. <rant> If it was truly a "win probability" chart, that means it's a forecast. Except it's a shit forecast because you're trying to predict really freaking far into the future (You don't even know how far because the game could end at any point in time). It also makes zero sense. Think about it, I say "you have a 30% chance of winning the game from this position". What does that even mean? If I play well I have a 30% chance of winning? If my opponent is of equal skill I have a 30% chance of winning? It's completely uninterpretable. There's a good reason for this though. It's not measuring "win probability", it's just maximizing the value of "winning" (I.e. predicting output = 1) given the current state. They even admit this in the article! "In addition to making decisions for the game AI, we also used the model to display the estimated win probability for a player over the course of the game" It's supposed to be a vague and un-interpretable value because it's generated by a black box neural net. So why do we continue to pretend that this is a human-friendly value? </rant>

TylerLives 6y ago
> Think about it, I say "you have a 30% chance of winning the game from this position". What does that even mean? If I play well I have a 30% chance of winning? If my opponent is of equal skill I have a 30% chance of winning? It's completely uninterpretable. It means that the AI that generated the data has 30% chance of winning, or at least that's what it's trying to estimate.
- ZephyrBlu 6y ago
  That doesn't make sense in the context of a game though. You can't freeze the game and say "I have a xx% chance to win here", because the state of the game continues evolving over time. The generated value is based on a snapshot of the game and isn't related to the final outcome at all, so I believe saying it's a "win probability" is completely wrong.
  - anonymoushn 6y ago
    If I have pocket aces and you have 7-2 off-suit, does a win probability exist for my aces against your 7-2? If Ryu is knocked down and he has to dragon punch to beat a meaty command grab and has to throw to beat an opponet who is attempting to block his dragon punch, and both players are at very low health, and these two options dominate all other options, such that the entire game hinges on a single round of a double-blind guessing game with only two possibilities, does a win probability exist for Ryu?
    - ZephyrBlu 6y ago
      You have created 2 scenarios with very small actions spaces. Most games have enormous action spaces, so you can't easily compute a probability. Take Chess for example. Chess engines evaluate millions of scenarios for every move. They don't generate a "win probability" value though, they evaluate board positions.
      - anonymoushn 6y ago
        > Most games have enormous action spaces, so you can't easily compute a probability. Saying you can't compute something by hand on a single sheet of paper is different from saying it doesn't exist. In Chess, your win probability is either 0 or 1 (you either can force a win or you cannot), but most chess engines have a real-valued estimate of their equity in the game that is between 0 and 1.
        ZephyrBlu 6y ago
        > but most chess engines have a real-valued estimate of their equity in the game that is between 0 and 1 My understanding is that Chess engines do not evaluate a 0-1 value to representing chance to win. They evaluate the perceived material advantage from the current board position. See: https://www.chess.com/blog/ChessMN16/how-to-read-engine-evaluations https://www.chess.com/blog/ChessMN16/how-to-read-engine-eval...
        deleted 6y ago
jwilber 6y ago
Disagree with basically everything written here. The neural net is a well-researched cnn variant - the activations are quite interpretable these days. The metric’s interpretability is independent from the neural net used for gameplay, so the black box comment doesn’t make sense (unless I’m misunderstanding the comment...). The metric is also very human friendly: think of it as 30% of players in the given position, at the given game’s stage and state, successfully continued on to a victory.
- ZephyrBlu 6y ago
  > The metric is also very human friendly: think of it as 30% of players in the given position, at the given game’s stage and state, successfully continued on to a victory What I am trying to say is that this is a meaningless piece of information, because that's not how games work. You can't freeze the game state and say "I have a xx% chance to win here", because the state of the game continues evolving over time. If you were actually estimating your chance to win, it would be a forecast. Except if it's anything like this kind of NN [0], then it is literally just estimating based on a snapshot of game state and not the context of the game. At best, this "win probability" is snapshot estimate of the current game state and has nothing to do with the actual outcome of the game. This value is also impossible to falsify, so it could be spitting out any random number and you can't say that it's wrong. [0] https://medium.com/analytics-vidhya/a-simple-neural-network-for-in-game-win-probability-modeling-of-ncaa-basketball-games-58ab4e3ca0f9 https://medium.com/analytics-vidhya/a-simple-neural-network-...
  - anonymoushn 6y ago
    What's "the context of the game" and why can't it be included in the encoding of the game state?
    - ZephyrBlu 6y ago
      Context of the game = previous events that occurred. Don't you think it's a bit odd to be forecasting a "win probability" value without taking the events of the game into account? > why can't it be included in the encoding of the game state? How do you encode time series or event data into singular values?
sdwr 6y ago
Like any metric, it's not perfect, but there's a lot of good information there. Dota added a win prediction graph a while back that's been valuable to look at. It's not really a black box, because you can see the win % change over time and get a feel for what it's weighting. For example, it takes character selection into account, and the initial prediction can get to 60/40 before the game even starts. The bit that I found the most interesting is when the prediction doesn't line up with my intuition of the game state. I've had a couple of comebacks from a predicted 1% win chance, and all those felt more like 20-30% comebacks, where we were down but definitely not out. A map is not the territory, and that's easiest to see when you fall off the edge of the map.
- ZephyrBlu 6y ago
  I'm aware of Dota's win prediction chart. Do you know about STRATZ? They also have a win probability chart on their website as well as predictions for each hero in the game. In their Discord, different people have asked the same questions about these "win probability" values multiple times because often they're very unintuitive (Because they are not actually probabilities). > It's not really a black box, because you can see the win % change over time and get a feel for what it's weighting If it wasn't a black box you would know why it's weighted at a particular value. I think it's mostly a curiosity and not that helpful if you're trying to analyze your games.

TylerLives 6y ago

> Think about it, I say "you have a 30% chance of winning the game from this position". What does that even mean? If I play well I have a 30% chance of winning? If my opponent is of equal skill I have a 30% chance of winning? It's completely uninterpretable. It means that the AI that generated the data has 30% chance of winning, or at least that's what it's trying to estimate.

ZephyrBlu 6y ago
That doesn't make sense in the context of a game though. You can't freeze the game and say "I have a xx% chance to win here", because the state of the game continues evolving over time. The generated value is based on a snapshot of the game and isn't related to the final outcome at all, so I believe saying it's a "win probability" is completely wrong.
- anonymoushn 6y ago
  If I have pocket aces and you have 7-2 off-suit, does a win probability exist for my aces against your 7-2? If Ryu is knocked down and he has to dragon punch to beat a meaty command grab and has to throw to beat an opponet who is attempting to block his dragon punch, and both players are at very low health, and these two options dominate all other options, such that the entire game hinges on a single round of a double-blind guessing game with only two possibilities, does a win probability exist for Ryu?
  - ZephyrBlu 6y ago
    You have created 2 scenarios with very small actions spaces. Most games have enormous action spaces, so you can't easily compute a probability. Take Chess for example. Chess engines evaluate millions of scenarios for every move. They don't generate a "win probability" value though, they evaluate board positions.
    - anonymoushn 6y ago
      > Most games have enormous action spaces, so you can't easily compute a probability. Saying you can't compute something by hand on a single sheet of paper is different from saying it doesn't exist. In Chess, your win probability is either 0 or 1 (you either can force a win or you cannot), but most chess engines have a real-valued estimate of their equity in the game that is between 0 and 1.
      - ZephyrBlu 6y ago
        > but most chess engines have a real-valued estimate of their equity in the game that is between 0 and 1 My understanding is that Chess engines do not evaluate a 0-1 value to representing chance to win. They evaluate the perceived material advantage from the current board position. See: https://www.chess.com/blog/ChessMN16/how-to-read-engine-evaluations https://www.chess.com/blog/ChessMN16/how-to-read-engine-eval...
        deleted 6y ago

jwilber 6y ago

Disagree with basically everything written here. The neural net is a well-researched cnn variant - the activations are quite interpretable these days. The metric’s interpretability is independent from the neural net used for gameplay, so the black box comment doesn’t make sense (unless I’m misunderstanding the comment...). The metric is also very human friendly: think of it as 30% of players in the given position, at the given game’s stage and state, successfully continued on to a victory.

ZephyrBlu 6y ago
> The metric is also very human friendly: think of it as 30% of players in the given position, at the given game’s stage and state, successfully continued on to a victory What I am trying to say is that this is a meaningless piece of information, because that's not how games work. You can't freeze the game state and say "I have a xx% chance to win here", because the state of the game continues evolving over time. If you were actually estimating your chance to win, it would be a forecast. Except if it's anything like this kind of NN [0], then it is literally just estimating based on a snapshot of game state and not the context of the game. At best, this "win probability" is snapshot estimate of the current game state and has nothing to do with the actual outcome of the game. This value is also impossible to falsify, so it could be spitting out any random number and you can't say that it's wrong. [0] https://medium.com/analytics-vidhya/a-simple-neural-network-for-in-game-win-probability-modeling-of-ncaa-basketball-games-58ab4e3ca0f9 https://medium.com/analytics-vidhya/a-simple-neural-network-...
- anonymoushn 6y ago
  What's "the context of the game" and why can't it be included in the encoding of the game state?
  - ZephyrBlu 6y ago
    Context of the game = previous events that occurred. Don't you think it's a bit odd to be forecasting a "win probability" value without taking the events of the game into account? > why can't it be included in the encoding of the game state? How do you encode time series or event data into singular values?

sdwr 6y ago

Like any metric, it's not perfect, but there's a lot of good information there. Dota added a win prediction graph a while back that's been valuable to look at. It's not really a black box, because you can see the win % change over time and get a feel for what it's weighting. For example, it takes character selection into account, and the initial prediction can get to 60/40 before the game even starts. The bit that I found the most interesting is when the prediction doesn't line up with my intuition of the game state. I've had a couple of comebacks from a predicted 1% win chance, and all those felt more like 20-30% comebacks, where we were down but definitely not out. A map is not the territory, and that's easiest to see when you fall off the edge of the map.

ZephyrBlu 6y ago
I'm aware of Dota's win prediction chart. Do you know about STRATZ? They also have a win probability chart on their website as well as predictions for each hero in the game. In their Discord, different people have asked the same questions about these "win probability" values multiple times because often they're very unintuitive (Because they are not actually probabilities). > It's not really a black box, because you can see the win % change over time and get a feel for what it's weighting If it wasn't a black box you would know why it's weighted at a particular value. I think it's mostly a curiosity and not that helpful if you're trying to analyze your games.