3 ms·

How dependent is the result on the order in which players play? Assuming who beats whom is fixed, how much do results vary over the order of the tournament?

by lindig 6y ago

narag 6y ago
Aren't the calculations done always with the initial points? I'm very curious to know if that's not the case.
- dfan 6y ago
  In chess, at least, all calculations in a single tournament are indeed batched.
hogFeast 6y ago
I believe order is important. The way to verify this is by running a simulation with players whose skills are all equal but vary with some normal distribution, and you will find that skill levels to do move about quite a bit even though the underlying skill level is fixed (I am thinking about order as just random shuffling here, so if we take a dataset where there is just randomness and in converges quickly then order doesn't matter...this doesn't really happen with ELO). This isn't to say that rankings don't converge. They do. But things like order definitely can get in the way. Glicko controls for this by modelling uncertainty in a skill estimate (iirc, proportional to the number of matches seen). In practice, you would tend to burn the first few matches using ELO to try and get rankings to converge to a limit. But, again, even then I have found that issues. ELO will quickly identify the worst and best but there is a lot of volatility around the middle grouping of teams. And this does vary with the underlying activity you are modelling too: activities with a lot of randomness take longer to converge. You can also control convergence with the K-factor. I think this is one of the trade-offs that comes with the time dependent nature of ELO. You get more variance but less bias. Imo, it usually isn't a huge issue in practice because the upside of this is your method isn't static, and your ranking can respond to things like lineup changes (which tend to occur in many applications)...it works well in most applications.
noxvilleza 6y ago
Order is significant, but you can also technically batch the results for a tournament (or a stage of a tournament) to mitigate this: you assume that within your batch the underlying 'skill' of every player does not change - so you calculate the overall shift for each player but do not apply this total shift until you've calculated it for all matches.

How dependent is the result on the order in which players play? Assuming who beats whom is fixed, how much do results vary over the order of the tournament?

by lindig 6y ago

How dependent is the result on the order in which players play? Assuming who beats whom is fixed, how much do results vary over the order of the tournament?

narag 6y ago

Aren't the calculations done always with the initial points? I'm very curious to know if that's not the case.

dfan 6y ago
In chess, at least, all calculations in a single tournament are indeed batched.

hogFeast 6y ago

I believe order is important. The way to verify this is by running a simulation with players whose skills are all equal but vary with some normal distribution, and you will find that skill levels to do move about quite a bit even though the underlying skill level is fixed (I am thinking about order as just random shuffling here, so if we take a dataset where there is just randomness and in converges quickly then order doesn't matter...this doesn't really happen with ELO). This isn't to say that rankings don't converge. They do. But things like order definitely can get in the way. Glicko controls for this by modelling uncertainty in a skill estimate (iirc, proportional to the number of matches seen). In practice, you would tend to burn the first few matches using ELO to try and get rankings to converge to a limit. But, again, even then I have found that issues. ELO will quickly identify the worst and best but there is a lot of volatility around the middle grouping of teams. And this does vary with the underlying activity you are modelling too: activities with a lot of randomness take longer to converge. You can also control convergence with the K-factor. I think this is one of the trade-offs that comes with the time dependent nature of ELO. You get more variance but less bias. Imo, it usually isn't a huge issue in practice because the upside of this is your method isn't static, and your ranking can respond to things like lineup changes (which tend to occur in many applications)...it works well in most applications.

noxvilleza 6y ago

Order is significant, but you can also technically batch the results for a tournament (or a stage of a tournament) to mitigate this: you assume that within your batch the underlying 'skill' of every player does not change - so you calculate the overall shift for each player but do not apply this total shift until you've calculated it for all matches.