6 ms·

all of this, plus an additional observation that i've had about games w/ tiers/divisions: player skill is assumed to be normally distributed when that is just s

all of this, plus an additional observation that i've had about games w/ tiers/divisions: player skill is assumed to be normally distributed when that is just so demonstrably not the case-- there is a fairly high skill floor to be able to play the game at all, and the right tail (high skill) of the distribution is WAY fatter than the left. especially with well-established, popular games-- Chess, League of Legends, Overwatch, etc. (where there is even a financial interest in being a top player to boot), the skill levels of the people at the absolute top simply profoundly dwarf players that would even superficially seem "comparable" by the standard of being in adjacent, or even within the same tier. in League of Legends, for example, it is often claimed that the differences between players in high/low Challenger, high/low Master's, and even high/low "high diamond" (low d2 vs high d1) all constitute distinct "tiers" of player quality that are as substantial as the full-tier jumps closer to the median (e.g. silver -> gold, gold -> platinum), but because of this shoehorned prior about skill distribution it leads to this compression at the very top.

_dps 6y ago
> player skill is assumed to be normally distributed when that is just so demonstrably not the case This is close but not exactly right, and the small difference matters. Elo does not assume that skill is normally distributed, but rather that "quality of play" in a single game is normally distributed around some average quality level for the player. Obviously this too is an approximation but it's a much smaller one.
- keeganpoppen 6y ago
  hmm, interesting. i did mean to say that this is a problem more in the context of games that add tiers/divisions to their ranked ladders, but i hadn't really thought about elo making assumption about the normal-distributed-ness of player deviation from their "true" skill level. does that not just fall out directly from the Central limit theorem (given the taking of large #s samples (game W/L observations vs. predicted P(win|my elo, their elo)) of means, etc.)?
  - deleted 6y ago
    [deleted]
Someone 6y ago
“player skill is assumed to be normally distributed” I would think player skill level (at best; there easily can be cases where P typically beats Q, Q beats R and R beats P) is an ordinal (https://en.wikipedia.org/wiki/Ordinal_data https://en.wikipedia.org/wiki/Ordinal_data), so one can’t say “player P is twice as good as player Q”, “player P is as much better than player Q as player R is better than player S”, and certainly can’t prove or disprove whether skill is being normally distributed. It is customary to assume that, though. Also, if one assigns numbers to skill levels, those can be normally distributed. It probably is possible to design an ELO-like system that, given enough games, guarantees that the set of skill level numbers of all players approaches being normally distributed.
Aerroon 6y ago
Another thing to consider with a lot of these games is that they're not static. The game changes and this can boost one player's rating up when their preferred champions/heroes/whatever are strong at that time. Even if the game didn't change, there are so many different characters that's play differently enough that the player's results with them could end up at a rather different rating.
Godel_unicode 6y ago
Here's season 13 of rocket league. Free red delicious apple for the first person to correctly identify the shape of the curve: https://2p1ipt36o1g23g1tt6ba5nou-wpengine.netdna-ssl.com/wp-content/uploads/2020/04/rocket-league-ranks-distribution-season-13-2020-standard-896x1024.png https://2p1ipt36o1g23g1tt6ba5nou-wpengine.netdna-ssl.com/wp-...
- keeganpoppen 6y ago
  looks gamma, but just like... slightly gamma. i guess my contention would be that it "probably" should be smushed/redistributed with even more mass on the right tail, but i couldn't tell you from personal experience whether that's true, as i'm pretty terrible at rocket league. i will say that the top Rocket League players (to my untrained eye, and jaw on the absolute floor) may have an even higher z-score than top players than any other game(s)... but rocket league is kind of unique in it's being a remake of a game that i guess a lot of (the same) people used to play.
- thaumasiotes 6y ago
  The X axis appears to be an ordinal number, some kind of proprietary rank. How much sense does it make to talk about the shape of a distribution over ordinal numbers? If we converted those to cardinals, the shape of the new, reality-based distribution could be pretty much anything.
  - thaumasiotes 6y ago
    To expand on this point, I recently made histograms of gaokao scores for several Chinese provinces. You can see Beijing data here: http://www.gaokao.com/e/20180623/5b2e13c43951c.shtml http://www.gaokao.com/e/20180623/5b2e13c43951c.shtml This report is nicer than many others I found in that scores are reported all the way down to 0, rather than cutting off at the threshold for university admission. It's much less nice than some in that, below the university admission threshold, scores are reported in brackets of 10 points rather than by individual score. (This, even though the document is called "一分一段表"...) Anyway, I imported this data into python and plotted it with matplotlib. The histogram you get from this is obviously, wildly flawed -- the ten-points-wide bar from 410 to 419 is also 710 people tall, dwarfing the actual mode of the distribution. To correct this problem, you need to divide the count for bracketed scores by 10 (the width of the bracket) -- the 710 people scoring 410-419 are 71 per score in that range, very comparable to the 70 people scoring 420, but not to the 182 people scoring 548. Without knowing the width of the rocket league rank brackets, that picture of the population of each rank doesn't tell us anything -- at all -- about the shape of the distribution.

_dps 6y ago

> player skill is assumed to be normally distributed when that is just so demonstrably not the case This is close but not exactly right, and the small difference matters. Elo does not assume that skill is normally distributed, but rather that "quality of play" in a single game is normally distributed around some average quality level for the player. Obviously this too is an approximation but it's a much smaller one.

keeganpoppen 6y ago
hmm, interesting. i did mean to say that this is a problem more in the context of games that add tiers/divisions to their ranked ladders, but i hadn't really thought about elo making assumption about the normal-distributed-ness of player deviation from their "true" skill level. does that not just fall out directly from the Central limit theorem (given the taking of large #s samples (game W/L observations vs. predicted P(win|my elo, their elo)) of means, etc.)?
- deleted 6y ago
  [deleted]

Someone 6y ago

“player skill is assumed to be normally distributed” I would think player skill level (at best; there easily can be cases where P typically beats Q, Q beats R and R beats P) is an ordinal (https://en.wikipedia.org/wiki/Ordinal_data https://en.wikipedia.org/wiki/Ordinal_data), so one can’t say “player P is twice as good as player Q”, “player P is as much better than player Q as player R is better than player S”, and certainly can’t prove or disprove whether skill is being normally distributed. It is customary to assume that, though. Also, if one assigns numbers to skill levels, those can be normally distributed. It probably is possible to design an ELO-like system that, given enough games, guarantees that the set of skill level numbers of all players approaches being normally distributed.

Aerroon 6y ago

Another thing to consider with a lot of these games is that they're not static. The game changes and this can boost one player's rating up when their preferred champions/heroes/whatever are strong at that time. Even if the game didn't change, there are so many different characters that's play differently enough that the player's results with them could end up at a rather different rating.

Godel_unicode 6y ago

Here's season 13 of rocket league. Free red delicious apple for the first person to correctly identify the shape of the curve: https://2p1ipt36o1g23g1tt6ba5nou-wpengine.netdna-ssl.com/wp-content/uploads/2020/04/rocket-league-ranks-distribution-season-13-2020-standard-896x1024.png https://2p1ipt36o1g23g1tt6ba5nou-wpengine.netdna-ssl.com/wp-...

keeganpoppen 6y ago
looks gamma, but just like... slightly gamma. i guess my contention would be that it "probably" should be smushed/redistributed with even more mass on the right tail, but i couldn't tell you from personal experience whether that's true, as i'm pretty terrible at rocket league. i will say that the top Rocket League players (to my untrained eye, and jaw on the absolute floor) may have an even higher z-score than top players than any other game(s)... but rocket league is kind of unique in it's being a remake of a game that i guess a lot of (the same) people used to play.
thaumasiotes 6y ago
The X axis appears to be an ordinal number, some kind of proprietary rank. How much sense does it make to talk about the shape of a distribution over ordinal numbers? If we converted those to cardinals, the shape of the new, reality-based distribution could be pretty much anything.
- thaumasiotes 6y ago
  To expand on this point, I recently made histograms of gaokao scores for several Chinese provinces. You can see Beijing data here: http://www.gaokao.com/e/20180623/5b2e13c43951c.shtml http://www.gaokao.com/e/20180623/5b2e13c43951c.shtml This report is nicer than many others I found in that scores are reported all the way down to 0, rather than cutting off at the threshold for university admission. It's much less nice than some in that, below the university admission threshold, scores are reported in brackets of 10 points rather than by individual score. (This, even though the document is called "一分一段表"...) Anyway, I imported this data into python and plotted it with matplotlib. The histogram you get from this is obviously, wildly flawed -- the ten-points-wide bar from 410 to 419 is also 710 people tall, dwarfing the actual mode of the distribution. To correct this problem, you need to divide the count for bracketed scores by 10 (the width of the bracket) -- the 710 people scoring 410-419 are 71 per score in that range, very comparable to the 70 people scoring 420, but not to the 182 people scoring 548. Without knowing the width of the rocket league rank brackets, that picture of the population of each rank doesn't tell us anything -- at all -- about the shape of the distribution.