9 ms·

Elo is great for what it was built for: ranking chess players. Chess is (1) extremely low-variance, (2) has an extremely high skill ceiling, and (3) is 1-on-1.

by dvt 6y ago

Elo is great for what it was built for: ranking chess players. Chess is (1) extremely low-variance, (2) has an extremely high skill ceiling, and (3) is 1-on-1. Elo works great for chess, but it would never work for something like Poker. Let's briefly go over these three points. Most games aren't chess -- where the only variance is picking who's black and who's white -- in fact, they might include dozens of RNG mechanics (from critical strikes to ability rolls, to spawn points). These mechanics (while fun and well-designed) might pollute your "idealized" model. There's also the problem of RPS (rock-paper-scissors) mechanics or pick-counter-pick mechanics which will also heavily skew win rates. For instance, given a slow combo Magic deck, you will most likely auto-concede to mono red aggro (regardless of skill level). If you're using Elo, this will pollute your model. (Hint: you shouldn't be using Elo.) Most games also don't have chess' high skill ceiling. Chess has such a high skill ceiling for a number of reasons -- it's one of the oldest games still being actively played, for one. Suppose your "game" is simply the flip of a coin (everyone wins 50% of the time). Zero skill involved. Trying to model win-loss-ratios using a sigmoid curve is silly. Obviously, no game is going to be a coin flip, but there's a world of difference between chess and DOTA. TruSkill attempts to fix (3) by using clever Bayesian updating on a player-by-player basis[1] but in reality, it's a shit-show. Using Elo (or variants thereof) for team-based games where the team isn't really a team (more like 3-5 random people plopped together for one match) is incredibly misguided, but continues to be implemented in just about every modern multiplayer game (to the players' frustration). Of course, mixing and matching pre-made groups with non pre-made groups creates as many issues as you might imagine. In short, why so many game devs are enamored with Elo when it comes to ranking is a bit bizarre. [1] https://www.microsoft.com/en-us/research/wp-content/uploads/2007/01/NIPS2006_0688.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/...

keeganpoppen 6y ago
all of this, plus an additional observation that i've had about games w/ tiers/divisions: player skill is assumed to be normally distributed when that is just so demonstrably not the case-- there is a fairly high skill floor to be able to play the game at all, and the right tail (high skill) of the distribution is WAY fatter than the left. especially with well-established, popular games-- Chess, League of Legends, Overwatch, etc. (where there is even a financial interest in being a top player to boot), the skill levels of the people at the absolute top simply profoundly dwarf players that would even superficially seem "comparable" by the standard of being in adjacent, or even within the same tier. in League of Legends, for example, it is often claimed that the differences between players in high/low Challenger, high/low Master's, and even high/low "high diamond" (low d2 vs high d1) all constitute distinct "tiers" of player quality that are as substantial as the full-tier jumps closer to the median (e.g. silver -> gold, gold -> platinum), but because of this shoehorned prior about skill distribution it leads to this compression at the very top.
- _dps 6y ago
  > player skill is assumed to be normally distributed when that is just so demonstrably not the case This is close but not exactly right, and the small difference matters. Elo does not assume that skill is normally distributed, but rather that "quality of play" in a single game is normally distributed around some average quality level for the player. Obviously this too is an approximation but it's a much smaller one.
  - keeganpoppen 6y ago
    hmm, interesting. i did mean to say that this is a problem more in the context of games that add tiers/divisions to their ranked ladders, but i hadn't really thought about elo making assumption about the normal-distributed-ness of player deviation from their "true" skill level. does that not just fall out directly from the Central limit theorem (given the taking of large #s samples (game W/L observations vs. predicted P(win|my elo, their elo)) of means, etc.)?
    - deleted 6y ago
      [deleted]
- Someone 6y ago
  “player skill is assumed to be normally distributed” I would think player skill level (at best; there easily can be cases where P typically beats Q, Q beats R and R beats P) is an ordinal (https://en.wikipedia.org/wiki/Ordinal_data https://en.wikipedia.org/wiki/Ordinal_data), so one can’t say “player P is twice as good as player Q”, “player P is as much better than player Q as player R is better than player S”, and certainly can’t prove or disprove whether skill is being normally distributed. It is customary to assume that, though. Also, if one assigns numbers to skill levels, those can be normally distributed. It probably is possible to design an ELO-like system that, given enough games, guarantees that the set of skill level numbers of all players approaches being normally distributed.
- Aerroon 6y ago
  Another thing to consider with a lot of these games is that they're not static. The game changes and this can boost one player's rating up when their preferred champions/heroes/whatever are strong at that time. Even if the game didn't change, there are so many different characters that's play differently enough that the player's results with them could end up at a rather different rating.
keeganpoppen 6y ago
also, what system(s) do you prefer / know of that handle multiplayer matchmaking well? it seems to me that a good system might be necessarily game-specific to some extent, although i'm sure the state of the art is much better than what i've experienced gaming to date xD.
- dvt 6y ago
  > also, what system(s) do you prefer / know of that handle multiplayer matchmaking well? None, and actually I don't think it's particularly healthy for the game. For example, I had plenty of fun casually pubbing Counter Strike in the early 2000s. When I wanted to take the game more seriously, I made a team and joined a league which might include group play, single/double elimination, and exhibition games. Actual competitive play (scrims, matches, tournaments) is fundamentally different than what today we call "matchmaking."
  - lemagedurage 6y ago
    Do you believe multiplayer games would be better off without matchmaking at all?
    - dvt 6y ago
      Yeah, 100%. I'm actually flabbergasted why you can't make a team in games like CSGO or Overwatch. And then play in tournaments or matches (against, you know, other teams). It makes no sense to have individual matchmaking in a team game. Game devs create this individual matchmaking system (which is paradoxically taken seriously by casual players, but totally ignored by actual competitive players), and the community and other organizers (enter FaceIT, ESEA, etc.) have to actually set up leagues, tournaments, and events.
      - keeganpoppen 6y ago
        this would probably work better if, in the case of Overwatch, the teams weren't six players (i personally have always felt like the game would be better @ like 4v4 anyway, because of how god damn frustrating dealing with 5 random players on your team every game is in a game that is balanced purely around teamwork and inability to solo carry w/o being much more skilled than everyone else in the game)
juped 6y ago
Elo, not ELO, after Árpád Élő.
- dvt 6y ago
  Correct, fixed :)
bsder 6y ago
I also suspect that Chess is exponential because of the "one mistake and you die" nature when playing good players. Ben Finegold (a Chess Grandmaster) talks about this all the time--"The reason why I'm higher rated than you is that I can play 100 moves without a major mistake and at some point you will hang a piece. The reason why Magnus Carlsson is rated higher than me is that he will play 100 moves that are slightly better than mine and I will lose."
Wohlf 6y ago
>Why so many game devs are enamored with Elo when it comes to ranking is a bit bizarre. Probably just Occam's Razor, they don't know of or care to make something better and can just pull Elo off the shelf.
- tmpz22 6y ago
  Another example would be competitive overwatch where the developer's stated goal was an equal distribution of rated players throughout the various ranks (bronze/silver/gold/platinum/diamond/masters/grandmasters). They tweaked the variables until they got their desired distribution, lumping the majority of players in gold and plat. Ranking up became an exercise in either playing hundreds of hours or starting a brand new account with fresh MMR. Predictably this led to an explosion in boosting and win-trading services.
  - nl 6y ago
    > the developer's stated goal was an equal distribution of rated players throughout the various ranks and > They tweaked the variables until they got their desired distribution, lumping the majority of players in gold and plat seem to be opposites. What am I missing here?
- canofbars 6y ago
  It also probably doesn't even matter since the main problem is players intentionally losing to rank down and then smash lower rank players.
CWuestefeld 6y ago
My wife was a champion table tennis player. This sport uses Elo as well, and I know from watching the sport over time that the rating system has real problems. It doesn't suffer from the weaknesses that you cite, but even so, the problem of "rating inflation" is widely discussed. It seems that much of the problem comes from rating points brought in by newbie players (and note that, contra TFA, the problem isn't with experienced players losing to newbies, but the opposite). A newbie is started off with some nominal rating; I forget the number, but let's say it's 800. Most likely that newbie is going to lose his first matches, and some proportion of those newbies will get frustrated and quit. For the ones that stay in the game, things probably work out in the long run. But for those that got discouraged and quit, in the course of their loss they caused a few points (not many, because they're likely way overmatched, but definitely more than 0) to be credited to their opponents. When they quit the sport, they're never going to reclaim any of the rating points that they lost initially. But those points are still in the system, having been added to their winning opponents. It's hard to quantify because the Elo system is the only objective comparison we have, but over the course of the almost 30 years I've been watching my wife play, the Elo rating enjoyed by a player of a given hypothetical skill level has increased dramatically. Many are saying that for someone of the upper echelons, their rating is maybe 200 points higher than it would have been 30 years ago. So back in 1991, my wife was in the top 30 women in the USA with a rating in the mid-1700s. Today, someone with that rating isn't even going to be in the top brackets of serious tournament. Despite all that, the usefulness of the rating system keeps it in use as a valuable tool. It seems that the ability to match players who have never seen each other before, ensuring interesting matches, is part of keeping the game competitive for those in it. And table tennis is also, because of this, one of what I believe is few sports where men and women often play head-to-head (even though men generally have much higher ratings, on account of the sport requiring far more strength than you might suspect).
- dvt 6y ago
  > It doesn't suffer from the weaknesses that you cite, but even so, the problem of "rating inflation" is widely discussed. Ah yes! Inflation is also a problem I've seen in competitive online games. Rating inflation was a serious issue with World of Warcraft PvP arenas circa 10 years ago (iirc Blizzard hard capped arena ratings at 3000 during WotLK). I don't follow chess much, and I'm not exactly sure how chess avoids it (or even if it does).
lmm 6y ago
> Most games aren't chess -- where the only variance is picking who's black and who's white -- in fact, they might include dozens of RNG mechanics (from critical strikes to ability rolls, to spawn points). These mechanics (while fun and well-designed) might pollute your "idealized" model. There's also the problem of RPS (rock-paper-scissors) mechanics or pick-counter-pick mechanics which will also heavily skew win rates. For instance, given a slow combo Magic deck, you will most likely auto-concede to mono red aggro (regardless of skill level). If you're using Elo, this will pollute your model. (Hint: you shouldn't be using Elo.) None of which matters? All that means is that the results of individual games are a bit higher variance. Elo handles that by design. If you lose a certain proportion of Magic games to less-skilled players then this should be considered a reflection of your skill, because the only reasonable definition of skill at the came is the rate at which you actually win it; anything else can be gamed and so should be ignored. > Most games also don't have chess' high skill ceiling. Chess has such a high skill ceiling for a number of reasons -- it's one of the oldest games still being actively played, for one. Suppose your "game" is simply the flip of a coin (everyone wins 50% of the time). Zero skill involved. Trying to model win-loss-ratios using a sigmoid curve is silly. Obviously, no game is going to be a coin flip, but there's a world of difference between chess and DOTA. That's also something that Elo handles just fine? If every game is a coin flip then everyone will end up with the same Elo. If player A has x more Elo points than player B, then they win y% of their games. If your game has a skill ceiling where even a complete beginner always wins, say, 20% of their games, then that just means no-one will ever be able to rise above a corresponding Elo rating.
- dvt 6y ago
  > That's also something that Elo handles just fine? If every game is a coin flip then everyone will end up with the same Elo. If player A has x more Elo points than player B, then they win y% of their games. If your game has a skill ceiling where even a complete beginner always wins, say, 20% of their games, then that just means no-one will ever be able to rise above a corresponding Elo rating. That's not how it works. The distribution you end up with will not be uniform, it will look like this (just ran Elo with a coinflip; 11 players, 1000 matches): https://imgur.com/9O82pRj https://imgur.com/9O82pRj On the long term, I think this will tend to a geometric distribution with a low p value.
Godel_unicode 6y ago
In your magic example you seem to be arguing that which kind of deck you pick is not part of your skill, which is of course totally incorrect. Picking "fun" decks over "obvious/OP" decks means you're worse at winning games. Or at least that you generally play with a handicap, which is easy to account for in elo. To your coin-flip example, if you model a league in excel you'll find that elo actually results in a rank distribution very consistent with what your intuition would expect (given enough players and enough matches, of course).
- dvt 6y ago
  > To your coin-flip example, if you model a league in excel you'll find that elo actually results in a rank distribution very consistent with what your intuition would expect (given enough players and enough matches, of course). This is incorrect. If you simulate Elo with a coin-flip, you'll get something that looks like this (11 players, 1000 matches): https://imgur.com/9O82pRj https://imgur.com/9O82pRj -- I think this will tend to a geometric distribution (not sure what the p is though, probably depends on the constants).
  - Godel_unicode 6y ago
    >> given enough players and enough matches, of course
    - dvt 6y ago
      Here's 1000 players with 100,000 matches: https://i.imgur.com/1Y08jUB.png https://i.imgur.com/1Y08jUB.png Feel free to try simulating it yourself, but even mathematically it makes no sense to end up with a uniform distribution as we tend to infinity.

by dvt 6y ago

keeganpoppen 6y ago

all of this, plus an additional observation that i've had about games w/ tiers/divisions: player skill is assumed to be normally distributed when that is just so demonstrably not the case-- there is a fairly high skill floor to be able to play the game at all, and the right tail (high skill) of the distribution is WAY fatter than the left. especially with well-established, popular games-- Chess, League of Legends, Overwatch, etc. (where there is even a financial interest in being a top player to boot), the skill levels of the people at the absolute top simply profoundly dwarf players that would even superficially seem "comparable" by the standard of being in adjacent, or even within the same tier. in League of Legends, for example, it is often claimed that the differences between players in high/low Challenger, high/low Master's, and even high/low "high diamond" (low d2 vs high d1) all constitute distinct "tiers" of player quality that are as substantial as the full-tier jumps closer to the median (e.g. silver -> gold, gold -> platinum), but because of this shoehorned prior about skill distribution it leads to this compression at the very top.

_dps 6y ago
> player skill is assumed to be normally distributed when that is just so demonstrably not the case This is close but not exactly right, and the small difference matters. Elo does not assume that skill is normally distributed, but rather that "quality of play" in a single game is normally distributed around some average quality level for the player. Obviously this too is an approximation but it's a much smaller one.
- keeganpoppen 6y ago
  hmm, interesting. i did mean to say that this is a problem more in the context of games that add tiers/divisions to their ranked ladders, but i hadn't really thought about elo making assumption about the normal-distributed-ness of player deviation from their "true" skill level. does that not just fall out directly from the Central limit theorem (given the taking of large #s samples (game W/L observations vs. predicted P(win|my elo, their elo)) of means, etc.)?
  - deleted 6y ago
    [deleted]
Someone 6y ago
“player skill is assumed to be normally distributed” I would think player skill level (at best; there easily can be cases where P typically beats Q, Q beats R and R beats P) is an ordinal (https://en.wikipedia.org/wiki/Ordinal_data https://en.wikipedia.org/wiki/Ordinal_data), so one can’t say “player P is twice as good as player Q”, “player P is as much better than player Q as player R is better than player S”, and certainly can’t prove or disprove whether skill is being normally distributed. It is customary to assume that, though. Also, if one assigns numbers to skill levels, those can be normally distributed. It probably is possible to design an ELO-like system that, given enough games, guarantees that the set of skill level numbers of all players approaches being normally distributed.
Aerroon 6y ago
Another thing to consider with a lot of these games is that they're not static. The game changes and this can boost one player's rating up when their preferred champions/heroes/whatever are strong at that time. Even if the game didn't change, there are so many different characters that's play differently enough that the player's results with them could end up at a rather different rating.

keeganpoppen 6y ago

also, what system(s) do you prefer / know of that handle multiplayer matchmaking well? it seems to me that a good system might be necessarily game-specific to some extent, although i'm sure the state of the art is much better than what i've experienced gaming to date xD.

dvt 6y ago
> also, what system(s) do you prefer / know of that handle multiplayer matchmaking well? None, and actually I don't think it's particularly healthy for the game. For example, I had plenty of fun casually pubbing Counter Strike in the early 2000s. When I wanted to take the game more seriously, I made a team and joined a league which might include group play, single/double elimination, and exhibition games. Actual competitive play (scrims, matches, tournaments) is fundamentally different than what today we call "matchmaking."
- lemagedurage 6y ago
  Do you believe multiplayer games would be better off without matchmaking at all?
  - dvt 6y ago
    Yeah, 100%. I'm actually flabbergasted why you can't make a team in games like CSGO or Overwatch. And then play in tournaments or matches (against, you know, other teams). It makes no sense to have individual matchmaking in a team game. Game devs create this individual matchmaking system (which is paradoxically taken seriously by casual players, but totally ignored by actual competitive players), and the community and other organizers (enter FaceIT, ESEA, etc.) have to actually set up leagues, tournaments, and events.
    - keeganpoppen 6y ago
      this would probably work better if, in the case of Overwatch, the teams weren't six players (i personally have always felt like the game would be better @ like 4v4 anyway, because of how god damn frustrating dealing with 5 random players on your team every game is in a game that is balanced purely around teamwork and inability to solo carry w/o being much more skilled than everyone else in the game)

juped 6y ago

Elo, not ELO, after Árpád Élő.

dvt 6y ago
Correct, fixed :)

bsder 6y ago

I also suspect that Chess is exponential because of the "one mistake and you die" nature when playing good players. Ben Finegold (a Chess Grandmaster) talks about this all the time--"The reason why I'm higher rated than you is that I can play 100 moves without a major mistake and at some point you will hang a piece. The reason why Magnus Carlsson is rated higher than me is that he will play 100 moves that are slightly better than mine and I will lose."

Wohlf 6y ago

>Why so many game devs are enamored with Elo when it comes to ranking is a bit bizarre. Probably just Occam's Razor, they don't know of or care to make something better and can just pull Elo off the shelf.

tmpz22 6y ago
Another example would be competitive overwatch where the developer's stated goal was an equal distribution of rated players throughout the various ranks (bronze/silver/gold/platinum/diamond/masters/grandmasters). They tweaked the variables until they got their desired distribution, lumping the majority of players in gold and plat. Ranking up became an exercise in either playing hundreds of hours or starting a brand new account with fresh MMR. Predictably this led to an explosion in boosting and win-trading services.
- nl 6y ago
  > the developer's stated goal was an equal distribution of rated players throughout the various ranks and > They tweaked the variables until they got their desired distribution, lumping the majority of players in gold and plat seem to be opposites. What am I missing here?
canofbars 6y ago
It also probably doesn't even matter since the main problem is players intentionally losing to rank down and then smash lower rank players.

CWuestefeld 6y ago

My wife was a champion table tennis player. This sport uses Elo as well, and I know from watching the sport over time that the rating system has real problems. It doesn't suffer from the weaknesses that you cite, but even so, the problem of "rating inflation" is widely discussed. It seems that much of the problem comes from rating points brought in by newbie players (and note that, contra TFA, the problem isn't with experienced players losing to newbies, but the opposite). A newbie is started off with some nominal rating; I forget the number, but let's say it's 800. Most likely that newbie is going to lose his first matches, and some proportion of those newbies will get frustrated and quit. For the ones that stay in the game, things probably work out in the long run. But for those that got discouraged and quit, in the course of their loss they caused a few points (not many, because they're likely way overmatched, but definitely more than 0) to be credited to their opponents. When they quit the sport, they're never going to reclaim any of the rating points that they lost initially. But those points are still in the system, having been added to their winning opponents. It's hard to quantify because the Elo system is the only objective comparison we have, but over the course of the almost 30 years I've been watching my wife play, the Elo rating enjoyed by a player of a given hypothetical skill level has increased dramatically. Many are saying that for someone of the upper echelons, their rating is maybe 200 points higher than it would have been 30 years ago. So back in 1991, my wife was in the top 30 women in the USA with a rating in the mid-1700s. Today, someone with that rating isn't even going to be in the top brackets of serious tournament. Despite all that, the usefulness of the rating system keeps it in use as a valuable tool. It seems that the ability to match players who have never seen each other before, ensuring interesting matches, is part of keeping the game competitive for those in it. And table tennis is also, because of this, one of what I believe is few sports where men and women often play head-to-head (even though men generally have much higher ratings, on account of the sport requiring far more strength than you might suspect).

dvt 6y ago
> It doesn't suffer from the weaknesses that you cite, but even so, the problem of "rating inflation" is widely discussed. Ah yes! Inflation is also a problem I've seen in competitive online games. Rating inflation was a serious issue with World of Warcraft PvP arenas circa 10 years ago (iirc Blizzard hard capped arena ratings at 3000 during WotLK). I don't follow chess much, and I'm not exactly sure how chess avoids it (or even if it does).

lmm 6y ago

> Most games aren't chess -- where the only variance is picking who's black and who's white -- in fact, they might include dozens of RNG mechanics (from critical strikes to ability rolls, to spawn points). These mechanics (while fun and well-designed) might pollute your "idealized" model. There's also the problem of RPS (rock-paper-scissors) mechanics or pick-counter-pick mechanics which will also heavily skew win rates. For instance, given a slow combo Magic deck, you will most likely auto-concede to mono red aggro (regardless of skill level). If you're using Elo, this will pollute your model. (Hint: you shouldn't be using Elo.) None of which matters? All that means is that the results of individual games are a bit higher variance. Elo handles that by design. If you lose a certain proportion of Magic games to less-skilled players then this should be considered a reflection of your skill, because the only reasonable definition of skill at the came is the rate at which you actually win it; anything else can be gamed and so should be ignored. > Most games also don't have chess' high skill ceiling. Chess has such a high skill ceiling for a number of reasons -- it's one of the oldest games still being actively played, for one. Suppose your "game" is simply the flip of a coin (everyone wins 50% of the time). Zero skill involved. Trying to model win-loss-ratios using a sigmoid curve is silly. Obviously, no game is going to be a coin flip, but there's a world of difference between chess and DOTA. That's also something that Elo handles just fine? If every game is a coin flip then everyone will end up with the same Elo. If player A has x more Elo points than player B, then they win y% of their games. If your game has a skill ceiling where even a complete beginner always wins, say, 20% of their games, then that just means no-one will ever be able to rise above a corresponding Elo rating.

dvt 6y ago
> That's also something that Elo handles just fine? If every game is a coin flip then everyone will end up with the same Elo. If player A has x more Elo points than player B, then they win y% of their games. If your game has a skill ceiling where even a complete beginner always wins, say, 20% of their games, then that just means no-one will ever be able to rise above a corresponding Elo rating. That's not how it works. The distribution you end up with will not be uniform, it will look like this (just ran Elo with a coinflip; 11 players, 1000 matches): https://imgur.com/9O82pRj https://imgur.com/9O82pRj On the long term, I think this will tend to a geometric distribution with a low p value.

Godel_unicode 6y ago

In your magic example you seem to be arguing that which kind of deck you pick is not part of your skill, which is of course totally incorrect. Picking "fun" decks over "obvious/OP" decks means you're worse at winning games. Or at least that you generally play with a handicap, which is easy to account for in elo. To your coin-flip example, if you model a league in excel you'll find that elo actually results in a rank distribution very consistent with what your intuition would expect (given enough players and enough matches, of course).

dvt 6y ago
> To your coin-flip example, if you model a league in excel you'll find that elo actually results in a rank distribution very consistent with what your intuition would expect (given enough players and enough matches, of course). This is incorrect. If you simulate Elo with a coin-flip, you'll get something that looks like this (11 players, 1000 matches): https://imgur.com/9O82pRj https://imgur.com/9O82pRj -- I think this will tend to a geometric distribution (not sure what the p is though, probably depends on the constants).
- Godel_unicode 6y ago
  >> given enough players and enough matches, of course
  - dvt 6y ago
    Here's 1000 players with 100,000 matches: https://i.imgur.com/1Y08jUB.png https://i.imgur.com/1Y08jUB.png Feel free to try simulating it yourself, but even mathematically it makes no sense to end up with a uniform distribution as we tend to infinity.