9 ms·
Implementing the Elo Rating System (2020)
- noir_lord 6y agoGlicko2 is a better system. https://en.wikipedia.org/wiki/Glicko_rating_system https://en.wikipedia.org/wiki/Glicko_rating_system
- andrewzah 6y agoThere's also the TrueSkill rating system from Microsoft (developed for Halo), which claims to be better than Glicko: "Glicko was developed as an extension of ELO and was thus naturally limited to two player matches which end in either win or loss. Glicko cannot update skill levels of players if they compete in multi-player events or even in teams. The logistic model would make it computationally expensive to deal with team and multi-player games. Moreover, chess is usually played in pre-set tournaments and thus matching the right opponents was not considered a relevant problem in Glicko. In contrast, the TrueSkill ranking system offers a way to measure the quality of a match between any set of players." https://www.microsoft.com/en-us/research/project/trueskill-ranking-system/ https://www.microsoft.com/en-us/research/project/trueskill-r...
- HeyImAlex 6y agoI noticed trueskill is pretty effective at matching people together, but the stickiness of rankings makes it frustrating if it’s user facing. People want to see the number next to their name change after 4 consecutive wins.
- postalrat 6y agoI think most people would want to see a significant boost if you won 40 out of 50 games. Winning 4 out of 4 doesn't mean much.
- monkeybutton 6y agoFirst, a good article about TrueSkill I found when looking for systems to rank pairs of foosball players: http://www.moserware.com/2010/03/computing-your-skill.html http://www.moserware.com/2010/03/computing-your-skill.html Second, apparently Tinder uses an ELO ranking system for their algorithm. I would add a link but everything I'm finding online right now is very clickbait-y.
- bmm6o 6y agoI'd be interested to read more about Tinder's application. Off the top of my head, i don't see how it would apply to a dating app.
- monkeybutton 6y ago>What was it, though? It was a part of our algorithm that considered how others engaged with your profile. ... this part of our algorithm compared Likes and Nopes, and was utilized to show you potential matches who may be a fit for you, based on similarities in the way others would engage with profiles. Based on those profile ratings you received, there was a “score” — in the sense that it was represented with a numeric value in our systems so that it could factor into the other facets in our algorithm. From: https://blog.gotinder.com/powering-tinder-r-the-method-behind-our-matching/ https://blog.gotinder.com/powering-tinder-r-the-method-behin... Its not a deep description at all, but it sounds like when someone viewing your profile is treated as a game, which way they swipe is a win or loss for you and your change in score depends on their score as well? Basically if a desirable person swipes right on your average profile, its indicating you are desirable and moves you up more than say the inverse scenario where an average profile swipes right on a desirable profile (I guess this would be akin to a low ranked player losing to a high ranked player). Edit: I guess the outcome would be people of equal (as perceived by others) attractiveness being matched.
- uxp100 6y agoIs the issue multiple players or is it teams? I didn't have any problem implementing ELO for multiple players in games where there can be multiple losers but one winner. I modeled it as simultaneous wins for the winner, with a K-factor divided by the number of players. But I wasn't concerned with being mathematically rigourus, maybe there is some glitch with this approach? It was for an EDH group. (BTW, don't do this. We no longer play EDH. It makes the spiky players even spikier, and then they get saltier when they're targeted as they have a far higher ELO)
- leetcrew 6y agoI think the issue is that, outside of formal tournaments, there are rarely stable teams competing against each other in typical online multiplayer. each team is a random assortment of players, where anywhere from zero to a full team are queuing together. people might queue with a friend of a very different skill level. you need to track the skill level of each individual player and then estimate the win probability for a team of players who may never have played with each other before. I play a lot of csgo, and while far from perfect, I'm surprised how good the matchmaking is. I win pretty close to 50% of my matches.
- reificator 6y ago> It was for an EDH group. (BTW, don't do this. We no longer play EDH. It makes the spiky players even spikier, and then they get saltier when they're targeted as they have a far higher ELO) Yes please keep tournaments and rankings away from EDH for the love of god. CEDH is a different story because it's a completely different mindset, but if I sat down for normal EDH and was told we were tracking Elo I'd run far, far away.
- uxp100 6y agoIt was mainly to track strength of different decks we all played, since you could check ELO of either. It was the strongest player who was upset by it, since once people realized he won 50% of 3 and 4 player games he was always targeted. But yeah, I wouldn't do it again.
- porphyra 6y agoFor certain types of multiplayer games the recent Elo-MMR is even better (faster, provable guarantees) [1][2]. Quote from the paper regarding TrueSkill: > The main disadvantage of TrueSkill is its complexity: originally developed by Microsoft for the popular Halo video game, TrueSkill performs approximate belief propagation on a factor graph, which is iterated until convergence. Aside from being less human-interpretable, this complexity means that, to our knowledge, there are no proofs of key properties such as runtime and incentive alignment. Even when these properties are discussed, no rigorous justification is provided. In addition, we are not aware of any work that extends TrueSkill to non-Gaussian performance models, which might be desirable to limit the influence of outlier performances. [1] https://github.com/EbTech/Elo-MMR https://github.com/EbTech/Elo-MMR [2] https://arxiv.org/abs/2101.00400 https://arxiv.org/abs/2101.00400
- andrewzah 6y agoIt is fascinating to see improvement in this space. People -really- care about good ranking systems. I personally saw this when I administrated rankings for a local fighting game scene. Back then we used Glicko since matches were 1-on-1.
- meheleventyone 6y agoTrueskill is also under patent. Which is considerably more off putting than its rigor.
- porphyra 6y agoGlicko2 has a fatal flaw for massively multiplayer games such as Pokemon Go. Basically, by intentionally losing certain games, you can increase the volatility in your rating, and then you can win a few games in a row against low ranked players and leapfrog people who are actually consistently trying to win. See: [1] Farming Volatility: How a major flaw in a well-known rating system takes over the GBL leaderboard. https://www.reddit.com/r/TheSilphRoad/comments/hwff2d/farming_volatility_how_a_major_flaw_in_a/ https://www.reddit.com/r/TheSilphRoad/comments/hwff2d/farmin...
- syspec 6y agoDoes this flaw only apply to massively multiplayer games?
- bitshiftfaced 6y agoThis further strengthens my belief that Glicko 1 is much more preferable to Glicko 2. It's simpler (no volatility parameter), nearly as predictive (if not just as predictive), and it wasn't specifically designed for a monthly chess schedule, so it fits online games much more easily.
- noxvilleza 6y agoDepends on the environment really. If you're looking at measuring performance where competitors are not able to game the system (like, in professional sports) then Glicko 2 does a slightly better (but statistically significant) job than Glicko 1.
- bluecalm 6y agoDepends what you mean by better. I agree Glicko approximates your true skill level better and faster. It's also not as fun as ELO in my view. With ELO when I win a few games in a row I get an exciting chance to play against much better players than usual. This is a fun challenge. It's necessary to learn new things. It gives you highs and lows. It's just fun. With Glicko, once it pinpoints your true skill level, it feels like 2 meters of mud around you. You will never get out of it and you will be forever matched vs the same group of players stuck near you. So yeah, I don't like Glicko. I have quitted services over it (even something like chess problem solving website). It's just not fun imo.
- bitshiftfaced 6y agoIt sounds like it's not that you don't like Glicko but rather you'd prefer more variety in matchmaking. You can accomplish that easily by changing the way matchmaking picks your opponent.
- carbocation 6y agoNot sure that I've seen this addressed, but I view Elo as an approximation to a PageRank-like stationary distribution. Have others thought about this more formally?
- nextos 6y agoMicrosoft Research developed TrueSkill [1], where they frame the problem as a Bayesian inference one. Incidentally, TrueSkill is implemented in Infer.NET, which IMHO doesn't get the attention it deserves. An open-source state-of-the-art factor graph engine with amazing performance. When dealing with many kinds of problems, including those involving discrete variables, I haven't found anything that compares in terms of speed and covergence. [1] https://www.microsoft.com/en-us/research/uploads/prod/2018/03/trueskill2.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0...
- morelandjs 6y agoThere are a lot of similarities between classic Elo and Bayesian statistics. You essentially construct a prior for each comparison and update your prior by its deviation from the observation. Does true skill just do this in a more rigorous fashion according to Bayes' theorem?
- deleted 6y ago[deleted]
- 1980phipsi 6y agoProbably could use something like this for looking at sports teams.
- LanceH 6y agoI've done this with various sports. In particular college football. The ratings were highly accurate compared to polls, except for undefeated teams not from major conferences. The polls involving humans would punish them terribly for not having the right pedigree. I also rated the whole season simultaneously as it's my opinion that a ranking should be based on the performance in a season versus a power ranking of who is best now. By my calculations, the people choosing teams for the playoffs are either stupid or corrupt in the sense of not treating all teams fairly. It's a popularity contest.
- lindig 6y agoHow dependent is the result on the order in which players play? Assuming who beats whom is fixed, how much do results vary over the order of the tournament?
- narag 6y agoAren't the calculations done always with the initial points? I'm very curious to know if that's not the case.
- dfan 6y agoIn chess, at least, all calculations in a single tournament are indeed batched.
- hogFeast 6y agoI believe order is important. The way to verify this is by running a simulation with players whose skills are all equal but vary with some normal distribution, and you will find that skill levels to do move about quite a bit even though the underlying skill level is fixed (I am thinking about order as just random shuffling here, so if we take a dataset where there is just randomness and in converges quickly then order doesn't matter...this doesn't really happen with ELO). This isn't to say that rankings don't converge. They do. But things like order definitely can get in the way. Glicko controls for this by modelling uncertainty in a skill estimate (iirc, proportional to the number of matches seen). In practice, you would tend to burn the first few matches using ELO to try and get rankings to converge to a limit. But, again, even then I have found that issues. ELO will quickly identify the worst and best but there is a lot of volatility around the middle grouping of teams. And this does vary with the underlying activity you are modelling too: activities with a lot of randomness take longer to converge. You can also control convergence with the K-factor. I think this is one of the trade-offs that comes with the time dependent nature of ELO. You get more variance but less bias. Imo, it usually isn't a huge issue in practice because the upside of this is your method isn't static, and your ranking can respond to things like lineup changes (which tend to occur in many applications)...it works well in most applications.
- noxvilleza 6y ago
- chairmanwow1 6y agoOne of my favorite Elo bugs I came across was a bug in a game called age of empires 2: DE. Whenever my friend and I played online together, we would sporadically get absolutely creamed. We started looking up the stats of the others players we were matched with and noticed that on those bad games the average Elo of our opponents would jump to >2k Elo vs our barely 1k. Both of us being software engineers, eventually the conversation turned to “What the hell do you think is going on here? How does a matchmaking bug like this crop up?” Eventually we came to the conclusion that they must be summing out Elo for the Elo of our matchmaking party rather than averaging it. Eventually the issue was patched and our worst suspicions were confirmed that indeed multiplayer party Elos were summed instead of averaged.
- jsight 6y agoHow would that make a difference? Or were the party sizes different between you and your opponents?
- ivarv 6y agoTwo players of similar skill would probably get creamed by a team with a large skill disparity.
- jsight 6y agoI guess, but (making up numbers here) two players with 750 skill would be 1500 or an average of 750. A sum might match them up against two players with 1300 skill & 200 skill due to the 1500 sum. OTOH, the average would match them up against... an average of 750, so the same people? I'm sure I'm just missing something really fundamental about how this all works.
- chairmanwow1 6y agoSo our opponents were queueing as individuals and were highly skilled. So their “party Elo” was genuinely 2k. Parties would be selected to take place in a match (2v2 / 3v3 / 4v4) of similar party Elos. We would either get paired with another couple 2k players against a full team of 2k (4v4 match) or in the worst case a 2v2 against 2k players. In the 2v2 case, because of the Elo difference the opposing side was expected to win >99.99% of the time.
- vmception 6y agoNice use for gaming Crowd-behavior ranking systems should be banned for human relationships, where the person doing the ranking didn't intend to send a message to the crowd
- sabas123 6y agoThere was a quote from day[9] that really struck with me and wonder if anybody has some intelligent opinion on it. "MMR [in context of systems like elo] is __not__ a measurement of skill, but of progress".
- noxvilleza 6y agoDepends on how quickly you calibrate to your 'true rating'. Elo-like systems take relatively quite long, since you can gain at most k points per game (but obviously fewer points per game if you're being matched with someone who's similar in skill to you). Chess GMs for example stream their 'new account to GM' runs, and it takes very many hours. Chess is a game where there's very high correlation with individual skill and the outcome, in more noisy/random games like Dota 2 for example (which also are ~40 minutes on average) your expected relative gain per match is much lower.
- PostThisTooFast 6y agoRating system for what? Titling failure.
- harpiaharpyja 6y ago> 3. Exponent Base (Seen as constant 10) > In the articles I’ve read I have not ever seen this changed nor was it mentioned that a change could be used, but if you’re designing a special system and perhaps looking for something else to tweak this might be an option. It can be easily shown that changing the base is equivalent to scaling the exponent, so if you're already scaling the exponent by a semi-arbitrary 1/400 chosen for aesthetic purposes, there really is nothing that is added by discussing the base of the exponent separately.
- souprock 6y agoUse it for class rank, replacing GPA. Being the best in a given classroom is a win, being at the median is a tie, being at the bottom is a loss. This would help with the problem of people avoiding harder classes due to GPA fear.