5 ms·

There have been four rating systems, including Glicko and TruSkill, received lots and lots of complaints for both those systems. This new system receives few co

There have been four rating systems, including Glicko and TruSkill, received lots and lots of complaints for both those systems. This new system receives few complaints. Tested across 135000 players. If the players had not complained so much, we would still be on Glicko. Those are the facts. The theories as to why that is are up to you.

jsnell 6y ago
Optimizing a rating system for minimal complains, maximum player engagement, or some similar metric is of course totally valid. It reminds me of Sirlin's story of being hired to design a rating system for Starcraft 2, and optimizing for totally different things that Blizzard wanted [0]. If the author (you?) had just described it in those terms, it'd be hard to object. But the article goes further, and makes claims about the system being more accurate due to a different rating curve. That's the claim that would need to be justified by actually comparing whether the predictions the new system makes really are better. [0] http://sirlingames.squarespace.com/blog/2010/7/24/analyzing-starcraft-2s-ranking-system.html http://sirlingames.squarespace.com/blog/2010/7/24/analyzing-...
- an_opabinia 6y ago
  Part of a matchmaking algorithm that increases user engagement is telling a story about how it's more fair though. Like our tolerance for losing is acquired. Most normal people losing in League for the first time stop playing, usually forever. Just randomly visit your friend's match histories in League, frequent players have many days of long losing streaks. If you're just conditioned to play despite losing, great, in a Darwinian way (surviving, being around to be measured) you will be representative of the average player in League. And there are so many League players with such long retention you cannot possibly argue that skill-based matchmaking is the core component of user engagement. His dataset is interesting because it will necessarily overrepresent people who kept playing despite the old system. That sort of refutes its importance - I mean sure people complain but they keep playing, so was it really that important? So what if complaints go down? Those are important goals, and also, it's still an interesting twist in multiplayer game design. You just gotta interpret it as a commentary on a whole system even if it doesn't narrowly talk about a scientific objective like performance prediction.
  - BSTRhino 6y ago
    Player retention was significantly lower before the new rating system was introduced. It wasn't just complaints, it's just that is a more direct metric because multiple changes may have affected the retention.
- Natsu 6y ago
  What you said makes me wonder about a totally different way of using a metric of how likely the person is to win a given match. That is, perhaps the system could be engineered to maintain a more even win/loss ratio so that people don't go on super-long win (or loss) streaks in general by adjusting who they get matched with. It probably wouldn't work that well towards the edges, but around the middle it might work well enough.
  - teen 6y ago
    This has what dota 2 does. Every win / loss is worth the same points, the games are extremely close in terms of player rank, so and it's a team game so you get more variance than just your individual skill. Players eventually settle close to 50 percent win rate
- mcnamaratw 6y ago
  The "different rating curve" appears to be the actual historical probability data, not a formula. I think. If that's right then it this estimate of the probability of winning is not a new discovery. On the other hand I suspect the historical data really is the "best fit" to the historical data.
- lanius 6y ago
  The funny thing about SC2 is that the player's MMR (matchmaking rating) is decoupled from rank, due to design decisions such as demotions not occurring midseason. So a gold league player with a low enough MMR may get matched against bronze league players, despite ostensibly being ranked higher. Actually for the longest time after release, player MMR was not visible. It took 6 years and 2 expansions before it was finally displayed in-game.
  - vkou 6y ago
    Not to mention that the bonus pool feature has long been removed. More importantly, though, your MMR has now been split up across the three factions that you can play as. If your Terran gameplay is much better than your Zerg gameplay, you'll be facing higher-ranked opponents when you play Terran, than when you play Zerg. This has made it easier for players to switch their factions up, without being punished for it.
  - ajuc 6y ago
    I like the league system in sc2, it allows you to see progress and to get to know other people's style (because you meet the same players several times in a quick succession). People build narratives around that (I remember that guy I lost vs him last time and now I had my revange). If somebody won against you with a particularly dirty strategy you can get back at him and you won't be fooled the next time. It's better than dota system where you're most likely never going to play vs the same people again (or at least it feels like that). Also the whole 5v5 game that lasts for 1 hour format is inherently very toxic - most of the time you know you lost after 10 minutes and yet you can't disconnect and have to keep playing for the next 30-50 minutes while everybody blames everybody else on your team. In sc2 if you know you lost after 5 minutes it's discrespectful to continue playing :)
    - de_watcher 6y ago
      Oh, I love those guys who "know you lost after 10 minutes" in Dota...
      - 6y ago
cortesoft 6y ago
You have to decide what the purpose of the rating system is. Using it as a reward system for players to feel accomplishment is a different use case then trying to correctly predict the likely outcome of a game. Personally, if I was designing a rating system, I would use two separate systems. One would be like the one in this game; publicly viewable, pleases the players, and gives a sense of accomplishment. Then, I would have a second, internal only rating system that players can't see but is used for matchmaking to make sure people are matched up to players with as close to equivalent skill as possible.
- nobodyandproud 6y ago
  Some games do this already, and players are very unpleased by the results.
  - afterburner 6y ago
    Probably because they get matched to tougher opponents if they're better?
    - wanderer2323 6y ago
      It's kinda disheartening being matched to plat players in your silver promos.
- dmos62 6y ago
  I find similar solutions experientially two-faced and frustrating. Imagine the matchmaking engine was a person behind a desk that you interact with. If he consistently told you one thing and then did something else you'd be displeased.
- ajuc 6y ago
  That's what xp is in sc2. For several years xp was the only number that was visible - your true mmr was hidden. Peple knew this and ignored xp altogether.

jsnell 6y ago

Optimizing a rating system for minimal complains, maximum player engagement, or some similar metric is of course totally valid. It reminds me of Sirlin's story of being hired to design a rating system for Starcraft 2, and optimizing for totally different things that Blizzard wanted [0]. If the author (you?) had just described it in those terms, it'd be hard to object. But the article goes further, and makes claims about the system being more accurate due to a different rating curve. That's the claim that would need to be justified by actually comparing whether the predictions the new system makes really are better. [0] http://sirlingames.squarespace.com/blog/2010/7/24/analyzing-starcraft-2s-ranking-system.html http://sirlingames.squarespace.com/blog/2010/7/24/analyzing-...

an_opabinia 6y ago
Part of a matchmaking algorithm that increases user engagement is telling a story about how it's more fair though. Like our tolerance for losing is acquired. Most normal people losing in League for the first time stop playing, usually forever. Just randomly visit your friend's match histories in League, frequent players have many days of long losing streaks. If you're just conditioned to play despite losing, great, in a Darwinian way (surviving, being around to be measured) you will be representative of the average player in League. And there are so many League players with such long retention you cannot possibly argue that skill-based matchmaking is the core component of user engagement. His dataset is interesting because it will necessarily overrepresent people who kept playing despite the old system. That sort of refutes its importance - I mean sure people complain but they keep playing, so was it really that important? So what if complaints go down? Those are important goals, and also, it's still an interesting twist in multiplayer game design. You just gotta interpret it as a commentary on a whole system even if it doesn't narrowly talk about a scientific objective like performance prediction.
- BSTRhino 6y ago
  Player retention was significantly lower before the new rating system was introduced. It wasn't just complaints, it's just that is a more direct metric because multiple changes may have affected the retention.
Natsu 6y ago
What you said makes me wonder about a totally different way of using a metric of how likely the person is to win a given match. That is, perhaps the system could be engineered to maintain a more even win/loss ratio so that people don't go on super-long win (or loss) streaks in general by adjusting who they get matched with. It probably wouldn't work that well towards the edges, but around the middle it might work well enough.
- teen 6y ago
  This has what dota 2 does. Every win / loss is worth the same points, the games are extremely close in terms of player rank, so and it's a team game so you get more variance than just your individual skill. Players eventually settle close to 50 percent win rate
mcnamaratw 6y ago
The "different rating curve" appears to be the actual historical probability data, not a formula. I think. If that's right then it this estimate of the probability of winning is not a new discovery. On the other hand I suspect the historical data really is the "best fit" to the historical data.
lanius 6y ago
The funny thing about SC2 is that the player's MMR (matchmaking rating) is decoupled from rank, due to design decisions such as demotions not occurring midseason. So a gold league player with a low enough MMR may get matched against bronze league players, despite ostensibly being ranked higher. Actually for the longest time after release, player MMR was not visible. It took 6 years and 2 expansions before it was finally displayed in-game.
- vkou 6y ago
  Not to mention that the bonus pool feature has long been removed. More importantly, though, your MMR has now been split up across the three factions that you can play as. If your Terran gameplay is much better than your Zerg gameplay, you'll be facing higher-ranked opponents when you play Terran, than when you play Zerg. This has made it easier for players to switch their factions up, without being punished for it.
- ajuc 6y ago
  I like the league system in sc2, it allows you to see progress and to get to know other people's style (because you meet the same players several times in a quick succession). People build narratives around that (I remember that guy I lost vs him last time and now I had my revange). If somebody won against you with a particularly dirty strategy you can get back at him and you won't be fooled the next time. It's better than dota system where you're most likely never going to play vs the same people again (or at least it feels like that). Also the whole 5v5 game that lasts for 1 hour format is inherently very toxic - most of the time you know you lost after 10 minutes and yet you can't disconnect and have to keep playing for the next 30-50 minutes while everybody blames everybody else on your team. In sc2 if you know you lost after 5 minutes it's discrespectful to continue playing :)
  - de_watcher 6y ago
    Oh, I love those guys who "know you lost after 10 minutes" in Dota...
    - 6y ago

cortesoft 6y ago

You have to decide what the purpose of the rating system is. Using it as a reward system for players to feel accomplishment is a different use case then trying to correctly predict the likely outcome of a game. Personally, if I was designing a rating system, I would use two separate systems. One would be like the one in this game; publicly viewable, pleases the players, and gives a sense of accomplishment. Then, I would have a second, internal only rating system that players can't see but is used for matchmaking to make sure people are matched up to players with as close to equivalent skill as possible.

nobodyandproud 6y ago
Some games do this already, and players are very unpleased by the results.
- afterburner 6y ago
  Probably because they get matched to tougher opponents if they're better?
  - wanderer2323 6y ago
    It's kinda disheartening being matched to plat players in your silver promos.
dmos62 6y ago
I find similar solutions experientially two-faced and frustrating. Imagine the matchmaking engine was a person behind a desk that you interact with. If he consistently told you one thing and then did something else you'd be displeased.
ajuc 6y ago
That's what xp is in sc2. For several years xp was the only number that was visible - your true mmr was hidden. Peple knew this and ignored xp altogether.