7 ms·

One of my favorite Elo bugs I came across was a bug in a game called age of empires 2: DE. Whenever my friend and I played online together, we would sporadical

One of my favorite Elo bugs I came across was a bug in a game called age of empires 2: DE. Whenever my friend and I played online together, we would sporadically get absolutely creamed. We started looking up the stats of the others players we were matched with and noticed that on those bad games the average Elo of our opponents would jump to >2k Elo vs our barely 1k. Both of us being software engineers, eventually the conversation turned to “What the hell do you think is going on here? How does a matchmaking bug like this crop up?” Eventually we came to the conclusion that they must be summing out Elo for the Elo of our matchmaking party rather than averaging it. Eventually the issue was patched and our worst suspicions were confirmed that indeed multiplayer party Elos were summed instead of averaged.

jsight 6y ago
How would that make a difference? Or were the party sizes different between you and your opponents?
- ivarv 6y ago
  Two players of similar skill would probably get creamed by a team with a large skill disparity.
  - jsight 6y ago
    I guess, but (making up numbers here) two players with 750 skill would be 1500 or an average of 750. A sum might match them up against two players with 1300 skill & 200 skill due to the 1500 sum. OTOH, the average would match them up against... an average of 750, so the same people? I'm sure I'm just missing something really fundamental about how this all works.
- chairmanwow1 6y ago
  So our opponents were queueing as individuals and were highly skilled. So their “party Elo” was genuinely 2k. Parties would be selected to take place in a match (2v2 / 3v3 / 4v4) of similar party Elos. We would either get paired with another couple 2k players against a full team of 2k (4v4 match) or in the worst case a 2v2 against 2k players. In the 2v2 case, because of the Elo difference the opposing side was expected to win >99.99% of the time.
  - bitshiftfaced 6y ago
    Since there are an equal number of players on both teams, why would it matter if the system used the average or the sum of the players' Elo? Wouldn't both methods have the same outcome?
- nwallin 6y ago
  Players A and B are friends in real life and play a game together. Player A has a rating of 900 and Player B has a rating of 1100. They queue up as a team for a 2v2 match. Players C and D and randos on the internet and solo queue for whatever. Player C has a rating of 1900 and Player D has a rating of 2100. The matchmaking system tries to find a match for Players A and B to play against. Their Team Rating is 900+1100 = 2000, so it looks for another team rated around 2000 to play against. The matchmaking system finds a team rated 1000, and that's no good, and a team rated 3000, and that's no good either. It can't find a team rated 2000, so it goes to the fallback plan. It just tosses the team into the solo queue and lets the solo queue matchmaking system handle it. The solo queue matchmaking system has a player with a rating of 1900, a player with a rating of 2100, and a ~~team~~ player with a rating of 2000. Perfect! It matches the ~~team~~ player with a 2000 rating against the 1900 and the 2100, which averages out to 2000.
bitshiftfaced 6y ago
> Eventually we came to the conclusion that they must be summing out Elo for the Elo of our matchmaking party rather than summing it. I think I'm missing something. Could you rephrase this?
- SamBam 6y ago
  *rather than averaging it, I assume.
  - chairmanwow1 6y ago
    Yes. Thank you. *Averaging. Thought I caught all the mistakes.
TameAntelope 6y ago
The sentiment you're expressing is the idea that one 2k player is so much better than two 1k rated players that it doesn't make sense to simply sum up each team's rating and keep it even, which is the false assumption that the AoE 2 devs made. Elo is not linear. However I don't think they'd have fixed it just by averaging the Elo ratings, they must have implemented something more complex, like a proximity based algorithm, otherwise if the teams were the same size you'd hit a similar problem with an average as with a sum. A 2k player + a 0 rated player would sum to 2k and average to 1k. You and your friend would also sum to 2k and average to 1k, again assuming even team sizes. That said, still very funny! Also, fun fact: Elo isn't an acronym/initialism, it's someone's name, Arpad Elo (the inventor). Also also, I don't think Elo is really a good idea for multiplayer (more than 1v1 that is) games generally. Don't know enough to say that authoritatively, but using it for something like AoE 2 is likely prone to all kinds of strange unexpected outcomes.
- lordnacho 6y ago
  Seems to me that you need to use the max of the team rather than the average. The better player can tell the less good player to do, up to a point.
  - maybeOneDay 6y ago
    This is what rocket league uses. While it can be unpleasant as a high ranked player to jump in a lobby with your much less skilled friend, overall the matchmaking system and equality of matches is in my opinion utterly fantastic
  - jerf 6y ago
    I'd expect "geometric average" to be one of the closest simple statistics for most games, assuming that you have some ranking that can be treated linearly. It would be generally unclear that rankings can be treated linearly. A subtle aspect of Elo is that as the disparity between two players increases, the Elo metric essentially becomes less and less certain that there is any relationship between the skill of the players at all, which is probably a bit deeper way of understanding "why" . Eventually you'll reach a point where if you round to integers, the winner winning will result in an increase of 0 points. (In fact one thing to watch out for is how you round; it's easy to end up with a wandering Elo basis if you truncate both sides rather than rounding properly, for instance.) It's fairly trivial to show that if you take a pool of the worst players and the best players, Elo will eventually converge to those two pools having about that amount of distance between them, but by inserting a fresh pool of middling players into the pool, the distance between the best & the worst will increase despite their skill not changing. If skill is linearly distributed... assuming you can even define that... Elo ratings might be somewhat reasonably treated linearly, but skill is very unlikely to be linearly distributed. Basically, I like Elo when used properly and I have used it to good effect a few times myself, but it is a bit dangerous in that it offers a number... but that number lacks many properties we associate with integers. (For instance, in another posting I point out you can shift the entire Elo pool by any constant you like without changing the system. Integers do not have this property.) They have a definite meaning, but it isn't what your intuition might suggest. You really shouldn't do any arithmetic on them. They only make sense in the context of the Elo computation itself.
jayd16 6y ago
Average just causes other issues. Teams should probably have their own rating not derived from the individual elo. But honestly, elo is a shit system for fun matchmaking anyhow.
- buzzerbetrayed 6y ago
  Why is it shit and what system is better?
- elcomet 6y ago
  Why do you say it's shit ?
  - jayd16 6y ago
    Once you reach the top there's much less reason to keep playing. It doesn't have a great way to handle multiplayer situations. It doesn't handle confidence as well as some other systems ie it takes a while to skill test a player. https://en.wikipedia.org/wiki/Elo_rating_system#Practical_issues https://en.wikipedia.org/wiki/Elo_rating_system#Practical_is... Trueskill and many others are often used instead. https://en.wikipedia.org/wiki/TrueSkill https://en.wikipedia.org/wiki/TrueSkill The biggest issue is fun is not the same fair and its a lot more complicated to tune for fun. You could, for example, play 100 matches where you were given the closest opponent but at a disadvantage for every single match. There's a lot of other scenarios around how you want to fold in ping based match making and how that smaller pool effects the fairness of the matchhing.
    - maybeOneDay 6y ago
      I accept your premise and the fact that matchmaking is hard but: > You could, for example, play 100 matches where you were given the closest opponent but at a disadvantage for every single match. not really, you'd derank aggressively and end up playing people at a lower level and presumably winning comfortably after some time.
      - deleted 6y ago
hwjwill 6y ago
I also noticed it when I play AoE: 2 DE as I only play multiplayer game. I initially thought it's just because there aren't enough players so they match us with someone else much higher. However I only notice it in Quick Play where ELO is not affected instead of Ranked games.
postalrat 6y ago
They shouldn't be summing or averaging. They should be using the max elo of the players in your party for every member of your party.

jsight 6y ago

How would that make a difference? Or were the party sizes different between you and your opponents?

ivarv 6y ago
Two players of similar skill would probably get creamed by a team with a large skill disparity.
- jsight 6y ago
  I guess, but (making up numbers here) two players with 750 skill would be 1500 or an average of 750. A sum might match them up against two players with 1300 skill & 200 skill due to the 1500 sum. OTOH, the average would match them up against... an average of 750, so the same people? I'm sure I'm just missing something really fundamental about how this all works.
chairmanwow1 6y ago
So our opponents were queueing as individuals and were highly skilled. So their “party Elo” was genuinely 2k. Parties would be selected to take place in a match (2v2 / 3v3 / 4v4) of similar party Elos. We would either get paired with another couple 2k players against a full team of 2k (4v4 match) or in the worst case a 2v2 against 2k players. In the 2v2 case, because of the Elo difference the opposing side was expected to win >99.99% of the time.
- bitshiftfaced 6y ago
  Since there are an equal number of players on both teams, why would it matter if the system used the average or the sum of the players' Elo? Wouldn't both methods have the same outcome?
nwallin 6y ago
Players A and B are friends in real life and play a game together. Player A has a rating of 900 and Player B has a rating of 1100. They queue up as a team for a 2v2 match. Players C and D and randos on the internet and solo queue for whatever. Player C has a rating of 1900 and Player D has a rating of 2100. The matchmaking system tries to find a match for Players A and B to play against. Their Team Rating is 900+1100 = 2000, so it looks for another team rated around 2000 to play against. The matchmaking system finds a team rated 1000, and that's no good, and a team rated 3000, and that's no good either. It can't find a team rated 2000, so it goes to the fallback plan. It just tosses the team into the solo queue and lets the solo queue matchmaking system handle it. The solo queue matchmaking system has a player with a rating of 1900, a player with a rating of 2100, and a ~~team~~ player with a rating of 2000. Perfect! It matches the ~~team~~ player with a 2000 rating against the 1900 and the 2100, which averages out to 2000.

bitshiftfaced 6y ago

> Eventually we came to the conclusion that they must be summing out Elo for the Elo of our matchmaking party rather than summing it. I think I'm missing something. Could you rephrase this?

SamBam 6y ago
*rather than averaging it, I assume.
- chairmanwow1 6y ago
  Yes. Thank you. *Averaging. Thought I caught all the mistakes.

TameAntelope 6y ago

The sentiment you're expressing is the idea that one 2k player is so much better than two 1k rated players that it doesn't make sense to simply sum up each team's rating and keep it even, which is the false assumption that the AoE 2 devs made. Elo is not linear. However I don't think they'd have fixed it just by averaging the Elo ratings, they must have implemented something more complex, like a proximity based algorithm, otherwise if the teams were the same size you'd hit a similar problem with an average as with a sum. A 2k player + a 0 rated player would sum to 2k and average to 1k. You and your friend would also sum to 2k and average to 1k, again assuming even team sizes. That said, still very funny! Also, fun fact: Elo isn't an acronym/initialism, it's someone's name, Arpad Elo (the inventor). Also also, I don't think Elo is really a good idea for multiplayer (more than 1v1 that is) games generally. Don't know enough to say that authoritatively, but using it for something like AoE 2 is likely prone to all kinds of strange unexpected outcomes.

lordnacho 6y ago
Seems to me that you need to use the max of the team rather than the average. The better player can tell the less good player to do, up to a point.
- maybeOneDay 6y ago
  This is what rocket league uses. While it can be unpleasant as a high ranked player to jump in a lobby with your much less skilled friend, overall the matchmaking system and equality of matches is in my opinion utterly fantastic
- jerf 6y ago
  I'd expect "geometric average" to be one of the closest simple statistics for most games, assuming that you have some ranking that can be treated linearly. It would be generally unclear that rankings can be treated linearly. A subtle aspect of Elo is that as the disparity between two players increases, the Elo metric essentially becomes less and less certain that there is any relationship between the skill of the players at all, which is probably a bit deeper way of understanding "why" . Eventually you'll reach a point where if you round to integers, the winner winning will result in an increase of 0 points. (In fact one thing to watch out for is how you round; it's easy to end up with a wandering Elo basis if you truncate both sides rather than rounding properly, for instance.) It's fairly trivial to show that if you take a pool of the worst players and the best players, Elo will eventually converge to those two pools having about that amount of distance between them, but by inserting a fresh pool of middling players into the pool, the distance between the best & the worst will increase despite their skill not changing. If skill is linearly distributed... assuming you can even define that... Elo ratings might be somewhat reasonably treated linearly, but skill is very unlikely to be linearly distributed. Basically, I like Elo when used properly and I have used it to good effect a few times myself, but it is a bit dangerous in that it offers a number... but that number lacks many properties we associate with integers. (For instance, in another posting I point out you can shift the entire Elo pool by any constant you like without changing the system. Integers do not have this property.) They have a definite meaning, but it isn't what your intuition might suggest. You really shouldn't do any arithmetic on them. They only make sense in the context of the Elo computation itself.

jayd16 6y ago

Average just causes other issues. Teams should probably have their own rating not derived from the individual elo. But honestly, elo is a shit system for fun matchmaking anyhow.

buzzerbetrayed 6y ago
Why is it shit and what system is better?
elcomet 6y ago
Why do you say it's shit ?
- jayd16 6y ago
  Once you reach the top there's much less reason to keep playing. It doesn't have a great way to handle multiplayer situations. It doesn't handle confidence as well as some other systems ie it takes a while to skill test a player. https://en.wikipedia.org/wiki/Elo_rating_system#Practical_issues https://en.wikipedia.org/wiki/Elo_rating_system#Practical_is... Trueskill and many others are often used instead. https://en.wikipedia.org/wiki/TrueSkill https://en.wikipedia.org/wiki/TrueSkill The biggest issue is fun is not the same fair and its a lot more complicated to tune for fun. You could, for example, play 100 matches where you were given the closest opponent but at a disadvantage for every single match. There's a lot of other scenarios around how you want to fold in ping based match making and how that smaller pool effects the fairness of the matchhing.
  - maybeOneDay 6y ago
    I accept your premise and the fact that matchmaking is hard but: > You could, for example, play 100 matches where you were given the closest opponent but at a disadvantage for every single match. not really, you'd derank aggressively and end up playing people at a lower level and presumably winning comfortably after some time.
    - deleted 6y ago

hwjwill 6y ago

I also noticed it when I play AoE: 2 DE as I only play multiplayer game. I initially thought it's just because there aren't enough players so they match us with someone else much higher. However I only notice it in Quick Play where ELO is not affected instead of Ranked games.

postalrat 6y ago

They shouldn't be summing or averaging. They should be using the max elo of the players in your party for every member of your party.