5 ms·

Creating a matchmaker for your multiplayer game

110 points by gafferongames 2y ago

nightowl_games 2y ago
Glenn Fiedler is the best blogger on game networking. We used your stuff to guide our netcode development ~6 years ago. We now have a great stack on top of Godot for multiplayer games on the web over webrtc or websocket and enet on mobile and native. We're a small time company (3 people) and we have several multiplayer games..our netcode is a serious competitive advantage largely due to unity's failure to provide a system to our competitors. Thanks for helping us, Glenn. Our matchmaker needs a rewrite, and these posts are nice.
- deleted 2y ago
  [deleted]
- gafferongames 2y ago
  > We now have a great stack on top of Godot for multiplayer games on the web over webrtc or websocket and enet on mobile and native. We're a small time company (3 people) and we have several multiplayer games..our netcode is a serious competitive advantage That's awesome! Well done
- lostmsu 2y ago
  What library do you use for WebRTC? The native libwebrtc?
  - Sean-Der 2y ago
    Godot uses libdatachannel. I remember them asking for mbedtls support
    - nightowl_games 2y ago
      Yeah we use libdatachannel. tbh we didnt see much (if any) network improvement using WebRTC over WebSockets, but our testing/analysis is pretty shallow (3 person company - whos got time for that, lets make products)
- intelVISA 2y ago
  > our netcode is a serious competitive advantage What do you sell that isn't already solved by XDP?
  - nightowl_games 2y ago
    Googling XDP tells me it's extreme diesel performance. I'm sure theyre engines are great but we make videogames.
    - remram 2y ago
      https://en.wikipedia.org/wiki/Express_Data_Path https://en.wikipedia.org/wiki/Express_Data_Path is a feature of recent Linux kernels allowing you to bypass/replace most of the normal Linux networking code, allowing for higher performance. Some network cards will even run your network code directly on their hardware. (not sure what GP meant, just providing the reference)
      - zarzavat 2y ago
        I think they were confused by the term “netcode”, which in game developer parlance means “code which implements networked gameplay in a multiplayer game”, rather than “code that implements networking in an OS”.
        nightowl_games 2y ago
        Yeah, our netcode forward predicts from the server state to the local time to allow the local player to see their own character without network artifacts, and see other players in approximately their true location. It's the 'server authoritative, with client side prediction and rollback' technique. Consult Gaffer On Games' blog posts to learn more.
- deleted 2y ago
  [deleted]
roes9627 2y ago
Great stuff!
tithe 2y ago
Is matchmaking the process of players "always [being] sent to the datacenters with the best chance of having low latency"? EDIT: No, it seems that it's literally grouping players together in multiplayer games (but doing so in a way where latency between players is minimized.)
- gafferongames 2y ago
  Yes, but much more too. The example here is really simplified to focus mostly on the finding datacenters with low latency problem, but it could also include things like matching players of similar skill together, finding a set of players that would make balanced teams, making sure that players who are partied up together play in the same match and so on. Basically, just like matchmaking in real life. It's the thing that works out groups of players who should play together in a match.
Aachen 2y ago
My understanding of the post: 1. Let the game client apply to servers in datacenters where they will get low latency if that's relevant for your game 2. Wait 1 second on the server, then shuffle the list of clients and group every set of 4 into a match. 3. If no match is found after 10 seconds, add the client to servers in suboptimal datacenters 4. If still no match after another 10 seconds, put them in all the queues and join them to the first game it finds so they can at least play at all, even if they'll have lag no matter where they go 5. Determine latency either using the same IP:port as the game server will have because the route might differ from another port (citation needed imo. The author links another blog post for further info which is mainly a statistics lecture about averages hiding outliers and doesn't back up this claim that the port number makes a significant difference for latency in any but the most exceptional of situations where you're sent eastward instead of westward around the globe or so) or by collecting statistics to find which server was best for a given location and ISP over the past 30 days (seems over-engineered with extra downsides, like that you will have gaps in the data and that it might have changed) Did I miss anything? The whole thing seems somewhere between "do the most naïve implication" (1–4) and doubtful (5). It's useful for devs to confirm the intuition they might have for the simplest implementation is correct but other comments saying this is a great blog surprise me when looking at these two posts The hardest part about matchmaking seems to me finding players well-matched in skill without making them wait long, especially during off-peak hours, or having off-putting loss streaks because they (50% chance for each game) got matched against a slightly stronger or equal team and lost. The article doesn't even mention these aspects exist, let alone offer advice on tackling them
- gafferongames 2y ago
  It's a shame that you missed the key point of the article in #5. Many games today have suboptimal inputs to their matchmaker, because like you, they refuse to believe that the internet can route a significant percentage of players incorrectly. Source: Ran a network accelerator with more than 50 million unique players for 5 years. Bad network performance is much more pervasive than you think. https://networknext.com https://networknext.com
  - 2y ago
jejejrjjrr 2y ago
[flagged]
bit_flipper 2y ago
I've written game matchmaking before for games and found it far more challenging to do the portions related to matchmaking groups by skill. If you're hyper latency sensitive then perhaps this blog post is really useful? But for the games I worked on we would trade 10ms worse average ping for 10% better skill pairings without question. If you have any advice on improving skill matchmaking I would be quite interested.
- gafferongames 2y ago
  > If you have any advice on improving skill matchmaking I would be quite interested. Samy Duc wrote a really good deep dive into Apex Legends matchmaking that covers skill and team balancing: https://www.ea.com/games/apex-legends/news/matchmaking-2023 https://www.ea.com/games/apex-legends/news/matchmaking-2023
  - bit_flipper 2y ago
    Thanks for the link, I hadn't read this one yet. Very interesting. I feel like as an industry we're still in the stone ages on how we do skill matching systems. A lot of the current and even future system they describe is really not great, and these flaws are definitely not unique to them (the systems I worked on are even more flawed)! Trying to matchmake dozens of players at a time is such a cool, challenging problem.
- bob1029 2y ago
  I really think it's a cursed mission to chase perfectly balanced matches. In terms of overall experience, the best approach in general is a purely random one. This is how you avoid the experience of getting trapped in a 20 game losing streak, even if you are a really good player. If you allow the natural balance of good players winning more often and bad players losing more often, you will find things get a lot less messy in between. This also provides your bad players an opportunity to occasionally witness what they could become. When you only play someone 1% stronger than yourself, you probably don't have a great idea of what the upper bound actually feels like. This can become a serious trap for players who are seeking to grow their skills.
smarkov 2y ago
> the matchmaker shuffles the queue to avoid the same players being matched together repeatedly This is actually something I hate about multiplayer games with matchmaking nowadays. I made the majority of my childhood friends only because we stayed on the same server and played together for hours on end. I don't think it's a stretch to say that a key reason for why we play multiplayer rather than single player games is to socialize. This has become increasingly more difficult when you just get a new set of people every 10-30 minutes.
- hlfshell 2y ago
  This is why smaller AA games still keep this alive - to build community. For instance: Squad, a 80-100 player multiplayer game which primarily depends upon privately hosted servers ran by communities that actually care about the community and game. You'll still run into jerks like any online game, but it's a world of difference to the one-match/single-serving strangers you usually play with in other games.
- tomr75 2y ago
  can't you queue with people
- gryn 2y ago
  Can't you make a group with people you like ? There a feature like that in rocket league. Getting grouped with the same bad team mates repeatedly would just make me quit the game.
  - rcxdude 2y ago
    You can queue with people you already know, and some games do have a 'stay as a team in the next queue' option, but there's no real opportunity to actually get familiar with anyone, let alone a community, and play with them more in the same way that you get with community servers
  - halfmatthalfcat 2y ago
    Group with the same team until the magic runs out in 3 games then everyone quits lol
szundi 2y ago
An option for not shuffling would be nice
- gafferongames 2y ago
  The full source code is here: https://github.com/mas-bandwidth/matchmaker https://github.com/mas-bandwidth/matchmaker You can turn off shuffling if you want.
RedlineTriad 2y ago
Google has Open-Match[1] for matchmaking where you just have to provide an API that takes a batch of players and returns groups, and it handles the surrounding stuff, also integrates with Agones[2] to automatically provision servers on k8s. [1] https://open-match.dev/ https://open-match.dev/ [2] https://agones.dev/ https://agones.dev/
placidpanda 2y ago
I'm confused about the using the greyscale map tiles to estimate ping. You don't want to have the users ping the servers themselves because those pings could be inaccurate or noisy, so you use historical average data for users in that region instead to get a nice simple number. But... how do you know where the user is? IP Geolocation? Can't that be wrong also? Isn't it better to have a direct measurement which could be a little wrong than an average of a guess which could be really wrong?

110 points by gafferongames 2y ago

Glenn Fiedler is the best blogger on game networking. We used your stuff to guide our netcode development ~6 years ago. We now have a great stack on top of Godot for multiplayer games on the web over webrtc or websocket and enet on mobile and native. We're a small time company (3 people) and we have several multiplayer games..our netcode is a serious competitive advantage largely due to unity's failure to provide a system to our competitors. Thanks for helping us, Glenn. Our matchmaker needs a rewrite, and these posts are nice.

deleted 2y ago
[deleted]
gafferongames 2y ago
> We now have a great stack on top of Godot for multiplayer games on the web over webrtc or websocket and enet on mobile and native. We're a small time company (3 people) and we have several multiplayer games..our netcode is a serious competitive advantage That's awesome! Well done
lostmsu 2y ago
What library do you use for WebRTC? The native libwebrtc?
- Sean-Der 2y ago
  Godot uses libdatachannel. I remember them asking for mbedtls support
  - nightowl_games 2y ago
    Yeah we use libdatachannel. tbh we didnt see much (if any) network improvement using WebRTC over WebSockets, but our testing/analysis is pretty shallow (3 person company - whos got time for that, lets make products)
intelVISA 2y ago
> our netcode is a serious competitive advantage What do you sell that isn't already solved by XDP?
- nightowl_games 2y ago
  Googling XDP tells me it's extreme diesel performance. I'm sure theyre engines are great but we make videogames.
  - remram 2y ago
    https://en.wikipedia.org/wiki/Express_Data_Path https://en.wikipedia.org/wiki/Express_Data_Path is a feature of recent Linux kernels allowing you to bypass/replace most of the normal Linux networking code, allowing for higher performance. Some network cards will even run your network code directly on their hardware. (not sure what GP meant, just providing the reference)
    - zarzavat 2y ago
      I think they were confused by the term “netcode”, which in game developer parlance means “code which implements networked gameplay in a multiplayer game”, rather than “code that implements networking in an OS”.
      - nightowl_games 2y ago
        Yeah, our netcode forward predicts from the server state to the local time to allow the local player to see their own character without network artifacts, and see other players in approximately their true location. It's the 'server authoritative, with client side prediction and rollback' technique. Consult Gaffer On Games' blog posts to learn more.
deleted 2y ago
[deleted]

roes9627 2y ago

Great stuff!

tithe 2y ago

Is matchmaking the process of players "always [being] sent to the datacenters with the best chance of having low latency"? EDIT: No, it seems that it's literally grouping players together in multiplayer games (but doing so in a way where latency between players is minimized.)

gafferongames 2y ago
Yes, but much more too. The example here is really simplified to focus mostly on the finding datacenters with low latency problem, but it could also include things like matching players of similar skill together, finding a set of players that would make balanced teams, making sure that players who are partied up together play in the same match and so on. Basically, just like matchmaking in real life. It's the thing that works out groups of players who should play together in a match.

Aachen 2y ago

My understanding of the post: 1. Let the game client apply to servers in datacenters where they will get low latency if that's relevant for your game 2. Wait 1 second on the server, then shuffle the list of clients and group every set of 4 into a match. 3. If no match is found after 10 seconds, add the client to servers in suboptimal datacenters 4. If still no match after another 10 seconds, put them in all the queues and join them to the first game it finds so they can at least play at all, even if they'll have lag no matter where they go 5. Determine latency either using the same IP:port as the game server will have because the route might differ from another port (citation needed imo. The author links another blog post for further info which is mainly a statistics lecture about averages hiding outliers and doesn't back up this claim that the port number makes a significant difference for latency in any but the most exceptional of situations where you're sent eastward instead of westward around the globe or so) or by collecting statistics to find which server was best for a given location and ISP over the past 30 days (seems over-engineered with extra downsides, like that you will have gaps in the data and that it might have changed) Did I miss anything? The whole thing seems somewhere between "do the most naïve implication" (1–4) and doubtful (5). It's useful for devs to confirm the intuition they might have for the simplest implementation is correct but other comments saying this is a great blog surprise me when looking at these two posts The hardest part about matchmaking seems to me finding players well-matched in skill without making them wait long, especially during off-peak hours, or having off-putting loss streaks because they (50% chance for each game) got matched against a slightly stronger or equal team and lost. The article doesn't even mention these aspects exist, let alone offer advice on tackling them

gafferongames 2y ago
It's a shame that you missed the key point of the article in #5. Many games today have suboptimal inputs to their matchmaker, because like you, they refuse to believe that the internet can route a significant percentage of players incorrectly. Source: Ran a network accelerator with more than 50 million unique players for 5 years. Bad network performance is much more pervasive than you think. https://networknext.com https://networknext.com
- 2y ago

jejejrjjrr 2y ago

[flagged]

bit_flipper 2y ago

I've written game matchmaking before for games and found it far more challenging to do the portions related to matchmaking groups by skill. If you're hyper latency sensitive then perhaps this blog post is really useful? But for the games I worked on we would trade 10ms worse average ping for 10% better skill pairings without question. If you have any advice on improving skill matchmaking I would be quite interested.

gafferongames 2y ago
> If you have any advice on improving skill matchmaking I would be quite interested. Samy Duc wrote a really good deep dive into Apex Legends matchmaking that covers skill and team balancing: https://www.ea.com/games/apex-legends/news/matchmaking-2023 https://www.ea.com/games/apex-legends/news/matchmaking-2023
- bit_flipper 2y ago
  Thanks for the link, I hadn't read this one yet. Very interesting. I feel like as an industry we're still in the stone ages on how we do skill matching systems. A lot of the current and even future system they describe is really not great, and these flaws are definitely not unique to them (the systems I worked on are even more flawed)! Trying to matchmake dozens of players at a time is such a cool, challenging problem.
bob1029 2y ago
I really think it's a cursed mission to chase perfectly balanced matches. In terms of overall experience, the best approach in general is a purely random one. This is how you avoid the experience of getting trapped in a 20 game losing streak, even if you are a really good player. If you allow the natural balance of good players winning more often and bad players losing more often, you will find things get a lot less messy in between. This also provides your bad players an opportunity to occasionally witness what they could become. When you only play someone 1% stronger than yourself, you probably don't have a great idea of what the upper bound actually feels like. This can become a serious trap for players who are seeking to grow their skills.

smarkov 2y ago

> the matchmaker shuffles the queue to avoid the same players being matched together repeatedly This is actually something I hate about multiplayer games with matchmaking nowadays. I made the majority of my childhood friends only because we stayed on the same server and played together for hours on end. I don't think it's a stretch to say that a key reason for why we play multiplayer rather than single player games is to socialize. This has become increasingly more difficult when you just get a new set of people every 10-30 minutes.

hlfshell 2y ago
This is why smaller AA games still keep this alive - to build community. For instance: Squad, a 80-100 player multiplayer game which primarily depends upon privately hosted servers ran by communities that actually care about the community and game. You'll still run into jerks like any online game, but it's a world of difference to the one-match/single-serving strangers you usually play with in other games.
tomr75 2y ago
can't you queue with people
gryn 2y ago
Can't you make a group with people you like ? There a feature like that in rocket league. Getting grouped with the same bad team mates repeatedly would just make me quit the game.
- rcxdude 2y ago
  You can queue with people you already know, and some games do have a 'stay as a team in the next queue' option, but there's no real opportunity to actually get familiar with anyone, let alone a community, and play with them more in the same way that you get with community servers
- halfmatthalfcat 2y ago
  Group with the same team until the magic runs out in 3 games then everyone quits lol

szundi 2y ago

An option for not shuffling would be nice

gafferongames 2y ago
The full source code is here: https://github.com/mas-bandwidth/matchmaker https://github.com/mas-bandwidth/matchmaker You can turn off shuffling if you want.

RedlineTriad 2y ago

Google has Open-Match[1] for matchmaking where you just have to provide an API that takes a batch of players and returns groups, and it handles the surrounding stuff, also integrates with Agones[2] to automatically provision servers on k8s. [1] https://open-match.dev/ https://open-match.dev/ [2] https://agones.dev/ https://agones.dev/

placidpanda 2y ago

I'm confused about the using the greyscale map tiles to estimate ping. You don't want to have the users ping the servers themselves because those pings could be inaccurate or noisy, so you use historical average data for users in that region instead to get a nice simple number. But... how do you know where the user is? IP Geolocation? Can't that be wrong also? Isn't it better to have a direct measurement which could be a little wrong than an average of a guess which could be really wrong?