3 ms·

My understanding of the post: 1. Let the game client apply to servers in datacenters where they will get low latency if that's relevant for your game 2. Wait

by Aachen 2y ago

My understanding of the post: 1. Let the game client apply to servers in datacenters where they will get low latency if that's relevant for your game 2. Wait 1 second on the server, then shuffle the list of clients and group every set of 4 into a match. 3. If no match is found after 10 seconds, add the client to servers in suboptimal datacenters 4. If still no match after another 10 seconds, put them in all the queues and join them to the first game it finds so they can at least play at all, even if they'll have lag no matter where they go 5. Determine latency either using the same IP:port as the game server will have because the route might differ from another port (citation needed imo. The author links another blog post for further info which is mainly a statistics lecture about averages hiding outliers and doesn't back up this claim that the port number makes a significant difference for latency in any but the most exceptional of situations where you're sent eastward instead of westward around the globe or so) or by collecting statistics to find which server was best for a given location and ISP over the past 30 days (seems over-engineered with extra downsides, like that you will have gaps in the data and that it might have changed) Did I miss anything? The whole thing seems somewhere between "do the most naïve implication" (1–4) and doubtful (5). It's useful for devs to confirm the intuition they might have for the simplest implementation is correct but other comments saying this is a great blog surprise me when looking at these two posts The hardest part about matchmaking seems to me finding players well-matched in skill without making them wait long, especially during off-peak hours, or having off-putting loss streaks because they (50% chance for each game) got matched against a slightly stronger or equal team and lost. The article doesn't even mention these aspects exist, let alone offer advice on tackling them

gafferongames 2y ago
It's a shame that you missed the key point of the article in #5. Many games today have suboptimal inputs to their matchmaker, because like you, they refuse to believe that the internet can route a significant percentage of players incorrectly. Source: Ran a network accelerator with more than 50 million unique players for 5 years. Bad network performance is much more pervasive than you think. https://networknext.com https://networknext.com
- remram 2y ago
  If it can route incorrectly, that seems like a strong signal not to rely on your maps. It doesn't matter if you know what the correct latency is, if the users only experience the incorrect one.
  - gafferongames 2y ago
    The solution is to pair the matchmaker using maps with a network acceleration product that fixes the bad routes, then the latency map and the actual latency experienced are very close. The reason you use the maps in the matchmaker is so you can look up the latency from client to datacenter in constant time and you don't have to waste any time on the client pinging ping servers to get the results. Plus, it costs money to run pings through the network accelerator, and they always (from experience) converge to the latency map anyway.
    - remram 2y ago
      How much time is wasted? 50 milliseconds at most? And you're about to send 50 packets per second for an hour during the game. Is sending one packet per datacenter really a cost worth mentioning?
      - gafferongames 2y ago
        [flagged]
        remram 2y ago
        [flagged]
        gafferongames 2y ago
        [flagged]
        2y ago
remram 2y ago
Players are coming in and out all the time, you can't "wait 1 second and shuffle the list". 1 second after what?
- gafferongames 2y ago
  The algorithm just has a tick rate at 1 second. Assume requests come in continuously, but only do work at some rate, eg. once every 1 second for a matchmaker that is meant to very quickly find games, and maybe once every 10-15 seconds for a higher quality matchmaker that is going to do a lot of work to match players together by skill etc (longer tick times means you'll have more players to consider in each iteration)
toast0 2y ago
> 5. Determine latency either using the same IP:port as the game server will have because the route might differ from another port (citation needed imo. No citation, but here's some things to think about. You can easily have two servers in the same facility, in the same rack, with very different ip addresses. Most of the time you'll get the same general route for all destinations within a /24 (v4) or /48 (v6), but not always. Two /24s at the same facility might get different routes, especially if there are capacity issues on the better route at any point. From same IP to same IP, anywhere there's a redundant link, which link to use is generally selected with a hash on the 5-tuple {protocol, destination ip, sender ip, destination port, sender port} (sometimes on a smaller tuple). If traffic is unbalanced or if one link is lossy, you can get drastically different experiences with a change in port number. If you've done traceroutes and seen multiple similar ips at a given hop number, it's probably the second effect; this is pretty common. Typical traceroute sends udp packets to the destination and modulates the sending port number. Sometimes, rarely, you can see drastically different routes within a traceroute. It's easier to do traffic engineering on a /24 level though. Traffic on redundant links is balanced by hashing flows rather than on a per packet basis so that packets within a flow tend to arrive in order; out of order arrival is very expensive for endpoints, so it's better to ocassionaly have unbalanced usage than to regularly have out of order arrival.

by Aachen 2y ago

gafferongames 2y ago

It's a shame that you missed the key point of the article in #5. Many games today have suboptimal inputs to their matchmaker, because like you, they refuse to believe that the internet can route a significant percentage of players incorrectly. Source: Ran a network accelerator with more than 50 million unique players for 5 years. Bad network performance is much more pervasive than you think. https://networknext.com https://networknext.com

remram 2y ago
If it can route incorrectly, that seems like a strong signal not to rely on your maps. It doesn't matter if you know what the correct latency is, if the users only experience the incorrect one.
- gafferongames 2y ago
  The solution is to pair the matchmaker using maps with a network acceleration product that fixes the bad routes, then the latency map and the actual latency experienced are very close. The reason you use the maps in the matchmaker is so you can look up the latency from client to datacenter in constant time and you don't have to waste any time on the client pinging ping servers to get the results. Plus, it costs money to run pings through the network accelerator, and they always (from experience) converge to the latency map anyway.
  - remram 2y ago
    How much time is wasted? 50 milliseconds at most? And you're about to send 50 packets per second for an hour during the game. Is sending one packet per datacenter really a cost worth mentioning?
    - gafferongames 2y ago
      [flagged]
      - remram 2y ago
        [flagged]
        gafferongames 2y ago
        [flagged]
        2y ago

remram 2y ago

Players are coming in and out all the time, you can't "wait 1 second and shuffle the list". 1 second after what?

gafferongames 2y ago
The algorithm just has a tick rate at 1 second. Assume requests come in continuously, but only do work at some rate, eg. once every 1 second for a matchmaker that is meant to very quickly find games, and maybe once every 10-15 seconds for a higher quality matchmaker that is going to do a lot of work to match players together by skill etc (longer tick times means you'll have more players to consider in each iteration)

toast0 2y ago

> 5. Determine latency either using the same IP:port as the game server will have because the route might differ from another port (citation needed imo. No citation, but here's some things to think about. You can easily have two servers in the same facility, in the same rack, with very different ip addresses. Most of the time you'll get the same general route for all destinations within a /24 (v4) or /48 (v6), but not always. Two /24s at the same facility might get different routes, especially if there are capacity issues on the better route at any point. From same IP to same IP, anywhere there's a redundant link, which link to use is generally selected with a hash on the 5-tuple {protocol, destination ip, sender ip, destination port, sender port} (sometimes on a smaller tuple). If traffic is unbalanced or if one link is lossy, you can get drastically different experiences with a change in port number. If you've done traceroutes and seen multiple similar ips at a given hop number, it's probably the second effect; this is pretty common. Typical traceroute sends udp packets to the destination and modulates the sending port number. Sometimes, rarely, you can see drastically different routes within a traceroute. It's easier to do traffic engineering on a /24 level though. Traffic on redundant links is balanced by hashing flows rather than on a per packet basis so that packets within a flow tend to arrive in order; out of order arrival is very expensive for endpoints, so it's better to ocassionaly have unbalanced usage than to regularly have out of order arrival.