7 ms·

Making Fast-Paced Multiplayer Networked Games Is Hard

109 points by Impossible 12y ago

ggambetta 12y ago
Another series of articles I wrote ~4 years ago about the same topic, but perhaps more detailed and with a live demo: http://www.gabrielgambetta.com/fpm1.html http://www.gabrielgambetta.com/fpm1.html The title of this article sounds vaguely familiar ;)
- cinskiy 12y ago
  Your articles were SUPER helpful when I added multiplayer to my game, you may be the reason I succeeded in it, alongside Valve documentation. Thank you so much for writing them!
  - ggambetta 12y ago
    Happy to hear my articles helped! But the reason you succeeded is you, not me :)
fredley 12y ago
Another reason [TagPro](http://tagpro.koalabeast.com http://tagpro.koalabeast.com) is an excellent game. It's implementation of this stuff is great - with a reasonable connection my ping is usually under 10ms, and the game is often won and lost with the smallest of margins making this kind of thing very important. Not to mention the excellent gameplay and mechanics - it's very simple to understand and learn, but very very difficult to play well!
- Luc 12y ago
  Not saying this is the case, but 10 ms is a very low ping that could hide a lot of issues with the implementation.
tinco 12y ago
The hardest part is that most game engines are not designed to be networked in this way. Find an open source physics engine that natively supports teleportation, easing and prediction, they're not there. UE4 is the first engine I've used that seems to have a very nice multiplayer API, and it's only been out for indie developers for a couple of months. So the hard part is not devising the networked scheme, it's building a whole game engine (or thoroughly modding one) afterwards, at least in my experience. I was working on a multiplayer racing game project (like GTA2), and my approach was to run 2 physics engines in parallel. One physics engine would always be authoritive, and be in sync with the server, but because of the lag would always be a frame or two (or more) behind. The other would be working on predicting what is going to happen. Every frame snapping back to the authority, and then applying the (predicted and user) inputs over that. The actual position the user would see would be an averaged position between the current predicted position, and its previous predicted position (to prevent too much jitter/snapback). If you've got a better scheme to do networked physics I'm all ears :)
- Cakez0r 12y ago
  Have had this problem too with making a racing game. We ended up making remote entities move independently of the physics engine (I.E. cars for remote players were just static objects that had their positions manually updated). This worked to some extent, but we had to do some fudging to make collisions work somewhat realistically.
  - Igglyboo 12y ago
    Are you saying that the client determined their own position? How did you account for people cheating?
    - ownagefool 12y ago
      Sounds like you never played Red Faction. I love that game but watching people fly around the map, clip through walls and shoot 1000 rockets a second (much to the dismay of my graphics card) wasn't the most pleasant experience.
no_future 12y ago
Fascinating, I've been interested in what kinds of serverside software real time online games(WoW, Call of Duty, etc) run for a while now, but haven't been able to find much info on it. Seems that they would need to be optimized for many very high stress concurrent connections with as little latency as possible, so I'd guess that they run C/C++ and/or Java? Do they use something like Websockets, or do UDP/TCP or some other persistent two way connection method? There don't seem to be any publicly available libraries focused on this kind of thing, so I assume that they develop their networking stuff mostly in-house. Anyone that knows about this stuff willing to share?
- pjc50 12y ago
  Generally it's UDP. Nobody would use websockets unless they had to run in a browser or similar VM. C++ would be the default choice, but Eve Online uses Stackless Python which seems to have worked well for them. The Quake3 server source (pure C) is available for your inspection, and there's a good analysis of how it works here: http://fabiensanglard.net/quake3/network.php http://fabiensanglard.net/quake3/network.php
- Cakez0r 12y ago
  WoW uses TCP the last time I checked (the overhead of TCP is acceptable for an RPG in most cases). I believe Diablo 3 also uses TCP and uses protobuf. EVE online have some very good tech posts and are generally quite open about their technology (although it's mostly about hardware and infrastructure). Valve also have some good articles on networking for Source games (someone posted the link elsewhere in this thread).
- dividuum 12y ago
  > There don't seem to be any publicly available libraries focused on this kind of thing, so I assume that they develop their networking stuff mostly in-house. Have a look at RakNet (http://www.jenkinssoftware.com/ http://www.jenkinssoftware.com/) or enet (http://enet.bespin.org/ http://enet.bespin.org/).
  - JabavuAdams 12y ago
teddyh 12y ago
See also: Distributed Virtual Reality – An Overview, Bernie Roehl, 1995: https://ece.uwaterloo.ca/~broehl/distrib.html https://ece.uwaterloo.ca/~broehl/distrib.html Characteristics of UDP Packet Loss: Effect of TCP Traffic, Hidenari Sawashima, Yoshiaki Hori, Hideki Sunahara, Yuji Oie, 1997: http://www.isoc.org/inet97/proceedings/F3/F3_1.HTM http://www.isoc.org/inet97/proceedings/F3/F3_1.HTM I Shot You First: Networking the Gameplay of HALO: REACH David Aldridge, 2011: http://www.gdcvault.com/play/1014345/I-Shot-You-First-Networking http://www.gdcvault.com/play/1014345/I-Shot-You-First-Networ...
- rlefebvre 12y ago
  This last link is a gem. They refer to the TRIBES Networking Model in the talk, explained here also: http://gamedevs.org/uploads/tribes-networking-model.pdf http://gamedevs.org/uploads/tribes-networking-model.pdf There is a beautiful implementation of this in the old OpenTNL that seems to be still alive in the newer Torque engine: https://github.com/GarageGames/Torque2D/tree/master/engine/source/network https://github.com/GarageGames/Torque2D/tree/master/engine/s... The original article does not discuss the aspect of scaling fast-paced games to handle large numbers of replicated objects and players under constrained network conditions but I found the approach proposed in the original TRIBES model to be the most (only?) credible so far. I don't see support for this in any of the popular, modern network libraries (RakNet, enet, lidgren, ...). They all seem to have taken the 'multiple reliable channels' direction but that just doesn't seem to scale to many connected players the way the TRIBES model does. I would love to hear from anyone who has had experience with that!
benihana 12y ago
I always found this article about how the Source Engine handles networking fascinating: https://developer.valvesoftware.com/wiki/Source_Multiplayer_Networking https://developer.valvesoftware.com/wiki/Source_Multiplayer_... Really shows how hard just a few of the challenges of making a multiplayer game are.
- squeaky-clean 12y ago
  I also love the Source Engine networking. Another one I like is The Torque Game Engine. It has a very similar networking model, I think it may even be a little better (it's the engine that powered the original Tribes game). I remember buying a license to the original TGE years ago and digging through the networking code, it was great. They've open-sourced the latest version of Torque under the MIT license, and I believe the networking code is nearly identical to the original code used for Tribes.
  - JabavuAdams 12y ago
    Yeah, there was a really interesting paper on the Tribes networking model in the '97 or '00 GDC conference proceedings.
toqueteos 12y ago
I've been using Bolt [1] for Unity3D for a fast-paced game and surprised me how far I got with almost no knowledge of the library. It's UDP all the way and super explicit about everything. [1]: www.boltengine.com
hellbanner 12y ago
In specific contexts like a 1v1 fighting game, you can get clever. GGPO*. Every move in the game has a specified "startup" time which is generally a) too fast to react to & b) consistent. When you jump & throw a punch you send the frame along with your attack. My client speeds up your character's game state to match what your client experiences. http://www.gamasutra.com/view/news/177508/The_lagfighting_techniques_behind_GGPOs_netcode.php http://www.gamasutra.com/view/news/177508/The_lagfighting_te...
Dove 12y ago
Just a few months ago, I did an overhaul of the network code for DXX-Retro[1] -- a source port of Descent. Descent worked much better over the (laggy, lossy, bursty) net than DOOM, and -- if you're looking to mimic old school games -- is really worth studying. Some quick technical commentary: The bandwidth calculation in the article is predicated on sending updates at 60 hz -- or what we'd in the Descent community call 60 PPS. Probably because the screen is refreshing at that rate? It's unnecessary. You want a high framerate for control of your own game, but you don't need it to see enemies smoothly. Remember, movies only run at 24 FPS. ;) The highest I allow in Descent is 30 PPS, and really . . . it's seen as a luxury. 20 is generally fine. Sometimes I play games at 10, and there you can definitely tell -- even with the smoothing (it sends velocity and acceleration, too, and interpolates) -- but it's perfectly playable. Which is something worth remembering. With old school games, "crappy but perfectly playable" is actually all they were able to achieve. No, the physics engines aren't perfectly locked, and how tolerable this is will depend on your game. In a simple FPS, this really isn't a big deal. You just lag lead (compensate in aim, both for the motion of your enemy, and the fact that the data is old). :) Some of how he's proposing to send the data is wasteful. He initially proposes sending "is weapon equipped" and "is firing" as 1 byte a piece -- and later concludes he can get those in 1 bit a piece. True. That's a savings by a factor of 8 right there. But I can do you one better -- don't send it every update. How often do those states really change? In Descent, we have two classes of data: position and orientation data that's sent at a steady rate (10-30 PPS), and event data that's sent . . . whenever it happens. Equipping weapons is definitely the second type; it happens extremely rarely -- like, seconds pass between weapon switches. :) One thing that may surprise you. We don't send "is firing" as a flag with every packet -- we send one packet per shot taken! Two reasons for this: one, it's actually lower bandwidth. Shots fire rarely -- our fastest gun fires 20 bullets per second, but the next fastest fires six. Then five, then four, then . . . one shot every two seconds. And you're not always firing, either. Sending one packet per shot saves bandwidth. But it also increases accuracy! We attach those shot packets to a position and orientation update, so -- even if the receiver has incorrectly interpolated your position -- the shot goes exactly where you intended it to. This is very important. :) As a player (and a programmer -- but mostly a player), I'm concerned about the article's conclusions about the value of prediction, and the recommendation to avoid position popping by applying smoothing -- to quote the article, "it ruins the virtual reality of the game". Ok, yes, it does. But the thing is, you have a choice here. You can present your players with something pretty and smooth that is fundamentally a lie, or with something jittery that is the best knowledge you have about where the enemy is. This is a fundamental tradeoff: versimiltude or accuracy. You can't have both. My players overwhelmingly prefer accuracy. To them, the avatars on the screen are targeting aids, and they understand that the data is old and a bit lossy and bursty sometimes, and they want the best data possible so they can take the best shot possible. :) I suppose your mileage may vary by audience. Mine's been playing this game 20 years and "crappy but playable" is both normal and good to them. :) But -- I can't imagine this would be different in another FPS -- taking a shot that you know is good on your screen, and have it not hit the other guy . . . that's rage-inducing right there! Yeah, networked games are hard. For sure. And there are fundamental hard tradeoffs involved in engineering them. For sure. But it's an interesting problem, and also worth it. :) [1] https://github.com/CDarrow/DXX-Retro https://github.com/CDarrow/DXX-Retro
Animats 12y ago
This problem was first described in Farmer and Morningstar's "The Lessons of Lucasfilms' Habitat" (1990), which was a MMORPG running on Commodore 64 machines over 300 baud dial-up modems. They referred to this problem as "surreal time". Back then, they had latencies of 100ms to 5000ms. The original article here says they can get latencies of 200ms from 99% of XBox connections. Not much progress in latency in 25 years. I can understand the LAN party nostalgia.
moomin 12y ago
As apparently, is being on the right side of the GamerGate fiasco. (They're down at the moment.)

109 points by Impossible 12y ago

Another series of articles I wrote ~4 years ago about the same topic, but perhaps more detailed and with a live demo: http://www.gabrielgambetta.com/fpm1.html http://www.gabrielgambetta.com/fpm1.html The title of this article sounds vaguely familiar ;)

cinskiy 12y ago
Your articles were SUPER helpful when I added multiplayer to my game, you may be the reason I succeeded in it, alongside Valve documentation. Thank you so much for writing them!
- ggambetta 12y ago
  Happy to hear my articles helped! But the reason you succeeded is you, not me :)

fredley 12y ago

Another reason [TagPro](http://tagpro.koalabeast.com http://tagpro.koalabeast.com) is an excellent game. It's implementation of this stuff is great - with a reasonable connection my ping is usually under 10ms, and the game is often won and lost with the smallest of margins making this kind of thing very important. Not to mention the excellent gameplay and mechanics - it's very simple to understand and learn, but very very difficult to play well!

Luc 12y ago
Not saying this is the case, but 10 ms is a very low ping that could hide a lot of issues with the implementation.

tinco 12y ago

The hardest part is that most game engines are not designed to be networked in this way. Find an open source physics engine that natively supports teleportation, easing and prediction, they're not there. UE4 is the first engine I've used that seems to have a very nice multiplayer API, and it's only been out for indie developers for a couple of months. So the hard part is not devising the networked scheme, it's building a whole game engine (or thoroughly modding one) afterwards, at least in my experience. I was working on a multiplayer racing game project (like GTA2), and my approach was to run 2 physics engines in parallel. One physics engine would always be authoritive, and be in sync with the server, but because of the lag would always be a frame or two (or more) behind. The other would be working on predicting what is going to happen. Every frame snapping back to the authority, and then applying the (predicted and user) inputs over that. The actual position the user would see would be an averaged position between the current predicted position, and its previous predicted position (to prevent too much jitter/snapback). If you've got a better scheme to do networked physics I'm all ears :)

Cakez0r 12y ago
Have had this problem too with making a racing game. We ended up making remote entities move independently of the physics engine (I.E. cars for remote players were just static objects that had their positions manually updated). This worked to some extent, but we had to do some fudging to make collisions work somewhat realistically.
- Igglyboo 12y ago
  Are you saying that the client determined their own position? How did you account for people cheating?
  - ownagefool 12y ago
    Sounds like you never played Red Faction. I love that game but watching people fly around the map, clip through walls and shoot 1000 rockets a second (much to the dismay of my graphics card) wasn't the most pleasant experience.

no_future 12y ago

Fascinating, I've been interested in what kinds of serverside software real time online games(WoW, Call of Duty, etc) run for a while now, but haven't been able to find much info on it. Seems that they would need to be optimized for many very high stress concurrent connections with as little latency as possible, so I'd guess that they run C/C++ and/or Java? Do they use something like Websockets, or do UDP/TCP or some other persistent two way connection method? There don't seem to be any publicly available libraries focused on this kind of thing, so I assume that they develop their networking stuff mostly in-house. Anyone that knows about this stuff willing to share?

pjc50 12y ago
Generally it's UDP. Nobody would use websockets unless they had to run in a browser or similar VM. C++ would be the default choice, but Eve Online uses Stackless Python which seems to have worked well for them. The Quake3 server source (pure C) is available for your inspection, and there's a good analysis of how it works here: http://fabiensanglard.net/quake3/network.php http://fabiensanglard.net/quake3/network.php
Cakez0r 12y ago
WoW uses TCP the last time I checked (the overhead of TCP is acceptable for an RPG in most cases). I believe Diablo 3 also uses TCP and uses protobuf. EVE online have some very good tech posts and are generally quite open about their technology (although it's mostly about hardware and infrastructure). Valve also have some good articles on networking for Source games (someone posted the link elsewhere in this thread).
dividuum 12y ago
> There don't seem to be any publicly available libraries focused on this kind of thing, so I assume that they develop their networking stuff mostly in-house. Have a look at RakNet (http://www.jenkinssoftware.com/ http://www.jenkinssoftware.com/) or enet (http://enet.bespin.org/ http://enet.bespin.org/).
- JabavuAdams 12y ago

teddyh 12y ago

See also: Distributed Virtual Reality – An Overview, Bernie Roehl, 1995: https://ece.uwaterloo.ca/~broehl/distrib.html https://ece.uwaterloo.ca/~broehl/distrib.html Characteristics of UDP Packet Loss: Effect of TCP Traffic, Hidenari Sawashima, Yoshiaki Hori, Hideki Sunahara, Yuji Oie, 1997: http://www.isoc.org/inet97/proceedings/F3/F3_1.HTM http://www.isoc.org/inet97/proceedings/F3/F3_1.HTM I Shot You First: Networking the Gameplay of HALO: REACH David Aldridge, 2011: http://www.gdcvault.com/play/1014345/I-Shot-You-First-Networking http://www.gdcvault.com/play/1014345/I-Shot-You-First-Networ...

rlefebvre 12y ago
This last link is a gem. They refer to the TRIBES Networking Model in the talk, explained here also: http://gamedevs.org/uploads/tribes-networking-model.pdf http://gamedevs.org/uploads/tribes-networking-model.pdf There is a beautiful implementation of this in the old OpenTNL that seems to be still alive in the newer Torque engine: https://github.com/GarageGames/Torque2D/tree/master/engine/source/network https://github.com/GarageGames/Torque2D/tree/master/engine/s... The original article does not discuss the aspect of scaling fast-paced games to handle large numbers of replicated objects and players under constrained network conditions but I found the approach proposed in the original TRIBES model to be the most (only?) credible so far. I don't see support for this in any of the popular, modern network libraries (RakNet, enet, lidgren, ...). They all seem to have taken the 'multiple reliable channels' direction but that just doesn't seem to scale to many connected players the way the TRIBES model does. I would love to hear from anyone who has had experience with that!

benihana 12y ago

I always found this article about how the Source Engine handles networking fascinating: https://developer.valvesoftware.com/wiki/Source_Multiplayer_Networking https://developer.valvesoftware.com/wiki/Source_Multiplayer_... Really shows how hard just a few of the challenges of making a multiplayer game are.

squeaky-clean 12y ago
I also love the Source Engine networking. Another one I like is The Torque Game Engine. It has a very similar networking model, I think it may even be a little better (it's the engine that powered the original Tribes game). I remember buying a license to the original TGE years ago and digging through the networking code, it was great. They've open-sourced the latest version of Torque under the MIT license, and I believe the networking code is nearly identical to the original code used for Tribes.
- JabavuAdams 12y ago
  Yeah, there was a really interesting paper on the Tribes networking model in the '97 or '00 GDC conference proceedings.

toqueteos 12y ago

I've been using Bolt [1] for Unity3D for a fast-paced game and surprised me how far I got with almost no knowledge of the library. It's UDP all the way and super explicit about everything. [1]: www.boltengine.com

hellbanner 12y ago

In specific contexts like a 1v1 fighting game, you can get clever. GGPO*. Every move in the game has a specified "startup" time which is generally a) too fast to react to & b) consistent. When you jump & throw a punch you send the frame along with your attack. My client speeds up your character's game state to match what your client experiences. http://www.gamasutra.com/view/news/177508/The_lagfighting_techniques_behind_GGPOs_netcode.php http://www.gamasutra.com/view/news/177508/The_lagfighting_te...

Dove 12y ago

Just a few months ago, I did an overhaul of the network code for DXX-Retro[1] -- a source port of Descent. Descent worked much better over the (laggy, lossy, bursty) net than DOOM, and -- if you're looking to mimic old school games -- is really worth studying. Some quick technical commentary: The bandwidth calculation in the article is predicated on sending updates at 60 hz -- or what we'd in the Descent community call 60 PPS. Probably because the screen is refreshing at that rate? It's unnecessary. You want a high framerate for control of your own game, but you don't need it to see enemies smoothly. Remember, movies only run at 24 FPS. ;) The highest I allow in Descent is 30 PPS, and really . . . it's seen as a luxury. 20 is generally fine. Sometimes I play games at 10, and there you can definitely tell -- even with the smoothing (it sends velocity and acceleration, too, and interpolates) -- but it's perfectly playable. Which is something worth remembering. With old school games, "crappy but perfectly playable" is actually all they were able to achieve. No, the physics engines aren't perfectly locked, and how tolerable this is will depend on your game. In a simple FPS, this really isn't a big deal. You just lag lead (compensate in aim, both for the motion of your enemy, and the fact that the data is old). :) Some of how he's proposing to send the data is wasteful. He initially proposes sending "is weapon equipped" and "is firing" as 1 byte a piece -- and later concludes he can get those in 1 bit a piece. True. That's a savings by a factor of 8 right there. But I can do you one better -- don't send it every update. How often do those states really change? In Descent, we have two classes of data: position and orientation data that's sent at a steady rate (10-30 PPS), and event data that's sent . . . whenever it happens. Equipping weapons is definitely the second type; it happens extremely rarely -- like, seconds pass between weapon switches. :) One thing that may surprise you. We don't send "is firing" as a flag with every packet -- we send one packet per shot taken! Two reasons for this: one, it's actually lower bandwidth. Shots fire rarely -- our fastest gun fires 20 bullets per second, but the next fastest fires six. Then five, then four, then . . . one shot every two seconds. And you're not always firing, either. Sending one packet per shot saves bandwidth. But it also increases accuracy! We attach those shot packets to a position and orientation update, so -- even if the receiver has incorrectly interpolated your position -- the shot goes exactly where you intended it to. This is very important. :) As a player (and a programmer -- but mostly a player), I'm concerned about the article's conclusions about the value of prediction, and the recommendation to avoid position popping by applying smoothing -- to quote the article, "it ruins the virtual reality of the game". Ok, yes, it does. But the thing is, you have a choice here. You can present your players with something pretty and smooth that is fundamentally a lie, or with something jittery that is the best knowledge you have about where the enemy is. This is a fundamental tradeoff: versimiltude or accuracy. You can't have both. My players overwhelmingly prefer accuracy. To them, the avatars on the screen are targeting aids, and they understand that the data is old and a bit lossy and bursty sometimes, and they want the best data possible so they can take the best shot possible. :) I suppose your mileage may vary by audience. Mine's been playing this game 20 years and "crappy but playable" is both normal and good to them. :) But -- I can't imagine this would be different in another FPS -- taking a shot that you know is good on your screen, and have it not hit the other guy . . . that's rage-inducing right there! Yeah, networked games are hard. For sure. And there are fundamental hard tradeoffs involved in engineering them. For sure. But it's an interesting problem, and also worth it. :) [1] https://github.com/CDarrow/DXX-Retro https://github.com/CDarrow/DXX-Retro

Animats 12y ago

This problem was first described in Farmer and Morningstar's "The Lessons of Lucasfilms' Habitat" (1990), which was a MMORPG running on Commodore 64 machines over 300 baud dial-up modems. They referred to this problem as "surreal time". Back then, they had latencies of 100ms to 5000ms. The original article here says they can get latencies of 200ms from 99% of XBox connections. Not much progress in latency in 25 years. I can understand the LAN party nostalgia.

moomin 12y ago

As apparently, is being on the right side of the GamerGate fiasco. (They're down at the moment.)