10 ms·

Fast-Paced Multiplayer (Part I): Client-Server Game Architecture

247 points by mkl95 5y ago

adamrezich 5y ago
wow, that interactive example at the end is fantastic and does a great job of visually explaining the concepts of the articles!
Sohcahtoa82 5y ago
I used this guide when implementing my own real-time multiplayer platforming game. It really understates the importance of authority. It blows my mind how many games, especially FPSes, give a lot of authority to the client. Lag compensation for shooting is one thing, but clients should not be able to shoot through walls or speed hack. These types of cheats are easy to prevent server-side, but developers don't because .... ? No idea. I've always guessed that it's to reduce server-side CPU load, but that's all I can guess.
- Thaxll 5y ago
  > should not be able to shoot through walls or speed hack. These types of cheats are easy to prevent server-side, but developers don't because .... "easy" so you think veterant c++ dev that invented those concepts are dumb or lazy and it's a solved problem? It's not easy and not solved because those are very complicated issues.
  - Sohcahtoa82 5y ago
    Baloney. Why is it hard to add server-side detection for line-of-sight when a player shoots at another player? The client should not be telling the server "I shot Player 2 and did X damage". Instead, the client should be saying "I shot in X direction at Y time" and the server should be reconstructing the world at Y time (As described in the Lag Compensation page of the article) to determine a hit. The server should be performing line-of-sight checks. In a game like PUBG, there's no reason a player should be popping headshots against players a mile away when there's a hill and/or several buildings between them. But the game allows clients to decide when they score a hit. At least, it did in the past.
    - Thaxll 5y ago
      You should read this: https://technology.riotgames.com/news/demolishing-wallhacks-valorants-fog-war https://technology.riotgames.com/news/demolishing-wallhacks-... In anycase it does not work like it should in Valorant.
      - Sohcahtoa82 5y ago
        This is a different issue than what I was talking about. Client-side hacks that control player input or give information to the player that they shouldn't have (ie, being able to see through walls) is hard to deal with. I'll give you that. But what is EASY to deal with is the client doing things they shouldn't be able to, like shoot through bullet-proof walls. And I'm not talking about cases of running behind cover but dying anyways because lag compensation meant that you saw yourself behind the wall, but the other player didn't because what they see is 100 ms behind what you see. I'm talking about cases where you were never exposed from cover to begin with, and the client is allowed to decide if a hit is done or not. In another comment, I linked several examples of players in PUBG scoring hits that were straight-up not possible due to lack of line-of-sight. Here's a good example: https://youtu.be/2Gn2Sixr5Wk https://youtu.be/2Gn2Sixr5Wk The type of hack being used in this video is 100% preventable via server-side checks.
        Gunax 5y ago
        Yes, that should be preventable. The reason we end up trusating the client to a degree is that we want a client-side hit to be registered as a hit. Even if you were to simulate the client on the server, i am not sure this is preventable in general. Sure, with a huge hill that cannot move, its impossible. But what if instead of a hill, the player is shooting through a truck that moves? Well, as the server you know you sent the vehicles location to the client at v1, v2, v3... And you know the player was at p1 when he shot. But you dont know at what time the truck crossed between the player and his victim because that depends on when the client received it. You dont even know IF he received it because you sent it by UDP. So maybe the client dropped the v2 packet. And then remember that the client can rewind history. Basically he can report he did something 100ms ago and the server has no way to distinguish that from a cheat or from lag. From a practical sense, i think devs are faced with a tricky situation: hacks are always possible. Even if you have perfect simulation, it doesn't prevent aim bots because those are theoretically possible. On top of that, you have to trust the client to some degree because of lag compensation. And running physics simulations server side might not be practical. So I think it's easy as a dev to sort of give up and try to solve it some other way. Maybe you just hope that anyone doing something this egregious will be banned manually. The second reason is that you dont want to instantly ban players because it lets the cheaters know exactly the mechanics that caught them. This is why there is always a delay before the real ban happens. Of ourse the side effect is that obliviously cheating players have to be tolerated for a bit before the ban hammer comes down.
  - jbluepolarbear 5y ago
    It is easy to prevent wall hacks and speed hacks. Prediction input resolution, server keeping an authoritative representation on the server, and limiting the synced objects to the current view of the player are all easy when done individually. Thinking about all these concepts at the same time can be confusing for people new to the topic, but each subject is relatively easy to understand and implement. I’d say that this article handles the speed hack issue because of authoritative server state, but doesn’t go far into limiting what is sent to the player or not. The issue arises when you didn’t make your game multiplayer from the start and adding prediction and sync visibility isn’t an option, that’s where you get player authoritative states that share their states with the other players. Client authoritative is easy to implement, impossible to prevent cheating.
    - Hikikomori 5y ago
      > limiting the synced objects to the current view of the player are all easy when done individually. Its so easy to do, but pretty much no game in the industry does it? I guess they must all be incompetent.
      - jbluepolarbear 5y ago
        Wall hacks aren’t the issues you think it is, you’re more thinking corner peeking which is a whole different monster. Wall hacks most competitive games solve, but because of extrapolation and latency corner peeking is inevitable. There are mitigations, but each solution has its pros and cons.
        Hikikomori 5y ago
        No, I'm not confused about this, I'm talking about regular old wallhacks/ESP that only requires the client to know about other players position and draw an overlay over the game. To my knowledge Valorant and csgo are the only ones that have done it. And it took what, almost 20 years for csgo to do it? In their cases it works because they have simple maps and a low amount of entities like players, so its not too much work for the server to do culling so they can hide positions of other players until its needed. But when in close proximity they can be seen through walls, like around a corner you can shoot through (at least csgo has penetration) and to avoid pop-in. And the other players still produce sound so information about other player positions are still sent to all clients and could be used for ESP. So no, this is not a solved problem, and is only partly so in very few games.
  - taneq 5y ago
    > "easy" so you think veterant c++ dev that invented those concepts are dumb or lazy and it's a solved problem? Game devs on big commercial games are often not seasoned veterans, in my experience. The industry largely runs on the fanatical, underpaid efforts of kids who will do anything to work on games.
- judge2020 5y ago
  Maybe if you just have a flat world where you move in any direction with a max speed, but this sort of anti-cheat becomes incredibly difficult once you add features like terrain, physics, vehicles, things like gliders (in BR games), speed boosts, etc. It's possible, but takes a lot more effort. For example, in Fortnite, if you toggle a lagswitch (as in, block Fortnite UDP for some time), then reconnecting to the server will return you to the same position you were at when you disconnected, assuming you were just trying to run on the ground. However, if you toggle a lagswitch in a vehicle, the servers don't want to do expensive, complicated physics server-sided, so your client becomes authoritative to your position and the sanity checks are simply making sure you're not teleporting around large chunks of the map in seconds. If you were to hack your client and give your vehicle moon gravity, the server generally wouldn't care about that.
  - jbluepolarbear 5y ago
    I’m pretty sure the car uses dead reckoning. So you position in the car is static (the last place you where), but the car is moving at speed and will continue even if you disconnect.
    - judge2020 5y ago
      Sure, but I’m talking about the control you have over the car when in the car. During the start of this battle pass season there was immense server issues, which I guessed was the tick rate dropping to super low values due to some bad code. During a few games where the servers degraded in this way, moving would be extremely slow on the ground as it kept pushing you back (since the server isn’t processing your ‘move forward’ commands as often) but getting in a car gave back free movement since I assume it’s a direct broadcast that doesn’t hit the game loop and doesn’t have any position-correcting sanity checks running.
      - jbluepolarbear 5y ago
        I’m saying a car is easier to predict because of dead reckoning. With less frequent updates your actions still feel fluid because the cars momentum can’t change a whole lot between updates. Say you’re driving a car in a straight line, there’s very good odds that in .25 seconds you’ll still be going straight so your predicted line will be very close to the real line. With players you can’t guarantee that, they could zig zag, turn around, jump, etc much more likely to change and require prediction reconciliation.
- mathgladiator 5y ago
  I believe part of the problem is that it is hard enough to build a game in the first place, and networking is a deep skill within itself. I've been doing large scale distributed systems for a decade, and I still feel like I'm learning especially as I retool for games. The way that I'm trying to make this simple is by inventing a new platform for "serverless game hosting" using my programming language (Adama : http://www.adama-lang.org/ http://www.adama-lang.org/ ). My focus on board game is the ultimate in server's role on authority since board games are more transactional between players and staggered (I play this card, that card enables another play to react, that reaction enables another play to augment, etc). I'm looking into what it could mean to escape the board-game pit, and one thing that I'm thinking about is latency. The question that I'm looking at is whether or not I can have a programming model be differentiated such that the client prediction code can be generated. A trivial way to exploit my language is to bring edge computing into the mix such that the edge compute runs a copy of the entire game, and then ships deltas quickly which will be overwritten from the authoritative server. This will work on updates to existing data (i.e. moving position), but it will fail at creating new items unless I move unique id generate to clients (which, is less than happy for me).
  - jdc 5y ago
    I'm no expert, but isn't turn-based multiplayer a completely different animal?
    - MrLeap 5y ago
      Hook a clock up to your turns, end your turns 64~ times a second and we're real time now baby.
      - TeMPOraL 5y ago
        Yeah, we are, but pretty inefficiently. All the complexity in real-time multiplayer comes from trying to pretend everyone is in lockstep, while not actually synchronizing them 60+ times per second. This makes it an open-ended problem - many of the tricks work under very specific assumptions (e.g. low latency, trustworthy clients), and fail when those assumptions no longer hold (e.g. people notice stutter, cheaters cheat). The job is to find the right set of tricks for your use case. And it starts with knowing the tricks in the first place (or inventing new ones), as it's pretty much a specialization of its own.
- skipants 5y ago
  Hell, I can forgive it in games -- it seems pretty easy to forget a detail that can accidentally give too much authority to a user in that space, especially when you're trying to crunch your frame time. What really surprises me is how how often I see this mistake in web development. "No, Bob, you should not increase the user's bank account based on that number passed in from the frontend React app."
  - tshaddox 5y ago
    I'm not sure exactly what you mean. If it's a web API request from a web client that says "transfer $1,000 between account A and account B," then what choice do you have but to "trust" that number? Obviously you have to check whether the request is authenticated and is authorized to transfer between those accounts, and check if account A has $1,000, but what about the client origin of the request do you need to check?
    - Tainnor 5y ago
      I mean since we're talking about banking: - Generate a transaction number and associate that number with the respective transaction details - Send this number to the customer's mobile phone with all those details, or other configured device - the transfer is only authorised if the customer has entered the transaction number That way, the customer is very likely to have verified the details of the transfer.
      - wibagusto 5y ago
        Do you really mean that?
        Tainnor 5y ago
        Yes of course. This is the standard way online banking has been operating in, I believe, much of Europe since practically forever (although it used to be physical lists before smartphones were popular).
    - 5y ago
- mysterydip 5y ago
  Latency and performance, if you have to wait for the server every time you move or fire. It's a tradeoff which things are client and which are server.
- gameswithgo 5y ago
  there are fun types of gameplay that can’t happen without client authority. you can make an aimbot either way so it isn’t a huge deal. like you i had a hard time accepting that pubg using client aide hit detection maee any sense, but after playing it a lot and thinking hard about it I changed my mind. That game is all about pixel perfect aiming. Lag compensation wouldn’t cut it.
  - Sohcahtoa82 5y ago
    https://youtu.be/2Gn2Sixr5Wk https://youtu.be/2Gn2Sixr5Wk IMO, This is clear-cut evidence that PUBG uses (or perhaps, in the past used) client-side hit checks. Notice there's not even bullet travel time. Just "pop pop" and a target that isn't in sight and never was inside instantly dies.
    - blibble 5y ago
      pretty much every game since Half-Life's patch in 2000 uses client side hit detection for hitscan weapons a few do client side prediction for projectiles too (Overwatch notably)
- flohofwoe 5y ago
  One important reason is that P2P games that only require central services for matchmaking but not for the actual gameplay sessions are a lot easier and cheaper to scale to high player counts.
- trissylegs 5y ago
  It was a big thing in World of Warcraft in particular because the internet was so unreliable when Wow release that they wanted the player to be able to do things when all your packets dropped for a few seconds (well maybe a whole minute) But they didn't really compensate for exploiting this to do things that were actually impossible. (ie speed hacks)
  - Arrath 5y ago
    Back when WoW launched I was still on dialup, and played far too much for my own good. Part of playing on dialup was getting used to the latency, like 'queuing' my follow-up spell casts when my current one was anywhere from a fraction of a second to 0.5s from completion, so the command would arrive at the server just as my current one finished. As a healer this was important to avoid dead time where I could be keeping the tank alive. It wasn't until into the 2nd expansion that I got broadband, and hoo boy was it hard to adjust to. Suddenly I didn't have to anticipate and plan ahead nearly as much, and I had a whole new learning curve to go through to get my timing on point again. It seemed to me at the time that the game was structured pretty decently to handle the huge differences in player system specs and connection capabilities.
- frozenlettuce 5y ago
  these checks would result in a more expensive cloud billing. it's cheaper to keep the server just as a message relay between clients
- Rd6n6 5y ago
  I suspect that a lot of that is for the sake of client side prediction: if an enemy runs out from behind a wall, you want as little delay as possible between it happening and the player seeing it in order to react. Cheaters love it, but it does help with the illusion that everyone is playing in the same game world in real-time with no delays
- thejosh 5y ago
  https://www.youtube.com/watch?v=W3aieHjyNvw https://www.youtube.com/watch?v=W3aieHjyNvw Have a watch of this.
- magicalhippo 5y ago
  From my recollection, the primary issue is, or at least was, consoles. Sony and Microsoft required the games to allow players to have ridiculously high pings (several hundred ms). If you play a first-person shooter with 300-400 ms ping and the game uses server side hit detection, you're not going to have much fun. By moving to client-side detection, players hit what they "should hit" from their perspective. At least that's how I recall it from when COD and BF moved to client-side hit detection.
- Gunax 5y ago
  I believe it's because there are 2 conflicting goals: 1. Authoritative server 2. Client accuracy By 'client accuracy' I mean that was happens on the client actually occured. Ie. If you shoot a player, he actually got shot. The conundrum is that these goals conflict. They are like Heisenberg uncertainty: you can have position or momentum, but not both. You can have server-side authority, or you can have client side accuracy, or somewhere in the middle. Just as an example, most shooters have the goal that 'if you shot him on your screen, it was a hit', even if according to server side knowledge the recipient of the shot had already moved. Otherwise when you're shooting you would have to guess where the player is going to be when you're bullet packets reach the server. So instead, we let the client decide what a hit was. Hopefully the server has some sanity checking. I shoot at time t, server receives the info at t+s, reporting that I shot at some trajectory at time t. But the client can erroneously edit history. What if the client sends t-0.1s instead of t? There is no check the server can do to prevent this... I appear like a client which has 150ms of latency even though I really only have 50.
hesdeadjim 5y ago
Highly recommend Glen Fiedler's articles for this topic. https://gafferongames.com/ https://gafferongames.com/
- jbluepolarbear 5y ago
  I’d also recommend this write up from JMP Waveran (really anything by JMP) https://mrelusive.com/publications/papers/The-DOOM-III-Network-Architecture.pdf https://mrelusive.com/publications/papers/The-DOOM-III-Netwo...
- lux 5y ago
  His writing has been such a great learning resource. Can't recommend highly enough!
tshaddox 5y ago
I highly recommend this hour-long video from a Call of Duty developer discussing not only methods they use to transmit game states between client and server, but also some interesting ways that they measure true perceived latency (like using high-speed cameras and LCD monitors to compare true input-to-display latencies between different builds). https://www.youtube.com/watch?v=EtLHLfNpu84 https://www.youtube.com/watch?v=EtLHLfNpu84
- magicalhippo 5y ago
  Battle(non)sense did some videos on Battlefield 4 regarding netcode and latency back in the day, including one regarding input latency[1]. He wired up a LED to the mouse button, so it lit up when the button was pressed. By filming both the monitor and the LED with a high-speed camera he could measure the true input lag. BF4 had quite horrible netcode on launch, but after a valiant effort by Viceral it got quite good, at least on PC. Sadly the game had lost momentum at that point, almost two years after launch, so not many players left to enjoy it. https://www.youtube.com/watch?v=4GnKsqDAmgY https://www.youtube.com/watch?v=4GnKsqDAmgY
ggambetta 5y ago
Hey, I wrote that thing! Always a pleasant surprise and a weird feeling to open Hacker News (for the 14308560234th time today) and read my name like that :)
- adamrezich 5y ago
  since you're here, it looks like the second paragraph under the second heading in the second article is an unfinished thought :) great work otherwise though!
  - ggambetta 5y ago
    Hmmm, you're right, great catch! Not sure what's going on there, I'll investigate.
- dested 5y ago
  I can't tell you how many times I find myself back on this page while working on multiplayer games. From the bottom of my heart I really can't thank you enough for this tutorial.
  - ggambetta 5y ago
    You're very welcome :)
- Rd6n6 5y ago
  This was one of my main resources for learning game networking. If you ever feel tempted to write more articles on the subject, or write a book, people will eagerly read it. Thanks for writing and hosting your articles Edit: I see you wrote a cg book! https://news.ycombinator.com/item?id=26013002 https://news.ycombinator.com/item?id=26013002
  - mrspeaker 5y ago
    Holy cow - I didn't know about this book. Everything Gabriel puts out is awesome: this is an instant-buy for me!
Raphaellll 5y ago
Another problem of predicting the movement of other players is that if they stop before you can see them, the trajectory would be interpolated further such that you can see them around the corner for a split second, knowing they are waiting there.
- ggambetta 5y ago
  The technique I describe in the article strictly interpolates between known positions; it never extrapolates further, so other players would never be seen or rendered beyond the point where they actually stopped.
  - Raphaellll 5y ago
    Yes, I got this. Just wanted to point out that a particular consequence of extrapolating would be that players can see enemys around corners. You already mention that the game would be unplayable if players get teleported every few ms, and above just describes why, aside from aesthetics, makes it unplayable.
nightpool 5y ago
> Here’s the crucial step. Since the server gets all the input with timestamps, it can authoritatively reconstruct the world at any instant in the past. In particular, it can reconstruct the world exactly as it looked like to any client at any point in time. Isn't there a really obvious exploit here? The client could lie about the timestamps and use that to take advantage of future information in making its decisions. For example, in a simplified fighting game where you have two attacks (top and bottom) and two blocks, it could always choose the perfect block to apply before the opponent attacks—simply by seeing the incoming attack packet, and then immediately sending a block packet with an "older" timestamp. This doesn't entirely break the concept of an authoritative server, but it seems to diminish it significantly. I know lots of fighting games use this "rollback-type" netcode, maybe they've developed workarounds for this?
- reitzensteinm 5y ago
  This absolutely is a concern with rollback networking (I have done a bunch of games in it). There are two ways you can cheat. Because you have the full simulation state and input, there is no possible precaution against map hacks. Some FPS games just won't send exact player positions until the client is about to be able to legitimately observe them. Not possible here. The other way is, as you say, you can build fake inputs. To do the timing attack as you describe, all of your input would have to be pretending to be slow. You can't send a t=10 I'm idle packet, then a t=8 I blocked. You'd need a fantastic connection to the server, because the delay you're exploiting is there to protect you from your lag. Your oponent would also need a fantastic connection, because you aren't seeing their current inputs, you're seeing their past inputs. My feeling is that it would be very challenging to blend human like behavior with superhuman reflexes when doing automated responses. But there's nothing you're missing. The attack is definitely possible.
- _--__--__ 5y ago
  All rollback fighting games are pure p2p, there is no server for gameplay, just matchmaking and leaderboards. I'm not sure if what you would describe would work, it would at minimum double the ping between the two players, and in games that set variable rollback windows based on ping that delay may grow exponentially until a disconnect/desync. Also I think no one has attempted to cheat like this because it's much easier to cheat with a program that 'reacts' to graphical cues (such auto-block bots are pretty common in the most recent Tekken and Street Fighter entries). There are some very rare moves that are impossible for a program to react to like this (usually '0-frame grabs') which hit on the exact frame the input is entered, I'm not sure what if anything the current cheaters do about those.
Rd6n6 5y ago
Another complication is stateful processes like abilities: exactly how to sync, predict, lag compensate, and prevent cheating etc for a multistage ability. Lots of little edge cases to deal with, but I also read several interviews by indie devs who claimed that this wasn’t the hard part: the hardest was all the other tasks like matchmaking, hosting/joining games, invites, sdk integration, chat, voice, etc. It was just a mountain of small tasks
- foota 5y ago
  Imo the reason here is that separately this aspect of networking is harder, but in total the others will be a death by a thousand cuts.
beebeepka 5y ago
Can't give enough credit to arQon for his work on Quake 3 CPMA and whoever wrote the unlagged code for OpenArena and its peers. Truly some of the best examples out there. That was well before the modern shooters kids play these days. I'd say things have not improved dramatically over the last 10 years or so. Sure CS servers now run at 100 or more FPS but other than that, not a whole lot of change. In fact, popular examples such as PUBG suck immensely as far as networking is concerned
johnisgood 5y ago
This is a pretty good write-up, very useful. I am currently working on a game so I bookmarked it. Already read a lot, it really is great, thank you for it! If you got anything new, I am all up for it! I have not checked yet but did you mention anti-cheating systems?
johnisgood 5y ago
Are there any articles similar to this, by the way? This is very useful for someone working on a game server.

247 points by mkl95 5y ago

wow, that interactive example at the end is fantastic and does a great job of visually explaining the concepts of the articles!

Sohcahtoa82 5y ago

I used this guide when implementing my own real-time multiplayer platforming game. It really understates the importance of authority. It blows my mind how many games, especially FPSes, give a lot of authority to the client. Lag compensation for shooting is one thing, but clients should not be able to shoot through walls or speed hack. These types of cheats are easy to prevent server-side, but developers don't because .... ? No idea. I've always guessed that it's to reduce server-side CPU load, but that's all I can guess.

Thaxll 5y ago
> should not be able to shoot through walls or speed hack. These types of cheats are easy to prevent server-side, but developers don't because .... "easy" so you think veterant c++ dev that invented those concepts are dumb or lazy and it's a solved problem? It's not easy and not solved because those are very complicated issues.
- Sohcahtoa82 5y ago
  Baloney. Why is it hard to add server-side detection for line-of-sight when a player shoots at another player? The client should not be telling the server "I shot Player 2 and did X damage". Instead, the client should be saying "I shot in X direction at Y time" and the server should be reconstructing the world at Y time (As described in the Lag Compensation page of the article) to determine a hit. The server should be performing line-of-sight checks. In a game like PUBG, there's no reason a player should be popping headshots against players a mile away when there's a hill and/or several buildings between them. But the game allows clients to decide when they score a hit. At least, it did in the past.
  - Thaxll 5y ago
    You should read this: https://technology.riotgames.com/news/demolishing-wallhacks-valorants-fog-war https://technology.riotgames.com/news/demolishing-wallhacks-... In anycase it does not work like it should in Valorant.
    - Sohcahtoa82 5y ago
      This is a different issue than what I was talking about. Client-side hacks that control player input or give information to the player that they shouldn't have (ie, being able to see through walls) is hard to deal with. I'll give you that. But what is EASY to deal with is the client doing things they shouldn't be able to, like shoot through bullet-proof walls. And I'm not talking about cases of running behind cover but dying anyways because lag compensation meant that you saw yourself behind the wall, but the other player didn't because what they see is 100 ms behind what you see. I'm talking about cases where you were never exposed from cover to begin with, and the client is allowed to decide if a hit is done or not. In another comment, I linked several examples of players in PUBG scoring hits that were straight-up not possible due to lack of line-of-sight. Here's a good example: https://youtu.be/2Gn2Sixr5Wk https://youtu.be/2Gn2Sixr5Wk The type of hack being used in this video is 100% preventable via server-side checks.
      - Gunax 5y ago
        Yes, that should be preventable. The reason we end up trusating the client to a degree is that we want a client-side hit to be registered as a hit. Even if you were to simulate the client on the server, i am not sure this is preventable in general. Sure, with a huge hill that cannot move, its impossible. But what if instead of a hill, the player is shooting through a truck that moves? Well, as the server you know you sent the vehicles location to the client at v1, v2, v3... And you know the player was at p1 when he shot. But you dont know at what time the truck crossed between the player and his victim because that depends on when the client received it. You dont even know IF he received it because you sent it by UDP. So maybe the client dropped the v2 packet. And then remember that the client can rewind history. Basically he can report he did something 100ms ago and the server has no way to distinguish that from a cheat or from lag. From a practical sense, i think devs are faced with a tricky situation: hacks are always possible. Even if you have perfect simulation, it doesn't prevent aim bots because those are theoretically possible. On top of that, you have to trust the client to some degree because of lag compensation. And running physics simulations server side might not be practical. So I think it's easy as a dev to sort of give up and try to solve it some other way. Maybe you just hope that anyone doing something this egregious will be banned manually. The second reason is that you dont want to instantly ban players because it lets the cheaters know exactly the mechanics that caught them. This is why there is always a delay before the real ban happens. Of ourse the side effect is that obliviously cheating players have to be tolerated for a bit before the ban hammer comes down.
- jbluepolarbear 5y ago
  It is easy to prevent wall hacks and speed hacks. Prediction input resolution, server keeping an authoritative representation on the server, and limiting the synced objects to the current view of the player are all easy when done individually. Thinking about all these concepts at the same time can be confusing for people new to the topic, but each subject is relatively easy to understand and implement. I’d say that this article handles the speed hack issue because of authoritative server state, but doesn’t go far into limiting what is sent to the player or not. The issue arises when you didn’t make your game multiplayer from the start and adding prediction and sync visibility isn’t an option, that’s where you get player authoritative states that share their states with the other players. Client authoritative is easy to implement, impossible to prevent cheating.
  - Hikikomori 5y ago
    > limiting the synced objects to the current view of the player are all easy when done individually. Its so easy to do, but pretty much no game in the industry does it? I guess they must all be incompetent.
    - jbluepolarbear 5y ago
      Wall hacks aren’t the issues you think it is, you’re more thinking corner peeking which is a whole different monster. Wall hacks most competitive games solve, but because of extrapolation and latency corner peeking is inevitable. There are mitigations, but each solution has its pros and cons.
      - Hikikomori 5y ago
        No, I'm not confused about this, I'm talking about regular old wallhacks/ESP that only requires the client to know about other players position and draw an overlay over the game. To my knowledge Valorant and csgo are the only ones that have done it. And it took what, almost 20 years for csgo to do it? In their cases it works because they have simple maps and a low amount of entities like players, so its not too much work for the server to do culling so they can hide positions of other players until its needed. But when in close proximity they can be seen through walls, like around a corner you can shoot through (at least csgo has penetration) and to avoid pop-in. And the other players still produce sound so information about other player positions are still sent to all clients and could be used for ESP. So no, this is not a solved problem, and is only partly so in very few games.
- taneq 5y ago
  > "easy" so you think veterant c++ dev that invented those concepts are dumb or lazy and it's a solved problem? Game devs on big commercial games are often not seasoned veterans, in my experience. The industry largely runs on the fanatical, underpaid efforts of kids who will do anything to work on games.
judge2020 5y ago
Maybe if you just have a flat world where you move in any direction with a max speed, but this sort of anti-cheat becomes incredibly difficult once you add features like terrain, physics, vehicles, things like gliders (in BR games), speed boosts, etc. It's possible, but takes a lot more effort. For example, in Fortnite, if you toggle a lagswitch (as in, block Fortnite UDP for some time), then reconnecting to the server will return you to the same position you were at when you disconnected, assuming you were just trying to run on the ground. However, if you toggle a lagswitch in a vehicle, the servers don't want to do expensive, complicated physics server-sided, so your client becomes authoritative to your position and the sanity checks are simply making sure you're not teleporting around large chunks of the map in seconds. If you were to hack your client and give your vehicle moon gravity, the server generally wouldn't care about that.
- jbluepolarbear 5y ago
  I’m pretty sure the car uses dead reckoning. So you position in the car is static (the last place you where), but the car is moving at speed and will continue even if you disconnect.
  - judge2020 5y ago
    Sure, but I’m talking about the control you have over the car when in the car. During the start of this battle pass season there was immense server issues, which I guessed was the tick rate dropping to super low values due to some bad code. During a few games where the servers degraded in this way, moving would be extremely slow on the ground as it kept pushing you back (since the server isn’t processing your ‘move forward’ commands as often) but getting in a car gave back free movement since I assume it’s a direct broadcast that doesn’t hit the game loop and doesn’t have any position-correcting sanity checks running.
    - jbluepolarbear 5y ago
      I’m saying a car is easier to predict because of dead reckoning. With less frequent updates your actions still feel fluid because the cars momentum can’t change a whole lot between updates. Say you’re driving a car in a straight line, there’s very good odds that in .25 seconds you’ll still be going straight so your predicted line will be very close to the real line. With players you can’t guarantee that, they could zig zag, turn around, jump, etc much more likely to change and require prediction reconciliation.
mathgladiator 5y ago
I believe part of the problem is that it is hard enough to build a game in the first place, and networking is a deep skill within itself. I've been doing large scale distributed systems for a decade, and I still feel like I'm learning especially as I retool for games. The way that I'm trying to make this simple is by inventing a new platform for "serverless game hosting" using my programming language (Adama : http://www.adama-lang.org/ http://www.adama-lang.org/ ). My focus on board game is the ultimate in server's role on authority since board games are more transactional between players and staggered (I play this card, that card enables another play to react, that reaction enables another play to augment, etc). I'm looking into what it could mean to escape the board-game pit, and one thing that I'm thinking about is latency. The question that I'm looking at is whether or not I can have a programming model be differentiated such that the client prediction code can be generated. A trivial way to exploit my language is to bring edge computing into the mix such that the edge compute runs a copy of the entire game, and then ships deltas quickly which will be overwritten from the authoritative server. This will work on updates to existing data (i.e. moving position), but it will fail at creating new items unless I move unique id generate to clients (which, is less than happy for me).
- jdc 5y ago
  I'm no expert, but isn't turn-based multiplayer a completely different animal?
  - MrLeap 5y ago
    Hook a clock up to your turns, end your turns 64~ times a second and we're real time now baby.
    - TeMPOraL 5y ago
      Yeah, we are, but pretty inefficiently. All the complexity in real-time multiplayer comes from trying to pretend everyone is in lockstep, while not actually synchronizing them 60+ times per second. This makes it an open-ended problem - many of the tricks work under very specific assumptions (e.g. low latency, trustworthy clients), and fail when those assumptions no longer hold (e.g. people notice stutter, cheaters cheat). The job is to find the right set of tricks for your use case. And it starts with knowing the tricks in the first place (or inventing new ones), as it's pretty much a specialization of its own.
skipants 5y ago
Hell, I can forgive it in games -- it seems pretty easy to forget a detail that can accidentally give too much authority to a user in that space, especially when you're trying to crunch your frame time. What really surprises me is how how often I see this mistake in web development. "No, Bob, you should not increase the user's bank account based on that number passed in from the frontend React app."
- tshaddox 5y ago
  I'm not sure exactly what you mean. If it's a web API request from a web client that says "transfer $1,000 between account A and account B," then what choice do you have but to "trust" that number? Obviously you have to check whether the request is authenticated and is authorized to transfer between those accounts, and check if account A has $1,000, but what about the client origin of the request do you need to check?
  - Tainnor 5y ago
    I mean since we're talking about banking: - Generate a transaction number and associate that number with the respective transaction details - Send this number to the customer's mobile phone with all those details, or other configured device - the transfer is only authorised if the customer has entered the transaction number That way, the customer is very likely to have verified the details of the transfer.
    - wibagusto 5y ago
      Do you really mean that?
      - Tainnor 5y ago
        Yes of course. This is the standard way online banking has been operating in, I believe, much of Europe since practically forever (although it used to be physical lists before smartphones were popular).
  - 5y ago
mysterydip 5y ago
Latency and performance, if you have to wait for the server every time you move or fire. It's a tradeoff which things are client and which are server.
gameswithgo 5y ago
there are fun types of gameplay that can’t happen without client authority. you can make an aimbot either way so it isn’t a huge deal. like you i had a hard time accepting that pubg using client aide hit detection maee any sense, but after playing it a lot and thinking hard about it I changed my mind. That game is all about pixel perfect aiming. Lag compensation wouldn’t cut it.
- Sohcahtoa82 5y ago
  https://youtu.be/2Gn2Sixr5Wk https://youtu.be/2Gn2Sixr5Wk IMO, This is clear-cut evidence that PUBG uses (or perhaps, in the past used) client-side hit checks. Notice there's not even bullet travel time. Just "pop pop" and a target that isn't in sight and never was inside instantly dies.
  - blibble 5y ago
    pretty much every game since Half-Life's patch in 2000 uses client side hit detection for hitscan weapons a few do client side prediction for projectiles too (Overwatch notably)
flohofwoe 5y ago
One important reason is that P2P games that only require central services for matchmaking but not for the actual gameplay sessions are a lot easier and cheaper to scale to high player counts.
trissylegs 5y ago
It was a big thing in World of Warcraft in particular because the internet was so unreliable when Wow release that they wanted the player to be able to do things when all your packets dropped for a few seconds (well maybe a whole minute) But they didn't really compensate for exploiting this to do things that were actually impossible. (ie speed hacks)
- Arrath 5y ago
  Back when WoW launched I was still on dialup, and played far too much for my own good. Part of playing on dialup was getting used to the latency, like 'queuing' my follow-up spell casts when my current one was anywhere from a fraction of a second to 0.5s from completion, so the command would arrive at the server just as my current one finished. As a healer this was important to avoid dead time where I could be keeping the tank alive. It wasn't until into the 2nd expansion that I got broadband, and hoo boy was it hard to adjust to. Suddenly I didn't have to anticipate and plan ahead nearly as much, and I had a whole new learning curve to go through to get my timing on point again. It seemed to me at the time that the game was structured pretty decently to handle the huge differences in player system specs and connection capabilities.
frozenlettuce 5y ago
these checks would result in a more expensive cloud billing. it's cheaper to keep the server just as a message relay between clients
Rd6n6 5y ago
I suspect that a lot of that is for the sake of client side prediction: if an enemy runs out from behind a wall, you want as little delay as possible between it happening and the player seeing it in order to react. Cheaters love it, but it does help with the illusion that everyone is playing in the same game world in real-time with no delays
thejosh 5y ago
https://www.youtube.com/watch?v=W3aieHjyNvw https://www.youtube.com/watch?v=W3aieHjyNvw Have a watch of this.
magicalhippo 5y ago
From my recollection, the primary issue is, or at least was, consoles. Sony and Microsoft required the games to allow players to have ridiculously high pings (several hundred ms). If you play a first-person shooter with 300-400 ms ping and the game uses server side hit detection, you're not going to have much fun. By moving to client-side detection, players hit what they "should hit" from their perspective. At least that's how I recall it from when COD and BF moved to client-side hit detection.
Gunax 5y ago
I believe it's because there are 2 conflicting goals: 1. Authoritative server 2. Client accuracy By 'client accuracy' I mean that was happens on the client actually occured. Ie. If you shoot a player, he actually got shot. The conundrum is that these goals conflict. They are like Heisenberg uncertainty: you can have position or momentum, but not both. You can have server-side authority, or you can have client side accuracy, or somewhere in the middle. Just as an example, most shooters have the goal that 'if you shot him on your screen, it was a hit', even if according to server side knowledge the recipient of the shot had already moved. Otherwise when you're shooting you would have to guess where the player is going to be when you're bullet packets reach the server. So instead, we let the client decide what a hit was. Hopefully the server has some sanity checking. I shoot at time t, server receives the info at t+s, reporting that I shot at some trajectory at time t. But the client can erroneously edit history. What if the client sends t-0.1s instead of t? There is no check the server can do to prevent this... I appear like a client which has 150ms of latency even though I really only have 50.

hesdeadjim 5y ago

Highly recommend Glen Fiedler's articles for this topic. https://gafferongames.com/ https://gafferongames.com/

jbluepolarbear 5y ago
I’d also recommend this write up from JMP Waveran (really anything by JMP) https://mrelusive.com/publications/papers/The-DOOM-III-Network-Architecture.pdf https://mrelusive.com/publications/papers/The-DOOM-III-Netwo...
lux 5y ago
His writing has been such a great learning resource. Can't recommend highly enough!

tshaddox 5y ago

I highly recommend this hour-long video from a Call of Duty developer discussing not only methods they use to transmit game states between client and server, but also some interesting ways that they measure true perceived latency (like using high-speed cameras and LCD monitors to compare true input-to-display latencies between different builds). https://www.youtube.com/watch?v=EtLHLfNpu84 https://www.youtube.com/watch?v=EtLHLfNpu84

magicalhippo 5y ago
Battle(non)sense did some videos on Battlefield 4 regarding netcode and latency back in the day, including one regarding input latency[1]. He wired up a LED to the mouse button, so it lit up when the button was pressed. By filming both the monitor and the LED with a high-speed camera he could measure the true input lag. BF4 had quite horrible netcode on launch, but after a valiant effort by Viceral it got quite good, at least on PC. Sadly the game had lost momentum at that point, almost two years after launch, so not many players left to enjoy it. https://www.youtube.com/watch?v=4GnKsqDAmgY https://www.youtube.com/watch?v=4GnKsqDAmgY

ggambetta 5y ago

Hey, I wrote that thing! Always a pleasant surprise and a weird feeling to open Hacker News (for the 14308560234th time today) and read my name like that :)

adamrezich 5y ago
since you're here, it looks like the second paragraph under the second heading in the second article is an unfinished thought :) great work otherwise though!
- ggambetta 5y ago
  Hmmm, you're right, great catch! Not sure what's going on there, I'll investigate.
dested 5y ago
I can't tell you how many times I find myself back on this page while working on multiplayer games. From the bottom of my heart I really can't thank you enough for this tutorial.
- ggambetta 5y ago
  You're very welcome :)
Rd6n6 5y ago
This was one of my main resources for learning game networking. If you ever feel tempted to write more articles on the subject, or write a book, people will eagerly read it. Thanks for writing and hosting your articles Edit: I see you wrote a cg book! https://news.ycombinator.com/item?id=26013002 https://news.ycombinator.com/item?id=26013002
- mrspeaker 5y ago
  Holy cow - I didn't know about this book. Everything Gabriel puts out is awesome: this is an instant-buy for me!

Raphaellll 5y ago

Another problem of predicting the movement of other players is that if they stop before you can see them, the trajectory would be interpolated further such that you can see them around the corner for a split second, knowing they are waiting there.

ggambetta 5y ago
The technique I describe in the article strictly interpolates between known positions; it never extrapolates further, so other players would never be seen or rendered beyond the point where they actually stopped.
- Raphaellll 5y ago
  Yes, I got this. Just wanted to point out that a particular consequence of extrapolating would be that players can see enemys around corners. You already mention that the game would be unplayable if players get teleported every few ms, and above just describes why, aside from aesthetics, makes it unplayable.

nightpool 5y ago

> Here’s the crucial step. Since the server gets all the input with timestamps, it can authoritatively reconstruct the world at any instant in the past. In particular, it can reconstruct the world exactly as it looked like to any client at any point in time. Isn't there a really obvious exploit here? The client could lie about the timestamps and use that to take advantage of future information in making its decisions. For example, in a simplified fighting game where you have two attacks (top and bottom) and two blocks, it could always choose the perfect block to apply before the opponent attacks—simply by seeing the incoming attack packet, and then immediately sending a block packet with an "older" timestamp. This doesn't entirely break the concept of an authoritative server, but it seems to diminish it significantly. I know lots of fighting games use this "rollback-type" netcode, maybe they've developed workarounds for this?

reitzensteinm 5y ago
This absolutely is a concern with rollback networking (I have done a bunch of games in it). There are two ways you can cheat. Because you have the full simulation state and input, there is no possible precaution against map hacks. Some FPS games just won't send exact player positions until the client is about to be able to legitimately observe them. Not possible here. The other way is, as you say, you can build fake inputs. To do the timing attack as you describe, all of your input would have to be pretending to be slow. You can't send a t=10 I'm idle packet, then a t=8 I blocked. You'd need a fantastic connection to the server, because the delay you're exploiting is there to protect you from your lag. Your oponent would also need a fantastic connection, because you aren't seeing their current inputs, you're seeing their past inputs. My feeling is that it would be very challenging to blend human like behavior with superhuman reflexes when doing automated responses. But there's nothing you're missing. The attack is definitely possible.
_--__--__ 5y ago
All rollback fighting games are pure p2p, there is no server for gameplay, just matchmaking and leaderboards. I'm not sure if what you would describe would work, it would at minimum double the ping between the two players, and in games that set variable rollback windows based on ping that delay may grow exponentially until a disconnect/desync. Also I think no one has attempted to cheat like this because it's much easier to cheat with a program that 'reacts' to graphical cues (such auto-block bots are pretty common in the most recent Tekken and Street Fighter entries). There are some very rare moves that are impossible for a program to react to like this (usually '0-frame grabs') which hit on the exact frame the input is entered, I'm not sure what if anything the current cheaters do about those.

Rd6n6 5y ago

Another complication is stateful processes like abilities: exactly how to sync, predict, lag compensate, and prevent cheating etc for a multistage ability. Lots of little edge cases to deal with, but I also read several interviews by indie devs who claimed that this wasn’t the hard part: the hardest was all the other tasks like matchmaking, hosting/joining games, invites, sdk integration, chat, voice, etc. It was just a mountain of small tasks

foota 5y ago
Imo the reason here is that separately this aspect of networking is harder, but in total the others will be a death by a thousand cuts.

beebeepka 5y ago

Can't give enough credit to arQon for his work on Quake 3 CPMA and whoever wrote the unlagged code for OpenArena and its peers. Truly some of the best examples out there. That was well before the modern shooters kids play these days. I'd say things have not improved dramatically over the last 10 years or so. Sure CS servers now run at 100 or more FPS but other than that, not a whole lot of change. In fact, popular examples such as PUBG suck immensely as far as networking is concerned

johnisgood 5y ago

This is a pretty good write-up, very useful. I am currently working on a game so I bookmarked it. Already read a lot, it really is great, thank you for it! If you got anything new, I am all up for it! I have not checked yet but did you mention anti-cheating systems?

johnisgood 5y ago

Are there any articles similar to this, by the way? This is very useful for someone working on a game server.