12 ms·

Never trust the client

vvanders 10y ago
They missed one key bit on CS's network model(which is what made it so great) is that they would actually re-wind the gamestate to do hitchecks so you could have a latency of 200ms+ and have a reasonable experience. Also the common term for resolving these types of things on the client is called "dead-reckoning". You've always got a diverging states from latency so you're continuously trying to reconcile this on the client. Simple Newtonian physics based games(Subspace, etc) are famous for being latency tolerant since the simulation is incredibly deterministic and are based on the players extrapolating where things will be in X time rather than twitch responses(which is also why fighting games are so hard without using a solution like Counter-Strike). You could play SubSpace on a 500ms dial-up connection and still be competitive without needing to lead for lag.
- Jare 10y ago
  A quick note, I think dead reckoning is not about correcting for latency. It is a conscious decision on the server to send less precise (cheaper), incremental data to the client most of the time. The client extrapolates from this imprecise data to still provide a fluid and continuous motion. The server performs the same extrapolation, and precisely tracks the error accumulated by the client to decide when to send a more expensive and precise packet that corrects the divergence.
  - vvanders 10y ago
    That's the traditional term, but it has specific meaning in game network programming. For instance we never send back the "error accumulated" to the server. Instead the server usually sends a PVS(potentially visible set) that the client handles displaying in a meaningful way(and reconciling and discrepancies). This PVS set is usually absolute values but at a discrete snapshot in time(of which the client can never be certain due to latency and there not being a single global clock). The client is basically guessing where the simulation is going and then smoothly handling(normally interpolation over your usual time-slice) differences from the server snapshot. That PVS is why you'll see map hacks where people can see through walls and whatnot. The server is doing less expensive checks(not doing a ray cast per-client to client which is n^2) and the cheat takes advantage of that data in memory.
- zubspace 10y ago
  There's something I always wondered about the gamestate rewind thing: Is it correct that the gamestate rewind only works for hitscan weapons (instant hit bullet)? What about slow moving projectiles or physically accurate bullets? Can the server spawn the projectile 200ms in the past at the time you pressed the button? But then other clients would receive the information about 400ms later after the initial keypress, which does not seem right...
  - vvanders 10y ago
    The projectile issue actually is much better. Since as a player you're trying to shoot the projectile to "predict" where it will intersect you're actually playing to the strength of latency and dead reckoning(that's the SubSpace case above). You want to segregate your action into twitch(reactionary: hitscan, parry, etc) and predictive(slow projectile, > 250ms ttl, etc) and only rewind/replay for the twitch case. There's also the case that when you rewind and do confirm a gamestate change you need to gracefully handle resolving it on all clients. That's why you'd see people warp back around corners when shot in Counter-Strike. Their player movement speed slowed after being hit and the server reconciles it and then the clients interpolate the result.
  - clevernickname 10y ago
    In Source Engine games, there is no lag compensation for projectiles. The lower your ping, the faster the projectile is spawned on the server. You can see this very clearly when playing on high ping: you click mouse1, you hear the rocket fire, but the rocket doesn't appear for a good fraction of a second. Since TF2 is very heavily based around projectiles (Soldier's rockets, Demo's pills and stickies, etc) many competitive players play with their cl_interp (determines how many ticks worth of game state are delayed before being displayed, interpolated, to the player) set lower than default (which is 2 ticks at 66 ticks per second by default). This results in jerky movement if packets are lost, but it ensures that they see threats as soon as possible so that they can fire their projectiles with minimal (real time) delay. I think it might also have some effect on client side input buffering; not sure, but that would be an even bigger reason for projectile users to want low interp.
- nhaehnle 10y ago
  The article does mention it, though without going into detail - it's called lag compensation.
wodenokoto 10y ago
Does anyone have a mirror for the "this is super bad" video?
- pgrote 10y ago
  I couldn't find it, but I did find a reddit thread with over video examples: https://www.reddit.com/r/thedivision/comments/4gblza/hackers_are_currently_making_the_dz_unplayable/ https://www.reddit.com/r/thedivision/comments/4gblza/hackers...
- onetimePete 10y ago
  Do you refer to the netcode analysis? https://www.youtube.com/watch?v=oc51gxwvyGc https://www.youtube.com/watch?v=oc51gxwvyGc Or the clientside thrust problem?
nicops 10y ago
I would think that in 2016 this should be something everybody knows....
- milesokeefe 10y ago
  Yeah, isn't this basically the first thing you learn in game networking?
  - cortesoft 10y ago
    Or in any sort of networking work where you don't control both ends of a connection.
    - neohaven 10y ago
      Even if you do, you still should never trust the client. Who knows when you might lose control of one end?
minimaxir 10y ago
The Division is an interesting game from a QA perspective. While client-side architecture is one thing, blocking bugs are present in the game and have been a strong deterrent to community stability. Case in point, it was recently discovered by the community that gear with the "Protection from Elites" bonus actually increased damage taken from Elites: https://reddit.com/r/thedivision/comments/4g6lnk/tested_confirmed_protection_from_elites_increases/ https://reddit.com/r/thedivision/comments/4g6lnk/tested_conf... And that's not even getting into the loot quality issues and the complete lack of a carrot-on-a-stick that has been codified in many, many games. (Massive actually made the game grindier because people were hitting end-game content too fast. Which only punished people who did not hit the content yet.)
- stefs 10y ago
  coincidentally i re-read the post about the riot games league of legends automated testing infrastructure yesterday. it's fascinating (and honestly, i'm even a bit envious!). in case you haven't read it yet: https://engineering.riotgames.com/news/automated-testing-league-legends https://engineering.riotgames.com/news/automated-testing-lea... guess it's down to the value of competitive multiplayer. e-sports are the focus of games like LOL (or, i guess, CS), instead of throw-away gaming. the division won't have international competitions in 3 years - and that's ok with ubi soft. because they'll have a couple of new games out by then and players will have bought those instead of still playing the division.
MustardTiger 10y ago
I'm still constantly shocked at how many games do this. In the MMOG world it is really easy to do everything right, UO came out in 1997 and the devs outright told everyone working on similar games "never trust the client" and "the client is in the hands of the enemy". Yet games like FFXI and WOW that came out many years later have the client setting the position of the player and the server just blindly accepting it, allowing for common and popular speed/pos hacks.
- throwaway13337 10y ago
  UO had rubberbanding rampant - that is, you'd move on the client, then warp back to an old position after the server said you actually didnt move there. It was terrible. Most all MMOs with WSDA movement trust the client to set the player's position and it mostly works. Cheating through warping or speed is easily detectable through other means of post-verification and banning players that fail that verification. It makes the game feel more fluid to the player while still not really allowing cheating. Everything else is generally handled serverside which is fine because it mostly doesn't need as high a reaction time as movement to feel natural. If speed/warp hacks are a problem, it's not because the developers that use client-set-movement aren't able to fix it, it's because they don't care.
  - TillE 10y ago
    Movement in UO worked ok most of the time, and that was with clients on 56k modems and servers powered by hamster wheels. > it's because they don't care That'd be nearly every company running an MMO then. You're quite right in principle: speed hacks and teleportation should be extremely easy to find by doing basic sanity checks every now and then. But in practice these things go on for months and months.
    - throwaway13337 10y ago
      Yes, it's true that they don't care. I know this from experience as one of 'they'. It's I guess oversimplified that we don't care. It's just that I, as a developer, would not be able to justify allotted time to fixing hacks that don't hurt the game much until they do.
MBCook 10y ago
Since this is a multi-player game this is obviously a big problem but I've always wondered how you'd handle this in a single player game with a leaderboard. Let's say you have a Bejeweled clone/match-3 game and a global high score board. What can you do to prevent fake scores? You can obviously obfuscate things, sign requests, etc but at some point the client needs to sign the score it's sending up so the client has the key. You could use this model of recording inputs and playing them back on the server but that seems like it would be a ton of work if your game is popular and an extraordinary cost for keeping a simple score board clean. Do you try and rely on subtle math tricks that mean that certain score numbers are de-facto invalid because no combination of scores could end up with that as a final total? Is there any way to handle the issue other than deleting scores that seem likely fake (large round numbers, for example, or those orders and orders of magnitude higher than other scores)?
- sythe2o0 10y ago
  You could send the score on a regular basis, and check to make sure it isn't increasing at an impossible rate?
  - GauntletWizard 10y ago
    You can send the replay of client actions taken, including the random seed to be sure the board and subsequent random results are accurate. A bejeweled game probably represents a few KB of data in this format. Here's the seed, a list of which two gems were switched, and when special abilities were activated. Done. Chess has a standard notation, and there's even tools to detect when human players are cheating by having chess engines play their moves by replaying parts of the game to those engines and checking for too many move similarities.
- iLoch 10y ago
  This is actually impossible without hardware encryption. Apple should have no problem with this since they have control over their hardware, but they don't do anything to prevent fake scores from showing up in their leaderboards. Xbox is a great example of being able to trust the client. The console tells the server that it has unlocked an achievement by using a signed request which was signed by the secure chip. The same chip verifies that the games you are playing are legitimate, etc. It's a very important piece of hardware and it would likely be the Xbox's demise if that secure chip suddenly became insecure. Running the game on the server is the only option as far as I know for securing the integrity of the data. You could add a private key to your app, but then you're shipping the private key to the client.
partycoder 10y ago
If you don't know this you should not be working on software. It's a latent risk to customers.
- rhinoceraptor 10y ago
  It's a video game, not a life critical system.
  - partycoder 10y ago
    A game can include a payment system, or processing of personally identifiable information. Security is important.
- deleted 10y ago
  [deleted]
morgante 10y ago
Obviously, this applies to all networked applications—not just games. It's embarrassing how often I see web apps which use something like Firebase with absolutely no validation or access control. Often the developers behind them don't even realize they have a problem. Their excuse is that "most users wouldn't have any reason to hack it, so why does it matter?" Developers need to realize their clients are inherently in the hands of hackers. Any security needs to be done on the server if it is going to succeed at all.
- cced 10y ago
  Can you recommend a go-to book for this kind of stuff?
awalton 10y ago
Console port gone horribly wrong? I mean, how else do you violate the most simple of game rules - never trust the client?
- minimaxir 10y ago
  The irony is that the console versions are the least-impacted, as the client-side hacks are only loadable on the PC version.
  - awalton 10y ago
    Trusting the client works on consoles since they're locked down. Porting that same idea to PCs gets you into trouble almost instantly. Hence my question - is this a console port gone horribly wrong?
  - csours 10y ago
    Walton's phrasing is kind of ambiguous, Console port could mean port from console or port to console. It could have been developed for console first.
    - awalton 10y ago
      "Console port" in the gaming world unambiguously means "a game that was originally developed for a console, then ported to the PC." The opposite would be "PC port."
- sleepybrett 10y ago
  Generally these days it isn't a port per-se it's simply one codebase compiled for different platforms. So I'm sure they just take the stance of 'this always worked on the console (where it's much harder to run modified code)'.
- h_ar 10y ago
  Actually, after extraction of data files, it points out that it is likely a PC port to console instead of the usual console port way.
xenadu02 10y ago
"Never trust the client" is good advice for every kind of application. Clients are filthy liars. Clients have bugs. Old clients don't upgrade. Packets arrive out of order. ACKs never make it to clients, causing clients to repeat what the client believes to be a failed operation. All of this is before even considering an attacker actively trying to subvert you. The server should always be the source of truth. It should always enforce all the consistency rules. Database schemas can be a critical tool here as well. Don't blindly slap whatever the client sends into a no-SQL database, then vomit it back out for queries.
- ksk 10y ago
  With caveats. What you're also implying is that clients should do no processing - because clients can't be trusted to do anything. If you apply this logic to web-browsing, you're asking the server to send the client a bitmap image of what the rendered page should look like.
  - JoshTriplett 10y ago
    Not at all. It's fine for the client to handle processing when the only thing they could hurt is the user of the client. If the client renders a web page incorrectly, only the user of the client gets hurt by that. But the server API must not allow a malicious client to do damage that affects other clients.
  - abraae 10y ago
    Your example is about the client misreporting server state to the end user. The big deal is instead about a badly behaved client being able to make "illegal" changes to the server state.
  - cortesoft 10y ago
    I think that is misinterpreting what the person you are replying to is saying. They aren't saying "don't let the clients do anything", they are saying "don't trust the values the clients send back" If the client messes up the rendering, the only person affected is the client. There is no 'trust' required, because you are not relying on anything the client has done. The idea of trust only comes into play if there is a consequence to that trust being broken.
bdavisx 10y ago
I doubt that the developer's didn't know not to trust the client - they likely made a business decision. I (used to a lot more) play Elite: Dangerous. A space sim that has multi-player. They made the decision to use (mostly) p2p/client networking to save money - they wouldn't need nearly as many servers. This has caused other issues in addition to cheating - a low limit on the number of players in the same "instance" of the universe for example. But it was a business decision - imo the wrong one, but what do I matter?
- mentos 10y ago
  Yea and I'll go out on a limb and say it was the correct one. The shelf life of a popular game like this is probably shorter than most people think so it makes sense to offer an amazing experience that you couldn't necessarily achieve with an authoritative server model and then by the time hackers have defeated the game players have probably moved on to the next.
  - garrettgrimsley 10y ago
    Initial Release Date: March 8, 2016
  - floodyberry- 10y ago
    Guaranteeing your multiplayer blockbuster game designed to have "infinite gameplay" will be dead in the water the instant hackers discover its highly flawed networking model doesn't sound like a "correct business decision".
    - mentos 10y ago
      It was marketed to have infinite gameplay. What game do you know that honestly has achieved that?
secoif 10y ago
Ubisoft appears to have some institutional problems. Their other recent Tom Clancy title, Rainbow Six Siege, is hands-down one of the most rewarding & intense shooters I've ever played, however it too is yet to realise its full potential due to amateur-hour quality issues and ongoing troubles with hackers.
forgotmypassw 10y ago
I thought it was a common sense to treat the client as merely the data representation and nothing else, sad to see that multi million dollar companies make rookie mistakes.
- abraae 10y ago
  I guess in the eyes of a corporate, it's only a mistake if it harms the bottom line. Sounds like this would though, if players can't enjoy a fair experience.
outworlder 10y ago
Trusting the client, in essence, turns a game into one of your childhood: shouting matches of "I hit you!" "No you did!" "I totally did".
andrewclunn 10y ago
I was not interested in this game, but now knowing that it can be hacked in cool ways makes me interested. I only care about playing with friends, and this would allow for some super cool custom game modes. It's not a bug it's a feature!
kristofferR 10y ago
The Division also uses TCP instead of UDP so they're clearly totally incompetent when it comes to networking.
trjordan 10y ago
I think there's a false dichotomy here. You can't simply ignore the client, so no matter what, you have to have a set of server-side checks that ensure the client's inputs are valid. Part of what this article offers is "you can't fix it by adding server-side checks", guessing that if they haven't done it already, it's impossible to do. I think it's entirely possible that their model is reasonable (takes movement, fire events, etc. instead of position, inventory, etc.), and they simply haven't guarded against malicious inputs. Adding server-side checks is _exactly_ what you need to do to make movement/firing inputs safe! The existence of a certain class of bug doesn't mean the whole architecture is broken. It may simply mean they shipped the game without trying to detect cheaters. That may be a bad idea, but if the design is right, it's certainly fixable.
- kcbanner 10y ago
  I think you might be misunderstanding the scope of an "input". It's something like "the player pressed the key/button that maps to move forward", not "move to position X". Teleport hacks wouldn't be possible in the former case because the player has to actually move their avatar to the desired position by traversing the game world through a series of inputs. The server would then process the movement inputs, and the avatar would say, bump into a wall, preventing it's movement. There would be no input for "my position is now X".
  - trjordan 10y ago
    Is an input always valid? "Firing once" is an input: you have to validate that it's been long enough since last firing that you should act on it (limit the rate). "Moving left" is an input, but by how much? Can I send "Left, 9999999" and replicate a teleport hack by pre-calculating these inputs? You could bound the inputs to a simple stream of enums (FIRE, LEFT, RIGHT, etc.), but it wouldn't be crazy to batch it up a little bit and send some magnitude data as well (FIRE 2, LEFT 50, etc.). It's a little harder to validate, but depending on the specifics of implementation, that could be a win.
    - ufo 10y ago
primigenus 10y ago
The server-side network model he describes here is the same architecture Meteor is designed with (see this page: https://www.meteor.com/why-meteor/features https://www.meteor.com/why-meteor/features). With Meteor, you get client side prediction and latency compensation (they call it "optimistic UI" now) for free. I've always been impressed they decided to build that, because I sure never would have myself. In fact the Meteor team has always said you need this kind of architecture in order to build true real-time applications. But I haven't seen other web-oriented platforms take a similar approach. Did the Meteor team just know something no one else has picked up on (despite it apparently being common practice in the games industry)? What gives?
- emerongi 10y ago
  Even though I don't use Meteor anymore, I still like DDP[1] a lot. It's simple (can be implemented in 50-100 lines), but it is the core of Meteor. There's a specification for it in their Github repo. Meteor's "optimistic UI" is basically an abstraction on top DDP, IMO. [1] https://www.meteor.com/ddp https://www.meteor.com/ddp
andersonmvd 10y ago
I'm surprised that "never trust the client" isn't among the basic concepts that everyone should know, at least on HN. It's such a basic premise.
- Aelinsaar 10y ago
  If you ignore the context, "Never trust the client/patient/enduser/etc..." is probably equally true.
stefs 10y ago
now that i think about it - doesn't GTA5 suffer from the exact same problem? cheaters spawing tanks above people practically must be a client game state problem.
- oridecon 10y ago
  Yes and that is why I think it's a new business model to make games not last long on the PC. At least the multiplayer part. They promote the shit out of games before release and expect them to expire a few weeks after. Reducing operational costs. They also don't have to develop and maintain an anti-cheat, a better server-side structure, and they can close down most servers. I think it was a unexpected consequence of a bad console port to pc that proved to be quite lucrative. And now they know people will keep buying no matter what, hence the new disposable pc game business model. Basically, if you play on a PC you're screwed. Another reason not to buy a console or support games like the division on the pc.
stefs 10y ago
another recent example of "never trust the client" is the amazon kindle unlimited debacle. in case you missed it: the client syncs the last page read. so you can buy a book, jump to the last side and amazon marks it as read, without checking for intermediate states. the problem is: amazon pays authors by "pages read". so people publish fake books with 3000 pages, let sweat shops try the book for free and just jump from the first to the last page, sync and it registers with "this customer read all 3k pages". now create 20 fake author accounts and 30 fake reader accounts, publish 20 fake 3k books (one for each fake author account and they're practically filled with 3k pages of randomly scraped web content) and you got 20303000 = 1.800.000 pages read. if you are an aspiring author and publish a novella with, say, 50 pages and 10.000 readers devour every single page of it, you've got 500.000 pages read. the faker can do this in a week and take about 4 times more than, even though it took you 2 months to write it. payouts are out of a pot shared between all authors. thus, authors lose in the short run (less money now), cheaters win big time (a lot money now), amazon doesn't lose money (now) because the payout is the same. in the long run it's still problematic because small authors are unhappy once they realize what's afoot (and earn 1k instead of 10k). big name authors don't care as much if they earn 1m or 1.1m. customers aren't likely to buy the fake books anyway (cheaters take them offline before the free trial period ends) so they're not really affected.
- chris_wot 10y ago
  That could reasonably be considered fraud. If I was Amazon, I'd pursue them, publicly.
notacoward 10y ago
I'll bet there was at least one developer who saw they were doing it wrong, and told them, and was overruled. I'd love to hear his/her account.
- majewsky 10y ago
  That type of story grows old after the 100th time you hear it.
  - notacoward 10y ago
    Also, get off my lawn.
rexpop 10y ago
How do we reconcile sentiments like "never trust the client" alongside this community's hope for decentralization, á la "The Internet Has Been Stolen From You. Take it Back Nonviolently"?
- zaphar 10y ago
  In decentralization you are always the server and everyone else is the client. You have ways to validate what they send to you. It works out because they all think of themselves as the server and everyone else as the client too. This is of course a vast simplification but the trust part is the same. No one trusts anyone but themselves and validates everything everyone sends them.
majewsky 10y ago
One can easily observe this "server is the actual game" property in Minecraft when running on an underpowered server. At home, Minecraft runs on my home server (with a small Athlon CPU) and it fails to keep up with the required speed of the game, so the server log will show messages like "server clock running behind, skipping 58 ticks" every so often. An observable effect of this is that you have to hold the right mouse button just a little bit longer than the actual animation when eating stuff. And when you look at the sun, you will see it moving forward continuously (as calculated by the client), but skipping back a tiny bit every few seconds (when the client adjusts for the server's time drift).

They missed one key bit on CS's network model(which is what made it so great) is that they would actually re-wind the gamestate to do hitchecks so you could have a latency of 200ms+ and have a reasonable experience. Also the common term for resolving these types of things on the client is called "dead-reckoning". You've always got a diverging states from latency so you're continuously trying to reconcile this on the client. Simple Newtonian physics based games(Subspace, etc) are famous for being latency tolerant since the simulation is incredibly deterministic and are based on the players extrapolating where things will be in X time rather than twitch responses(which is also why fighting games are so hard without using a solution like Counter-Strike). You could play SubSpace on a 500ms dial-up connection and still be competitive without needing to lead for lag.

Jare 10y ago
A quick note, I think dead reckoning is not about correcting for latency. It is a conscious decision on the server to send less precise (cheaper), incremental data to the client most of the time. The client extrapolates from this imprecise data to still provide a fluid and continuous motion. The server performs the same extrapolation, and precisely tracks the error accumulated by the client to decide when to send a more expensive and precise packet that corrects the divergence.
- vvanders 10y ago
  That's the traditional term, but it has specific meaning in game network programming. For instance we never send back the "error accumulated" to the server. Instead the server usually sends a PVS(potentially visible set) that the client handles displaying in a meaningful way(and reconciling and discrepancies). This PVS set is usually absolute values but at a discrete snapshot in time(of which the client can never be certain due to latency and there not being a single global clock). The client is basically guessing where the simulation is going and then smoothly handling(normally interpolation over your usual time-slice) differences from the server snapshot. That PVS is why you'll see map hacks where people can see through walls and whatnot. The server is doing less expensive checks(not doing a ray cast per-client to client which is n^2) and the cheat takes advantage of that data in memory.
zubspace 10y ago
There's something I always wondered about the gamestate rewind thing: Is it correct that the gamestate rewind only works for hitscan weapons (instant hit bullet)? What about slow moving projectiles or physically accurate bullets? Can the server spawn the projectile 200ms in the past at the time you pressed the button? But then other clients would receive the information about 400ms later after the initial keypress, which does not seem right...
- vvanders 10y ago
  The projectile issue actually is much better. Since as a player you're trying to shoot the projectile to "predict" where it will intersect you're actually playing to the strength of latency and dead reckoning(that's the SubSpace case above). You want to segregate your action into twitch(reactionary: hitscan, parry, etc) and predictive(slow projectile, > 250ms ttl, etc) and only rewind/replay for the twitch case. There's also the case that when you rewind and do confirm a gamestate change you need to gracefully handle resolving it on all clients. That's why you'd see people warp back around corners when shot in Counter-Strike. Their player movement speed slowed after being hit and the server reconciles it and then the clients interpolate the result.
- clevernickname 10y ago
  In Source Engine games, there is no lag compensation for projectiles. The lower your ping, the faster the projectile is spawned on the server. You can see this very clearly when playing on high ping: you click mouse1, you hear the rocket fire, but the rocket doesn't appear for a good fraction of a second. Since TF2 is very heavily based around projectiles (Soldier's rockets, Demo's pills and stickies, etc) many competitive players play with their cl_interp (determines how many ticks worth of game state are delayed before being displayed, interpolated, to the player) set lower than default (which is 2 ticks at 66 ticks per second by default). This results in jerky movement if packets are lost, but it ensures that they see threats as soon as possible so that they can fire their projectiles with minimal (real time) delay. I think it might also have some effect on client side input buffering; not sure, but that would be an even bigger reason for projectile users to want low interp.
nhaehnle 10y ago
The article does mention it, though without going into detail - it's called lag compensation.

wodenokoto 10y ago

Does anyone have a mirror for the "this is super bad" video?

pgrote 10y ago
I couldn't find it, but I did find a reddit thread with over video examples: https://www.reddit.com/r/thedivision/comments/4gblza/hackers_are_currently_making_the_dz_unplayable/ https://www.reddit.com/r/thedivision/comments/4gblza/hackers...
onetimePete 10y ago
Do you refer to the netcode analysis? https://www.youtube.com/watch?v=oc51gxwvyGc https://www.youtube.com/watch?v=oc51gxwvyGc Or the clientside thrust problem?

nicops 10y ago

I would think that in 2016 this should be something everybody knows....

milesokeefe 10y ago
Yeah, isn't this basically the first thing you learn in game networking?
- cortesoft 10y ago
  Or in any sort of networking work where you don't control both ends of a connection.
  - neohaven 10y ago
    Even if you do, you still should never trust the client. Who knows when you might lose control of one end?

minimaxir 10y ago

The Division is an interesting game from a QA perspective. While client-side architecture is one thing, blocking bugs are present in the game and have been a strong deterrent to community stability. Case in point, it was recently discovered by the community that gear with the "Protection from Elites" bonus actually increased damage taken from Elites: https://reddit.com/r/thedivision/comments/4g6lnk/tested_confirmed_protection_from_elites_increases/ https://reddit.com/r/thedivision/comments/4g6lnk/tested_conf... And that's not even getting into the loot quality issues and the complete lack of a carrot-on-a-stick that has been codified in many, many games. (Massive actually made the game grindier because people were hitting end-game content too fast. Which only punished people who did not hit the content yet.)

stefs 10y ago
coincidentally i re-read the post about the riot games league of legends automated testing infrastructure yesterday. it's fascinating (and honestly, i'm even a bit envious!). in case you haven't read it yet: https://engineering.riotgames.com/news/automated-testing-league-legends https://engineering.riotgames.com/news/automated-testing-lea... guess it's down to the value of competitive multiplayer. e-sports are the focus of games like LOL (or, i guess, CS), instead of throw-away gaming. the division won't have international competitions in 3 years - and that's ok with ubi soft. because they'll have a couple of new games out by then and players will have bought those instead of still playing the division.

MustardTiger 10y ago

I'm still constantly shocked at how many games do this. In the MMOG world it is really easy to do everything right, UO came out in 1997 and the devs outright told everyone working on similar games "never trust the client" and "the client is in the hands of the enemy". Yet games like FFXI and WOW that came out many years later have the client setting the position of the player and the server just blindly accepting it, allowing for common and popular speed/pos hacks.

throwaway13337 10y ago
UO had rubberbanding rampant - that is, you'd move on the client, then warp back to an old position after the server said you actually didnt move there. It was terrible. Most all MMOs with WSDA movement trust the client to set the player's position and it mostly works. Cheating through warping or speed is easily detectable through other means of post-verification and banning players that fail that verification. It makes the game feel more fluid to the player while still not really allowing cheating. Everything else is generally handled serverside which is fine because it mostly doesn't need as high a reaction time as movement to feel natural. If speed/warp hacks are a problem, it's not because the developers that use client-set-movement aren't able to fix it, it's because they don't care.
- TillE 10y ago
  Movement in UO worked ok most of the time, and that was with clients on 56k modems and servers powered by hamster wheels. > it's because they don't care That'd be nearly every company running an MMO then. You're quite right in principle: speed hacks and teleportation should be extremely easy to find by doing basic sanity checks every now and then. But in practice these things go on for months and months.
  - throwaway13337 10y ago
    Yes, it's true that they don't care. I know this from experience as one of 'they'. It's I guess oversimplified that we don't care. It's just that I, as a developer, would not be able to justify allotted time to fixing hacks that don't hurt the game much until they do.

MBCook 10y ago

Since this is a multi-player game this is obviously a big problem but I've always wondered how you'd handle this in a single player game with a leaderboard. Let's say you have a Bejeweled clone/match-3 game and a global high score board. What can you do to prevent fake scores? You can obviously obfuscate things, sign requests, etc but at some point the client needs to sign the score it's sending up so the client has the key. You could use this model of recording inputs and playing them back on the server but that seems like it would be a ton of work if your game is popular and an extraordinary cost for keeping a simple score board clean. Do you try and rely on subtle math tricks that mean that certain score numbers are de-facto invalid because no combination of scores could end up with that as a final total? Is there any way to handle the issue other than deleting scores that seem likely fake (large round numbers, for example, or those orders and orders of magnitude higher than other scores)?

sythe2o0 10y ago
You could send the score on a regular basis, and check to make sure it isn't increasing at an impossible rate?
- GauntletWizard 10y ago
  You can send the replay of client actions taken, including the random seed to be sure the board and subsequent random results are accurate. A bejeweled game probably represents a few KB of data in this format. Here's the seed, a list of which two gems were switched, and when special abilities were activated. Done. Chess has a standard notation, and there's even tools to detect when human players are cheating by having chess engines play their moves by replaying parts of the game to those engines and checking for too many move similarities.
iLoch 10y ago
This is actually impossible without hardware encryption. Apple should have no problem with this since they have control over their hardware, but they don't do anything to prevent fake scores from showing up in their leaderboards. Xbox is a great example of being able to trust the client. The console tells the server that it has unlocked an achievement by using a signed request which was signed by the secure chip. The same chip verifies that the games you are playing are legitimate, etc. It's a very important piece of hardware and it would likely be the Xbox's demise if that secure chip suddenly became insecure. Running the game on the server is the only option as far as I know for securing the integrity of the data. You could add a private key to your app, but then you're shipping the private key to the client.

partycoder 10y ago

If you don't know this you should not be working on software. It's a latent risk to customers.

rhinoceraptor 10y ago
It's a video game, not a life critical system.
- partycoder 10y ago
  A game can include a payment system, or processing of personally identifiable information. Security is important.
deleted 10y ago
[deleted]

morgante 10y ago

Obviously, this applies to all networked applications—not just games. It's embarrassing how often I see web apps which use something like Firebase with absolutely no validation or access control. Often the developers behind them don't even realize they have a problem. Their excuse is that "most users wouldn't have any reason to hack it, so why does it matter?" Developers need to realize their clients are inherently in the hands of hackers. Any security needs to be done on the server if it is going to succeed at all.

cced 10y ago
Can you recommend a go-to book for this kind of stuff?

awalton 10y ago

Console port gone horribly wrong? I mean, how else do you violate the most simple of game rules - never trust the client?

minimaxir 10y ago
The irony is that the console versions are the least-impacted, as the client-side hacks are only loadable on the PC version.
- awalton 10y ago
  Trusting the client works on consoles since they're locked down. Porting that same idea to PCs gets you into trouble almost instantly. Hence my question - is this a console port gone horribly wrong?
- csours 10y ago
  Walton's phrasing is kind of ambiguous, Console port could mean port from console or port to console. It could have been developed for console first.
  - awalton 10y ago
    "Console port" in the gaming world unambiguously means "a game that was originally developed for a console, then ported to the PC." The opposite would be "PC port."
sleepybrett 10y ago
Generally these days it isn't a port per-se it's simply one codebase compiled for different platforms. So I'm sure they just take the stance of 'this always worked on the console (where it's much harder to run modified code)'.
h_ar 10y ago
Actually, after extraction of data files, it points out that it is likely a PC port to console instead of the usual console port way.

xenadu02 10y ago

"Never trust the client" is good advice for every kind of application. Clients are filthy liars. Clients have bugs. Old clients don't upgrade. Packets arrive out of order. ACKs never make it to clients, causing clients to repeat what the client believes to be a failed operation. All of this is before even considering an attacker actively trying to subvert you. The server should always be the source of truth. It should always enforce all the consistency rules. Database schemas can be a critical tool here as well. Don't blindly slap whatever the client sends into a no-SQL database, then vomit it back out for queries.

ksk 10y ago
With caveats. What you're also implying is that clients should do no processing - because clients can't be trusted to do anything. If you apply this logic to web-browsing, you're asking the server to send the client a bitmap image of what the rendered page should look like.
- JoshTriplett 10y ago
  Not at all. It's fine for the client to handle processing when the only thing they could hurt is the user of the client. If the client renders a web page incorrectly, only the user of the client gets hurt by that. But the server API must not allow a malicious client to do damage that affects other clients.
- abraae 10y ago
  Your example is about the client misreporting server state to the end user. The big deal is instead about a badly behaved client being able to make "illegal" changes to the server state.
- cortesoft 10y ago
  I think that is misinterpreting what the person you are replying to is saying. They aren't saying "don't let the clients do anything", they are saying "don't trust the values the clients send back" If the client messes up the rendering, the only person affected is the client. There is no 'trust' required, because you are not relying on anything the client has done. The idea of trust only comes into play if there is a consequence to that trust being broken.

bdavisx 10y ago

I doubt that the developer's didn't know not to trust the client - they likely made a business decision. I (used to a lot more) play Elite: Dangerous. A space sim that has multi-player. They made the decision to use (mostly) p2p/client networking to save money - they wouldn't need nearly as many servers. This has caused other issues in addition to cheating - a low limit on the number of players in the same "instance" of the universe for example. But it was a business decision - imo the wrong one, but what do I matter?

mentos 10y ago
Yea and I'll go out on a limb and say it was the correct one. The shelf life of a popular game like this is probably shorter than most people think so it makes sense to offer an amazing experience that you couldn't necessarily achieve with an authoritative server model and then by the time hackers have defeated the game players have probably moved on to the next.
- garrettgrimsley 10y ago
  Initial Release Date: March 8, 2016
- floodyberry- 10y ago
  Guaranteeing your multiplayer blockbuster game designed to have "infinite gameplay" will be dead in the water the instant hackers discover its highly flawed networking model doesn't sound like a "correct business decision".
  - mentos 10y ago
    It was marketed to have infinite gameplay. What game do you know that honestly has achieved that?

secoif 10y ago

Ubisoft appears to have some institutional problems. Their other recent Tom Clancy title, Rainbow Six Siege, is hands-down one of the most rewarding & intense shooters I've ever played, however it too is yet to realise its full potential due to amateur-hour quality issues and ongoing troubles with hackers.

forgotmypassw 10y ago

I thought it was a common sense to treat the client as merely the data representation and nothing else, sad to see that multi million dollar companies make rookie mistakes.

abraae 10y ago
I guess in the eyes of a corporate, it's only a mistake if it harms the bottom line. Sounds like this would though, if players can't enjoy a fair experience.

outworlder 10y ago

Trusting the client, in essence, turns a game into one of your childhood: shouting matches of "I hit you!" "No you did!" "I totally did".

andrewclunn 10y ago

I was not interested in this game, but now knowing that it can be hacked in cool ways makes me interested. I only care about playing with friends, and this would allow for some super cool custom game modes. It's not a bug it's a feature!

kristofferR 10y ago

The Division also uses TCP instead of UDP so they're clearly totally incompetent when it comes to networking.

trjordan 10y ago

I think there's a false dichotomy here. You can't simply ignore the client, so no matter what, you have to have a set of server-side checks that ensure the client's inputs are valid. Part of what this article offers is "you can't fix it by adding server-side checks", guessing that if they haven't done it already, it's impossible to do. I think it's entirely possible that their model is reasonable (takes movement, fire events, etc. instead of position, inventory, etc.), and they simply haven't guarded against malicious inputs. Adding server-side checks is _exactly_ what you need to do to make movement/firing inputs safe! The existence of a certain class of bug doesn't mean the whole architecture is broken. It may simply mean they shipped the game without trying to detect cheaters. That may be a bad idea, but if the design is right, it's certainly fixable.

kcbanner 10y ago
I think you might be misunderstanding the scope of an "input". It's something like "the player pressed the key/button that maps to move forward", not "move to position X". Teleport hacks wouldn't be possible in the former case because the player has to actually move their avatar to the desired position by traversing the game world through a series of inputs. The server would then process the movement inputs, and the avatar would say, bump into a wall, preventing it's movement. There would be no input for "my position is now X".
- trjordan 10y ago
  Is an input always valid? "Firing once" is an input: you have to validate that it's been long enough since last firing that you should act on it (limit the rate). "Moving left" is an input, but by how much? Can I send "Left, 9999999" and replicate a teleport hack by pre-calculating these inputs? You could bound the inputs to a simple stream of enums (FIRE, LEFT, RIGHT, etc.), but it wouldn't be crazy to batch it up a little bit and send some magnitude data as well (FIRE 2, LEFT 50, etc.). It's a little harder to validate, but depending on the specifics of implementation, that could be a win.
  - ufo 10y ago

primigenus 10y ago

The server-side network model he describes here is the same architecture Meteor is designed with (see this page: https://www.meteor.com/why-meteor/features https://www.meteor.com/why-meteor/features). With Meteor, you get client side prediction and latency compensation (they call it "optimistic UI" now) for free. I've always been impressed they decided to build that, because I sure never would have myself. In fact the Meteor team has always said you need this kind of architecture in order to build true real-time applications. But I haven't seen other web-oriented platforms take a similar approach. Did the Meteor team just know something no one else has picked up on (despite it apparently being common practice in the games industry)? What gives?

emerongi 10y ago
Even though I don't use Meteor anymore, I still like DDP[1] a lot. It's simple (can be implemented in 50-100 lines), but it is the core of Meteor. There's a specification for it in their Github repo. Meteor's "optimistic UI" is basically an abstraction on top DDP, IMO. [1] https://www.meteor.com/ddp https://www.meteor.com/ddp

andersonmvd 10y ago

I'm surprised that "never trust the client" isn't among the basic concepts that everyone should know, at least on HN. It's such a basic premise.

Aelinsaar 10y ago
If you ignore the context, "Never trust the client/patient/enduser/etc..." is probably equally true.

stefs 10y ago

now that i think about it - doesn't GTA5 suffer from the exact same problem? cheaters spawing tanks above people practically must be a client game state problem.

oridecon 10y ago
Yes and that is why I think it's a new business model to make games not last long on the PC. At least the multiplayer part. They promote the shit out of games before release and expect them to expire a few weeks after. Reducing operational costs. They also don't have to develop and maintain an anti-cheat, a better server-side structure, and they can close down most servers. I think it was a unexpected consequence of a bad console port to pc that proved to be quite lucrative. And now they know people will keep buying no matter what, hence the new disposable pc game business model. Basically, if you play on a PC you're screwed. Another reason not to buy a console or support games like the division on the pc.

stefs 10y ago

another recent example of "never trust the client" is the amazon kindle unlimited debacle. in case you missed it: the client syncs the last page read. so you can buy a book, jump to the last side and amazon marks it as read, without checking for intermediate states. the problem is: amazon pays authors by "pages read". so people publish fake books with 3000 pages, let sweat shops try the book for free and just jump from the first to the last page, sync and it registers with "this customer read all 3k pages". now create 20 fake author accounts and 30 fake reader accounts, publish 20 fake 3k books (one for each fake author account and they're practically filled with 3k pages of randomly scraped web content) and you got 20303000 = 1.800.000 pages read. if you are an aspiring author and publish a novella with, say, 50 pages and 10.000 readers devour every single page of it, you've got 500.000 pages read. the faker can do this in a week and take about 4 times more than, even though it took you 2 months to write it. payouts are out of a pot shared between all authors. thus, authors lose in the short run (less money now), cheaters win big time (a lot money now), amazon doesn't lose money (now) because the payout is the same. in the long run it's still problematic because small authors are unhappy once they realize what's afoot (and earn 1k instead of 10k). big name authors don't care as much if they earn 1m or 1.1m. customers aren't likely to buy the fake books anyway (cheaters take them offline before the free trial period ends) so they're not really affected.

chris_wot 10y ago
That could reasonably be considered fraud. If I was Amazon, I'd pursue them, publicly.

notacoward 10y ago

I'll bet there was at least one developer who saw they were doing it wrong, and told them, and was overruled. I'd love to hear his/her account.

majewsky 10y ago
That type of story grows old after the 100th time you hear it.
- notacoward 10y ago
  Also, get off my lawn.

rexpop 10y ago

How do we reconcile sentiments like "never trust the client" alongside this community's hope for decentralization, á la "The Internet Has Been Stolen From You. Take it Back Nonviolently"?

zaphar 10y ago
In decentralization you are always the server and everyone else is the client. You have ways to validate what they send to you. It works out because they all think of themselves as the server and everyone else as the client too. This is of course a vast simplification but the trust part is the same. No one trusts anyone but themselves and validates everything everyone sends them.

majewsky 10y ago

One can easily observe this "server is the actual game" property in Minecraft when running on an underpowered server. At home, Minecraft runs on my home server (with a small Athlon CPU) and it fails to keep up with the required speed of the game, so the server log will show messages like "server clock running behind, skipping 58 ticks" every so often. An observable effect of this is that you have to hold the right mouse button just a little bit longer than the actual animation when eating stuff. And when you look at the sun, you will see it moving forward continuously (as calculated by the client), but skipping back a tiny bit every few seconds (when the client adjusts for the server's time drift).