7 ms·
I don't know enough about 'real time' netcode for games. However I have read several HN articles over the years so I've got at least a basic understanding. Wh
by mjevans 2y ago
I don't know enough about 'real time' netcode for games. However I have read several HN articles over the years so I've got at least a basic understanding.
Why can't the servers distrust the clients? What should a 'client side anti cheat' actually prevent?
The way I think I'd tackle such things is to have multiple copies of each character model moving in different locations and different ways. Such that trying to spy on the state of the game from one client's viewpoint yields mostly false data. New 'threads' would fork off of the existing threads and would only be culled when there are too many or they're about to make a side effect that would be visible if they were real. In that way the server would be responsible for feeding misinformation to clients but maintaining the state of the true game as a secret to itself.
- Iridescent_ 2y agoNot trusting the clients and redoing all calculations server-side would require massive processing on the server side. Your idea then multiplies the load on the server.
- worthless-trash 2y agoHow much though, i mean you're basically doing vector work, and only when its visible within range, and then when its visible within the player and the player hitbox is.... oh I see.
- Ekaros 2y agoWe do have massive processing on all sides. And the actual processing done for the game stuff is not really that big. Mostly it goes to graphics, sounds and so on.
- kachapopopow 2y agoOverwatch does this just fine and deadlock actually.
- totaa 2y ago> deadlock actually how do you know? has there been a technical deep dive published?
- kachapopopow 2y agoOpening the console and seeing rollback net-code pretty much says enough. Also the game itself feels great to play even with 160ms ping which is only really possible when using architecture similar to overwatch.
- claudex 2y agoThe client need to have more state info than the player to render accurately, for example, to render an opponent passing through a window without lag. And also, there are also cheat that doesn't need to spy on the state, like aim assist tool or HUD improvement.
- mjevans 2y ago* Pass through a window without lag - That's why the server is sending multiple copies of potential movements and paths through the level for each character, but terminating the ones that are about to reveal their effects (no longer be culled by walls / objects) when they'd send false information to non-cheating players. * Aim Assist - what's that supposed to work with for the assist? I guess it might help someone target a player once they're exposed, or once they've locked on. For that I think that extremely top tier players might behave within fuzzing distance of tool assist, at least some of the time. Dodging might have similar issues. I could even see ML assisting inputs just based on frame-grabs off the screen video output. -- So I'm not sure what client side anti-cheat is supposed to do here. * HUD improvements - like what?
- oerdier 2y agoRegarding HUD improvements, this monitor shows where enemies are likely to appear. The monitor does this autonomously; the OS isn't in the loop. https://www.tomshardware.com/monitors/msis-ai-powered-gaming-monitor-helps-you-cheat-at-league-of-legends-looks-great-doing-it https://www.tomshardware.com/monitors/msis-ai-powered-gaming...
- mjevans 2y agoThat monitor seems to dynamically do things based on data the game legitimately shows a player. For some data, like the health bars, a skill / accessibility leveling feature might be to just let the user pick HOW the game displays that data, to customize the UI layout to their needs. Enemy position highlight based on the minimap vs present location? Yeah, that crosses a clear line, but it's abusing some data the game probably shouldn't have told the player to begin with. What if the minimap reflected the known shape of the world, but only updated with the visible area (standard 'fog of war' mechanic)? Again, it might be within accessibility features to highlight enemies within sight, so I don't see too much issue if the minimap's render state is restricted to the immediate area + what the camera direction could see.
- Animats 2y ago> Why can't the servers distrust the clients? What should a 'client side anti cheat' actually prevent? There are two issues. One is the user seeing things that the server is hiding, such as enemies hidden behind obstacles, by going into "wireframe mode". The other is superhuman performance via computer assistance, or "aimbot hacks". The first is a performance issue. The server can do some occlusion culling to avoid telling the client about invisible enemies, but that adds to the server workload. The second is becoming impossible to fix, since at this point you can have a program looking at the actual video output and helping to aim. (You can now get that in real-world guns.[1]) Attempts to crack down on people whose aim is "too good" result in loud screams from players whose aim really is that good. [1] https://talonprecisionoptics.com/technology/how-it-works/ https://talonprecisionoptics.com/technology/how-it-works/
- sadeshmukh 2y agoThe only feasible solution is to have high-level players compete in physical tournaments or at verified centers, where the authenticity of the player is replaced with some authority. At a high enough level, there is no way to distinguish a really good player from a cheater.
- Xeamek 2y agoDisagree. But it's not really feasible to argue since you need to be on such high level in the first place to honestly engage in 'is this player chesting' conversation. And it's on case-by-case basis
- Draiken 2y agoCan you expand on the disagree? I've watched professional games in SC, CS and DOTA for decades and I definitely agree that pros are indistinguishable from a good cheater (not a rage hacker). One of the issues around this is cheating within pros too. People that are actually good at the game, but use cheats to get even further ahead. These players are already statistical anomalies and even from an experienced player's perspective, you can't tell if they have an amazing game sense (many really do) or he's wall hacking, as an example.
- tonyhart7 2y agowell because realtime online multiplayer game need to be "FAST", I mean really fast sure if you develop platform today we can check token user now with hashtable we have in database but in games ?? You cant verify calculated damage numbers users gave, not fast enough
- raincole 2y agoSorry but I don't think you worked for a multiplayer game in the past 15 years. Verifying damage numbers is no-brainer. The programmers won't discuss "should we verify damage numbers" at all. It's the norm today.
- Xeamek 2y agoYou absolutely can do that and (almost) all games do that already. This type of cheats are DECADES in the past. Today is all about a) enhancing normal behavior with artificial precision, not making any 'illegal' (from game perspective) actions. b) giving player information he isn't supposed to have but that is passed to client for latency sake
- TheFragenTaken 2y agoThe first rule in any software backed by a server, but especially multiplayer games is, you never trust the client. You could have a perfectly deterministic game where every action is validated on the server, be defeated by running the game at half speed.
- DaiPlusPlus 2y agoA bullet-time video game, you say?
- raincole 2y agoBecause the popular cheats aren't "the client says the player shot the enemy". The popular cheats are "the client says the player just clicked at (1030, 534) on the screen", which is a totally valid move, except it's calculated by the cheat instead of the player.
- forrestthewoods 2y ago<sarcasm> oh dang you should be a multiplayer engineer. Sounds like with barely even thinking about the problem space you've solved what thousands of extremely talented and knowledgeable engineers never could! </sarcasm> Servers very much distrust the client. Obviously. That's literally rule #1. Don't trust the client! Comments like yours are extremely irritating. Please don't behave this way with your co-workers. Anyhow, there's all kinds of types of cheats for different kinds of games. There's a variety of mitigations for each kind. I don't think there's a multiplayer shooter on the planet that has fully solved aimbots. For however clever you think you are I promise the cheat makers are much, much more clever. :)
- cabbageicefruit 2y agoTouch grass buddy. In their first sentence they openly admit this isn’t their area of expertise. Hence them asking a question to people who know more than them. “Don’t behave this way to your co-workers” is much better advice for your comment than for GP’s.
- forrestthewoods 2y agoWent outside with my dogs and successfully touched grass. It’s growing back in nicely after a week of rain. “Why can’t you just” guys are extremely irritating. I implore OP to not be a “why can’t you just” guy at work. What is a WCYJGuy? Someone who has no knowledge of a domain but proposes solutions under the implication that there is a simple solution that they are oh so clever to have instantly discovered. It takes a lot of time and effort to explain “no you can not just” to someone who doesn’t have the pre-requisite knowledge.
- deznu 2y agoIt was an honest and inquisitive question, calm down.
- forrestthewoods 2y agoHis arguing bad solutions in other replies suggests otherwise. If you want to inquire then inquire. Don’t propose a bad solution as if it was an easy problem that you solved with nary a thought.
- KK7NIL 2y ago> I don't know enough about X. However I have read several HN articles over the years so I've got at least a basic understanding. I'm sorry but this really does read like the start of a troll post.
- burnished 2y agoIts just regular intellectual curiosity.
- KK7NIL 2y agoSure, but you'll need to do more than "read several HN articles over the years" (presumably completely unrelated to anti-cheats?) to get even a basic understanding of how anti-cheats work, as he went on to demonstrate. I also just thought it was unintentionally funny, like a comedic setup for a stereotypically cocky HN user to comment with great confidence on something way outside of their field of expertise. (not saying that's the case for mjevans)
- burnished 2y agoYes, that stereotype irritates me because it stigmatizes people asking the obvious questions.
- dijit 2y agoDelaying UI interaction until it has been verified by a server that runs at 20fps (60 is uncommon on servers unless theres no AI), with a RTT of 60 ms, means your hitmarker will take 110ms instead of 6ms if rendered locally. Apply that to every interaction that the server has to be authoritative about, movement, reloading. Your game will be unplayable. And if you want to combat aimbotting: your viewport and hit point would have to be server authoritative too. Basically: unless its Stadia or geforce now, this wont work.
- mjevans 2y agoNot delaying UI interaction; though conflict resolution (there are at least two involved clients, each with it's own lagged view of the other, and a server that knows it's own truth) might change the outcome of events. THAT is the part of multiplayer net code I know the least about, mostly because I don't think there is a perfect solution but I am not a subject expert on what works well as an approximation.
- dijit 2y agoThen you have rollbacks and rubberbanding, and UI elements that fired which aren't valid once the physics interactions have resolved serverside. Jt doesn't take much for people to feel like the UI is untrustworthy and “broken”. No game wants to be a jank piece of ass, but theres no good solution here, believe me, we’ve tried.
- mook 2y agoMy understanding is that it's popular now to use rollbacks in fighting games (in combination with delays so the rollback doesn't get too far). Perhaps something like that would be useful, though of course that would depend on the game (and how much data it needs to send between players).
- ThatPlayer 2y agoI think the difference is fighting games are easier to simulate. Part of rollback is to rewind 6 frames and resimulate those 6 frames again with the new input. This basically requires you to be able to run your game at 6x speed consistently. It's also increased memory requirement, because you need to have the game state from those 6 frames ago in memory. These are also reasons you cannot do too many rollback frames without adding delay. I believe the Nintendo Switch never got the rollback update for BlazBlue: Cross Tag Battle because of performance reasons. Fighting games have two (maybe 4 with assists) characters generally at 60fps. That's relatively easy to do. A worse case would be an RTS game: in a fight when each unit's attack needs to be calculated repeatedly. Valorant runs at 128 ticks/second. For the same latency compensation as 6 frames in a fighting game, you would need 13 frames, so you need to be able to simulate the game at 13x speed. And rollback still has janky visuals when conflicts happen. The games I've played will let you choose between smoother visuals with more delay or rollback artifacts with less delay. Generally the default setting is the former.
- psgdev 2y agoSending copies of fake character data isn't a thing because there eventually has to be a flag that tells the client to not render that character that the client hack could simply read. It should be clear that servers already do not trust the client, they do many checks hence you don't see teleportation hacks in games like Counter strike or Valorant. There used to be cheats in the counter strike games like "nospread" where you could have 100% pixel perfect aiming but that was because the the client was trusted however now in most games with some randomness in bullet spray patterns the random seed is different between the client and server so something like "nospread" are no longer possible. You might be stumbling upon "fog of war" that is not sending data to a client unless the enemy player is close to visible which is a thing. It's widely used and I'd say effective in MOBA/MMORPG/RTS games however in FPS games fog of war is many times more computationally expensive which matters at the scale of games these days. It has been a thing for a long time in counter strike with server plugins like "SMAC anti wall hack" or "server side occlusion culling" however the implementations sometimes have not been perfect and require significantly stronger servers. https://github.com/87andrewh/CornerCullingSourceEngine https://github.com/87andrewh/CornerCullingSourceEngine Riot games also implements fog of war at scale in Valorant and has a blog post covering some of the issues they overcame. One thing you can see the gif at the end of the blog post, even though fog of war is effective it is only effective in reducing the effectiveness of wall hacks and wall hacks still provide a significant advantage. https://technology.riotgames.com/news/demolishing-wallhacks-valorants-fog-war https://technology.riotgames.com/news/demolishing-wallhacks-...
- mjevans 2y agoThere would be no such flag. The clients would cull the characters that are out of position. Yes that's some client load for the culling, but it's probably less overhead than 'anti cheat'. The important reason I suggested MULTIPLE clones of a character and only forking new paths off of existing characters in the world is that it should eliminate any information oracle about which of those is the real character.
- psgdev 2y agoThere is a high level of server load for this as well, not only placing these fake characters but making them move and act like real human players so they are believed and then culling them (server would cull them not the client to be clear because how would the client know to cull them without a flag?) only right before they are visible while taking account of lag (ping), interp, packet loss etc.. I definitely could see someone games doing this as a one-off to just catch specific cheaters they are suspicious of to confirm they are cheating (Many 3rd party anti-cheats in counter strike and the 1st party valorant anti-cheat do manual bans based on replay reviews) but also since they already do fog of war someone with wall hack seeing an enemy player pop in for 1 frame before disappearing would make it not effective on a wide scale.
- Hikikomori 2y agoThen all the cheater would need to do is move close to the area of these ghosts and find out which the real player is. Its also going to be very taxing for the server to create realistic ghost players that move around dynamically.
- tumetab1 2y agoThe reason multiplayer servers implicitly trust clients is because it's a cheaper and proven (less risk) solution. The traditional anti-cheat can be just slapped after the game is developed in most games. If the game is very successful then you can just update the game with extra paid protections provided by the anti-cheat tool. The alternative is local game engine that works with a partial game state which is a challenge on it self. If you still can make it work, you will still have to deal with people "modding" the client to gain an advantage. E.g.: enemies are painted red instead of camouflage.
- dijit 2y agoAs someone working in AAA game development, I come across comments like these often, and they never fail to get under my skin. It’s like watching that infamous "Two idiots, one keyboard" scene from CSI—full of confidence, but completely detached from reality. I don’t mean to sound harsh, but it’s tough to tackle this kind of misconception because it’s stated with such certainty that others, who also might not know any better, just take it as fact. Here’s the thing: Multiplayer servers trust clients mainly for performance reasons. In AAA game development, anti-cheat isn’t something we focus on right from the start. It typically becomes a priority post-alpha (and by alpha, I’m talking about an internal milestone that usually spans about a year—not the "alpha" most people think of which is usually closer to an internal "beta", and "public beta" is more like release candidate 1). During that time, the tech team is constantly working on ways to secure the game. (make it work, make it correct*, make it fast). If we were to bake in anti-cheat measures from the very beginning of a project, it would force us to scale back our ambitions. Some might argue that’s a good thing, but the truth is, we’d also risk missing critical milestones like First-Playable or Vertical Slice. You simply can’t tackle everything at once—focus is a measure primarily of what you are not doing, after all. Back when I was working on The Division, we had some deep discussions about using player analytics and even early forms of machine learning to detect "too good" players in real-time. This was in 2014, well before the AI boom. The industry's interest in new anti-cheat methods has only grown since then, I promise you this. At the end of the day, games are all about delivering an experience. That’s the priority, and a solid anti-cheat system is key to ensuring it. Endpoint security is currently the best solution we have because it doesn’t bog down the client with delays or force awkward mechanics like rollbacks or lock-step processing. Plus, it lines up with the (very heavy) optimisations we already do for consoles. Nobody in this industry wants to install a rootkit on your PC if we can avoid it. It’s just the best trade-off (for all parties, especially gamers) given the circumstances. And let's be clear—these solutions are far from cheap. We pay a lot to implement them, even if some marketing material might suggest otherwise.