8 ms·
> We’ve been doing delay-based lockstep networking architecture for 25 years. Quake came out and we switched to a server-client architecture where the server te
by andrewmcwatters 4y ago
> We’ve been doing delay-based lockstep networking architecture for 25 years. Quake came out and we switched to a server-client architecture where the server tells everyone what they should be seeing and everyone interpolates it in order to make it look smooth. When this happened, lockstep networking kind of stopped evolving. It’s my opinion that rollback is the next step for lockstep networking and, just like every other discipline in video game development, it’s time for lockstep implementations to evolve. The time investment is just super worth it to me.
Rollback has been in the industry for years now. The whole article reads like there's basically large swathes of the fighting game subset of the gaming industry who don't know anything about multiplayer game networking.
Which is understandable considering the game industry at-large doesn't know a thing about multiplayer game networking. Go checkout an out-of-the-box build of Unreal Engine 5. Yeah, they've got Nanite wow!, but do they have Actors who don't rubberband? No, and it's stupid and frustrating. It makes the whole engine feel like a joke. Especially compared to Source 1 which had client-side prediction and rollback support over dial-up. Just plain dumb.
If you think that's bad, Unity does basically nothing for you, and Godot pretends to have multiplayer support by saying it has ENet bindings. Wow, thanks!
It's disappointing that firms can chase physically based rendering, and invest so many man-hours into asset development but can't recreate what's existed for literal decades.
Also, AFAIK QuakeWorld was THE client-side prediction implementation to match. Maybe it was developed prior to John Carmack, but I don't know of any major software that did so with a timestep. Everything else followed. This guy doesn't know what he's talking about.
Counter-Strike: Source had rollback in 2004 and Yahn Bernier wrote about it in "Latency Compensating Methods in Client/Server In-game Protocol Design and Optimization" in 2001.[1] Keep up dude.
[1]: https://www.gamedevs.org/uploads/latency-compensation-in-client-server-protocols.pdf https://www.gamedevs.org/uploads/latency-compensation-in-cli...
- icegreentea2 4y agoTo be slightly fair to the fighting game community, this article is from 2019. My understanding is that most top-line fighting games now are rollback enabled (the pandemic driven shift to online tournies put significant pressure on developers).
- djtango 4y agoMy history is a little hazy but wasn't rollback as a solution driven by the community. Maybe it was SF3 online emulation or something? Or was it Smash Melee
- cma 4y agoAt least 2006 https://en.m.wikipedia.org/wiki/GGPO https://en.m.wikipedia.org/wiki/GGPO
- djtango 4y agoAh yes this is what I had in mind
- whateveracct 4y agoRollback was added by the community to Melee. With in-game matchmaking. It's a combination of a modded Dolphin + cheats to mod the ISO itself.
- djtango 4y agoSounds about right. Never was part of the Melee community but I was always impressed at their ingenuity to figure out multiplayer in the face of Nintendo's active community hostility and bonkers multiplayer UX
- hyperhopper 4y agoNo, the first major game to use it and push it was Skullgirls. An exemplary example of how to make a great game. Unfortunately it has been though a series of unfortunate events and never gets the recognition it deserves
- bskrobisz 4y ago"Series of unfortunate events" is an understatement, but I would say it gets due recognition--it's an indie game over a decade old and still earns its slot at major fighting game tournaments and (after said events' hopeful end) still has support from developers and community alike.
- djtango 4y agoThe big difference is in the net code. While I can remember and justify all of my decisions about networking from DOOM through Quake, the bottom line is that I was working with the wrong basic assumptions for doing a good internet game. My original design was targeted at <200ms connection latencies. People that have a digital connection to the internet through a good provider get a pretty good game experience. Unfortunately, 99% of the world gets on with a slip or ppp connection over a modem, often through a crappy overcrowded ISP. This gives 300+ ms latencies, minimum. Client. User's modem. ISP's modem. Server. ISP's modem. User's modem. Client. God, that sucks. Ok, I made a bad call. I have a T1 to my house, so I just wasn't familliar with PPP life. I'm adressing it now. The first move was to scrap the current net code. It was based on a reliable stream as its original primitive (way back in qtest), then was retrofited to have an unreliable sideband to make internet play feasable. It was a big mess, so I took it out and shot it. The new code has the unreliable packet as its basic primitive, and all the complexities that entails is now visible to the main code instead of hidden under the net api. This is A Good Thing. Goodbye phantom unconnected players, messages not getting through, etc It was Carmack - I've always loved this article https://fabiensanglard.net/quakeSource/johnc-log.aug.htm https://fabiensanglard.net/quakeSource/johnc-log.aug.htm
- andrewmcwatters 4y agoYep, and because of his .plan files, and no other historical details around the technology, I think it's safe to say he invented it. In 1999, QuakeWorld was the first, to my knowledge, implementation of client-side prediction in a first-person shooter. But I'm almost certain there were rudimentary client-side prediction models that existed before John Carmack, which is why I say I think he was the first to "invent" it, but frankly I'm not so sure about that. I mean think about terminals, as an example. I do think with near certainty he was the first to invent the client-server with client-side prediction architecture in time stepped software, or more specifically in an FPS.
- egb 4y agoIs there any good template/library for modern multiplayer netcode for Unreal or Unity or anything else, or is it truly everybody rolls their own as the secret sauce in their AAA games?
- andrewmcwatters 4y agoNo, people literally still roll their own multiplayer networking in 2023. Seriously. Unreal Engine 5's out-of-the-box multiplayer code isn't production ready, and they're probably the "best" of the major engines people use today. It pales in comparison to older engines from other game companies. It's effectively unusable.
- bee_rider 4y agoThis seems wild in the free-to-play era, where multiplayer with in game purchases is, like, the main business model. Maybe Unreal figures the multiplayer implementation is the metric on which the game studios compete, so there’s no hope to commoditizing it, or something like that?
- andrewmcwatters 4y agoIt's also why so many multiplayer games are bad. Wanna know what's worse? Unreal Engine's architecture is unreliable by design.[1] Basically, Tim Sweeney's approach for Actor movement wasn't to replicate state accurately, but just trust that clients would eventually get the right data. This isn't the same as eventual consistency mind you, because game state constantly changes in fast-paced games. You can have whole time spans where Actors have absolutely no accurate player position replication. [1]: https://docs.google.com/document/d/1KGLbEfHsWANTTgUqfK6rkpFYDGvnZYj-BN18sxq6LPY/edit https://docs.google.com/document/d/1KGLbEfHsWANTTgUqfK6rkpFY...
- bob1029 4y ago> It makes the whole engine feel like a joke. 1000%. Multiplayer in UE5 (especially the documentation) feels like it's bad on purpose. Nevermind the actually crappy net code - Just try writing your own online services provider... I've tried on many occasions to get excited about these engines and started a project with them. Unfortunately, when the entire realm of multiplayer feels half-assed my hopes are dashed catastrophically. Right now, I am spending every moment of my free time developing my own game/engine from scratch. I honestly don't give a shit about Nanite, AAA graphics, et. al. Complex multiplayer experiences are far more compelling to me than raytraced graphics and sub-pixel triangles. I think the exact nature of the net code is much more intimately linked to the game concept than anything in the rendering pipeline (for a multiplayer experience like Quake or Counterstrike). You will never get a perfect fit if you outsource this.
- andrewmcwatters 4y agoYes! I completely agree. You're not alone. There's a lot of us out here slogging through writing portions of game software we don't want to only because it's necessary to create the experiences we want that can't be delivered in a timely fashion with existing technology.
- rollcat 4y ago> 1000%. Multiplayer in UE5 (especially the documentation) feels like it's bad on purpose. Nevermind the actually crappy net code - Just try writing your own online services provider... Intriguing. Makes even more sense that Frost Giant Studios (who are using UE5 for their upcoming RTS) have written their own networking code (with rollback!), and thrown away other big parts of the engine.
- bob1029 4y agoI'd be very interested in reading more about FGS and what they've been up to around UE5. The idea of incrementally defenestrating UE5 in favor of in-house components is compelling to me. This path definitely works a LOT better in a team setting where you have actual artists looking for things to work on. I don't know how the licensing actually works out around this though... Assume you went full ship of Theseus on UE5 and replaced 100% of it with in-house. Will Epic Games be happy walking away from this, or is there some fundamental legal component that we know we'd never screw with?
- whstl 4y agoI know I'm going against the grain and this is a super-unpopular opinion, but I really feel like there's room for more specialized engines and more independent engine middleware/tooling in 2023. The "game engines are impossibly hard, so we won't get new ones" discussion only ever talks about Unreal and Unity, where editor, engine and technology are all vertically integrated and the flexibility requires a lot of compromises. Plus you can't really replicate those two, even if you have a great team and 10 years of work on your hands (Godot). This oligopoly is super unhealthy for the industry, IMO.
- andrewmcwatters 4y agoI don't think it's unpopular. We need more and better game software, what exists today is unsatisfactory for a plethora of reasons. I'm personally interested in this because I ship Planimeter Game Engine 2D, which comes with client-side prediction standard, and it's the largest pure-Lua game engine on GitHub.[1] [1]: https://github.com/Planimeter/game-engine-2d https://github.com/Planimeter/game-engine-2d
- throwaway17_17 4y agoI think in most game development related forums, tutorials, etc, you will be correct about the unpopularity of building a new game engine. However, it’s definitely not a lost cause in general. I also agree that a vast majority of the ‘engines are impossible’ are predicated on Unreal/Unity as a baseline. I look at Arch Systemworks and see them using Unreal for their heavily stylized games (Guilty Gear, Drgaonball FighterZ, etc) and watch the various talks and articles about them having to bend over backward and work to fit Unreal’s expectations to make the engine render and animate in the manner they require and always wonder if a custom engine would not make workflow and maintenance simpler. I wonder the same for more specific genres, like RTS and HD2D-style JRPGs, where a custom engine would conceivably be a better long term option for developers. I always go back to Mike Acton talking at the first HandmadeCon, where he is talking about small dev teams and their benefits in specializing tools and wonder if that logic should be applied to engines as a whole.
- whstl 4y ago
- BoorishBears 4y agoOne of my biggest dings against Godot is not having support for Photon Networking's newest products Photon has a new group of network libraries that are essentially the evolution of "rubber band networking" that's been default for networking middleware for the last decade One is especially interesting in that it uses IL weaving to provide a completely deterministic engine for simulating gameplay based on inputs only: https://www.photonengine.com/quantum# https://www.photonengine.com/quantum#
- andrewmcwatters 4y agoDeterminism from inputs was basically a part of QuakeWorld, and how the "VCR" system in id Tech 1 worked. You could replay sessions because the inputs were what were networked, not other state. They're selling 24-year-old tech. :)
- BoorishBears 4y agoIf you can make 24 year old tech turnkey, you have a business
- andrewmcwatters 4y agoYes! I think this is proven very much by the industry widely still being far behind the curve of what is possible.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- rychco 4y agoRollback hasn’t been in fighting games for very long at all. It’s a relatively recent addition to the genre, and has resulted in dramatically improved online play. It’s still new enough that it’s even touted as a feature & selling point in marketing. Within the fighting game community, online play/tournaments have typically been regarded as a joke & not representative of skill at all because delay-based networking woes. However nowadays, with rollback, online tournaments are held regularly, more accessible than ever, and regarded more highly than before. Rollback is greatly helping the genre stay alive.
- anonymousab 4y agoFighting games have had rollback netcode since at least SF2 HD Remix in 2008. In terms of online fighting games, it has been around for more than half of their existence. The problem has been the lack of adoption/prioritization, and probably more than a little NotInventedHere-syndrome.
- benzor 4y agoIt's indeed sad to see. Game networking techniques have been known for multiple decades and yet still haven't made their into any of the major publicly available engines (notably Unreal and Unity). What gives? As a game dev of 10+ years myself, I have a few theories: 1. Multiplayer networking is the "secret sauce" that creates moats/barriers to entry for incumbent studios. Think Rocket League or Fortnite or StarCraft 2. The tech exists, it's functionally well understood, but hard to implement at a high level of quality. Why not keep it to yourself? There is more money in holding onto your game revenue monopoly than trying to sell the network tech. 2. Game genres differ greatly in their networking needs. Rollback is great for fighting games but that's about it. StarCraft uses delay based deterministic lockstep. FPS games use (typically) ad-hoc server authoritative state syncs with client side prediction. Other complex games use full determinism with rollback, sending only inputs over the wire. Some cheapo indie games using client authoritative models (open to cheating but easy to implement, arguably fine for coop games). There are even more variants and blended approaches but you get the picture: there is no one size fits all approach and many are mutually exclusive so it's harder to package into an engine as a comprehensive solution. 3. Networking a game properly involves very leaky abstractions. It is impossible to write gameplay code for a networked game without understanding the nuance of the network model. This makes it substantially harder to develop the game, and this hurts the major game engines' marketability, with both major players Unity and Unreal guilty of selling themselves as "look ma no code required" solutions. Similar to point 1, not worth the money to sell this. I don't see a great way out of this unfortunately. The only good networking middleware I know of is Photon and they're not exactly an easy to use product either. Hopefully we see better open source tooling in the future.
- andrewmcwatters 4y agoI agree. Sell the physically based rendering sizzle, keep the multiplayer networking sauce for yourself. I don't buy the different genres differ greatly in their networking needs, though. Dota 2 uses the same networking architecture as Counter-Strike. Both of those games use the same underlying networking techniques as RuneScape 2 (Old School Runescape). Those are a MOBA, FPS, and an MMORPG respectively. You use lag compensation (now people refer to it as rollback) in FPS games too. Yahn famously talked about it in his paper. In fact, any type of granular timing-dependent gameplay requires lag compensation, otherwise you end up with situations like players leading their shots or missing entirely (Halo 1 PC). In Halo 1 PC, they actually broadcast a hit sound from the server if you actually did make bullet contact. And those weren't trace-based (hitscan) either. Halo 1 bullets actually had travel time. So, Bungie basically didn't do lag compensation correctly, and hacked a solution on top that gave players feedback to help smooth over multiplayer. Because bullets had travel time, too, it just felt natural. But make no mistake. It was still wrong, because when bullets landed on your client, they didn't land on your target on the server. You might have hit the player on the server, heard the feedback, but saw dirt fly up from the ground. Basically, all real-time games have the same networking requirements. If you made a 2D chess game today, you'd still have a game loop and networking would be a part of it, sending payloads probably only when players took their turns. That same basic design applies to FPS games. Chess doesn't need client-side prediction, but as soon as you want to allow other players to see you moving pieces around based on cursor position, you're sending real-time data by payload over a time stepped game loop. You just don't need to predict anything, nor do you need to interpolate or anything else.