12 ms·

How fighting games use delay-based and rollback netcode (2019)

121 points by fzeindl 4y ago

notpushkin 4y ago
I guess this one is another victim of HN's title filter?
- Veuxdo 4y ago
  Yep. Makes it read like this is breaking news or something.
andrewmcwatters 4y ago
> We’ve been doing delay-based lockstep networking architecture for 25 years. Quake came out and we switched to a server-client architecture where the server tells everyone what they should be seeing and everyone interpolates it in order to make it look smooth. When this happened, lockstep networking kind of stopped evolving. It’s my opinion that rollback is the next step for lockstep networking and, just like every other discipline in video game development, it’s time for lockstep implementations to evolve. The time investment is just super worth it to me. Rollback has been in the industry for years now. The whole article reads like there's basically large swathes of the fighting game subset of the gaming industry who don't know anything about multiplayer game networking. Which is understandable considering the game industry at-large doesn't know a thing about multiplayer game networking. Go checkout an out-of-the-box build of Unreal Engine 5. Yeah, they've got Nanite wow!, but do they have Actors who don't rubberband? No, and it's stupid and frustrating. It makes the whole engine feel like a joke. Especially compared to Source 1 which had client-side prediction and rollback support over dial-up. Just plain dumb. If you think that's bad, Unity does basically nothing for you, and Godot pretends to have multiplayer support by saying it has ENet bindings. Wow, thanks! It's disappointing that firms can chase physically based rendering, and invest so many man-hours into asset development but can't recreate what's existed for literal decades. Also, AFAIK QuakeWorld was THE client-side prediction implementation to match. Maybe it was developed prior to John Carmack, but I don't know of any major software that did so with a timestep. Everything else followed. This guy doesn't know what he's talking about. Counter-Strike: Source had rollback in 2004 and Yahn Bernier wrote about it in "Latency Compensating Methods in Client/Server In-game Protocol Design and Optimization" in 2001.[1] Keep up dude. [1]: https://www.gamedevs.org/uploads/latency-compensation-in-client-server-protocols.pdf https://www.gamedevs.org/uploads/latency-compensation-in-cli...
- icegreentea2 4y ago
  To be slightly fair to the fighting game community, this article is from 2019. My understanding is that most top-line fighting games now are rollback enabled (the pandemic driven shift to online tournies put significant pressure on developers).
  - djtango 4y ago
    My history is a little hazy but wasn't rollback as a solution driven by the community. Maybe it was SF3 online emulation or something? Or was it Smash Melee
    - cma 4y ago
      At least 2006 https://en.m.wikipedia.org/wiki/GGPO https://en.m.wikipedia.org/wiki/GGPO
      - djtango 4y ago
        Ah yes this is what I had in mind
    - whateveracct 4y ago
      Rollback was added by the community to Melee. With in-game matchmaking. It's a combination of a modded Dolphin + cheats to mod the ISO itself.
      - djtango 4y ago
        Sounds about right. Never was part of the Melee community but I was always impressed at their ingenuity to figure out multiplayer in the face of Nintendo's active community hostility and bonkers multiplayer UX
    - hyperhopper 4y ago
      No, the first major game to use it and push it was Skullgirls. An exemplary example of how to make a great game. Unfortunately it has been though a series of unfortunate events and never gets the recognition it deserves
mellosouls 4y ago
"Explaining how" prefix (maybe autoclipped?) and: (2019)
kibwen 4y ago
Note that having rollback in mind from the very beginning can also impact how a fighting game is designed. The happy path of rollback involves "hiding" network jitter in the startup animations of various moves. But startup animations exist for a reason, to telegraph intent to an opponent and give them a chance to react. For moves with very little startup animation, a rollback correction can update the state of the game to several frames into the hit itself, making many moves unreactable! This feels about as bad as being shot despite ducking behind a wall in an FPS (with basically the same cause). The "solution" is to have a policy whereby you design the game such that you increase the amount of startup animation across the board, to smooth out these wrinkles. In the worst case, this is even used to justify massive built-in input delay on every input! This optimizes the game for being played online, at the expense of being played in-person. The irony being that, while the vast majority of play will happen online, all the most prestigious events (funded by the developers themselves to advertise their game as an e-sport) are in-person.
- haolez 4y ago
  Well, if the ping is high, a fighting game will get extremely deteriorated regardless. It must work really well for low ping scenarios.
- FoomFries 4y ago
  Many people cannot react to several frames, it is usually a predictive choice instead of a reaction. A great example of this is the Millie Blocker written in flash some time ago. It tests the reaction time of a move with a startup of several frames, and as it turns out is not so easy to react to without predicting. http://www.teyah.net/MilliaBlocker_v0.3.swf http://www.teyah.net/MilliaBlocker_v0.3.swf
  - kibwen 4y ago
    Regardless of where we draw the line of reactability, with a sufficient number of rollback frames, a move that was on the reactable side will shift to the unreactable side.
- Blackthorn 4y ago
NelsonMinar 4y ago
I've been playing Path of Exile again lately and it has the most straightforward approach to network code, it's sort of refreshing. The game works via a TCP stream to a server. By default the screen only updates when your client receives a packet from the server; if there's a hiccup the client freezes. This means it's immediately obvious when a network problem is causing trouble and you're never misled by prediction. Also looks crappy on a bad connection. There's a "Client predictive" mode you can enable but it's off by default. Details: https://www.poewiki.net/wiki/Networking_mode https://www.poewiki.net/wiki/Networking_mode I'm on Starlink so I get a lot of hiccups. It's pretty annoying. But then, it's also transparent. Lately the game is working as a diagnostic of Starlink's various failures. It helps the game has a very simple lag meter. https://imgur.com/a/S0VMjUM https://imgur.com/a/S0VMjUM PoE is a lot less demanding than a fighting game; it's mostly single player. Occasional 500ms latency and rubberbanding isn't game-destroying (just annoying). Just an interesting different approach.
- candiddevmike 4y ago
  I don't see what benefit TCP would have over UDP in your example, aside from increasing the lag over your unstable starlink connection. A constant UDP stream of updates is typically the best way to ensure all clients have the correct state.
- nearlyepic 4y ago
  I don't think it's literally TCP, but you're right that it's very much TCP-like behavior. I kept having stuttering in PoE, and eventually realized that NetworkManager was doing wifi channel scans way more often than it should be! Genuinely useful as a tool for diagnosing network conditions.
  - spmurrayzzz 4y ago
    PoE oddly enough does use TCP for all of its netcode, its somewhat unique in this way compared to similar titles. I know they use a few different outgoing ports so its possible they deal with head-of-line blocking issues with parallel streams (not sure about that, just conjecture).
  - 4y ago
umvi 4y ago
Rollback net code sounds just like a modern CPU and branch prediction. So... I guess we can expect spectre/meltdown Mortal Kombat variants?
- andrewmcwatters 4y ago
  One exploit with rewinding game state is making sure game clients don't forge that they were "first" with timing abuse. But the standard strategy is to ignore commands too far back in time. So yes, the strategy does create an exploit vector, but it's easily addressed and attempting to use it as a cheater can have negative effects on your gameplay anyway, so it's not without its cons.
  - andrewmcwatters 4y ago
    Too late to edit my post, but this was achieved with a technique called lag switching[1] back in the day. Now I feel old. :( Modern cheats just do it in software. These days, I think the technique is antiquated in any strategy due to its limited success with modern multiplayer implementations. [1]: https://www.google.com/search?q=lag+switching https://www.google.com/search?q=lag+switching
yazzku 4y ago
Does anybody have a good resource on netcode? Graphics are already arcane enough, netcode seems to be on another level of obscurity.
- andrewmcwatters 4y ago
  Yeah, don't use the term "netcode" which originated from gamers. Look up resources on multiplayer client-side prediction, interpolation, reconciliation, and lag compensation. Together those techniques constitute what laypersons consider "netcode," which ignores still fundamental details like payload construction, serialization, deserialization, broadcasting, heartbeats, etc. For instance, networked visibility affects bandwidth consumption, but no one talks about it or worries about it until you're playing with a cheater with wallhacks. But wait! Modern rendering techniques cull on the GPU. So you have, you guessed it, more and more potential for modern games to broadcast all player positions at all times, unless the developers know to segment player visibility and not network their presence behind non-translucent surfaces.
- 33985868 4y ago
  https://gafferongames.com/categories/game-networking/ https://gafferongames.com/categories/game-networking/ https://fabiensanglard.net/quakeSource/quakeSourceNetWork.php https://fabiensanglard.net/quakeSource/quakeSourceNetWork.ph... https://fabiensanglard.net/quakeSource/quakeSourcePrediction.php https://fabiensanglard.net/quakeSource/quakeSourcePrediction... https://developer.valvesoftware.com/wiki/Source_Multiplayer_Networking https://developer.valvesoftware.com/wiki/Source_Multiplayer_...
- CyberDildonics 4y ago
  Forget 'netcode' - that came from people who don't know about game development. A good networked game will be made to work well at a foundation level. There is no such thing as 'netcode'. It isn't like a shader that has it's own language and is sectioned off from the rest of the program.
nomilk 4y ago
A nice 101 explainer video: https://www.youtube.com/watch?v=1RI5scXYhK0 https://www.youtube.com/watch?v=1RI5scXYhK0
kevinfat 4y ago
Question why is rollback networking good for games? To play devil's advocate with the lag hidden wouldn't this lead me to make decisions without considering the possibility that the opponent has already made a decision that is opposite of the prediction I'm seeing is. With no rollback code then I feel the lag and can properly make decisions based on not only what I see right now but the exact amount of lag I'm feeling. How I play the game should be adjusted based on the latency of the lag. With rollback that feedback of the latency is hidden.
- andrewmcwatters 4y ago
  You're referring to "leading the shot." This was common for games in the late 90s and early 2000s. The problem is that leading the shot has two side effects: 1. what you shoot at in game isn't what you're hitting on the server, (you shot a player, but dirt kicked up from the ground several feet in front of them) and 2. some gameplay mechanics rely on accurate replication. If you throw a grenade at a player and "lead the shot" maybe it sticks to them on the server. It kills them. But on your side, it missed because you personally lag compensated, stuck to a vehicle, and now the vehicle blew up instead and flew off a cliff. Now the game sends back updated entity positions and the vehicle and its occupants are back on the ground and everything just snapped for you. You can rewind game state and make it not visible to players, then run hit calculations and do things like kill players and destroy objects without snapping. Proper lag compensation implementations make it such that infrequently do you actually notice lag, even in exceptionally poor connection environments.
  - kevinfat 4y ago
    So consider a first person shooter with no rollback you're probably going to be aiming slightly ahead of the person to compensate for latency lag. And maybe you'll aim to the other side of the person if you think they changed direction before you saw it actually happen. Here because you feel the lag of your own movements you can take that into account in knowing how far ahead to shoot on the left side or the right side of your opponent. With rollback if you guess they are going to continue moving in the same direction you just aim right on where you see them because of the prediction. If you think they are moving in the other direction you need to guess when they started moving in the other direction. That we are using rollback networking can't get rid of the need for you to guess this. To make that guess you'd like to know how far back they could have changed direction so you have some time interval in which to guess. From your perspective that time interval is based on the difference in time between the server's state now and the last server state you received. But my point is doesn't rollback make it impossible for you to know that while playing the game. Whether you're using rollback or not you as the player need to know intuitively what is the time difference between the server's state and the latest server state received.

121 points by fzeindl 4y ago

notpushkin 4y ago

I guess this one is another victim of HN's title filter?

Veuxdo 4y ago
Yep. Makes it read like this is breaking news or something.

andrewmcwatters 4y ago

> We’ve been doing delay-based lockstep networking architecture for 25 years. Quake came out and we switched to a server-client architecture where the server tells everyone what they should be seeing and everyone interpolates it in order to make it look smooth. When this happened, lockstep networking kind of stopped evolving. It’s my opinion that rollback is the next step for lockstep networking and, just like every other discipline in video game development, it’s time for lockstep implementations to evolve. The time investment is just super worth it to me. Rollback has been in the industry for years now. The whole article reads like there's basically large swathes of the fighting game subset of the gaming industry who don't know anything about multiplayer game networking. Which is understandable considering the game industry at-large doesn't know a thing about multiplayer game networking. Go checkout an out-of-the-box build of Unreal Engine 5. Yeah, they've got Nanite wow!, but do they have Actors who don't rubberband? No, and it's stupid and frustrating. It makes the whole engine feel like a joke. Especially compared to Source 1 which had client-side prediction and rollback support over dial-up. Just plain dumb. If you think that's bad, Unity does basically nothing for you, and Godot pretends to have multiplayer support by saying it has ENet bindings. Wow, thanks! It's disappointing that firms can chase physically based rendering, and invest so many man-hours into asset development but can't recreate what's existed for literal decades. Also, AFAIK QuakeWorld was THE client-side prediction implementation to match. Maybe it was developed prior to John Carmack, but I don't know of any major software that did so with a timestep. Everything else followed. This guy doesn't know what he's talking about. Counter-Strike: Source had rollback in 2004 and Yahn Bernier wrote about it in "Latency Compensating Methods in Client/Server In-game Protocol Design and Optimization" in 2001.[1] Keep up dude. [1]: https://www.gamedevs.org/uploads/latency-compensation-in-client-server-protocols.pdf https://www.gamedevs.org/uploads/latency-compensation-in-cli...

icegreentea2 4y ago
To be slightly fair to the fighting game community, this article is from 2019. My understanding is that most top-line fighting games now are rollback enabled (the pandemic driven shift to online tournies put significant pressure on developers).
- djtango 4y ago
  My history is a little hazy but wasn't rollback as a solution driven by the community. Maybe it was SF3 online emulation or something? Or was it Smash Melee
  - cma 4y ago
    At least 2006 https://en.m.wikipedia.org/wiki/GGPO https://en.m.wikipedia.org/wiki/GGPO
    - djtango 4y ago
      Ah yes this is what I had in mind
  - whateveracct 4y ago
    Rollback was added by the community to Melee. With in-game matchmaking. It's a combination of a modded Dolphin + cheats to mod the ISO itself.
    - djtango 4y ago
      Sounds about right. Never was part of the Melee community but I was always impressed at their ingenuity to figure out multiplayer in the face of Nintendo's active community hostility and bonkers multiplayer UX
  - hyperhopper 4y ago
    No, the first major game to use it and push it was Skullgirls. An exemplary example of how to make a great game. Unfortunately it has been though a series of unfortunate events and never gets the recognition it deserves

mellosouls 4y ago

"Explaining how" prefix (maybe autoclipped?) and: (2019)

kibwen 4y ago

Note that having rollback in mind from the very beginning can also impact how a fighting game is designed. The happy path of rollback involves "hiding" network jitter in the startup animations of various moves. But startup animations exist for a reason, to telegraph intent to an opponent and give them a chance to react. For moves with very little startup animation, a rollback correction can update the state of the game to several frames into the hit itself, making many moves unreactable! This feels about as bad as being shot despite ducking behind a wall in an FPS (with basically the same cause). The "solution" is to have a policy whereby you design the game such that you increase the amount of startup animation across the board, to smooth out these wrinkles. In the worst case, this is even used to justify massive built-in input delay on every input! This optimizes the game for being played online, at the expense of being played in-person. The irony being that, while the vast majority of play will happen online, all the most prestigious events (funded by the developers themselves to advertise their game as an e-sport) are in-person.

haolez 4y ago
Well, if the ping is high, a fighting game will get extremely deteriorated regardless. It must work really well for low ping scenarios.
FoomFries 4y ago
Many people cannot react to several frames, it is usually a predictive choice instead of a reaction. A great example of this is the Millie Blocker written in flash some time ago. It tests the reaction time of a move with a startup of several frames, and as it turns out is not so easy to react to without predicting. http://www.teyah.net/MilliaBlocker_v0.3.swf http://www.teyah.net/MilliaBlocker_v0.3.swf
- kibwen 4y ago
  Regardless of where we draw the line of reactability, with a sufficient number of rollback frames, a move that was on the reactable side will shift to the unreactable side.
Blackthorn 4y ago

NelsonMinar 4y ago

I've been playing Path of Exile again lately and it has the most straightforward approach to network code, it's sort of refreshing. The game works via a TCP stream to a server. By default the screen only updates when your client receives a packet from the server; if there's a hiccup the client freezes. This means it's immediately obvious when a network problem is causing trouble and you're never misled by prediction. Also looks crappy on a bad connection. There's a "Client predictive" mode you can enable but it's off by default. Details: https://www.poewiki.net/wiki/Networking_mode https://www.poewiki.net/wiki/Networking_mode I'm on Starlink so I get a lot of hiccups. It's pretty annoying. But then, it's also transparent. Lately the game is working as a diagnostic of Starlink's various failures. It helps the game has a very simple lag meter. https://imgur.com/a/S0VMjUM https://imgur.com/a/S0VMjUM PoE is a lot less demanding than a fighting game; it's mostly single player. Occasional 500ms latency and rubberbanding isn't game-destroying (just annoying). Just an interesting different approach.

candiddevmike 4y ago
I don't see what benefit TCP would have over UDP in your example, aside from increasing the lag over your unstable starlink connection. A constant UDP stream of updates is typically the best way to ensure all clients have the correct state.
nearlyepic 4y ago
I don't think it's literally TCP, but you're right that it's very much TCP-like behavior. I kept having stuttering in PoE, and eventually realized that NetworkManager was doing wifi channel scans way more often than it should be! Genuinely useful as a tool for diagnosing network conditions.
- spmurrayzzz 4y ago
  PoE oddly enough does use TCP for all of its netcode, its somewhat unique in this way compared to similar titles. I know they use a few different outgoing ports so its possible they deal with head-of-line blocking issues with parallel streams (not sure about that, just conjecture).
- 4y ago

umvi 4y ago

Rollback net code sounds just like a modern CPU and branch prediction. So... I guess we can expect spectre/meltdown Mortal Kombat variants?

andrewmcwatters 4y ago
One exploit with rewinding game state is making sure game clients don't forge that they were "first" with timing abuse. But the standard strategy is to ignore commands too far back in time. So yes, the strategy does create an exploit vector, but it's easily addressed and attempting to use it as a cheater can have negative effects on your gameplay anyway, so it's not without its cons.
- andrewmcwatters 4y ago
  Too late to edit my post, but this was achieved with a technique called lag switching[1] back in the day. Now I feel old. :( Modern cheats just do it in software. These days, I think the technique is antiquated in any strategy due to its limited success with modern multiplayer implementations. [1]: https://www.google.com/search?q=lag+switching https://www.google.com/search?q=lag+switching

yazzku 4y ago

Does anybody have a good resource on netcode? Graphics are already arcane enough, netcode seems to be on another level of obscurity.

andrewmcwatters 4y ago
Yeah, don't use the term "netcode" which originated from gamers. Look up resources on multiplayer client-side prediction, interpolation, reconciliation, and lag compensation. Together those techniques constitute what laypersons consider "netcode," which ignores still fundamental details like payload construction, serialization, deserialization, broadcasting, heartbeats, etc. For instance, networked visibility affects bandwidth consumption, but no one talks about it or worries about it until you're playing with a cheater with wallhacks. But wait! Modern rendering techniques cull on the GPU. So you have, you guessed it, more and more potential for modern games to broadcast all player positions at all times, unless the developers know to segment player visibility and not network their presence behind non-translucent surfaces.
33985868 4y ago
https://gafferongames.com/categories/game-networking/ https://gafferongames.com/categories/game-networking/ https://fabiensanglard.net/quakeSource/quakeSourceNetWork.php https://fabiensanglard.net/quakeSource/quakeSourceNetWork.ph... https://fabiensanglard.net/quakeSource/quakeSourcePrediction.php https://fabiensanglard.net/quakeSource/quakeSourcePrediction... https://developer.valvesoftware.com/wiki/Source_Multiplayer_Networking https://developer.valvesoftware.com/wiki/Source_Multiplayer_...
CyberDildonics 4y ago
Forget 'netcode' - that came from people who don't know about game development. A good networked game will be made to work well at a foundation level. There is no such thing as 'netcode'. It isn't like a shader that has it's own language and is sectioned off from the rest of the program.

nomilk 4y ago

A nice 101 explainer video: https://www.youtube.com/watch?v=1RI5scXYhK0 https://www.youtube.com/watch?v=1RI5scXYhK0

kevinfat 4y ago

Question why is rollback networking good for games? To play devil's advocate with the lag hidden wouldn't this lead me to make decisions without considering the possibility that the opponent has already made a decision that is opposite of the prediction I'm seeing is. With no rollback code then I feel the lag and can properly make decisions based on not only what I see right now but the exact amount of lag I'm feeling. How I play the game should be adjusted based on the latency of the lag. With rollback that feedback of the latency is hidden.

andrewmcwatters 4y ago
You're referring to "leading the shot." This was common for games in the late 90s and early 2000s. The problem is that leading the shot has two side effects: 1. what you shoot at in game isn't what you're hitting on the server, (you shot a player, but dirt kicked up from the ground several feet in front of them) and 2. some gameplay mechanics rely on accurate replication. If you throw a grenade at a player and "lead the shot" maybe it sticks to them on the server. It kills them. But on your side, it missed because you personally lag compensated, stuck to a vehicle, and now the vehicle blew up instead and flew off a cliff. Now the game sends back updated entity positions and the vehicle and its occupants are back on the ground and everything just snapped for you. You can rewind game state and make it not visible to players, then run hit calculations and do things like kill players and destroy objects without snapping. Proper lag compensation implementations make it such that infrequently do you actually notice lag, even in exceptionally poor connection environments.
- kevinfat 4y ago
  So consider a first person shooter with no rollback you're probably going to be aiming slightly ahead of the person to compensate for latency lag. And maybe you'll aim to the other side of the person if you think they changed direction before you saw it actually happen. Here because you feel the lag of your own movements you can take that into account in knowing how far ahead to shoot on the left side or the right side of your opponent. With rollback if you guess they are going to continue moving in the same direction you just aim right on where you see them because of the prediction. If you think they are moving in the other direction you need to guess when they started moving in the other direction. That we are using rollback networking can't get rid of the need for you to guess this. To make that guess you'd like to know how far back they could have changed direction so you have some time interval in which to guess. From your perspective that time interval is based on the difference in time between the server's state now and the last server state you received. But my point is doesn't rollback make it impossible for you to know that while playing the game. Whether you're using rollback or not you as the player need to know intuitively what is the time difference between the server's state and the latest server state received.