9 ms·

How fighting games use delay-based and rollback netcode (2019)

297 points by Kinrany 6y ago

vxNsr 6y ago
I didn’t understand what the author meant by “fighting game” until the end of the first page. Maybe a definition of that term would have helped those of us who aren’t so familiar with different types of gameplay.
- frutiger 6y ago
  The author of any piece should be permitted a certain set of assumptions about their audience. I think as long as a term has a relatively unambiguous Google result (which is true in this case), it’s fair game to be used in public texts.
  - stubish 6y ago
    The arguments the author made make no sense without some sort of a baseline though. 'The majority always play online' for instance points to a fairly tight definition, and may even be tautological (No True fighting game is designed for offline play).
    - shalmanese 6y ago
      The author isn't writing for a hypothetical future HN audience. They're writing for their blog audience who, presumably, 99.9% know what it means.
    - chowells 6y ago
      Every fighting game is designed for offline play, and has been for 25+ years. But that doesn't mean that's where the majority of players are, and that's the point the author is making.
- Impossible 6y ago
  The issue with this is if you don't know what a fighting game is, explaining the genre would take up a lot of space in an article targeted at gamers that know the genre. Sure they could have opened up with "A fighting game, like Street Fighter or Mortal Kombat", but you might not know what those are either. This is like an article about React getting posted and complaining that you don't know what a UI or Javascript is. At some point Google is probably better at filling gaps in assumed information than the article itself.
hn8788 6y ago
If you prefer the same info in a video format, here's an ~8 min video by the owner of a lan center in Korea. https://www.youtube.com/watch?v=0NLe4IpdS1w https://www.youtube.com/watch?v=0NLe4IpdS1w
shalmanese 6y ago
Prediction code for rollback is somewhat akin to branch prediction code in that the dumbest solution works surprisingly well but there's incremental efficiency gains to be had. I wonder if any fighting games have thought to train a neural network per player to try and predict the player's actions N frames ahead. The neural nets could be used for smoother netcode but if the accuracy got high enough, they could, eg: allow for play after one player disconnects, or used to estimate ELO by having the neural nets play each other before the match or be AIs you could play against in offline mode.
- tines 6y ago
  That's an amazing idea. I wonder how long a player would have to play before you could train a neural net to play like them. In a single-player game you could also create an enemy NPC that uses that same neural net, for sort of a "Dark Link" effect where you have to play against yourself. Would be awesome for chess also. Lots of possibilities.
- thaumasiotes 6y ago
  > I wonder if any fighting games have thought to train a neural network per player to try and predict the player's actions N frames ahead. The entire point of playing a fighting game is to attempt to solve this problem. A good player, by necessity, can't be accurately predicted; if they could, they'd be a bad player.
  - Itsdijital 6y ago
    A good player can't be accurately predicted by a human
    - thaumasiotes 6y ago
      First, a good player can't be accurately predicted at all; the conclusion from game theory is direct and clear. This is a case where a strategy involving picking moves at random is superior to any deterministic strategy. Second, your rebuttal is not especially good support for the idea that we should be trying to solve the problem with a technology specifically designed to imitate humans.
Jyaif 6y ago
This "Infil" dude just stole the entirety of the article from Ars Technica (https://arstechnica.com/gaming/2019/10/explaining-how-fighting-games-use-delay-based-and-rollback-netcode/ https://arstechnica.com/gaming/2019/10/explaining-how-fighti...)
- Mavvie 6y ago
  This is sarcasm right?
- shalmanese 6y ago
  > Ricky "Infil" Pusch is a long-time fighting game fan and content creator.
- shakna 6y ago
  > This article has been cross-posted on Ars Technica. - The infil site > Ricky "Infil" Pusch is a long-time fighting game fan and content creator. He wrote The Complete Killer Instinct Guide, an interactive and comprehensive website for learning about Killer Instinct. This article was originally published there. - ArsTechnica Both sites clearly reference where the original source came from. Nothing has been stolen.
  - Jyaif 6y ago
    Thanks. Don't know how I missed the reference to Ars Technica in the article.
- mahathu 6y ago
  Turns out one plus one can be one! https://youtu.be/rOOmQEsdaxg https://youtu.be/rOOmQEsdaxg
zucker42 6y ago
Super smash bros melee is having a bit of an online renaissance due to a combination of the COVID 19 pandemic and the recent implementation of rollback netcode by a community member that quit his job.
- whateveracct 6y ago
  The rollback is truly something else. Although - and I may be crazy - I swear it isn't quite as good as playing local people with delay-based netplay (such that the network delay can "hide" behind the inherent lag of the Gamecube/game itself) But outside of that edge case, it's outstanding.
  - CarVac 6y ago
    The fact that they were able to optimize two frames of input lag away to make the online gameplay have the same latency as offline is itself incredible. As far as I know, even other rollback-equipped fighting games still add a little bit of latency in order to play online without requiring constant rollbacks. That said, Melee is also a game that is very hard on the rollback, because movement and attack startup are far faster than in many fighting games. Dashdancing with rollback leads to some annoying camera judder.
    - zucker42 6y ago
      AFAIK they didn't do anything special to save two frames, just a CRT (which is what's used on LAN) has 2 frames of lag.
      - enneff 6y ago
        CRTs don’t traditionally introduce any latency. It’s digital displays like LCDs that use buffers to pre process images and that add latency.
        Blackthorn 6y ago
        Yes, we know. And melee is almost always played on CRTs locally.
whateveracct 6y ago
Relevant: GGPO, the age-old gold standard of rollback netcode, was recently open-sourced! https://github.com/pond3r/ggpo https://github.com/pond3r/ggpo I haven't used it myself yet - mostly because I'd want Haskell bindings first hah
- reitzensteinm 6y ago
  If your game state is entirely immutable, you're actually most of the way there. I built a rollback multiplayer system in C#, and most of work goes towards having classes that are mutable, but remember their previous state so they can be rewound to any frame in the past second or so. This involves enforcing invariants, persistent collection types, generating code to efficiently walk your entire game state, etc. In Haskell, you just have a list of game states at the end of every frame, pick what you want and go. Of course, you're not going to get C# performance this way, but you should still be able to build something pretty advanced if you give it a dedicated CPU core. There are over a thousand people at this moment playing my indie web games built with rollback networking - if anybody has any questions, happy to answer them!
  - xeromal 6y ago
    Did you follow a pattern for recording and tracking those class states? I stick to the business realm of our industry and Iv'e used a pattern called event sourcing that sounds kind of similar. Every update to an entity is an event and you can rewind/fastfoward any entity to a moment in time. It consumes a lot of memory/db space and if you don't snap shot occasionally, it's time consuming to build the events into a model so I'm curious what you use. Might give me some ideas.
    - reitzensteinm 6y ago
      Event sourcing is exactly what this is. The game state is built from a combination of initial setup parameters (random seed, level etc), and a list of player inputs. I snapshot after every recent frame. The snapshots are stored in the game entities themselves via generated code. Each user accessible entity has a bunch of shadow slots it can copy from or write to. Then a higher level system sends each entity commands such as "store your state in slot 5" or "copy slot 3 over your current state". Old snapshots aren't useful and are discarded. The server itself runs a few seconds behind the client states, does no rewinding, and any player that falls behind that will receive a fresh copy of the server's state.
btown 6y ago
This is an amazing writeup, and something I've often wondered about! For those who want to skip to the code, the article links to these resources: https://www.ggpo.net/ https://www.ggpo.net/ https://drive.google.com/file/d/1nRa3cRBQmKj0-SEyrT_1VNOkPOJWNhVI/view https://drive.google.com/file/d/1nRa3cRBQmKj0-SEyrT_1VNOkPOJ... - a writeup on the GGPO library including code samples of how it works For an example of what GGPO feels like when implemented in a game from day one, this Reddit thread illuminates what's possible: https://www.reddit.com/r/Games/comments/lolukg/guilty_gear_strive_beta_extended_to_feb23_due_to/go6qxi2/?context=3 https://www.reddit.com/r/Games/comments/lolukg/guilty_gear_s... - "I was playing matches with a friend in Denmark from the east coast of the US without any issues whatsoever. It's like black magic. If this kind of netcode is what we have to look forward to with GGS, I literally won't play any other fighting game that doesn't have it from here on." Taking a step back from gaming applications though: I'd love to see a convergence between those researching CRDTs and those who have implemented rollback netcode in the wild. This article from September 2020, by one of the contributors to Google Wave back in the day, is a great overview: https://josephg.com/blog/crdts-are-the-future/ https://josephg.com/blog/crdts-are-the-future/ There's a fascinating confluence of talents needed to generally solve the problem of "how do you keep people in a flow-state when collaborating in the presence of network delay" - which is more applicable now than ever. There's psychology, user experience design, user interface design, deep domain knowledge, fundamental CRDT research, the types of artistry that go into really good gaming netcode, people who have implemented undo stacks in massive desktop applications and know all the warts that arise there, security researchers, distributed systems researchers (the latter two because this will enable decentralized applications in a huge way)... All these people will come together in the coming years to make computing seemingly defy the laws of physics. It's an exciting time to be a software engineer.
rdescartes 6y ago
Does anyone know how Fall Guy's net code was implemented? It's very smooth with 100 participants.
- Jaxkr 6y ago
  Based on Fall Guy's hiring page, which asked for experience with Photon, it's probably this: https://www.photonengine.com/en/pun https://www.photonengine.com/en/pun
  - djmips 6y ago
    Could be using Photon but I don't think Photon solves this particular problem as far as I know. Did you see something in particular that mentions that as a feature?
- 8192kjshad09- 6y ago
  I don't know anything about their net code, but I suspect this is more about the fact that the game is less dependent on precise timing then something like a shooter or Smash.
- monocularvision 6y ago
  60*
- nightowl_games 6y ago
  Fall Guys doesnt heavily use server authority, hence why various hacks exist for it.
thewarrior 6y ago
This piece was really interesting. That you can hide network delays by building a “fake” model of the other player that replicates what a real player would do and rolling back when this prediction fails. So you can train some neural nets on what players do in such situations and you get a certain accuracy. And yet you can keep making this “fake” player better and better until it’s indistinguishable from fighting a human and we arrive a nice little paradox. These kinds of games and algorithms are really good proving grounds for AI. And I begin to get why the author is so passionate about this stuff and it’s possibilities.
- andai 6y ago
  Where did you get the thing about neural networks? I read your comment before reading the article and was really disappointed the prediction "algorithm" (pioneered by GGPO in 2006 [0] and still used today) is literally "assume nothing changed", ie. the opponent is still holding down the same keys as the last frame. [0]: magazine article by the author of GGPO [pdf] https://drive.google.com/file/d/1cV0fY8e_SC1hIFF5E1rT8XRVRzPjU8W9/view https://drive.google.com/file/d/1cV0fY8e_SC1hIFF5E1rT8XRVRzP...
  - reitzensteinm 6y ago
    You shouldn't be disappointed. Nothing has changed is overwhelmingly the correct answer. It is incredibly jarring to assume a remote player takes an action, display them taking that action, then roll back when you realize they didn't. From your local perspective, it looks like they blocked for a few frames, which makes you assume they're going to block, then they flash back to being defenseless, and your attack weirdly goes through even though you anticipated that it did not. Even if you can get your fancy neural net to figure out that the enemy is likely to block you - which is a feat worth writing some papers about - you're still going to be wrong about the frame on which they do it. Was their reaction time 160ms? 176ms? 182ms? If you're right about the action they take and wrong about the frame, that's going to cause each action in the game to have weird timing. You anticipate a block, then it doesn't come so you roll it back, then wait, it actually has come it was just late! The remote player flays around like they don't know what the hell they're doing, and it's not clear to you when you land your hit whether the timer started from when they first telegraphed their block or when the glitch occurred. Your punch appears to land at random. And blocking is an insignificant action - what if you're playing something like DayZ and the neural net decides that some random other neutral player is likely to try to attack you, say because they happened to mouse over you qucikly. It looks like they just shot you for a few frames, but weirdly your health goes down and springs back up again, but you're not going to figure out it is the netcode playing tricks on you. Instead you unload your magazine at the other player that's clearly trying to kill you. And since you are actually shooting now, of course they're going to return fire. Your prediction algorithm just caused two peaceful players to fight to the death. Just because the algorithm is simple doesn't mean it's possible to do better.
syspec 6y ago
Here's another great video on the subject: 8 Frames in 16ms: Rollback Networking in Mortal Kombat and Injustice 2 https://www.youtube.com/watch?v=7jb0FOcImdg https://www.youtube.com/watch?v=7jb0FOcImdg One of my favorite topics, partially because there's not that much literature on the subject but it's so important
- mxxx 6y ago
  Thanks for the share. Very interesting video!
choeger 6y ago
Does that article really end with: > Good netcode matters, period. So let’s talk about it. Because at that point I didn't read anything about netcode, except how important it is.
- no_protocol 6y ago
  Allow 3rd party resources, it has visual aids and buttons.
- barbegal 6y ago
  There are more pages, try clicking the next button at the bottom.
superjan 6y ago
It sounds like rollback could be used to cheat by artificially delaying your outgoing network traffic: the other player has less time to respond. But I doubt I’m the first to get that idea.
- jmgao 6y ago
  Not rolling back can be used to cheat too, by artifically delaying traffic while you're getting comboed to throw people's timing off.
- nightowl_games 6y ago
  If your input is delayed then you have less time to respond too.
esturk 6y ago
Does anyone know if rollback netcode works well in a 20+ player shooter game? I curious because the current craze of Battle Royals seems really taxing to the whole re-simulation of multiple frames within the window of a single frame. Without being any less subtle, games like Apex Legends is known for really bad server performance, latency, and among other things.
- jmgao 6y ago
  There's generally not all that much point to rollback-style netcode in first person shooters, because you can just run asynchronously and do client-side prediction. If the prediction was wrong, it doesn't matter that much: you might have missed a shot and they're not quite where you expected they were. In fighting games, that's completely infeasible, because each player's actions are completely determined by the previous ones: either I hit you, and I'm going to continue into a combo that's very timing specific, comparatively, or you blocked the hit, and I'm doing something completely different.
- SkyPuncher 6y ago
  I'm a pretty avid Call of Duty fan and have been playing ther Battle Royale, Warzone, a lot lately. Prior to this article, I've suspected there was some sort of predictive algorithm that helps make things smooth overall. I've noticed on a few occasions, I'll get shot by someone pre-firing a corner I haven't gone around yet (like sprinting to a corner to make a play). Several times, the kill cam has shown me fully around the corner for the other player. I also have poor latency (my ISP routes poorly) and playing with friends that are geographically far - with servers even further away from me (though relatively close to them). It's a bummer that it happens, but I'm up to 200ms behind realtime. I suspect the game is predicting my movements - some of which put me in very bad positions, unintentionally.
  - meheleventyone 6y ago
    Most FPS games suffer from “peekers advantage” due to the client side simulation of the player moving. Basically they can pop out or move around a corner and shoot before the movement has time to replicate from the server to you. The worse your latency the more advantage another player has.
schmorptron 6y ago
I'm glad this has been done more recently, and even been expanded to other genres, like the upcoming Knockout City which will apparently feature some sort of rollback netcode[1]. I've been playing Lethal League Blaze, and playing against people with 80 to 150 ping is possible, and doesn't feel horrible which is wild, since in other genres that have to rely on dedicated servers to be somewhat decent like CS:GO or Valorant, anything above 50 or 60 ping (to the server, not the other players) is pretty jarring to experience. [1]https://www.nintendoenthusiast.com/knockout-city-is-the-wildest-game-of-dodgeball-youve-ever-seen/ https://www.nintendoenthusiast.com/knockout-city-is-the-wild... (only mentioned in one sentence)
zeknife 6y ago
I was linked here by a friend. There's a lot of thoughts in the discussion about how and if input prediction could be made more sophisticated, maybe using machine learning. This is actually the topic of a degree project I've been working on for the past year, and I hope to finalize the report next month and publish the code on github soon. The tldr is that it seems to take a lot of additional complexity for the improvement you get, and it is currently unknown how "false positive" predictions (ie when the model predicts a new buttonpress that doesn't happen) affect the user experience (but intuitively it seems like it would be worse than the false negatives we currently experience). In other words, we don't know if it would actually feel any better to play even if, say, the prediction accuracy is slightly higher. That said, this is only a first attempt and who knows how much it could be improved. I'll probably post more about it as @zeknife on twitter next month.
hrishi 6y ago
It's amazing how many comments suggest training a neutral net for prediction. Additional fitting through elo ratings I believe are suggested, as is dynamic tuning. To me, having played games with rollback, this takes away one of the key benefits of simplistic rollback: predictability and consistency. Sure it's not the same as offline, but your brain gets pretty good at understanding when it happens and even at predicting it. If an opponent is stationary a few more frames than feels right, or keeps moving, you know to correct for an actions they've already sent - and it's a good estimation most of the time. Over fitting with neural nets IMO removes this consistency without providing much benefit, plus if you've got a strong neural net you might as well train locally against that first.
shaharz195 6y ago
Article didn’t mention conflicts with the rollback approach, but aren’t there are cases where a history re-write causes the local player’s past input to be illegal? For example, game rules state a player can only make a move when opponents is in state A. Local player sees a prediction of Remote player in state A and executes the move, but really the player was in state B. When the rollback happens, there is an illegal game state of local player executing the move against opponent in state B. The situation is similar to a write-write conflict in a multi-leader DB.
- Kinrany 6y ago
  I think the inputs should be sent at the level independent from context: "player pressed SPACE", not "player jumped".
nwmcsween 6y ago
All these comments and this article relies on things that will actively break user experience, why not just limit the players to 'good' connections and have somewhat simple netcode? Having another player 'fast forward' with variable animations sounds like a recipe for frustration

297 points by Kinrany 6y ago

I didn’t understand what the author meant by “fighting game” until the end of the first page. Maybe a definition of that term would have helped those of us who aren’t so familiar with different types of gameplay.

frutiger 6y ago
The author of any piece should be permitted a certain set of assumptions about their audience. I think as long as a term has a relatively unambiguous Google result (which is true in this case), it’s fair game to be used in public texts.
- stubish 6y ago
  The arguments the author made make no sense without some sort of a baseline though. 'The majority always play online' for instance points to a fairly tight definition, and may even be tautological (No True fighting game is designed for offline play).
  - shalmanese 6y ago
    The author isn't writing for a hypothetical future HN audience. They're writing for their blog audience who, presumably, 99.9% know what it means.
  - chowells 6y ago
    Every fighting game is designed for offline play, and has been for 25+ years. But that doesn't mean that's where the majority of players are, and that's the point the author is making.
Impossible 6y ago
The issue with this is if you don't know what a fighting game is, explaining the genre would take up a lot of space in an article targeted at gamers that know the genre. Sure they could have opened up with "A fighting game, like Street Fighter or Mortal Kombat", but you might not know what those are either. This is like an article about React getting posted and complaining that you don't know what a UI or Javascript is. At some point Google is probably better at filling gaps in assumed information than the article itself.

hn8788 6y ago

If you prefer the same info in a video format, here's an ~8 min video by the owner of a lan center in Korea. https://www.youtube.com/watch?v=0NLe4IpdS1w https://www.youtube.com/watch?v=0NLe4IpdS1w

shalmanese 6y ago

Prediction code for rollback is somewhat akin to branch prediction code in that the dumbest solution works surprisingly well but there's incremental efficiency gains to be had. I wonder if any fighting games have thought to train a neural network per player to try and predict the player's actions N frames ahead. The neural nets could be used for smoother netcode but if the accuracy got high enough, they could, eg: allow for play after one player disconnects, or used to estimate ELO by having the neural nets play each other before the match or be AIs you could play against in offline mode.

tines 6y ago
That's an amazing idea. I wonder how long a player would have to play before you could train a neural net to play like them. In a single-player game you could also create an enemy NPC that uses that same neural net, for sort of a "Dark Link" effect where you have to play against yourself. Would be awesome for chess also. Lots of possibilities.
thaumasiotes 6y ago
> I wonder if any fighting games have thought to train a neural network per player to try and predict the player's actions N frames ahead. The entire point of playing a fighting game is to attempt to solve this problem. A good player, by necessity, can't be accurately predicted; if they could, they'd be a bad player.
- Itsdijital 6y ago
  A good player can't be accurately predicted by a human
  - thaumasiotes 6y ago
    First, a good player can't be accurately predicted at all; the conclusion from game theory is direct and clear. This is a case where a strategy involving picking moves at random is superior to any deterministic strategy. Second, your rebuttal is not especially good support for the idea that we should be trying to solve the problem with a technology specifically designed to imitate humans.

Jyaif 6y ago

This "Infil" dude just stole the entirety of the article from Ars Technica (https://arstechnica.com/gaming/2019/10/explaining-how-fighting-games-use-delay-based-and-rollback-netcode/ https://arstechnica.com/gaming/2019/10/explaining-how-fighti...)

Mavvie 6y ago
This is sarcasm right?
shalmanese 6y ago
> Ricky "Infil" Pusch is a long-time fighting game fan and content creator.
shakna 6y ago
> This article has been cross-posted on Ars Technica. - The infil site > Ricky "Infil" Pusch is a long-time fighting game fan and content creator. He wrote The Complete Killer Instinct Guide, an interactive and comprehensive website for learning about Killer Instinct. This article was originally published there. - ArsTechnica Both sites clearly reference where the original source came from. Nothing has been stolen.
- Jyaif 6y ago
  Thanks. Don't know how I missed the reference to Ars Technica in the article.
mahathu 6y ago
Turns out one plus one can be one! https://youtu.be/rOOmQEsdaxg https://youtu.be/rOOmQEsdaxg

zucker42 6y ago

Super smash bros melee is having a bit of an online renaissance due to a combination of the COVID 19 pandemic and the recent implementation of rollback netcode by a community member that quit his job.

whateveracct 6y ago
The rollback is truly something else. Although - and I may be crazy - I swear it isn't quite as good as playing local people with delay-based netplay (such that the network delay can "hide" behind the inherent lag of the Gamecube/game itself) But outside of that edge case, it's outstanding.
- CarVac 6y ago
  The fact that they were able to optimize two frames of input lag away to make the online gameplay have the same latency as offline is itself incredible. As far as I know, even other rollback-equipped fighting games still add a little bit of latency in order to play online without requiring constant rollbacks. That said, Melee is also a game that is very hard on the rollback, because movement and attack startup are far faster than in many fighting games. Dashdancing with rollback leads to some annoying camera judder.
  - zucker42 6y ago
    AFAIK they didn't do anything special to save two frames, just a CRT (which is what's used on LAN) has 2 frames of lag.
    - enneff 6y ago
      CRTs don’t traditionally introduce any latency. It’s digital displays like LCDs that use buffers to pre process images and that add latency.
      - Blackthorn 6y ago
        Yes, we know. And melee is almost always played on CRTs locally.

whateveracct 6y ago

Relevant: GGPO, the age-old gold standard of rollback netcode, was recently open-sourced! https://github.com/pond3r/ggpo https://github.com/pond3r/ggpo I haven't used it myself yet - mostly because I'd want Haskell bindings first hah

reitzensteinm 6y ago
If your game state is entirely immutable, you're actually most of the way there. I built a rollback multiplayer system in C#, and most of work goes towards having classes that are mutable, but remember their previous state so they can be rewound to any frame in the past second or so. This involves enforcing invariants, persistent collection types, generating code to efficiently walk your entire game state, etc. In Haskell, you just have a list of game states at the end of every frame, pick what you want and go. Of course, you're not going to get C# performance this way, but you should still be able to build something pretty advanced if you give it a dedicated CPU core. There are over a thousand people at this moment playing my indie web games built with rollback networking - if anybody has any questions, happy to answer them!
- xeromal 6y ago
  Did you follow a pattern for recording and tracking those class states? I stick to the business realm of our industry and Iv'e used a pattern called event sourcing that sounds kind of similar. Every update to an entity is an event and you can rewind/fastfoward any entity to a moment in time. It consumes a lot of memory/db space and if you don't snap shot occasionally, it's time consuming to build the events into a model so I'm curious what you use. Might give me some ideas.
  - reitzensteinm 6y ago
    Event sourcing is exactly what this is. The game state is built from a combination of initial setup parameters (random seed, level etc), and a list of player inputs. I snapshot after every recent frame. The snapshots are stored in the game entities themselves via generated code. Each user accessible entity has a bunch of shadow slots it can copy from or write to. Then a higher level system sends each entity commands such as "store your state in slot 5" or "copy slot 3 over your current state". Old snapshots aren't useful and are discarded. The server itself runs a few seconds behind the client states, does no rewinding, and any player that falls behind that will receive a fresh copy of the server's state.

btown 6y ago

This is an amazing writeup, and something I've often wondered about! For those who want to skip to the code, the article links to these resources: https://www.ggpo.net/ https://www.ggpo.net/ https://drive.google.com/file/d/1nRa3cRBQmKj0-SEyrT_1VNOkPOJWNhVI/view https://drive.google.com/file/d/1nRa3cRBQmKj0-SEyrT_1VNOkPOJ... - a writeup on the GGPO library including code samples of how it works For an example of what GGPO feels like when implemented in a game from day one, this Reddit thread illuminates what's possible: https://www.reddit.com/r/Games/comments/lolukg/guilty_gear_strive_beta_extended_to_feb23_due_to/go6qxi2/?context=3 https://www.reddit.com/r/Games/comments/lolukg/guilty_gear_s... - "I was playing matches with a friend in Denmark from the east coast of the US without any issues whatsoever. It's like black magic. If this kind of netcode is what we have to look forward to with GGS, I literally won't play any other fighting game that doesn't have it from here on." Taking a step back from gaming applications though: I'd love to see a convergence between those researching CRDTs and those who have implemented rollback netcode in the wild. This article from September 2020, by one of the contributors to Google Wave back in the day, is a great overview: https://josephg.com/blog/crdts-are-the-future/ https://josephg.com/blog/crdts-are-the-future/ There's a fascinating confluence of talents needed to generally solve the problem of "how do you keep people in a flow-state when collaborating in the presence of network delay" - which is more applicable now than ever. There's psychology, user experience design, user interface design, deep domain knowledge, fundamental CRDT research, the types of artistry that go into really good gaming netcode, people who have implemented undo stacks in massive desktop applications and know all the warts that arise there, security researchers, distributed systems researchers (the latter two because this will enable decentralized applications in a huge way)... All these people will come together in the coming years to make computing seemingly defy the laws of physics. It's an exciting time to be a software engineer.

rdescartes 6y ago

Does anyone know how Fall Guy's net code was implemented? It's very smooth with 100 participants.

Jaxkr 6y ago
Based on Fall Guy's hiring page, which asked for experience with Photon, it's probably this: https://www.photonengine.com/en/pun https://www.photonengine.com/en/pun
- djmips 6y ago
  Could be using Photon but I don't think Photon solves this particular problem as far as I know. Did you see something in particular that mentions that as a feature?
8192kjshad09- 6y ago
I don't know anything about their net code, but I suspect this is more about the fact that the game is less dependent on precise timing then something like a shooter or Smash.
monocularvision 6y ago
60*
nightowl_games 6y ago
Fall Guys doesnt heavily use server authority, hence why various hacks exist for it.

thewarrior 6y ago

This piece was really interesting. That you can hide network delays by building a “fake” model of the other player that replicates what a real player would do and rolling back when this prediction fails. So you can train some neural nets on what players do in such situations and you get a certain accuracy. And yet you can keep making this “fake” player better and better until it’s indistinguishable from fighting a human and we arrive a nice little paradox. These kinds of games and algorithms are really good proving grounds for AI. And I begin to get why the author is so passionate about this stuff and it’s possibilities.

andai 6y ago
Where did you get the thing about neural networks? I read your comment before reading the article and was really disappointed the prediction "algorithm" (pioneered by GGPO in 2006 [0] and still used today) is literally "assume nothing changed", ie. the opponent is still holding down the same keys as the last frame. [0]: magazine article by the author of GGPO [pdf] https://drive.google.com/file/d/1cV0fY8e_SC1hIFF5E1rT8XRVRzPjU8W9/view https://drive.google.com/file/d/1cV0fY8e_SC1hIFF5E1rT8XRVRzP...
- reitzensteinm 6y ago
  You shouldn't be disappointed. Nothing has changed is overwhelmingly the correct answer. It is incredibly jarring to assume a remote player takes an action, display them taking that action, then roll back when you realize they didn't. From your local perspective, it looks like they blocked for a few frames, which makes you assume they're going to block, then they flash back to being defenseless, and your attack weirdly goes through even though you anticipated that it did not. Even if you can get your fancy neural net to figure out that the enemy is likely to block you - which is a feat worth writing some papers about - you're still going to be wrong about the frame on which they do it. Was their reaction time 160ms? 176ms? 182ms? If you're right about the action they take and wrong about the frame, that's going to cause each action in the game to have weird timing. You anticipate a block, then it doesn't come so you roll it back, then wait, it actually has come it was just late! The remote player flays around like they don't know what the hell they're doing, and it's not clear to you when you land your hit whether the timer started from when they first telegraphed their block or when the glitch occurred. Your punch appears to land at random. And blocking is an insignificant action - what if you're playing something like DayZ and the neural net decides that some random other neutral player is likely to try to attack you, say because they happened to mouse over you qucikly. It looks like they just shot you for a few frames, but weirdly your health goes down and springs back up again, but you're not going to figure out it is the netcode playing tricks on you. Instead you unload your magazine at the other player that's clearly trying to kill you. And since you are actually shooting now, of course they're going to return fire. Your prediction algorithm just caused two peaceful players to fight to the death. Just because the algorithm is simple doesn't mean it's possible to do better.

syspec 6y ago

Here's another great video on the subject: 8 Frames in 16ms: Rollback Networking in Mortal Kombat and Injustice 2 https://www.youtube.com/watch?v=7jb0FOcImdg https://www.youtube.com/watch?v=7jb0FOcImdg One of my favorite topics, partially because there's not that much literature on the subject but it's so important

mxxx 6y ago
Thanks for the share. Very interesting video!

choeger 6y ago

Does that article really end with: > Good netcode matters, period. So let’s talk about it. Because at that point I didn't read anything about netcode, except how important it is.

no_protocol 6y ago
Allow 3rd party resources, it has visual aids and buttons.
barbegal 6y ago
There are more pages, try clicking the next button at the bottom.

superjan 6y ago

It sounds like rollback could be used to cheat by artificially delaying your outgoing network traffic: the other player has less time to respond. But I doubt I’m the first to get that idea.

jmgao 6y ago
Not rolling back can be used to cheat too, by artifically delaying traffic while you're getting comboed to throw people's timing off.
nightowl_games 6y ago
If your input is delayed then you have less time to respond too.

esturk 6y ago

Does anyone know if rollback netcode works well in a 20+ player shooter game? I curious because the current craze of Battle Royals seems really taxing to the whole re-simulation of multiple frames within the window of a single frame. Without being any less subtle, games like Apex Legends is known for really bad server performance, latency, and among other things.

jmgao 6y ago
There's generally not all that much point to rollback-style netcode in first person shooters, because you can just run asynchronously and do client-side prediction. If the prediction was wrong, it doesn't matter that much: you might have missed a shot and they're not quite where you expected they were. In fighting games, that's completely infeasible, because each player's actions are completely determined by the previous ones: either I hit you, and I'm going to continue into a combo that's very timing specific, comparatively, or you blocked the hit, and I'm doing something completely different.
SkyPuncher 6y ago
I'm a pretty avid Call of Duty fan and have been playing ther Battle Royale, Warzone, a lot lately. Prior to this article, I've suspected there was some sort of predictive algorithm that helps make things smooth overall. I've noticed on a few occasions, I'll get shot by someone pre-firing a corner I haven't gone around yet (like sprinting to a corner to make a play). Several times, the kill cam has shown me fully around the corner for the other player. I also have poor latency (my ISP routes poorly) and playing with friends that are geographically far - with servers even further away from me (though relatively close to them). It's a bummer that it happens, but I'm up to 200ms behind realtime. I suspect the game is predicting my movements - some of which put me in very bad positions, unintentionally.
- meheleventyone 6y ago
  Most FPS games suffer from “peekers advantage” due to the client side simulation of the player moving. Basically they can pop out or move around a corner and shoot before the movement has time to replicate from the server to you. The worse your latency the more advantage another player has.

schmorptron 6y ago

I'm glad this has been done more recently, and even been expanded to other genres, like the upcoming Knockout City which will apparently feature some sort of rollback netcode[1]. I've been playing Lethal League Blaze, and playing against people with 80 to 150 ping is possible, and doesn't feel horrible which is wild, since in other genres that have to rely on dedicated servers to be somewhat decent like CS:GO or Valorant, anything above 50 or 60 ping (to the server, not the other players) is pretty jarring to experience. [1]https://www.nintendoenthusiast.com/knockout-city-is-the-wildest-game-of-dodgeball-youve-ever-seen/ https://www.nintendoenthusiast.com/knockout-city-is-the-wild... (only mentioned in one sentence)

zeknife 6y ago

I was linked here by a friend. There's a lot of thoughts in the discussion about how and if input prediction could be made more sophisticated, maybe using machine learning. This is actually the topic of a degree project I've been working on for the past year, and I hope to finalize the report next month and publish the code on github soon. The tldr is that it seems to take a lot of additional complexity for the improvement you get, and it is currently unknown how "false positive" predictions (ie when the model predicts a new buttonpress that doesn't happen) affect the user experience (but intuitively it seems like it would be worse than the false negatives we currently experience). In other words, we don't know if it would actually feel any better to play even if, say, the prediction accuracy is slightly higher. That said, this is only a first attempt and who knows how much it could be improved. I'll probably post more about it as @zeknife on twitter next month.

hrishi 6y ago

It's amazing how many comments suggest training a neutral net for prediction. Additional fitting through elo ratings I believe are suggested, as is dynamic tuning. To me, having played games with rollback, this takes away one of the key benefits of simplistic rollback: predictability and consistency. Sure it's not the same as offline, but your brain gets pretty good at understanding when it happens and even at predicting it. If an opponent is stationary a few more frames than feels right, or keeps moving, you know to correct for an actions they've already sent - and it's a good estimation most of the time. Over fitting with neural nets IMO removes this consistency without providing much benefit, plus if you've got a strong neural net you might as well train locally against that first.

shaharz195 6y ago

Article didn’t mention conflicts with the rollback approach, but aren’t there are cases where a history re-write causes the local player’s past input to be illegal? For example, game rules state a player can only make a move when opponents is in state A. Local player sees a prediction of Remote player in state A and executes the move, but really the player was in state B. When the rollback happens, there is an illegal game state of local player executing the move against opponent in state B. The situation is similar to a write-write conflict in a multi-leader DB.

Kinrany 6y ago
I think the inputs should be sent at the level independent from context: "player pressed SPACE", not "player jumped".

nwmcsween 6y ago

All these comments and this article relies on things that will actively break user experience, why not just limit the players to 'good' connections and have somewhat simple netcode? Having another player 'fast forward' with variable animations sounds like a recipe for frustration