9 ms·

Reminds me of the excellent post about the architecture of multi-player in the original Age of Empires game: https://www.gamasutra.com/view/feature/131503/1500

Reminds me of the excellent post about the architecture of multi-player in the original Age of Empires game: https://www.gamasutra.com/view/feature/131503/1500_archers_on_a_288_network_.php https://www.gamasutra.com/view/feature/131503/1500_archers_o... You had high-ping 28.8k internet, and you wanted to be able to have immersive battles in a world with hundreds of units controlled by several players. There's no way you could send real-time information about the status of every object in the game. So they didn't. They made the game simulation completely deterministic and then simply shipped user input around on the network. User input was processed on a slight (couple hundred milliseconds) delay (to allow for network transmission speeds) and executed simultaneously in "game time" on everybody's machine, so the simulation stayed in perfect sync. Voila, no worrying about lag.

PaulStatezny 6y ago
StarCraft 2, the modern "king of RTS games" so to speak, still uses that model! Determinism based on user input, plus a deliberate lag (~150ms?) in handling user input to allow time for player inputs to be received.
- AnotherGoodName 6y ago
  And the original Starcraft 1 was the pioneer of the technique fwiw. They even let you set the input delay in the options to account for lag.
- hinkley 6y ago
  Aren't we saying that all of the popular 'Real Time Strategy" games are in fact turn-based, but there are 7 turns per second?
  - MattRix 6y ago
    Hah, nope. A delay of 150ms doesn't mean that events only happen every 150ms. Also even if events were sent in 150ms chunks, the offset of time within that chunk would still matter. With that said, most games do have a maximum tick rate, so in a very misleading way you could say they are turn-based ;)
JRKrause 6y ago
If we keep a complete record of all player inputs we can also replay the exact game after the fact, a great feature that arises as a consequence of this design choice.
- malexw 6y ago
  A great feature not only for players, but for developers as well. If every change to game state goes through an event system you can reproduce any bug or crash as long as you're careful about making sure that the event stream is recorded before the process is killed.
- hinkley 6y ago
  This ended up being an enabling technology for e-sports too, didn't it? The observer gets the replay log (cheap), and broadcasts the pixels out (expensive) to the world with very little impact on the game play for the active participants.
  - setr 6y ago
    No one watches sc2 matches by loading up the game and running a live replay -- it's all video streaming/streamcasting. Casters do use a live (delayed) replay, letting them poke at any arbitrary object in the game, but a video stream of that is what gets broadcasted. Low bandwidth broadcast isn't really relevant I think to esport success -- the "complete" replay aspect to even 1 viewer (the caster) however might be
    - xsmasher 6y ago
      I think you're in agreement with the post you're replying to. You both say that the game state is sent to a non-player machine that creates the actual video stream seen by the viewers.
      - setr 6y ago
        Sort of -- im saying that for streaming, this doesnt much; deterministic lockstop does little for the general viewership, and saves relatively little processing compared to recording the stream directly on the player's machines. Much more notable is the impact it has on casting, where it's presence is clearly felt -- casters are granted much more freedom, and can be much more thorough and descriptive, when complete replays are available (versus simple dumb video). We agree about the technogoly -- we disagree on its impact
EamonnMR 6y ago
I built an implementation of this architecture in Python if you want to see how it works in code: https://github.com/eamonnmr/OpenLockStep https://github.com/eamonnmr/OpenLockStep
algorithmsRcool 6y ago
I recall that Supreme Commander used the same technique, a fully deterministic sim run in lockstep by all multiplayer peers. For it's era, it had very very high scale for an RTS game. (thousands of units and enormous game maps (the biggest of which took 15+ mins to send naval units across) If I recall correctly, there was another benefit. You could save a multiplayer game by just having all peers dump the current game state to disk and resume it later. Since a big SupCom game could take HOURS this was a great feature. However, there were 2 issues with SupCom's implementation. 1. It wasn't perfect and sometime hours deep into a big game, you would get a "desync" error and it was unrecoverable. 2. Everyone needed to be able to run the sim at the same speed. So the game could only progress at the speed of the slowest player's computer. And SupCom was VERY compute intensive.
- planetsmashr 6y ago
  I had no idea that supcom worked that way! Thanks for the insight. Your comment brings back fond memories of playing huge supcom games at a local LAN cafe. Those were some of the best times I've had gaming.
  - algorithmsRcool 6y ago
    I couldn't count the number of hours I spent playing that game with friends. Never played anything else like it!
- SolarNet 6y ago
  > For it's era, it had very very high scale for an RTS game > So the game could only progress at the speed of the slowest player's computer. And SupCom was VERY compute intensive. It might surprise you to learn that these statements are still true! SupCom:FA (still playable by buying a copy and through the third party FAF launcher; they have even fixed most of the desyncs, it's very stable these days) is probably still the largest scale RTS out there (even planetary annihilation only feels larger). And it still requires a top-tier gaming rig to play. The reason for this is pretty straightforward, CPUs actually haven't gotten faster for well programmed, cache aware, tight-looped CPU bound operations in the last 15 years. And in fact there are arguments for why something like SupCom:FA would be difficult to make today given the industry's reliance on game engines. And games are still rarely threaded more than SupCom was (e.g. game logic thread, render thread), so until someone can get back to the low level excellence of SupCom:FA and then design and implement a concurrent game logic loop for it that actually parallelizes well, I think it will remain the king of high scale RTS for quite a while. https://youtu.be/DEhp5eCwgSI?t=3304 https://youtu.be/DEhp5eCwgSI?t=3304 (to see why this game is awesome)
gameswithgo 6y ago
That is what you try to do with most games even today, just to make it harder to cheat, and various other reasons (easy to replay games, its the natural thing to do, etc) There are times gameplay needs don't allow that model to be perfect, but it is the norm.
- setr 6y ago
  Afaik its only the norm in RTS games -- almost everything else is client/server, with the server providing (and authority of) full game state. In which case the client is just a view into server simulation -- logic moved to the client only so far as to enable faster input response through local interpolation (with provisions to undo, if the server decides thats actually wasn't allowed), and that's also where information may be leaked to enable cheating. The main reason for the deterministic lockstep architecture is the cost of sending game state -- in a game with 10 player characters and some particles, it's sufficiently cheap. In a game with 10,000 player characters, not so much. Determinism is a bitch to maintain, so afaik, no one tries to do so unless they must.
nemetroid 6y ago
Fighting games take this one step further through "rollback" netcode. Instead of delaying the game while waiting, the game predicts what the opponent's input is going to be. If it turns out when the input arrives that the prediction was wrong, the game state is rolled back, the correct input is applied, and the game fast-forwarded to where it was. Here's an extensive writeup about the concept: https://ki.infil.net/w02-netcode.html https://ki.infil.net/w02-netcode.html
- exlurker 6y ago
  Many realtime online games use the same techniques today. For a very understandable walkthrough of the concept, I recommend Gabriel Gambretta's excellent 4 part series: https://gabrielgambetta.com/client-server-game-architecture.html https://gabrielgambetta.com/client-server-game-architecture....
  - monocasa 6y ago
    Interestingly enough, your brain does the similar things, playing games with temporal relationships and fixing it all back up after the fact in a way that you normally can't notice. https://en.wikipedia.org/wiki/Chronostasis https://en.wikipedia.org/wiki/Chronostasis
    - lioeters 6y ago
      I was talking to someone about a related topic a couple weeks ago, how the brain "fudges" with the experience of time to compensate for the latency of perception. But I couldn't remember what it was called. Thanks to the link above, I found the exact phrase: neural antedating/backdating. Ah, it's nice to have resolved that question.
- smabie 6y ago
  Except Super Smash Bros, which has the worst netcode and online play possible. Nintendo is this weird combination of incompetent and brilliant that I don't really understand.
rgoulter 6y ago
Somewhat related, one of the developers discussed some of the issues he faced when working with the re-mastered "Age of Empires: Definitive Edition". https://richg42.blogspot.com/2018/02/some-lessons-learned-while-developing.html https://richg42.blogspot.com/2018/02/some-lessons-learned-wh...
infinite8s 6y ago
Nice! I used this approach for a turn based gamed engine, but with the input processing lag it totally makes sense for real-time games as well!

PaulStatezny 6y ago

StarCraft 2, the modern "king of RTS games" so to speak, still uses that model! Determinism based on user input, plus a deliberate lag (~150ms?) in handling user input to allow time for player inputs to be received.

AnotherGoodName 6y ago
And the original Starcraft 1 was the pioneer of the technique fwiw. They even let you set the input delay in the options to account for lag.
hinkley 6y ago
Aren't we saying that all of the popular 'Real Time Strategy" games are in fact turn-based, but there are 7 turns per second?
- MattRix 6y ago
  Hah, nope. A delay of 150ms doesn't mean that events only happen every 150ms. Also even if events were sent in 150ms chunks, the offset of time within that chunk would still matter. With that said, most games do have a maximum tick rate, so in a very misleading way you could say they are turn-based ;)

JRKrause 6y ago

If we keep a complete record of all player inputs we can also replay the exact game after the fact, a great feature that arises as a consequence of this design choice.

malexw 6y ago
A great feature not only for players, but for developers as well. If every change to game state goes through an event system you can reproduce any bug or crash as long as you're careful about making sure that the event stream is recorded before the process is killed.
hinkley 6y ago
This ended up being an enabling technology for e-sports too, didn't it? The observer gets the replay log (cheap), and broadcasts the pixels out (expensive) to the world with very little impact on the game play for the active participants.
- setr 6y ago
  No one watches sc2 matches by loading up the game and running a live replay -- it's all video streaming/streamcasting. Casters do use a live (delayed) replay, letting them poke at any arbitrary object in the game, but a video stream of that is what gets broadcasted. Low bandwidth broadcast isn't really relevant I think to esport success -- the "complete" replay aspect to even 1 viewer (the caster) however might be
  - xsmasher 6y ago
    I think you're in agreement with the post you're replying to. You both say that the game state is sent to a non-player machine that creates the actual video stream seen by the viewers.
    - setr 6y ago
      Sort of -- im saying that for streaming, this doesnt much; deterministic lockstop does little for the general viewership, and saves relatively little processing compared to recording the stream directly on the player's machines. Much more notable is the impact it has on casting, where it's presence is clearly felt -- casters are granted much more freedom, and can be much more thorough and descriptive, when complete replays are available (versus simple dumb video). We agree about the technogoly -- we disagree on its impact

EamonnMR 6y ago

I built an implementation of this architecture in Python if you want to see how it works in code: https://github.com/eamonnmr/OpenLockStep https://github.com/eamonnmr/OpenLockStep

algorithmsRcool 6y ago

I recall that Supreme Commander used the same technique, a fully deterministic sim run in lockstep by all multiplayer peers. For it's era, it had very very high scale for an RTS game. (thousands of units and enormous game maps (the biggest of which took 15+ mins to send naval units across) If I recall correctly, there was another benefit. You could save a multiplayer game by just having all peers dump the current game state to disk and resume it later. Since a big SupCom game could take HOURS this was a great feature. However, there were 2 issues with SupCom's implementation. 1. It wasn't perfect and sometime hours deep into a big game, you would get a "desync" error and it was unrecoverable. 2. Everyone needed to be able to run the sim at the same speed. So the game could only progress at the speed of the slowest player's computer. And SupCom was VERY compute intensive.

planetsmashr 6y ago
I had no idea that supcom worked that way! Thanks for the insight. Your comment brings back fond memories of playing huge supcom games at a local LAN cafe. Those were some of the best times I've had gaming.
- algorithmsRcool 6y ago
  I couldn't count the number of hours I spent playing that game with friends. Never played anything else like it!
SolarNet 6y ago
> For it's era, it had very very high scale for an RTS game > So the game could only progress at the speed of the slowest player's computer. And SupCom was VERY compute intensive. It might surprise you to learn that these statements are still true! SupCom:FA (still playable by buying a copy and through the third party FAF launcher; they have even fixed most of the desyncs, it's very stable these days) is probably still the largest scale RTS out there (even planetary annihilation only feels larger). And it still requires a top-tier gaming rig to play. The reason for this is pretty straightforward, CPUs actually haven't gotten faster for well programmed, cache aware, tight-looped CPU bound operations in the last 15 years. And in fact there are arguments for why something like SupCom:FA would be difficult to make today given the industry's reliance on game engines. And games are still rarely threaded more than SupCom was (e.g. game logic thread, render thread), so until someone can get back to the low level excellence of SupCom:FA and then design and implement a concurrent game logic loop for it that actually parallelizes well, I think it will remain the king of high scale RTS for quite a while. https://youtu.be/DEhp5eCwgSI?t=3304 https://youtu.be/DEhp5eCwgSI?t=3304 (to see why this game is awesome)

gameswithgo 6y ago

That is what you try to do with most games even today, just to make it harder to cheat, and various other reasons (easy to replay games, its the natural thing to do, etc) There are times gameplay needs don't allow that model to be perfect, but it is the norm.

setr 6y ago
Afaik its only the norm in RTS games -- almost everything else is client/server, with the server providing (and authority of) full game state. In which case the client is just a view into server simulation -- logic moved to the client only so far as to enable faster input response through local interpolation (with provisions to undo, if the server decides thats actually wasn't allowed), and that's also where information may be leaked to enable cheating. The main reason for the deterministic lockstep architecture is the cost of sending game state -- in a game with 10 player characters and some particles, it's sufficiently cheap. In a game with 10,000 player characters, not so much. Determinism is a bitch to maintain, so afaik, no one tries to do so unless they must.

nemetroid 6y ago

Fighting games take this one step further through "rollback" netcode. Instead of delaying the game while waiting, the game predicts what the opponent's input is going to be. If it turns out when the input arrives that the prediction was wrong, the game state is rolled back, the correct input is applied, and the game fast-forwarded to where it was. Here's an extensive writeup about the concept: https://ki.infil.net/w02-netcode.html https://ki.infil.net/w02-netcode.html

exlurker 6y ago
Many realtime online games use the same techniques today. For a very understandable walkthrough of the concept, I recommend Gabriel Gambretta's excellent 4 part series: https://gabrielgambetta.com/client-server-game-architecture.html https://gabrielgambetta.com/client-server-game-architecture....
- monocasa 6y ago
  Interestingly enough, your brain does the similar things, playing games with temporal relationships and fixing it all back up after the fact in a way that you normally can't notice. https://en.wikipedia.org/wiki/Chronostasis https://en.wikipedia.org/wiki/Chronostasis
  - lioeters 6y ago
    I was talking to someone about a related topic a couple weeks ago, how the brain "fudges" with the experience of time to compensate for the latency of perception. But I couldn't remember what it was called. Thanks to the link above, I found the exact phrase: neural antedating/backdating. Ah, it's nice to have resolved that question.
smabie 6y ago
Except Super Smash Bros, which has the worst netcode and online play possible. Nintendo is this weird combination of incompetent and brilliant that I don't really understand.

rgoulter 6y ago

Somewhat related, one of the developers discussed some of the issues he faced when working with the re-mastered "Age of Empires: Definitive Edition". https://richg42.blogspot.com/2018/02/some-lessons-learned-while-developing.html https://richg42.blogspot.com/2018/02/some-lessons-learned-wh...

infinite8s 6y ago

Nice! I used this approach for a turn based gamed engine, but with the input processing lag it totally makes sense for real-time games as well!