7 ms·

1500 Archers on a 28.8: Network Programming in Age of Empires and Beyond (2001)

196 points by larsiusprime 5y ago

deleted 5y ago
[deleted]
netfl0 5y ago
I’ve been waiting 20 years for this article. I used to have 28.8. I’d play this game for hours on there! I was always in awe this ever worked, even more so over time as software has become more complex….
- larsiusprime 5y ago
  Appropriate, since it was written 20 years ago :)
orionblastar 5y ago
I remember that BBSes added SLIRP and PPP in order to dial in and access their Internet connection. It was mostly pay BBSes with MajorBBS or some commercial software. We played Doom that way as well.
- bombcar 5y ago
  The mention of Doom reminds me - games with this form of perfect synchronization allowed for VERY small replays - basically all it needed to record was keypresses and time stamps.
  - skocznymroczny 5y ago
    Unfortunately it also struggles with patching, because due to balance changes the commands don't work anymore as they are supposed to. For example back in the day Starcraft balanced Spawning Pool to cost 200 minerals instead of 150, to delay early zergling rushes by several seconds (which makes a massive difference). But if watching an old replay, it will still try to build the Spawning Pool at 150 minerals and from there everything breaks.
    - bombcar 5y ago
      Some games handled this with “rule sets” but is was usually handled by referencing the version it was played at.
  - setr 5y ago
    A replay is just another client to sync with — just arbitrarily far behind.. :)
- ericbarrett 5y ago
  Did anybody else here play Doom on on DWANGO [0]? My claim to fame was a 4-way deathmatch with Thresh [1], myself, and two other randos (like me) on Doom 2, I think the map E2M7? The rules were first to a score of 100, rocket launchers only. Final score: Thresh, 100; me, 7; other opponents -3 and -6. [0] https://doomwiki.org/wiki/DWANGO https://doomwiki.org/wiki/DWANGO [1] https://en.m.wikipedia.org/wiki/Thresh_(gamer) https://en.m.wikipedia.org/wiki/Thresh_(gamer)
  - aidenn0 5y ago
    I played a couple of times at my friend's house; I wasn't allowed to tie up the phone line
  - serf 5y ago
    thanks for reminding me of DWANGO. I did.
- jabl 5y ago
  I recall using SLIRP back in the day with an dialup ISP that provided shell accounts in the base package, but charged extra for PPP (IIUC this was a fairly common business model). With SLIRP I was able to access internet directly from my PC without paying for PPP access.
the__alchemist 5y ago
I love it. I wonder how much of this they applied to AOE2 DE, which is currently thriving, and uses the same basic game mechanics, artwork, and balance as "AOK".
- timpattinson 5y ago
  It's the same engine with some updates, except game messages are routed through a centralised server. All of the stuff about lockstep simulation still applies.
  - fakedang 5y ago
    Isn't the engine modified to accommodate for the new physics elements?
    - paavohtl 5y ago
      If you're referring to building destruction, they are just baked animations; the physics simulation was done in an external 3D modelling program. Which is why the enhanced graphics pack for the game is about 30 gigabytes.
      - fakedang 5y ago
        Thanks. That makes a ton of sense now.
orangeoxidation 5y ago
(2001) - Not some strange content collector, Gamasutra rebranded Definitely worth a read. The deterministic simulation is easy to understand (though not to do), clever and allows for great networking performance. Afaik this approach is still useful and used today (e.g. in fighting games).
- halvnykterist 5y ago
  While lockstep is useful for some games, it was never really a good choice for fighting games due to the fast and timing sensitive nature. Rollback is an improvement over it that adds speculative execution for remote inputs, this is what should be the standard, although Japanese developers have only recently gotten on board with it.
  - dilap 5y ago
    I guess it depends on how high a level you're looking at it from. Both lockstep and rollback are variants of the general approach of "shared deterministic simulation" where you're mostly just sending inputs. ("Mostly" 'cuz on reconnect or desync or whatever you'll probably want to send full state.) I really enjoyed this GDC talk on rollback networking: https://www.youtube.com/watch?v=7jb0FOcImdg&t=536s https://www.youtube.com/watch?v=7jb0FOcImdg&t=536s (From 2019.)
  - PicassoCTs 5y ago
    One improvement is, to train a model on the player inputs and thus predict behavior. It takes some time, but if done right, the game after a while feels really "instantaneous" even when it comes to fast paced high-apm action.
29athrowaway 5y ago
[2001]
codetrotter 5y ago
The article seems to have been cut short. When I viewed it, it ends at the beginning of a sentence starting with the words "None of". Here's an older snapshot of the article from back when the site was still called Gamasutra, that includes the full text of the article: https://web.archive.org/web/20180719170411/https://www.gamasutra.com/view/feature/131503/1500_archers_on_a_288_network_.php?print=1 https://web.archive.org/web/20180719170411/https://www.gamas...
- Macha 5y ago
  Oh, didn't realise they had renamed themselves until I saw this post. Reading the article on why, the reason makes sense, though I never made that connection until reading the article. They could have picked a less generic new name though, feels a bit like the naming equivalent of the flat logos which currently have an article on the front page too.
- larsiusprime 5y ago
  I'll contact the editor in chief and see if it can be fixed.
  - codetrotter 5y ago
    Thank you :)
bawolff 5y ago
Its amazing what people can do with very limited constraints (8 player real time multiplayer on 28.8 modem!) when they have to. I probably can't even do a (modern) google search on a 28.8 k modem without it feeling absolutely painful.
- vvanders 5y ago
  Subspace was another impressive one from around that era. 50-100 concurrent players on 28.8/56k. They used a totally different technique(dead reckoning + game design that was about prediction) but it was pretty smooth given what average pings where back then.
  - docflabby 5y ago
    It's also amazingly still going - I've played since 2003 (beta launch was 1996) - probably the oldest "action" MMO https://store.steampowered.com/app/352700/Subspace_Continuum/ https://store.steampowered.com/app/352700/Subspace_Continuum... The techniques used in the game ended up being replicated in many other games and apps. Its networking was also key to the low lag it is entirety based on a custom UDP network stack that can send both reliable and unreliable packets.
- rytcio 5y ago
  It’s more sad than amazing. The amount of computing resources we waste for no real reaso. We have crazy fast technology that runs software just as slow as they did in the 2000s
  - fakedang 5y ago
    Unfortunately as much as I love it, the new AoE2 DE has fallen to the same trap. Old AoE2 uses a fraction of the power (and disk space) compared to the new one.
    - jcelerier 5y ago
      Eh I disagree. On the same computer original aoe2 would lag like crazy even on a LAN with 4-5 players and pop. limit to 200, with AoE2 DE it's muuuch smoother
aetherspawn 5y ago
I found RTS3 had a lot more network issues than AoE2. It would more consistently fail to hole punch (fail to join a lobby game with certain people for no discernible rhyme or reason) and/or drop connection during play. I could play RTS3 with most people around the world, it seemed, except a few of my best friends. They could individually play with random people all over the world, but not each other or myself, depending on the day of the week or something like that. Perhaps their decision to move away from DirectPlay was a mistake or premature optimization!
- Agentlien 5y ago
  For anyone wondering, RTS3 refers to Age of Mythology from 2002.
  - aetherspawn 5y ago
    Ah, I thought they meant AoE 3. My bad.
- jeroenhd 5y ago
  Age of Mythology always worked for me on LAN and usually WAN without too much port forwarding. AoE (both 1 and 2) never seemed to work well over the internet for me without loads of manually forwarded ports. The rewrites/updates have definitely helped in this situation but I think the AoM code was built for and tested on networking hardware that was a lot more error prone than modern routers are, which in turn affects how it behaves even today. Clever NAT bypasses definitely work better now than they did fifteen or twenty years ago, but they're definitely not guaranteed to work still. The rise in CGNAT usage also paints a grim picture for anyone playing video games together. I've also had my fair share of peering issues between ISPs. At one point I discovered that I could send UDP packets to my friend, but my friend could not send UDP packets to me, no matter the port forwards and even after forcing the modem into bridge mode. TCP worked fine in both directions, though. Something upstream seemed to filter the packets out, I'm guessing to prevent DDoS attacks, so whenever we played P2P games one of us needed to run some kind of VPN. IPv6 can solve many of the problems I remember having to deal with setting up games back in the day, but most games have become cloud-only anyway. The age of custom servers and P2P gaming is over, killed by lootboxes and shitty NAT implementations.
aligray 5y ago
For anyone looking for further reading on the subject, this article popped up on hn a few months ago which I thought was fascinating: https://ki.infil.net/w02-netcode.html https://ki.infil.net/w02-netcode.html
oreally 5y ago
This article is wayy outdated in the context of today's games. Look to https://gafferongames.com/post/deterministic_lockstep/ https://gafferongames.com/post/deterministic_lockstep/ instead, you get bonus animated gifs there.
- reitzensteinm 5y ago
  What specifically is outdated? If you look under the hood of modern RTS games they're all going to look pretty similar, and this article has a bunch of info Gaffer's doesn't. I would read both.
  - oreally 5y ago
    For example you really don't need to enforce a 200ms turn rate, and their notion of what makes a 'communications turn' is unclear given the terms gamedevs use nowadays.
    - reitzensteinm 5y ago
      Yeah, I used 30hz for my turn rate, but the principles are the same. The terminology is dated, but there's some real gold in there. The bits about deer facing snowballing out of sync errors, out of simulation code corrupting simulation state, comparing state dumps... this is exactly what your life will be like if you build your games this way. I know this because I have (although complicated by the addition of rollback). The Gaffer article barely scratches the surface.
- TacticalCoder 5y ago
  This article paved the way. FWIW I had written a fully deterministic (non networked though) game engine around... 1991. HN user "dfan" here wrote a fully deterministic game engine for the DOS game Terra Nova in 1996. There were probably others but resources about deterministic game engines were very rare for nearly nobody had one so this article about AoE was a gold mine for many. Some terms may have changed, but the techniques are basically identical. Blizzard in 2002 with Warcraft III had basically solved that issue for RTS once and forever too (and maybe for Starcraft before that?) and not much has changed since then.
blunte 5y ago
I don't know if Paul was the only guy working on the network code, but I believe he was responsible for most or all of it. He wrote that code when he was 18 or 19 years old I believe. But unlike some of the other devs at Ensemble, he never had a big ego, and he was just generally nice and humble. He (much) later created the early mobile hit, Words With Friends, which had many copycats later.
ddek 5y ago
As a fairly high rated AoE2 player (~1800), I can attest that the network architecture has not stood the test of time. Lockstep causes tonnes of problems. Desynchronisation is still possible, when that happens the simulation needs to backtrack (I think?), which can cause units to jump multiple tiles. Because your machine is resolving the movements of other players, you run into problems with more players. The wire protocol is widely known and easily hacked, in a way that's hard to detect in game. AoE2 DE is still lockstep, but through a server. All players are connected to the same server. This has slightly improved on the previous P2P, but not really addressed anything.
- Jare 5y ago
  > Desynchronisation is still possible, when that happens the simulation needs to backtrack (I think? Sounds more like client-side prediction to smooth things, than actual simulation desync. I have a hard time believing a deterministic game with such a large state was able to backtrack and resync the sim back to determinism. I did not think that lockstep RTS games would need client-side prediction (the indirect and long-term commands in RTS helps hides latency), but I guess if your gameplay lends itself to high APMs then it becomes necessary.
  - marginalia_nu 5y ago
    Well you could just have state snapshots taken at regular intervals and then verify that both sides' hashes agree. It's only a couple of thousand entities so it's really not so bad. For you can probably ignore those that haven't deviated from the previous snapshot (and that would account for a state reconstruction taking time). RTS games have a replay mechanisms at least as far back as StarCraft: Brood War, so a journal of player inputs are likely going to be recorded anyway.
    - Jare 5y ago
      Yes the journal of player inputs sure, but the intermediate states is a different matter. AoE2's state size is peanuts for a modern machine, but I would say that at the time, it was quite significant and it would be too costly to store it on the fly. I certainly did not dare try that in the RTS-like deterministic games I worked on (Commandos and Praetorians)
Agentlien 5y ago
For those who may not be aware, the "upcoming" RTS3 mentioned in the article is Age of Mythology from 2002.
gvv 5y ago
Very cool. Just a remark, I'm using Brave and the website won't load until I remove the shield.
Ronll 5y ago
"Part of the difficulty was conceptual -- programmers were not used to having to write code that used the same number of calls to random within the simulation (yes, the random numbers were seeded and synchronized as well)." What do they mean, "same number of calls to random within the simulation", didn't get it
- thatjoeoverthr 5y ago
  The random number generator needs to give the same outcomes on both ends so that you don’t need to transfer results. To this end, you have to use the same deterministic generator code, seed and ensure you call it the same number of times on each system. This last point is because queries to a random number generator also change its state. If system A and B start with some seed, but system A calls it 30 times and system B 29 times, the next call will yield a different result on each system.
- transpostmeta 5y ago
  If you call a pseudo-random number generator, you get the next random number in a strictly defined sequence of numbers derived from the seed. So the generator has state, and needs to be called exactly the same number of times in the same order on all clients to make sure that the random events happen the same on all clients.
- short_sells_poo 5y ago
  I suspect it pertains to the fact that’s most RNGs tend to be pseudo-random number generators that maintain an internal state. Each call of the RNG mutates the internal state. If you have two processes, one calls the RNG 2 times and the other calls it 5 times, the process states will have diverged and the simulation is no longer consistent between them. This is true even if the simulation ends up using 1 value out of those RNG calls, because the RNG state itself is implicitly part of the simulation.
EamonnMR 5y ago
I've built a couple of demos using this article as a guide to understanding how the net architecture works. It's pretty fun to run through. Making your sim deterministic is a neat challenge.
triska 5y ago
Complementing this article, there's also a great post mortem article about Age of Empires: https://www.gamedeveloper.com/pc/the-game-developer-archives-postmortem-ensemble-s-age-of-empires- https://www.gamedeveloper.com/pc/the-game-developer-archives... In the section "Things That Went Wrong Or We Could Have Done Better", it mentions as the final point an insight I often think about: "8. We didn’t take enough advantage of automated testing. In the final weeks of development, we set up the game to automatically play up to eight computers against each other. Additionally, a second computer containing the development platform and debugger could monitor each computer that took part. These games, while randomly generated, were logged so that if anything happened, we could reproduce the exact game over and over until we isolated the problem. The games themselves were allowed to run at an accelerated speed and were left running overnight. This was a great success and helped us in isolating very hard to reproduce problems. Our failure was in not doing this earlier in development; it could have saved us a great deal of time and effort. All of our future production plans now include automated testing from Day One."
- R0b0t1 5y ago
  Very clever but it only tests actions the AI takes.
  - alternatetwo 5y ago
    The AI actions are the same actions that players would take manually, they just get computed by the AI instead of a brain. Afaik the AI logic gets processed on the host instance, and the rest just follow the commands sent through the network.
  - shaggyfrog 5y ago
    Based on my personal experience of automated testing in AAA games, even if you "only" count the overlap with the actions humans take, it's a huge overlap in practice. And while there are edge cases only humans find, automated testing finds bugs humans don't. The point of that section is that the earlier you find bugs, the quicker you fix them, and you end up with less overall pain for the length of the project.
  - 5y ago
nulptr 5y ago
If each command would only be executed two communication turns in the future, and a communication turn is ~200ms, isn't there ~600ms of lag for each command? > For RTS games, 250 milliseconds of command latency was not even noticed -- between 250 and 500 msec was very playable, and beyond 500 it started to be noticeable. The article mentions this as well, so was ~600ms latency just expected? Why doesn't it work to execute commands _one_ communication turn in the future?

196 points by larsiusprime 5y ago

deleted 5y ago

[deleted]

I’ve been waiting 20 years for this article. I used to have 28.8. I’d play this game for hours on there! I was always in awe this ever worked, even more so over time as software has become more complex….

larsiusprime 5y ago
Appropriate, since it was written 20 years ago :)

orionblastar 5y ago

I remember that BBSes added SLIRP and PPP in order to dial in and access their Internet connection. It was mostly pay BBSes with MajorBBS or some commercial software. We played Doom that way as well.

bombcar 5y ago
The mention of Doom reminds me - games with this form of perfect synchronization allowed for VERY small replays - basically all it needed to record was keypresses and time stamps.
- skocznymroczny 5y ago
  Unfortunately it also struggles with patching, because due to balance changes the commands don't work anymore as they are supposed to. For example back in the day Starcraft balanced Spawning Pool to cost 200 minerals instead of 150, to delay early zergling rushes by several seconds (which makes a massive difference). But if watching an old replay, it will still try to build the Spawning Pool at 150 minerals and from there everything breaks.
  - bombcar 5y ago
    Some games handled this with “rule sets” but is was usually handled by referencing the version it was played at.
- setr 5y ago
  A replay is just another client to sync with — just arbitrarily far behind.. :)
ericbarrett 5y ago
Did anybody else here play Doom on on DWANGO [0]? My claim to fame was a 4-way deathmatch with Thresh [1], myself, and two other randos (like me) on Doom 2, I think the map E2M7? The rules were first to a score of 100, rocket launchers only. Final score: Thresh, 100; me, 7; other opponents -3 and -6. [0] https://doomwiki.org/wiki/DWANGO https://doomwiki.org/wiki/DWANGO [1] https://en.m.wikipedia.org/wiki/Thresh_(gamer) https://en.m.wikipedia.org/wiki/Thresh_(gamer)
- aidenn0 5y ago
  I played a couple of times at my friend's house; I wasn't allowed to tie up the phone line
- serf 5y ago
  thanks for reminding me of DWANGO. I did.
jabl 5y ago
I recall using SLIRP back in the day with an dialup ISP that provided shell accounts in the base package, but charged extra for PPP (IIUC this was a fairly common business model). With SLIRP I was able to access internet directly from my PC without paying for PPP access.

the__alchemist 5y ago

I love it. I wonder how much of this they applied to AOE2 DE, which is currently thriving, and uses the same basic game mechanics, artwork, and balance as "AOK".

timpattinson 5y ago
It's the same engine with some updates, except game messages are routed through a centralised server. All of the stuff about lockstep simulation still applies.
- fakedang 5y ago
  Isn't the engine modified to accommodate for the new physics elements?
  - paavohtl 5y ago
    If you're referring to building destruction, they are just baked animations; the physics simulation was done in an external 3D modelling program. Which is why the enhanced graphics pack for the game is about 30 gigabytes.
    - fakedang 5y ago
      Thanks. That makes a ton of sense now.

orangeoxidation 5y ago

(2001) - Not some strange content collector, Gamasutra rebranded Definitely worth a read. The deterministic simulation is easy to understand (though not to do), clever and allows for great networking performance. Afaik this approach is still useful and used today (e.g. in fighting games).

halvnykterist 5y ago
While lockstep is useful for some games, it was never really a good choice for fighting games due to the fast and timing sensitive nature. Rollback is an improvement over it that adds speculative execution for remote inputs, this is what should be the standard, although Japanese developers have only recently gotten on board with it.
- dilap 5y ago
  I guess it depends on how high a level you're looking at it from. Both lockstep and rollback are variants of the general approach of "shared deterministic simulation" where you're mostly just sending inputs. ("Mostly" 'cuz on reconnect or desync or whatever you'll probably want to send full state.) I really enjoyed this GDC talk on rollback networking: https://www.youtube.com/watch?v=7jb0FOcImdg&t=536s https://www.youtube.com/watch?v=7jb0FOcImdg&t=536s (From 2019.)
- PicassoCTs 5y ago
  One improvement is, to train a model on the player inputs and thus predict behavior. It takes some time, but if done right, the game after a while feels really "instantaneous" even when it comes to fast paced high-apm action.

29athrowaway 5y ago

[2001]

codetrotter 5y ago

The article seems to have been cut short. When I viewed it, it ends at the beginning of a sentence starting with the words "None of". Here's an older snapshot of the article from back when the site was still called Gamasutra, that includes the full text of the article: https://web.archive.org/web/20180719170411/https://www.gamasutra.com/view/feature/131503/1500_archers_on_a_288_network_.php?print=1 https://web.archive.org/web/20180719170411/https://www.gamas...

Macha 5y ago
Oh, didn't realise they had renamed themselves until I saw this post. Reading the article on why, the reason makes sense, though I never made that connection until reading the article. They could have picked a less generic new name though, feels a bit like the naming equivalent of the flat logos which currently have an article on the front page too.
larsiusprime 5y ago
I'll contact the editor in chief and see if it can be fixed.
- codetrotter 5y ago
  Thank you :)

bawolff 5y ago

Its amazing what people can do with very limited constraints (8 player real time multiplayer on 28.8 modem!) when they have to. I probably can't even do a (modern) google search on a 28.8 k modem without it feeling absolutely painful.

vvanders 5y ago
Subspace was another impressive one from around that era. 50-100 concurrent players on 28.8/56k. They used a totally different technique(dead reckoning + game design that was about prediction) but it was pretty smooth given what average pings where back then.
- docflabby 5y ago
  It's also amazingly still going - I've played since 2003 (beta launch was 1996) - probably the oldest "action" MMO https://store.steampowered.com/app/352700/Subspace_Continuum/ https://store.steampowered.com/app/352700/Subspace_Continuum... The techniques used in the game ended up being replicated in many other games and apps. Its networking was also key to the low lag it is entirety based on a custom UDP network stack that can send both reliable and unreliable packets.
rytcio 5y ago
It’s more sad than amazing. The amount of computing resources we waste for no real reaso. We have crazy fast technology that runs software just as slow as they did in the 2000s
- fakedang 5y ago
  Unfortunately as much as I love it, the new AoE2 DE has fallen to the same trap. Old AoE2 uses a fraction of the power (and disk space) compared to the new one.
  - jcelerier 5y ago
    Eh I disagree. On the same computer original aoe2 would lag like crazy even on a LAN with 4-5 players and pop. limit to 200, with AoE2 DE it's muuuch smoother

aetherspawn 5y ago

I found RTS3 had a lot more network issues than AoE2. It would more consistently fail to hole punch (fail to join a lobby game with certain people for no discernible rhyme or reason) and/or drop connection during play. I could play RTS3 with most people around the world, it seemed, except a few of my best friends. They could individually play with random people all over the world, but not each other or myself, depending on the day of the week or something like that. Perhaps their decision to move away from DirectPlay was a mistake or premature optimization!

Agentlien 5y ago
For anyone wondering, RTS3 refers to Age of Mythology from 2002.
- aetherspawn 5y ago
  Ah, I thought they meant AoE 3. My bad.
jeroenhd 5y ago
Age of Mythology always worked for me on LAN and usually WAN without too much port forwarding. AoE (both 1 and 2) never seemed to work well over the internet for me without loads of manually forwarded ports. The rewrites/updates have definitely helped in this situation but I think the AoM code was built for and tested on networking hardware that was a lot more error prone than modern routers are, which in turn affects how it behaves even today. Clever NAT bypasses definitely work better now than they did fifteen or twenty years ago, but they're definitely not guaranteed to work still. The rise in CGNAT usage also paints a grim picture for anyone playing video games together. I've also had my fair share of peering issues between ISPs. At one point I discovered that I could send UDP packets to my friend, but my friend could not send UDP packets to me, no matter the port forwards and even after forcing the modem into bridge mode. TCP worked fine in both directions, though. Something upstream seemed to filter the packets out, I'm guessing to prevent DDoS attacks, so whenever we played P2P games one of us needed to run some kind of VPN. IPv6 can solve many of the problems I remember having to deal with setting up games back in the day, but most games have become cloud-only anyway. The age of custom servers and P2P gaming is over, killed by lootboxes and shitty NAT implementations.

aligray 5y ago

For anyone looking for further reading on the subject, this article popped up on hn a few months ago which I thought was fascinating: https://ki.infil.net/w02-netcode.html https://ki.infil.net/w02-netcode.html

oreally 5y ago

This article is wayy outdated in the context of today's games. Look to https://gafferongames.com/post/deterministic_lockstep/ https://gafferongames.com/post/deterministic_lockstep/ instead, you get bonus animated gifs there.

reitzensteinm 5y ago
What specifically is outdated? If you look under the hood of modern RTS games they're all going to look pretty similar, and this article has a bunch of info Gaffer's doesn't. I would read both.
- oreally 5y ago
  For example you really don't need to enforce a 200ms turn rate, and their notion of what makes a 'communications turn' is unclear given the terms gamedevs use nowadays.
  - reitzensteinm 5y ago
    Yeah, I used 30hz for my turn rate, but the principles are the same. The terminology is dated, but there's some real gold in there. The bits about deer facing snowballing out of sync errors, out of simulation code corrupting simulation state, comparing state dumps... this is exactly what your life will be like if you build your games this way. I know this because I have (although complicated by the addition of rollback). The Gaffer article barely scratches the surface.
TacticalCoder 5y ago
This article paved the way. FWIW I had written a fully deterministic (non networked though) game engine around... 1991. HN user "dfan" here wrote a fully deterministic game engine for the DOS game Terra Nova in 1996. There were probably others but resources about deterministic game engines were very rare for nearly nobody had one so this article about AoE was a gold mine for many. Some terms may have changed, but the techniques are basically identical. Blizzard in 2002 with Warcraft III had basically solved that issue for RTS once and forever too (and maybe for Starcraft before that?) and not much has changed since then.

blunte 5y ago

I don't know if Paul was the only guy working on the network code, but I believe he was responsible for most or all of it. He wrote that code when he was 18 or 19 years old I believe. But unlike some of the other devs at Ensemble, he never had a big ego, and he was just generally nice and humble. He (much) later created the early mobile hit, Words With Friends, which had many copycats later.

ddek 5y ago

As a fairly high rated AoE2 player (~1800), I can attest that the network architecture has not stood the test of time. Lockstep causes tonnes of problems. Desynchronisation is still possible, when that happens the simulation needs to backtrack (I think?), which can cause units to jump multiple tiles. Because your machine is resolving the movements of other players, you run into problems with more players. The wire protocol is widely known and easily hacked, in a way that's hard to detect in game. AoE2 DE is still lockstep, but through a server. All players are connected to the same server. This has slightly improved on the previous P2P, but not really addressed anything.

Jare 5y ago
> Desynchronisation is still possible, when that happens the simulation needs to backtrack (I think? Sounds more like client-side prediction to smooth things, than actual simulation desync. I have a hard time believing a deterministic game with such a large state was able to backtrack and resync the sim back to determinism. I did not think that lockstep RTS games would need client-side prediction (the indirect and long-term commands in RTS helps hides latency), but I guess if your gameplay lends itself to high APMs then it becomes necessary.
- marginalia_nu 5y ago
  Well you could just have state snapshots taken at regular intervals and then verify that both sides' hashes agree. It's only a couple of thousand entities so it's really not so bad. For you can probably ignore those that haven't deviated from the previous snapshot (and that would account for a state reconstruction taking time). RTS games have a replay mechanisms at least as far back as StarCraft: Brood War, so a journal of player inputs are likely going to be recorded anyway.
  - Jare 5y ago
    Yes the journal of player inputs sure, but the intermediate states is a different matter. AoE2's state size is peanuts for a modern machine, but I would say that at the time, it was quite significant and it would be too costly to store it on the fly. I certainly did not dare try that in the RTS-like deterministic games I worked on (Commandos and Praetorians)

Agentlien 5y ago

For those who may not be aware, the "upcoming" RTS3 mentioned in the article is Age of Mythology from 2002.

gvv 5y ago

Very cool. Just a remark, I'm using Brave and the website won't load until I remove the shield.

Ronll 5y ago

"Part of the difficulty was conceptual -- programmers were not used to having to write code that used the same number of calls to random within the simulation (yes, the random numbers were seeded and synchronized as well)." What do they mean, "same number of calls to random within the simulation", didn't get it

thatjoeoverthr 5y ago
The random number generator needs to give the same outcomes on both ends so that you don’t need to transfer results. To this end, you have to use the same deterministic generator code, seed and ensure you call it the same number of times on each system. This last point is because queries to a random number generator also change its state. If system A and B start with some seed, but system A calls it 30 times and system B 29 times, the next call will yield a different result on each system.
transpostmeta 5y ago
If you call a pseudo-random number generator, you get the next random number in a strictly defined sequence of numbers derived from the seed. So the generator has state, and needs to be called exactly the same number of times in the same order on all clients to make sure that the random events happen the same on all clients.
short_sells_poo 5y ago
I suspect it pertains to the fact that’s most RNGs tend to be pseudo-random number generators that maintain an internal state. Each call of the RNG mutates the internal state. If you have two processes, one calls the RNG 2 times and the other calls it 5 times, the process states will have diverged and the simulation is no longer consistent between them. This is true even if the simulation ends up using 1 value out of those RNG calls, because the RNG state itself is implicitly part of the simulation.

EamonnMR 5y ago

I've built a couple of demos using this article as a guide to understanding how the net architecture works. It's pretty fun to run through. Making your sim deterministic is a neat challenge.

triska 5y ago

Complementing this article, there's also a great post mortem article about Age of Empires: https://www.gamedeveloper.com/pc/the-game-developer-archives-postmortem-ensemble-s-age-of-empires- https://www.gamedeveloper.com/pc/the-game-developer-archives... In the section "Things That Went Wrong Or We Could Have Done Better", it mentions as the final point an insight I often think about: "8. We didn’t take enough advantage of automated testing. In the final weeks of development, we set up the game to automatically play up to eight computers against each other. Additionally, a second computer containing the development platform and debugger could monitor each computer that took part. These games, while randomly generated, were logged so that if anything happened, we could reproduce the exact game over and over until we isolated the problem. The games themselves were allowed to run at an accelerated speed and were left running overnight. This was a great success and helped us in isolating very hard to reproduce problems. Our failure was in not doing this earlier in development; it could have saved us a great deal of time and effort. All of our future production plans now include automated testing from Day One."

R0b0t1 5y ago
Very clever but it only tests actions the AI takes.
- alternatetwo 5y ago
  The AI actions are the same actions that players would take manually, they just get computed by the AI instead of a brain. Afaik the AI logic gets processed on the host instance, and the rest just follow the commands sent through the network.
- shaggyfrog 5y ago
  Based on my personal experience of automated testing in AAA games, even if you "only" count the overlap with the actions humans take, it's a huge overlap in practice. And while there are edge cases only humans find, automated testing finds bugs humans don't. The point of that section is that the earlier you find bugs, the quicker you fix them, and you end up with less overall pain for the length of the project.
- 5y ago

nulptr 5y ago

If each command would only be executed two communication turns in the future, and a communication turn is ~200ms, isn't there ~600ms of lag for each command? > For RTS games, 250 milliseconds of command latency was not even noticed -- between 250 and 500 msec was very playable, and beyond 500 it started to be noticeable. The article mentions this as well, so was ~600ms latency just expected? Why doesn't it work to execute commands _one_ communication turn in the future?