5 ms·

For the DOS game Terra Nova (1996) I made our game engine fully deterministic. I guess this was a year before Age of Empires but well after you. It was amazing

by dfan 5y ago

For the DOS game Terra Nova (1996) I made our game engine fully deterministic. I guess this was a year before Age of Empires but well after you. It was amazing how much easier this made debugging. The effect was so great that I can't even put a multiplier on it, because it moved bugs from the "we'll never reproduce this" category to the "just see what happened and fix it" category. Some of these replays represented more than half an hour of gameplay too. One thing that surprised me is that it also found a bunch of bugs waiting to happen (uninitalized variables / dangling pointer sort of stuff) that would trigger an error when replaying from a file didn't produce the same results as the original play (we had a checksum of game state that we could check).

TacticalCoder 5y ago
That is really cool! There may have been others, earlier ones: just not that I knew of. You came with the idea yourselves or you knew about other game developers doing that? > Some of these replays represented more than half an hour of gameplay too. Same... That was really my main motivation: sometimes needing to play for 20 minutes before the bug would show up. > (we had a checksum of game state that we could check) Ooooh I love that: that is plain bad---! So you not only had your deterministic engine, but a way to directly identify any discrepancy between the original state and the replayed one. I didn't think about that! It's amazing that it let you identify bugs before they even stroke.
- dfan 5y ago
  I don't remember hearing about other developers doing the same thing. I think it was just that I had a lot of experience debugging deterministic programs (like command-line tools) and it was infinitely more pleasant than trying to debug an interactive graphical program, so it was worth seeing whether we could make the game itself deterministic. I've never made a system that ambitious again, but one thing I've learned from that experience is to never ever call global rand(); always always create your own RNGs that you can run explicitly, and if you have multiple systems that can be disabled independently (e.g., we were able to turn off graphics during our replays), give them each their own RNG.
tomlagier 5y ago
Is there any downside to writing deterministic games? It seems like the only sane way to do it, but I guess it adds some complexity overhead for the initial write otherwise everyone would do it by default?
- munificent 5y ago
  There aren't many significant downsides I know of, it's just really easy to accidentally make your game not deterministic. You have to entirely isolate the game state from any sources of non-determinism. The latter can include: subtle CPU timing issues, GPU timing, other GPU artifacts, the system clock, timing from IO operations. If you want the state to be deterministic across machines (useful for debugging multiplayer stuff) then you also need to include floating point operations (some chips behave differently on some boundary cases, I think) as well as some graphics operations (thinks like texture operations and rounding are always bitwise identical across GPUs). If any bit of non-determinism sneaks in from one of these, it will wander through and pollute any other operations and data that depend on it. Flushing out non-determinism bugs can almost feel as difficult as debugging a non-deterministic engine. I was at EA when the Madden team refactored the engine to be deterministic. It took a full cycle of bug hunting, but it was marvelous once it got there. QA could just send over a replay file and any dev could simply load up the replay and repro the bugs. In fact, the user-facing replay system in the game ("Let's watch that play again in slow mo!") was simply restarting the engine and then replaying the user inputs deterministically to resimulate the whole game again.
  - skykooler 5y ago
    There's a bunch of excellent blog posts from the Factorio developers about tracking down tiny bits of non-determinism (since it uses deterministic multiplayer, as there is way too much dynamic world state to update constantly over the network).
- learc83 5y ago
  Yeah it’s just harder and potentially less performant. There are different levels—deterministic across the same architecture, different architectures etc… The simplest example I can think of is being deterministic across different frame rates. Imagine you move a player by adding x to it’s position each frame. You adjust x based on the frame rate so that you don’t move faster on a faster computer. So on a computer tuning at 30 FPS you move 10 pixels each update. But on a computer running at 60 FPS you move 5. You have walls that are 6 pixels wide. On the 60fps computer it works fine, but on the 30fps machine you can teleport through the walls.

by dfan 5y ago

TacticalCoder 5y ago

That is really cool! There may have been others, earlier ones: just not that I knew of. You came with the idea yourselves or you knew about other game developers doing that? > Some of these replays represented more than half an hour of gameplay too. Same... That was really my main motivation: sometimes needing to play for 20 minutes before the bug would show up. > (we had a checksum of game state that we could check) Ooooh I love that: that is plain bad---! So you not only had your deterministic engine, but a way to directly identify any discrepancy between the original state and the replayed one. I didn't think about that! It's amazing that it let you identify bugs before they even stroke.

dfan 5y ago
I don't remember hearing about other developers doing the same thing. I think it was just that I had a lot of experience debugging deterministic programs (like command-line tools) and it was infinitely more pleasant than trying to debug an interactive graphical program, so it was worth seeing whether we could make the game itself deterministic. I've never made a system that ambitious again, but one thing I've learned from that experience is to never ever call global rand(); always always create your own RNGs that you can run explicitly, and if you have multiple systems that can be disabled independently (e.g., we were able to turn off graphics during our replays), give them each their own RNG.

tomlagier 5y ago

Is there any downside to writing deterministic games? It seems like the only sane way to do it, but I guess it adds some complexity overhead for the initial write otherwise everyone would do it by default?

munificent 5y ago
There aren't many significant downsides I know of, it's just really easy to accidentally make your game not deterministic. You have to entirely isolate the game state from any sources of non-determinism. The latter can include: subtle CPU timing issues, GPU timing, other GPU artifacts, the system clock, timing from IO operations. If you want the state to be deterministic across machines (useful for debugging multiplayer stuff) then you also need to include floating point operations (some chips behave differently on some boundary cases, I think) as well as some graphics operations (thinks like texture operations and rounding are always bitwise identical across GPUs). If any bit of non-determinism sneaks in from one of these, it will wander through and pollute any other operations and data that depend on it. Flushing out non-determinism bugs can almost feel as difficult as debugging a non-deterministic engine. I was at EA when the Madden team refactored the engine to be deterministic. It took a full cycle of bug hunting, but it was marvelous once it got there. QA could just send over a replay file and any dev could simply load up the replay and repro the bugs. In fact, the user-facing replay system in the game ("Let's watch that play again in slow mo!") was simply restarting the engine and then replaying the user inputs deterministically to resimulate the whole game again.
- skykooler 5y ago
  There's a bunch of excellent blog posts from the Factorio developers about tracking down tiny bits of non-determinism (since it uses deterministic multiplayer, as there is way too much dynamic world state to update constantly over the network).
learc83 5y ago
Yeah it’s just harder and potentially less performant. There are different levels—deterministic across the same architecture, different architectures etc… The simplest example I can think of is being deterministic across different frame rates. Imagine you move a player by adding x to it’s position each frame. You adjust x based on the frame rate so that you don’t move faster on a faster computer. So on a computer tuning at 30 FPS you move 10 pixels each update. But on a computer running at 60 FPS you move 5. You have walls that are 6 pixels wide. On the 60fps computer it works fine, but on the 30fps machine you can teleport through the walls.