4 ms·

Always enjoy these articles. I do think that the simulation strategy deserves a bit more discussion, and the pitfalls of replays are glossed over slightly. Sim

Always enjoy these articles. I do think that the simulation strategy deserves a bit more discussion, and the pitfalls of replays are glossed over slightly. Simulating players gives you the benefit of end-to-end testing much more easily (for example, it tests your login flow and disconnect flow, both of which are often written without much care for performance). It also practically demands that you write a scripting console for your game, which is an invaluable resource for error testing, debugging and development both by your engine and content developers. It's true that human players are more unpredictable than bots, but that's not a drawback, because you want to stress-test the hottest part of your game. For these purposes, you can write a small test level (think an arena) and have bots spawn at either end and run towards each other, attacking until they die. This isn't at all representative of how players will play the game but it will very definitely catch performance regressions. Replays depend on two factors, deterministic playback and unchanging content. Deterministic playback is possible (though extremely difficult) to achieve, but it really has to be a priority from the beginning, and it has to have its own testing suite and fixing regressions has to be a priority. It's impossible to retro-fit it onto an existing game (sometimes by design, as non-determinism occasionally is mistaken for randomness). Unchanging content is, obviously, unlikely to happen during development and is much more likely to be possible during betas, but during those time periods you have much more time to devote programmer and content developer time and attention to ancillary features like expanding bot programming.

kevingadd 14y ago
The upside for the simulation problem is that many multiplayer games have something approaching bot AI to begin with - enemy AI - so you can use that as a starting point for those sorts of tests. I'm not sure if that was ever done at scale for GW testing, but it would have been possible... My understanding is that the replays were possible because the game engine was architected from the beginning with the server 100% in control of game events and the clients relying on message-passing to communicate with the server. This means that the recording captured on the server is a definitive recording of what actually happened in the game. Clients could desync, so a replay might not reproduce that, but otherwise they were pretty accurate. I'm not sure how the unchanging content problem was solved. I think everything was revision-controlled aggressively enough that it would have been possible to set up a local server instance using old content, and it's also possible that most ordinary changes (altering a texture, a map, etc) wouldn't actually change the results of a replay. Definitely a challenge, though. Replays came in handy later on for other features (like letting players watch others' PvP matches in-game) so they probably ended up carrying their weight in terms of development effort. I do wonder if they felt like a burden initially, though, before their value became clear...
- endianswap 14y ago
  Dealing with the many scenarios that could break replays was problematic, but being able to run several instances of the same game at the same time and compare them occasionally for desyncs can help alleviate that. For example, using a pointer value (address) as a hash table or sort key means that depending on allocation order, iteration of game entities may be different from one run to another.

kevingadd 14y ago

The upside for the simulation problem is that many multiplayer games have something approaching bot AI to begin with - enemy AI - so you can use that as a starting point for those sorts of tests. I'm not sure if that was ever done at scale for GW testing, but it would have been possible... My understanding is that the replays were possible because the game engine was architected from the beginning with the server 100% in control of game events and the clients relying on message-passing to communicate with the server. This means that the recording captured on the server is a definitive recording of what actually happened in the game. Clients could desync, so a replay might not reproduce that, but otherwise they were pretty accurate. I'm not sure how the unchanging content problem was solved. I think everything was revision-controlled aggressively enough that it would have been possible to set up a local server instance using old content, and it's also possible that most ordinary changes (altering a texture, a map, etc) wouldn't actually change the results of a replay. Definitely a challenge, though. Replays came in handy later on for other features (like letting players watch others' PvP matches in-game) so they probably ended up carrying their weight in terms of development effort. I do wonder if they felt like a burden initially, though, before their value became clear...

endianswap 14y ago
Dealing with the many scenarios that could break replays was problematic, but being able to run several instances of the same game at the same time and compare them occasionally for desyncs can help alleviate that. For example, using a pointer value (address) as a hash table or sort key means that depending on allocation order, iteration of game entities may be different from one run to another.