3 ms·

In twenty years in game development I have never worked on a game which had real unit tests or even integration tests. I’ve seen engines which used them, but n

by mewse 5y ago

In twenty years in game development I have never worked on a game which had real unit tests or even integration tests. I’ve seen engines which used them, but not games. The rationale was that it was just too hard, which always felt like a cop-out to me. I would dearly love to have more automated tests in the game I’m working on now, but I’ve never seen a model of it working well which I could copy, and part of me suspects that it’d be a huge investment of time to figure it out entirely on my own when I’m already vastly overworked as it is. If anybody has references to indepth case studies of making game engines more friendly toward automated tests, I’d be super interested to see whether there were lessons I could apply toward my own situation!

dsego 5y ago
Not TDD but here is a talk about automated testing at Croteam Continuous integration and testing pipelines in games - case studies of The Talos Principle and Serious Sam https://www.youtube.com/watch?v=YGIvWT-NBHk https://www.youtube.com/watch?v=YGIvWT-NBHk
tarcon 5y ago
Interesting. I always assumed that to get the balancing feel right, a game would have to run huge parameterized test-suits to make sure win/lose results to user inputs are in balance.
- beckingz 5y ago
  Good luck doing this for a huge modern AAA game.
  - mywittyname 5y ago
    I've always assumed they did this for games like LoL. Bots already exist, so the foundation for automated play testing is in place. Take the basic AI and add some functionality to track the effectiveness of various skills or loadout across plays. Using A/B/n testing to choose the most effective character strategy would probably highlight overpowered loadouts within a few thousand game-test-hours. They could probably take analytics from real players and do what's outlined above and get a reasonable idea of the impact a change will have.
    - anchpop 5y ago
      The issue with that is that LOL bots are laughably bad, and making them not bad requires a substantial investment. But something like what you describe is possible. OpenAI has a paper where they evaluate chess variants by training AlphaZero on them and seeing which ones generate more balanced games between white vs black, and which ones lead to more dynamic games with fewer stalemates etc.
      - 8note 5y ago
        Stalemate a are great fun; it's "the players agreed to a draw" and "draw by repetition" that are boring
    - wbc 5y ago
      I dont think they used bots to test, just rpc endpoints, at least as of 2016 blog: https://technology.riotgames.com/news/automated-testing-league-legends https://technology.riotgames.com/news/automated-testing-leag...
  - kempbellt 5y ago
    This is likely why "Game Tester" is an actual job, or at least used to be. 14 year old me didn't believe it was a real thing... "Early Access" seems to be the more popular route now. People are happy to pay to test a game and feel like they participated in its development. Check out Star Citizen if you want to be amazed and depressed by this truth at the same time. Sometimes it works well. Sometimes it doesn't. Seems largely dependent on how engaged the devs are with their community.
teamonkey 5y ago
It's not exactly what you mean, but automated gameplay testing is fairly common in the AAA space, although maybe not taken seriously enough. A simple example might be simply starting and closing the engine after the automated build & package process to make sure the game actually runs, but I've seen things like using bots to emulate player behaviour to smoke test gameplay functionality. No Man's Sky used automated tools to evaluate the procedural-generation algorithms[1]. Here's a more comprehensive example of automated gameplay testing[2]. [1] https://youtu.be/sCRzxEEcO2Y?t=3100 https://youtu.be/sCRzxEEcO2Y?t=3100 [2] https://www.youtube.com/watch?v=VVq_hgaX8MQ https://www.youtube.com/watch?v=VVq_hgaX8MQ
- mewse 5y ago
  I’ve definitely seen some automated “pretend to be a player” systems which either spammed controller inputs at random or was able to take some very constrained actions within some portion of a game (undirected driving of a car, for example). Those are neat and helpful, but don’t really have the rigor of proper testing where you know that the same thing is being tested every time. One of my stumbling blocks at the moment is that the machines in my CI setup don’t have graphics cards in them (or monitors attached or etc); they actually can’t run the game, and I’d have trouble detecting the difference between “runs successfully” and “runs, but only displays a black screen” or other critical graphical issues.

by mewse 5y ago

dsego 5y ago

Not TDD but here is a talk about automated testing at Croteam Continuous integration and testing pipelines in games - case studies of The Talos Principle and Serious Sam https://www.youtube.com/watch?v=YGIvWT-NBHk https://www.youtube.com/watch?v=YGIvWT-NBHk

tarcon 5y ago

Interesting. I always assumed that to get the balancing feel right, a game would have to run huge parameterized test-suits to make sure win/lose results to user inputs are in balance.

beckingz 5y ago
Good luck doing this for a huge modern AAA game.
- mywittyname 5y ago
  I've always assumed they did this for games like LoL. Bots already exist, so the foundation for automated play testing is in place. Take the basic AI and add some functionality to track the effectiveness of various skills or loadout across plays. Using A/B/n testing to choose the most effective character strategy would probably highlight overpowered loadouts within a few thousand game-test-hours. They could probably take analytics from real players and do what's outlined above and get a reasonable idea of the impact a change will have.
  - anchpop 5y ago
    The issue with that is that LOL bots are laughably bad, and making them not bad requires a substantial investment. But something like what you describe is possible. OpenAI has a paper where they evaluate chess variants by training AlphaZero on them and seeing which ones generate more balanced games between white vs black, and which ones lead to more dynamic games with fewer stalemates etc.
    - 8note 5y ago
      Stalemate a are great fun; it's "the players agreed to a draw" and "draw by repetition" that are boring
  - wbc 5y ago
    I dont think they used bots to test, just rpc endpoints, at least as of 2016 blog: https://technology.riotgames.com/news/automated-testing-league-legends https://technology.riotgames.com/news/automated-testing-leag...
- kempbellt 5y ago
  This is likely why "Game Tester" is an actual job, or at least used to be. 14 year old me didn't believe it was a real thing... "Early Access" seems to be the more popular route now. People are happy to pay to test a game and feel like they participated in its development. Check out Star Citizen if you want to be amazed and depressed by this truth at the same time. Sometimes it works well. Sometimes it doesn't. Seems largely dependent on how engaged the devs are with their community.

teamonkey 5y ago

It's not exactly what you mean, but automated gameplay testing is fairly common in the AAA space, although maybe not taken seriously enough. A simple example might be simply starting and closing the engine after the automated build & package process to make sure the game actually runs, but I've seen things like using bots to emulate player behaviour to smoke test gameplay functionality. No Man's Sky used automated tools to evaluate the procedural-generation algorithms[1]. Here's a more comprehensive example of automated gameplay testing[2]. [1] https://youtu.be/sCRzxEEcO2Y?t=3100 https://youtu.be/sCRzxEEcO2Y?t=3100 [2] https://www.youtube.com/watch?v=VVq_hgaX8MQ https://www.youtube.com/watch?v=VVq_hgaX8MQ

mewse 5y ago
I’ve definitely seen some automated “pretend to be a player” systems which either spammed controller inputs at random or was able to take some very constrained actions within some portion of a game (undirected driving of a car, for example). Those are neat and helpful, but don’t really have the rigor of proper testing where you know that the same thing is being tested every time. One of my stumbling blocks at the moment is that the machines in my CI setup don’t have graphics cards in them (or monitors attached or etc); they actually can’t run the game, and I’d have trouble detecting the difference between “runs successfully” and “runs, but only displays a black screen” or other critical graphical issues.