5 ms·

I wonder how common TDD is in game development land, especially when you’re using things like Unity or Unreal. I feel like testing your behaviors is pretty har

I wonder how common TDD is in game development land, especially when you’re using things like Unity or Unreal. I feel like testing your behaviors is pretty hard, and even if you unit-test your behaviors, there’s still integration tests. I only write games as a hobby and never use TDD, even if I’d like to, since the tooling is just either poorly documented or too slow, or both. Usually this ends with me being frustrated with the slow development cycle and pushes me towards more unconventional methods of developing games in Javascript using Mocha to run the tests directly in the browser.

exdsq 5y ago
Anecdotal but I think it's pretty rare - my friend worked as a game developer for Epic and didn't know what TDD (or SQL for that matter) actually meant.
- tobyhinloopen 5y ago
  Given how common it is for bugs to reappear in Fortnite, I’m pretty confident their testing suite is either incomplete or not present at all
mschuster91 5y ago
Testing? In games? There is no such thing on a wide scale, not anymore since the cost of distributing patches essentially became a small budget line for a CDN. Modern games are notorious for using the first, most loyal customers as beta testers (hello Fallout 76...). The reason is two-fold... while you absolutely can test some parts of the engine (e.g. collision detection, networking) you can't really "test" stuff that needs a human eye to see if it's working as intended (anything that's rendered) or involves randomness (e.g. fire, fog, water, opponent spawning, loot). That means you have to hire lots of skilled (!) humans, provide them with expensive rigs, and give them time. Which is incredibly expensive.
- gmueckl 5y ago
  Just to give you some perspective: my current employer's main product isn't called a game, but it has an engine at its core that is a game engine in all aspect except its name. And we test the sh*t out of it. We have thousands of automated and very sensitive tests on that stuff. Our test suite goes as far as testing for pixel perfect output. And this involves stuff that is "random". It took us some effort to be random in a perfectly reproduceable way, but we got there. Game QA is more involved than that, of course. Content needs to go through a signoff and QA process that involves humans (we do that, too).
- kempbellt 5y ago
  This is definitely true for some games, and Early Access is increasingly popular, but for others, QA and testing is definitely a part of the development process. I interviewed at a game company a few years ago where one of my daily tasks would be to spend an hour just playing the game and seeing if I spot any bugs. I didn't end up taking the job so I didn't see how involved their actual code testing process was, but it was apparent that they actually cared a bit about quality control.
deleted 5y ago
[deleted]
Danieru 5y ago
Factorio is perhaps the only game I am aware of using TDD. Lots of engine teams use extensive automated testing. Only Factorio is applying it to game logic of any major game I know.
- Glowbox 5y ago
  https://twitter.com/playartifact/status/1051964775658217473?lang=en https://twitter.com/playartifact/status/1051964775658217473?... Artifact did it too (which is no longer being worked on though).
- mywittyname 5y ago
  Factorio seems especially well suited to TDD. The core gameplay loop involves automating away manual tasks. So I have to imagine that the test cases leverage blueprints a lot, i.e., create a blueprint for a feature, pipe resources into it via conveyor belt, test output rates on conveyor belts.
CodeGlitch 5y ago
I was in the industry from early 2000 to early 2010. It was only towards the end that Unit Testing was a thing. At the start we didn't even do code reviews or use a sensible source-control system. Yeah it was a painful experience, but I survived.
mewse 5y ago
In twenty years in game development I have never worked on a game which had real unit tests or even integration tests. I’ve seen engines which used them, but not games. The rationale was that it was just too hard, which always felt like a cop-out to me. I would dearly love to have more automated tests in the game I’m working on now, but I’ve never seen a model of it working well which I could copy, and part of me suspects that it’d be a huge investment of time to figure it out entirely on my own when I’m already vastly overworked as it is. If anybody has references to indepth case studies of making game engines more friendly toward automated tests, I’d be super interested to see whether there were lessons I could apply toward my own situation!
- dsego 5y ago
  Not TDD but here is a talk about automated testing at Croteam Continuous integration and testing pipelines in games - case studies of The Talos Principle and Serious Sam https://www.youtube.com/watch?v=YGIvWT-NBHk https://www.youtube.com/watch?v=YGIvWT-NBHk
- tarcon 5y ago
  Interesting. I always assumed that to get the balancing feel right, a game would have to run huge parameterized test-suits to make sure win/lose results to user inputs are in balance.
  - beckingz 5y ago
    Good luck doing this for a huge modern AAA game.
    - mywittyname 5y ago
      I've always assumed they did this for games like LoL. Bots already exist, so the foundation for automated play testing is in place. Take the basic AI and add some functionality to track the effectiveness of various skills or loadout across plays. Using A/B/n testing to choose the most effective character strategy would probably highlight overpowered loadouts within a few thousand game-test-hours. They could probably take analytics from real players and do what's outlined above and get a reasonable idea of the impact a change will have.
ashtonkem 5y ago
I feel like game dev is probably an area where TDD has some value. My team writes distributed systems, which drastically reduces the value of a TDD approach. There’s only so far you can take the technique with a database backed api before it just becomes absurd.
Thaxll 5y ago
Video game client don't have tests.
jayd16 5y ago
So there's a few reasons tests aren't as ubiquitous in games as they are in non-game dev. You need a large QA team (relative to your team size) to test for fun anyway. The game is constantly getting tested and bugs will get logged. The marginal benefit of automated tests is less than other places because of this. Games have no specification. You have almost no idea of even the genre of game you'll end up with at the end of the dev cycle unless you're making a sequel that has to fit into a mold. Sure you can write tests as you go along and test that enemies with negative health die. The next day someone will suggest "what if they stay alive for a period of time and then explode!" The definition of correct is constantly changing. Tests ossify functionality. It makes it harder to change things because at the very least you need to also change the test. If you're just changing tests whenever you want to suit your new desires then its hard to build trust in the tests. Games don't need to be correct. They just need to be fun. This also decreases the marginal benefit of tests compared to other industries. That said, it would be natural to unit test some data structure or some well defined system. Also, once your game is done, a la Factorio, you can go back and write tests for some refactor because you know the full design specs.
- indeedmug 5y ago
  I don't know if "correct" and "fun" are at odds with each other. There are very famous examples of games failing at the start because of bugs like Cyberpunk. There is a point where the game is too broken to enjoy. You want games where the correct behavior is the fun behavior. (However, there are counter examples like Goat Simulator.) To be fair, I don't pretend to know how CDProject developed their games. They might already have testing and the timetable was the problem.

exdsq 5y ago

Anecdotal but I think it's pretty rare - my friend worked as a game developer for Epic and didn't know what TDD (or SQL for that matter) actually meant.

tobyhinloopen 5y ago
Given how common it is for bugs to reappear in Fortnite, I’m pretty confident their testing suite is either incomplete or not present at all

mschuster91 5y ago

Testing? In games? There is no such thing on a wide scale, not anymore since the cost of distributing patches essentially became a small budget line for a CDN. Modern games are notorious for using the first, most loyal customers as beta testers (hello Fallout 76...). The reason is two-fold... while you absolutely can test some parts of the engine (e.g. collision detection, networking) you can't really "test" stuff that needs a human eye to see if it's working as intended (anything that's rendered) or involves randomness (e.g. fire, fog, water, opponent spawning, loot). That means you have to hire lots of skilled (!) humans, provide them with expensive rigs, and give them time. Which is incredibly expensive.

gmueckl 5y ago
Just to give you some perspective: my current employer's main product isn't called a game, but it has an engine at its core that is a game engine in all aspect except its name. And we test the sh*t out of it. We have thousands of automated and very sensitive tests on that stuff. Our test suite goes as far as testing for pixel perfect output. And this involves stuff that is "random". It took us some effort to be random in a perfectly reproduceable way, but we got there. Game QA is more involved than that, of course. Content needs to go through a signoff and QA process that involves humans (we do that, too).
kempbellt 5y ago
This is definitely true for some games, and Early Access is increasingly popular, but for others, QA and testing is definitely a part of the development process. I interviewed at a game company a few years ago where one of my daily tasks would be to spend an hour just playing the game and seeing if I spot any bugs. I didn't end up taking the job so I didn't see how involved their actual code testing process was, but it was apparent that they actually cared a bit about quality control.

deleted 5y ago

[deleted]

Danieru 5y ago

Factorio is perhaps the only game I am aware of using TDD. Lots of engine teams use extensive automated testing. Only Factorio is applying it to game logic of any major game I know.

Glowbox 5y ago
https://twitter.com/playartifact/status/1051964775658217473?lang=en https://twitter.com/playartifact/status/1051964775658217473?... Artifact did it too (which is no longer being worked on though).
mywittyname 5y ago
Factorio seems especially well suited to TDD. The core gameplay loop involves automating away manual tasks. So I have to imagine that the test cases leverage blueprints a lot, i.e., create a blueprint for a feature, pipe resources into it via conveyor belt, test output rates on conveyor belts.

CodeGlitch 5y ago

I was in the industry from early 2000 to early 2010. It was only towards the end that Unit Testing was a thing. At the start we didn't even do code reviews or use a sensible source-control system. Yeah it was a painful experience, but I survived.

mewse 5y ago

In twenty years in game development I have never worked on a game which had real unit tests or even integration tests. I’ve seen engines which used them, but not games. The rationale was that it was just too hard, which always felt like a cop-out to me. I would dearly love to have more automated tests in the game I’m working on now, but I’ve never seen a model of it working well which I could copy, and part of me suspects that it’d be a huge investment of time to figure it out entirely on my own when I’m already vastly overworked as it is. If anybody has references to indepth case studies of making game engines more friendly toward automated tests, I’d be super interested to see whether there were lessons I could apply toward my own situation!

dsego 5y ago
Not TDD but here is a talk about automated testing at Croteam Continuous integration and testing pipelines in games - case studies of The Talos Principle and Serious Sam https://www.youtube.com/watch?v=YGIvWT-NBHk https://www.youtube.com/watch?v=YGIvWT-NBHk
tarcon 5y ago
Interesting. I always assumed that to get the balancing feel right, a game would have to run huge parameterized test-suits to make sure win/lose results to user inputs are in balance.
- beckingz 5y ago
  Good luck doing this for a huge modern AAA game.
  - mywittyname 5y ago
    I've always assumed they did this for games like LoL. Bots already exist, so the foundation for automated play testing is in place. Take the basic AI and add some functionality to track the effectiveness of various skills or loadout across plays. Using A/B/n testing to choose the most effective character strategy would probably highlight overpowered loadouts within a few thousand game-test-hours. They could probably take analytics from real players and do what's outlined above and get a reasonable idea of the impact a change will have.

ashtonkem 5y ago

I feel like game dev is probably an area where TDD has some value. My team writes distributed systems, which drastically reduces the value of a TDD approach. There’s only so far you can take the technique with a database backed api before it just becomes absurd.

Thaxll 5y ago

Video game client don't have tests.

jayd16 5y ago

So there's a few reasons tests aren't as ubiquitous in games as they are in non-game dev. You need a large QA team (relative to your team size) to test for fun anyway. The game is constantly getting tested and bugs will get logged. The marginal benefit of automated tests is less than other places because of this. Games have no specification. You have almost no idea of even the genre of game you'll end up with at the end of the dev cycle unless you're making a sequel that has to fit into a mold. Sure you can write tests as you go along and test that enemies with negative health die. The next day someone will suggest "what if they stay alive for a period of time and then explode!" The definition of correct is constantly changing. Tests ossify functionality. It makes it harder to change things because at the very least you need to also change the test. If you're just changing tests whenever you want to suit your new desires then its hard to build trust in the tests. Games don't need to be correct. They just need to be fun. This also decreases the marginal benefit of tests compared to other industries. That said, it would be natural to unit test some data structure or some well defined system. Also, once your game is done, a la Factorio, you can go back and write tests for some refactor because you know the full design specs.

indeedmug 5y ago
I don't know if "correct" and "fun" are at odds with each other. There are very famous examples of games failing at the start because of bugs like Cyberpunk. There is a point where the game is too broken to enjoy. You want games where the correct behavior is the fun behavior. (However, there are counter examples like Goat Simulator.) To be fair, I don't pretend to know how CDProject developed their games. They might already have testing and the timetable was the problem.