6 ms·

Ask HN: Is TDD or BDD used in AAA gaming industry?

• taspeotis 9y ago
  Hi, this article [1] isn't quite what you're after but you might be interested in it anyway. [1] https://engineering.riotgames.com/news/automated-testing-league-legends https://engineering.riotgames.com/news/automated-testing-lea...
  • je42 9y ago
    Nice ! Looks, like they also automatically create tickets for defects ! !
• brad0 9y ago
  In my experience no. Testing was not even considered. I worked on mobile games 5 years ago so things may have changed. Most of the time we had strict timelines with feature milestones. We were building mobile apps of licenced titles. The lack of TDD was replaced with full time QA, sometimes multiple on the one game.
• stuntkite 9y ago
  I worked for a AAA game company in the game server department and I would say that with the schedule and how the shop was directed it was nearly impossible to do TDD. I can't speak for the engine side, but my guess is that it was not possible over there either. I bet like anything through, if you go to shops that have strong direction, can work with a small team and over deliver, those standards paradigms are used. Check out the tests in the Quake3 Source Code. It is a shining example for sure. https://github.com/id-Software/Quake-III-Arena https://github.com/id-Software/Quake-III-Arena
• gleenn 9y ago
  Having never worked at a gaming company but having a friend who was the head of testing at one very recognizable shop, the answer was definitely no, at least not there. Its definitely a big enough industry that I'm sure some does though. Also, last time I wrote any 3D code, it at least appeared to be incredibly hard to test that some visual thing was just so. Of course there are still things like business logic that surely could be tested, but it's just not going to be as straight forward as having some code generate some HTML string and then asserting things against it. Those last-mile things you might want to test are always a challenge IMHO. I learned a lot of my coding chops at a Rails shop that was very in to testing, so that's definitely my perspective, YMMV.
• ZenoArrow 9y ago
  I'm not in the game industry, but from what little I know, this is probably going to be a tricky question to answer. Reason being, a high proportion of AAA game development is done using off-the-shelf game engines. Of course it's still necessary to write code for the game logic and features, but because of the stable base this code is written against, it might not be seen as necessary to follow a strict TDD/BDD approach.
  • clemParis 9y ago
    A high proportion of web development is done using somehow "off-the-shelf" web frameworks, and yet people write unit test/functional tests for the application logic and features. Despite how stable the base this code is written against.
  • maccard 9y ago
    > a high proportion of AAA game development is done using off-the-shelf game engines. I don't think this is particularly true. A high proportion of game development is, but many AAA studios are using either home-rolled engines, or started with an existing engine (Unreal or iDTech) and what they're currently running looks absolutely nothing like the original engine.
    • sk0g 9y ago
      I agree with both of you, sorta? The problem is, most of the major game dev companies have their own engines. Unreal Engine is massive in AAA games, but most EA titles use Frostbite, Valve: Source, Ubisoft: Anvil, Crytek: CryEngine, Bethesda: Gamebryo, and the CoD devs use IW, which is a fork of id tech engine. However, this doesn't mean that they aren't using off-the-shelf engines. As long as you count in-house "shelves".
• purrcat259 9y ago
  Factorio is a good example of a test driven developed game: https://www.factorio.com/blog/post/fff-186 https://www.factorio.com/blog/post/fff-186
  • Lapsa 9y ago
    factorio doesn't really qualify as coming from AAA gaming industry. and factorio devs are on another level.
    • purrcat259 9y ago
      Yes, I agree. I forgot the AAA part after reading several comments before I posted mine.
  • hacker_9 9y ago
    Is that TDD? Looks like automated tests run after each commit.
    • falcolas 9y ago
      I agree; automated integration tests. Still incredibly useful (per the devs), and not quite the industry standard.
• dfabulich 9y ago
  In my experience, no, they don't write tests first, and in fact AAA game companies write almost no automated tests at all. Instead, AAA teams have large staffs of cheap manual testers--high school and college kids who think they're getting paid to play video games. The managers in charge of these QA teams work their manual testers to the bone, encouraging them to work late and do unpaid overtime. These testers inevitably burn out, and but there's always a new batch of kids coming in to replace them. When manual QA is that cheap, it turns the economics of automated testing on its head. Why maintain an automated regression suite when you have humans who will do the testing for you? There's a similar effect on the dev side. A lot of inexperienced developers are willing to take a significant pay cut to work at a AAA game company, just for the dream of working on a AAA game. The devs, too, are notoriously managed as an expendable human resource. "You'd better keep working overtime, or I'll just replace you with another kid who'll work harder than you for less money." As a result, AAA game developers burn out pretty quickly, too, leaving the game industry for other software companies that both pay better and offer a better work/life balance. As a result, you'll find that a lot of AAA teams are relatively inexperienced. Sure, there's a core of long-time devs on every team, technical leads who can point the way, but it's not an environment where you can get solid mentoring on the best practices of software development. Even if one team does pick up TDD, who knows if the same team will even be there in another year or two? Good software practices are regularly "rediscovered" on AAA dev teams, and then forgotten, as the team turns over and their tribal knowledge fades away.
  • microcolonel 9y ago
    In addition to the steady supply of college-age fools who think that life as a video game tester would be fulfilling; it is also borderline impossible to test many things in video games adequately without human intervention. Aside from critical technical elements like netplay protocol and file parsing, the only reliable signal you'll get from automated testing is whether the code crashes or not under normal circumstances on the test platform with the tested driver and operating system patch release, most properties of the game are subjective or emergent from the code itself.
    • stickfigure 9y ago
• EngineerBetter 9y ago
  As a one-time indie game developer who went to work at a 250-person games studio, my answer based on limited experience is "probably not". BDD in particular is tricky, because beyond the level of abstraction of the engine, you don't really know what you're building. How do you express that a thing should be fun, or weapons reloads feel punchy, in a BDD spec? How do you automatically test that? Games code exists purely to elicit reactions in human brains. We don't have the technology yet to examine the desired state change in the target system. Regular apps have the same problem, but with less expensive assets and less iteration. "As a user, I should feel that the login transition is slick" is a spec I can imagine that is UX-related. However, we can probably all agree on what slick is, and failing that you could user-test it to prove that the implementation is acceptable. A game is made of many complex adaptive interacting systems, so every change needs subjective validation. On the subject of complex adaptive systems: many games feature them and so exhibit fundamentally unpredictable emergent behaviour. This can be hard to test for. The most testing I've done in a games project is to figure what it is you're trying to build, then unit test the implementation when you've worked out if it's fun.
• fuwafuwa 9y ago
  Short answer: No. Gamedevs have to keep showing testing zealots the door. Long answer: The problems with the approach are manyfold. There are a few points where testing is automatable, but they don't describe entire game projects well. * What are you testing when you add gameplay? You are testing for a whole set of design concerns across the project, not just a technical specification or quality of service metric. You cannot do it in isolation and get the results you want at the speed you need, because you need tons of feedback to discover what a complex game is currently strong or weak at. The only reasonable way to gain the necessary feedback is to allow prototyping code to drive near-term changes, manually playtest it, and then factor out the most stable parts of the resulting soup where you can. This works against test-first because you don't really know what you're testing. You throw in a feature on a hunch - "maybe this will give the experience we're looking for" - and see how it behaves hands-on. If it works, there are still usually ramifications and elaborations that didn't surface up-front. And once it does work it's hard to break out into a sandbox because the interesting part of the behavior is in the coupling to other systems and features. * Games deal with vast quantities of mutable state strapped onto relatively straightforward data pipelines: the parts that are most testable are engine and toolchain core elements like asset builds and rendering systems. These parts do see unit tests and integration tests, although there isn't total consensus on how much testing is needed or how it should be done. They are the most like business and application software, though, since their development can be more structured around technical goals and quality metrics. * Many bugs are ultimately data bugs. Some designer or artist set up the wrong binding of assets or made a valid yet broken combination of parameters. Games tend to fly apart very suddenly when they operate on bad data. But the code is doing exactly what it should do for that set of inputs - and if it's a failure it's at the level of process and tooling, not the runtime code.
• speps 9y ago
  If you want to experience that in the AAA game industry, I work at Rare Ltd (UK, part of Microsoft Studios). We've been doing TDD and Continuous Integration for about 5-6 years now starting with Kinect Sports Rivals. For Sea of Thieves[0] (our current project), we've extended that to Continuous Delivery and it's been just a complete game changer in regards to overtime and crunch. For example, the onboarding exercise is basically a TDD kata that has to use our coding guidelines and build farm so you get to know how we work without having to dive right away into Unreal Engine 4. We use TeamCity for our CI solution. And yes, we are recruiting : https://www.rare.co.uk/careers https://www.rare.co.uk/careers [0]: https://www.seaofthieves.com https://www.seaofthieves.com
  • SCdF 9y ago
    Can you give some examples of what you're actually testing? Perhaps an example of a specific test statement, the kind of level the tests sit at etc, what gets covered and what doesn't etc? Are you testing the rendering engine? Lighting / physics / shaders / collision detect? Are you testing scripting? Win conditions? AI (probably not for SoT I guess)? Are you testing the core engine? File mgmt? memory mgmt? input reading etc? As someone who is a developer and a gamer but has never combined the two, the idea of TDD in that scenario is fascinating.
    • speps 9y ago
      We are unit testing all new code that can be unit tested, trying to do dependency injection when possible so mocking is easier. We are testing the rendering engine, lighting, shaders, etc. in a visual test framework we created ourselves that compares each run of the tests with the previous and uses SSIM to output a comparison metric. At the moment, that's still checked by someone every week but could be automated. However, when changing rendering code or shaders, we do require that a test result is shown during code review. For physics and collision detection, we do have tests covering that too, a lot of them involving our FFT water simulation as well. We are testing the whole flow of the game from start menu to entering a session, matchmaking, loading a map etc. I think AI also has some good coverage, notably for the navmesh which is essential. The core engine is Unreal Engine 4. It's not an engine made for testing unfortunately, even though the vanilla version has some (but not nearly enough). We've stuck with a specific version for a while now as upgrading usually meant that we'd also get new bugs from the new version. All the core is covered by all the existing tests, we've added tests where we thought it was critical and where we fixed or improved internal code of the engine. Before joining Rare, I really wasn't aware of any testing whatsoever in the video games industry. It's been an eye opener and I wouldn't do it any other way. Reading other comments, we might not follow TDD all the time or to the letter but the point is that testing itself is a very part of everything we do as engineers. That allows our QA team to focus on stuff that is actually broken and can't be tested (such as if a feature is actually fun).
• moomin 9y ago
  My biggest observation with AAA games is that the coding is actually a fairly small part of the work. Most of it is one huge asset pipeline. And I don't think we have particularly good tech for testing what "looks right". Maybe computer vision will get that good eventually, but it's not even a thing right now. The same applies to the gross properties of renderers. Even in big-spec-up-front triple-A games, there's a lot of exploratory work where you honestly don't know what you want. BDD and TDD don't help there either, rapid prototyping does. (TDD presupposes you know what you want, BDD presupposes someone does and you need to find and talk to them.) None of which is to say that there aren't many places where test friendly design and regression tests wouldn't help. But the surface isn't the same as the average business app.
• drawkbox 9y ago
  Game development projects are usually focused on tech/data/network/engine, asset creation/pipeline, behaviors and gameplay. In the tech/data/network/engine areas there may be TDD/BDD if the engine is to run multiple titles beyond the current project, if not the budget may not even allow for time. Possibly other areas may have tests in the behavior/gameplay areas: components, AI etc, this depends on the current dev culture usually. Really the main thing in game development is shipping and building fun, if tests help you do that faster/better then you do them. I think most devs developing anything that is not game specific but tech/data/network/engine related likes to have tests to rely on, time permitting. Most studios I have worked with that do some TDD/BDD/automated testing it usually happens on the "tech" or core/engine teams rather than the "production" or "live" teams. The tech teams are working on engine/network/data while the live/production teams are using the engine/tools/scripts to make assets and gameplay. The tech team isn't as locked to game launch dates/crunches and has more time to be thorough because most of their code will last across multiple titles. For the most part, in the asset creation/pipeline, behaviors and gameplay area, more of that is visual and QA based. There are testing companies and departments setup just for the gameplay/feature testing. So much of game development is going for fun and a good mechanic that actually playing it and testing it live is more effective than automated tests in some areas. Although there are some automated test tools in Unity[1], Unreal[2] and custom engines maybe for very rote areas of gameplay (physics engine, collectables, asset loading, very common unchanging actions etc). Many times on the production/live side of game development, gameplay and behaviors change so much that any automated testing becomes a lag on dev/iteration time, or aren't updated and stagnate as the changes happen so quickly day to day. This is especially true in iterative development during prototyping or pre/post-production, since it is mostly visual the benefits are outweighed by the speed required to ship. [1] https://unity3d.com/unity/qa/test-tools https://unity3d.com/unity/qa/test-tools [2] https://docs.unrealengine.com/latest/INT/Programming/Automation/ https://docs.unrealengine.com/latest/INT/Programming/Automat...
  • je42 9y ago
    > Many times on the production/live side of game development, gameplay and behaviors change so much that any automated testing becomes a lag on dev/iteration time, or aren't updated and stagnate as the changes happen so quickly day to day. This is especially true in iterative development during prototyping or pre/post-production, since it is mostly visual the benefits are outweighed by the speed required to ship. So you think, you if you do testing you are overall slower ? Skipping tests makes youufaster short term. Long term, you are slow as hell because of the lack of coverage and manual effort. Also, especially test help you with large scale changes. Because, these tell you what breaks what shouldn't have been broken. Further, test is not the same as test. In order to do efficiently testing you need to apply the test-pyramid. i.e. have lots of unittest. less integration test. less system tests. less e2e tests. another guiding principle is to focus automated testing on the things are most curicial for the success of the product, or the most difficult to do manually. also, you need to chose the tools that support your testing stratgey. if you work against your tooling. oh boy, you have a journey ahead. (For example unity unittesting framework was pretty crappy last time i check -- 3 years ago )
• Nysa 9y ago
  I am not working directly in gamedev but I have some insights. I've seen growing interest in automated testing from mobile apps. Basically guys were writing regression testing with framework like Selenium (UI testing) that goes throughout most common scenarios. (I think that was described by huuugegames.com on some presentation) The main idea was to let QA focus on "exploratory" tests. (Testing rules of the game, design decisions or performance) TDD & BDD can be used gamedev but not in places you will think of first. I will simplify it vastly: programmers make tools for designers to use. Tools are made by programmers - tools can be done with TDD and/or BDD. Games are made by designer - they use tools to make rules, systems etc. Therefore gameplay is make mostly out of data (designer's parameters set in engine/tools) - you can't test your data with TDD/BDD.
• Nysa 9y ago
  I am not working directly in gamedev but I have some insights. I've seen growing interest in automated testing from mobile apps. Basically guys were writing regression testing with framework like Selenium (UI testing) that goes throughout most common scenarios. (I think that was described by huuugegames.com on some presentation) The main idea was to let QA focus on "exploratory" tests. (Testing rules of the game, design decisions or performance) TDD & BDD can be used gamedev but not in places you will think of first. I will simplify it vastly: programmers make tools for designers to use. Tools are made by programmers - tools can be done with TDD and/or BDD. Games are made by designer - they use tools to make rules, systems etc. Therefore gameplay is make mostly out of data (designer's parameters set in engine/tools) - you can't test your data with TDD/BDD.
• wolfgke 9y ago
  Croteam applied lots of automated testing by bots in The Talos Principle to ensure that changes in level design cannot lead to a situation where the game cannot be completed anymore: > https://venturebeat.com/2014/11/04/the-talos-principle-underwent-15000-hours-of-playtesting-but-not-by-humans-interview/ https://venturebeat.com/2014/11/04/the-talos-principle-under... > http://www.gdcvault.com/play/1022784/Fast-Iteration-Tools-in-the http://www.gdcvault.com/play/1022784/Fast-Iteration-Tools-in...
  • sk0g 9y ago
    That's not TDD, really, but it's a clever way of testing both the AI, and level changes, to a certain extent. If you drop a fucking boulder and you can't cross it yeah, it will light you up with red floodlights, but if a certain corner drops you into the nether... Unless the AI also knows to check each possible interaction, movement, etc.
    • majewsky 9y ago
      Can you clarify why you do not consider this TDD?
      • wolfgke 9y ago
        I think that if one is meticulous, sk0g is actually right (source: https://en.wikipedia.org/w/index.php?title=Test-driven_development&oldid=790275486#Test-driven_development_cycle https://en.wikipedia.org/w/index.php?title=Test-driven_devel...): In TDD one begins anything (in particular any new feature) with writing a test for it, then one runs all tests to see if something fails and only then starts to write code if necessary until all tests pass.
      • sk0g 9y ago
        In TDD you only write code if it makes a red light green, so to speak. First you make the test, then you run it. It will obviously fail, and then you implement code that starts passing tests.
• stonewhite 9y ago
  A colleauge working at a AAA Soccer game gave some insight about that a few years back. Their automated testing was mostly bruteforce stuff like, * Placing a character at penalty spot and making it shoot with every possible angle and strength, making sure the ball didn't launch to space. * They would leave a computer vs. computer game running for hours / days to make sure no kind of random crash would happen.
• Negitivefrags 9y ago
  I work on a game called Path of Exile. At our studio we don't do TDD, but we do continuous integration with a lot of asset testing. In the early days, we tried doing unit tests, but honestly, it's very hard in a game. We had little success testing gameplay code. What we have a huge amount of is asset tests. We have 4 times more non-programmers than programmers working on the game, so most of what is created isn't code anyway. Every time anyone commits anything, it goes through our continuous integration pipeline, and any asset that changed, or is dependent on an asset that changed, is loaded and tested. This includes test loading entire levels. Test loading every asset catches most types of failures. We try, where possible, to add specific tests for common mistakes in order to catch things as early as possible. For example, if an animation has "walk" or "run" in the name, then we test it also contains footstep events in the animation somewhere. If a model is used as a monster we test that it has an attachment point called "frame_upper" onto which various effects can be attached. We look for things like bounding box of an object being far away from the co-ordinate origin (a common mistake when making a big maya scene with lots of exported objects in it). We also do a test start of the servers, which includes loading all of their data, and a test start of the client. All of this means that a build that goes through green is unlikely to cause crashes for other people trying to work on the game. We do, of course, also have an excellent QA team.
  • skibz 9y ago
    Was the difficulty of unit testing game code due to the complexity and unusual nature of PoE's gameplay? (Specifically, the whole spell/attack/support gem system and various other item modifiers)
  • existencebox 9y ago
    Let me take this opportunity to say "you guys do some damn good work". Certainly one of the more impressive games I play, stability/longevity/feature growth wise, especially considering the size of GGG. Topical question; however, since the thread on a whole has focused on the more visible but more finniky-to-test aspects of these games. Do your same statements apply more or less to the backend/networking infra? I remember the decent ball of work you guys wrote about a while back when introducing the lockstep change, was curious what went on behind the scenes to support and validate that from an ops/maintainability perspective, if you can speak on that at all.
• oscarlgz 9y ago
  Kind of related: http://www.eurogamer.net/articles/2017-07-19-a-devs-eye-view-of-bugs http://www.eurogamer.net/articles/2017-07-19-a-devs-eye-view...
• grogenaut 9y ago
  I worked on a AAA game. I controlled the networking code (web services really) down and the entire server infra for our project. All of that was full TDD but it was also all greenfield by me who had done heavy TDD before and working with an external team who was all TDD. I even wrote integration tests from a stub client that ran on the playstation as part of the build to hit my server and a stub server to do wire testing for the client as part of my builds. Also had a full load regression test as part of my build that ran every checkin on a smaller slower cluster. The main game had no tests... except that you need to realize that we had 480GB of asset compiler integration tests and 32GB of game engine integration tests. As a heavy TDD person it took me a long time to realize that this can be very sufficient though it's a longer turn around loop. The test coverage just from being able to boot the game up is quite large. This is something that many non-game type programs have real issues with, they can't easily just "run all the user data". I know we had one project where we did a lot of TDD but it wasn't actually as good as just running the entire input set every build which I eventually switched us to. This was a non-game data processing system. Doing this + unit testing gave us great coverage and few bugs. Generally major bugs were found within 15 minutes by another engineer, eg the next person to sync, build or run the engine or art compiler. I do think we could have unit tested the core collections libraries in the engine to great effect and faster turn around but the build was not really segregated in a way that we would take advantage of this. It was just a monolithic build; the collections objects were heavily header included so there really would have been a build step to built them into their own test exe, run them then build the game. So I'm not sure how much benefit we would have gotten. I think in 4 years I saw one collections issue that lasted past just launching the engine and not having it crash.