11 ms·

Addition with flamethrowers – why game devs don't unit test

44 points by jgalecki 2y ago

jrockway 2y ago
Bugs are kind of the fun part of games. If every subroutine worked perfectly, you wouldn't have the chaos of real life. Some of players favorite mechanics are just bugs. (Overwatch example: Mercy's super jump, now a legitimate predictable mechanic that everyone can do, not just people that read the forums and watch YouTube videos about bugs. It started out as a bug, and it was so cool and skill-ceiling increasing that now it's just part of the game.) Having said that, sometimes you need unit tests. Overwatch had this bug where there is an ultimate ability called "amplification matrix" that is a window that you shoot through and the bullets do twice as much damage. One patch, that stopped working. This kind of issue is pretty easy to miss in play testing; if you're hitting headshots, then the bullets are doing the 2x damage they would if they were body shots that got properly amplified. If is very hard to tell damage numbers while play testing (as evidenced by how many patches are "we made character X do 1 more damage per bullet", and it smooths things out over the scale of millions of matches, but isn't really that noticeable to players unless breakpoints change). So for this reason, write an integration test where you set up this window thingie, put an enemy behind it, fire a bullet at a known point, and require.Equals(damage, 200). Ya just do it, so you don't ship the bug, make real people lose real MMR, and then have to "git stash" that cool thing you're working on today, check out the release branch, and uncomment the code that makes the amp matrix actually work. Games are just software engineering. Fun software engineering. But it's the same shit that your business logic brothers and sisters are working on. (Overwatch also had a really neat bug, that the community believes was due to a x == 0 check instead of an x < 0 check. If you pressed the right buttons while using Bastion's ultimate, you had infinite ammo. Normally it fires 3 air strikes, but if you got that counter to decrement twice and skip the == 0 check, then you got to do it an infinite number of times. (Well, actually 2^32 or 2^64 times. Eventually you'd integer overflow and have the chance to hit 0 again. Anyway, this was absolutely hilarious whenever it happened in game. The entire map would turn into a bunch of targets for artillery shells, the noise to alert you of incoming missiles would play 100 times more than normal, and it was total chaos as everyone on your team died. And not even that gamebreaking; both teams have the option to run the same characters, so you could just do it back to your opponent. Very fun, but they fixed the bug quickly. Follow up follow up: all of these silly bugs are in ultimates, which come up the least often of all abilities in the games. That's what happens with playtesting. You don't get test coverage where you need it. A test you write covers the stuff you're most scared about. A careful engineer that likes testing would have never shipped these.)
- AndyPa32 2y ago
  > Games are just software engineering. Fun software engineering. I do question the "fun" part. Midnight crunches, unpaid overtime and - as far as I have read - some of the worst working conditions in all of software engineering. I pass.
  - jrockway 2y ago
    That is probably true, but you know you suffer for your art and all that. People don't really like software, but they love games. We know that games are just software, but it's so fun, that people forget that. It's pretty cool. Though to me, I kind of like getting 8 hours of sleep a night and playing other people's games. While getting paid more :/
  - cableshaft 2y ago
    I've done both gamedev and webdev professionally. I don't miss the long hours or the low pay of gamedev, but I do miss what I worked on. Enough that I'm still working on my own games in my spare time. I still play the games I used to work on professionally from time to time, and like to show them off to people, even though none were financially successful (a few I worked on solo were at least popular, but we made some 'bad decisions in hindsight' or got really unlucky with release timing for all the games I worked on professionally). I don't care hardly at all about all the webdev projects I worked on professionally. They each had some intresting problem solving or coding challenges, and some of them were used WAY more than the games I made were played, but I'm still more into the games I made, and those are most likely to still be around after I'm dead, in a Flash game or ROM archive or torrent somewhere. Also, the worst long hours I ever had was actually in webdev. I once worked in software for a call center, and whenever the phone systems went down, I was expected to be on a call to help fix it (as there was 100+ call center employees that couldn't make calls and we were losing lots of money every minute they were down), sometimes for 16 hours or more sitting on a Zoom call, mostly waiting for people on the server teams to figure things out. And there was a data center migration that had some unexpected problems and required three of us to work for almost three days straight (I literally worked a 24 hour shift, then had 4 hours of sleep, then worked an 18 hour shift right after it). But that's not the norm in webdev at least. My current webdev job I only worked >40 hours a couple weeks to rework some issues with some junior dev's code for a feature before a big demo.
shaftway 2y ago
I don't buy this argument. Most game developers I know have said that unit tests are a waste of time so they never use them, but they're struggling with making changes to utility code and making sure that it doesn't do the wrong thing. Y'know, what unit tests are for. I think the key here is that the perceived cost / benefit ratio is too high. It's the perception that drives their behavior though. I'm in a company now that has zero unit tests, because they just don't see the value in it (and in their case they may be right for a whole slew of reasons). Also, remember that games are not very long-lived pieces of software. You build it, release it, maybe patch it, and move on. If the game moves to version 2 then you're probably going to re-write most of the game from scratch. When you support software for a decade then the code is what's valuable, and unit tests keep institutional knowledge about the code. But with disposable software like games, the mechanics of the game and IP are what's valuable. Why would you write a unit test for something you know you're going to throw away in 6 months?
- Atotalnoob 2y ago
  I am curious as to why your current company does not have unit tests. Do you mind sharing?
  - shaftway 2y ago
    We produce a library that gets included in software made by our clients, and we have several thousand clients. The uptake on new releases is low (most of the clients believe in "if it ain't broke, don't fix it"). So every release has the potential to live in the wild and need support for a long time. We're also in an industry with a ton of competitors. On top of that, the company was founded by some very junior engineers. for most of them this was their first or second job out of college. Literally every anti-pattern is in our codebase, and a lot of them are considered best practices by them. Unit tests were perceived as a cost with little benefit, so none were written. New engineers were almost always new grads to save on money. These facts combined make for an interesting environment. For starters, leadership is afraid to ship new code, or even refactor existing code. Partially because nobody knows how it works, partially because they don't have unit tests to verify that things are going well. All new code has to be gated by feature flags (there's an experiment right now to switch from try-finally to try-with-resources). If there isn't a business reason to add code, it gets rejected (I had a rejected PR that removed a "synchronized" block from around "return boolValue;"). And it's hard to say they're wrong. If we push out a bad release, there's a very real chance that our customers will pack up and migrate to one of our competitors. Why risk it? And the team's experience level plays a role too. With so many junior engineers and so much coding skill in-breeding, "best practices" have become pretty painful. Code is written without an eye towards future maintainability, and the classes are a gas factory mixed with a god object. It's not uncommon to trace a series of calls through a dozen classes, looping back to classes that you've already looked at. And trying to isolate chunks of the code is difficult. I recently tried to isolate 6 classes and I ended up with an interface that used 67 methods from the god object, ranging from logging, to thread management, to http calls, to state manipulation. And because nobody else on the team has significant experience elsewhere, nobody else really sees the value of unit tests. They've all been brought up in this environment where unit test are not mentioned, and so it has ingrained this idea that they're useless. So the question is how do you fix this and move forward? Ideally we'd start by refactoring a couple of these classes so that they could be isolated and tested. While management doesn't see significant value in unit tests, they're not strictly against them, but they are against refactoring code. So we can't really add unit tests on the risky code. The only places that you can really add them without pushback would be in the simplest utility classes, which would benefit from them the least, and in doing so prove to management that unit tests aren't really valuable. And I mean the SIMPLEST utility classes. Most of our utility classes require the god object so that we can log and get feature flags. I say we take off and nuke the entire site from orbit (start over from scratch with stronger principles). It's the only way to be sure. But there's no way I'm convincing management to let the entire dev team have the year they'd need to do that with feature parity, and leadership would only see it as a massive number of bugs to fix. In the meantime developer velocity is slowing, but management seems to see that as a good thing. Slower development translates into more stable code in their minds. And the company makes enough that it pays well and can't figure out what to do with the excess money. So nobody really sees a problem. Our recruiters actually make this a selling point, making fun of other companies that say their code is "well organized".
whatasaas 2y ago
Seems like an excuse that might be fine for small indie teams for a while. The blog certainly blurs the lines between unit and functional tests. In the end, even modest code coverage can pay off. Tests help with code review, understanding the codebase, and can provide an easy map for debugging. But if you’re in an environment where everyone is constantly demanding changes and only testing the happy path, then good luck.
fwlr 2y ago
Arguments against testing tend to fall prey to the von Neumann Objection: they insist there is something tests can’t catch, and then they tell you precisely what it is that tests can’t catch… so you can always imagine writing tests for that specific thing. E.g. this article uses an example of removing the number 5, causing the developer to have to implement a base-9 numbering system. Unit tests that confirm this custom base number system is working as expected would be extremely reassuring to have. Alternatively, you could keep the base-10 system everyone is familiar with, and just have logic to eliminate or transform any 5s. This would normally be far too risky, but high coverage testing could provide strong enough assurance to trust that your “patched base-10” isn’t letting any 5s through. The same is true for the other examples - unit testing feels like the first thing I’d reach for when told about flaming numbers.
- magoghm 2y ago
  Tests can't catch race conditions in multithreaded code. Now that I told you what the tests can't catch, can you imagine writing tests for that specific thing?
  - a_t48 2y ago
    I've written tests around multithreaded code, but they typically catch them in a statistical manner - either running a bit of code many times over to try and catch an edge condition, or by overloading the system to persuade rarer orderings to occur. There's also https://clang.llvm.org/docs/ThreadSafetyAnalysis.html https://clang.llvm.org/docs/ThreadSafetyAnalysis.html which can statically catch some threading issues, though I've not used it much myself.
  - jonex 2y ago
    tsan will catch a bunch of potential race conditions for you, under the condition that you run it somehow. How to make sure it's run? Well, add a test for the relevant code and add it to your tsan run in your CI and you'll certainly catch a bunch of race conditions over time. This has saved me a bunch of times when I've be doing work in code with proneness to those kind of issues. Sometimes it will just lead to a flaky test, but the investigation of the flake will usually find the root cause in the end.
kelseydh 2y ago
Most video game bugs are subtle and not things that are easy to catch with unit testing because they are dynamic systems with many interacting parts. The interaction is where the bugs come from. QA processes do a good job catching the rest.
- lionkor 2y ago
  I would bet that a lot of those bugs come from utility code that is testable
  - Aerroon 2y ago
    Perhaps in development, but the stuff that tends to make it into the release of games seems to be gameplay related. Npc behaviors not lining up, the developer literally not implementing certain stats in the game (looking at you, Diablo 4), graphical bugs caused by something not loading or loading too slowly, performance issues from something loading 1000 copies of itself etc.
    - hahn-kev 2y ago
      And where do you think many of those bugs come from? Usually utility code that isn't working in some edge case, those types of things can be easy to pull out and test in their own independent of the game
      - sfn42 2y ago
        I don't think I've ever seen someone argue against tests without giving the clear impression that they've never written a test in their life.
riffraff 2y ago
I am not convinced of the argument that games change a lot. I do buy the argument that the trade off between effort and value is different, but that's because it's harder to unit test user interactions than it is to unit test a physics engine. It's more or less the reason in the early life of the web few did end to end testing involving browsers, or unit tested iOS apps in the first releases of the iPhone.
mistercow 2y ago
I’ve found that the one thing you can always count on engineers to do is to dismiss sensible tools from adjacent domains using flimsy, post hoc justifications. All product development involves poorly defined boundaries where the product meets the user, where requirements shift frequently, and where the burdens of test maintenance have to be weighed against the benefits. You don’t throw out all of unit testing because it doesn’t work well for a subset of your code. You throw out all of unit testing because writing tests is annoying, none of your coworkers have set it up, and the rest of your industry doesn’t do it, so you feel justified in not doing it either.
- stouset 2y ago
  Right. And because the rest of the industry isn’t doing it, there’s no institutional knowledge of how to do it well. So someone tries it, they do a crap job of it out of understandable ignorance, and rather than taking forward any lessons learned the effort is discarded as a waste of time.
- jayd16 2y ago
  So wait, who's being dismissive of who's practices here?
  - mistercow 2y ago
    I didn’t say engineers are dismissive of other engineers’ practices. The general pattern is “that makes sense for your field, but we can’t use it because…” followed by silly reasons. I was guilty of this myself back when I was an indie dev. It took me an embarrassingly long time, for example, to admit that git wasn’t just something teams needed to coordinate, and that I should be using it as the sole developer of a project.
    - eviks 2y ago
      Interesting, was the impetus for change some big issue you've run into? Or just a gradual accumulation of knowledge about other people experiences made you reconsider? Or something else?
rkachowski 2y ago
It's an interesting idea, but here you have the game designer taking the place of the product manager stereotype - coming up with bizarre unfeasible ideas and the programmer is to make it happen. In any games company I've worked for the designer is responsible for mapping and balancing the rules and mechanics of the game, they would provide a specification of what "red vs blue numbers" would look like and a balanced idea of how to remove the number 5 from the game (balancing and changing the rules like this being entirely within the domain of game design). incidentally any game company I've worked at has had an extensive set of test suites.
epgui 2y ago
What nonsense that is… The idea that intentional mathematical design / correct, well-specified behaviour doesn’t apply to games is absurd.
SillyUsername 2y ago
Ok I know which AAA game studio this might be because I interviewed with them and had to sign an NDA. In their case their flagship game is full of bugs, and they had to ship their product asap pre-aquistion when they were a startup. Because of the mentality of the managers, and weak minded devs, they don't write unit tests, and instead spend the vast majority of their days fighting bugs, so much so they have to hire dedicated staff for their (single game) backlog as they were struggling to keep up "with its success". This is BS of course, I saw their backlog and it was a shit show, with Devs expected to work overtime free of charge to get actual features out (funny how this works isn't it, never affects the business execs' time/life who make the demands of no tests). I was asked what I would bring to the company to help them support their now AAA game, and I stated up front "more unit tests" and indirectly criticised their lack of them. I got a call later that day that (the manager thought) "I would not be a good fit". I got a lead job elsewhere that has the company's highest performing team, literally because of testing practices being well balanced between time and effectiveness (i.e. don't bother with low value tests, add tests if you find a bug etc, if an integration test takes too long leave it and use unit tests). I think back to that interview every time I interview at games studios now, and wonder if I shouldn't push unit tests if they're missing. I'd still do it. The managers at that job were assholes to their developers, and I now recognise the trait in a company.
Joel_Mckay 2y ago
1. Most game engines have the horrible compatibility layers abstracted away, and already fully tested under previous mass deployments 2. Anything primarily visual, audio, and control input based is extremely hard to reliably automate testing. Thus, if the clipping glitches are improbable and hardly noticeable... no one cares. Some people did get around the finer game-play issues by simply allowing their AI character to cheat. Mortal Kombat II was famous for the impossible moves and combos the AI would inflict on players... yet the release was still super popular, as people just assumed they needed more practice with the game. Have fun out there, =)
throwaway115 2y ago
Having written a 3d game engine from scratch, I had automated tests, but they were more comparable to "golden" tests, which are popular in the UI test world. Basically, my renderer needed to produce a pixel-perfect frame. If a pixel didn't match, an image diff was produced. This saved my butt numerous times when I broke subtle parts of the renderer.
vmaurin 2y ago
> why game devs don't unit test Sources ?
PlunderBunny 2y ago
I wrote a game using a 'bottom up' design (i.e. IO and business logic first), and I wrote unit tests for the business logic as I went. With no UI, I effectively tested and stepped through my code with unit tests. I had the luxury of working by myself at my own pace. I have a reasonably clean separation between the UI and the rest of the code, but I don't have any unit tests for the UI (I think - correct me if I'm wrong here - that would require integration tests rather than unit tests?) What I'm trying to say is that, if you don't do it this way around, and/or you have multiple programmers writing the game at once, and/or you _really_ optimise for performance, I can imagine that would make it much harder to write unit tests.
smokel 2y ago
"It's never good to be dogmatic." In some situations unit tests can be very effective and useful, such as in testing complex algorithms, or in code bases where some serious refactoring is required, and where one don't want to break existing behavior. In backend development, where user facing output is limited, there is typically no other practical way to check that things are working properly. However, in games, and typical front-end development, especially in its early stages, it can be beneficial to be as flexible as possible. And however way you put it, unit tests simply make your code more rigid. In the latter situation, some people prefer guard rails and find that they are more flexible with unit tests in place. Others prefer not to care about unit tests and attain higher productivity without them. Only when an application grows to a certain size where a developer does not naturally inspect typical behavior all day, and if quality is important, it starts to make sense to put in automated testing, because it is simply more cost effective. Similar reasoning goes for dynamic vs static typing. It seems that some people think that everyone should always use the same approach for any kind of software development, because it worked for them at some point in time. Over time I have grown a preference to avoid working with such people.
follower 2y ago
For an alternative perspective on testing & game development, here's a video I've seen from few years ago: * "Automated Testing of Gameplay Features in 'Sea of Thieves'": https://www.youtube.com/embed/X673tOi8pU8?si=uj_lcMEC9nvMpa6C https://www.youtube.com/embed/X673tOi8pU8?si=uj_lcMEC9nvMpa6... ~via https://www.gdcvault.com/play/1026366/Automated-Testing-of-Gameplay-Features https://www.gdcvault.com/play/1026366/Automated-Testing-of-G... : "Automated testing of gameplay features has traditionally not been embraced by the industry, due to the perceived time required and difficulty in creating reliable tests. Sea of Thieves however was created with automated testing for gameplay from the start. This session explains why automated testing was the right choice for Sea of Thieves and how it could benefit your game. It shows the framework that was built by Rare to let team members create automated tests quickly and easily, the different test types we built, and what level of test coverage was found to be appropriate. The session also contains best practices for making tests work reliably and efficiently, using clear worked through examples." Looks like there's also related talks in later years (which may or may not be currently available as free-to-view--I've not watched these ones): * "Lessons Learned in Adapting the 'Sea of Thieves' Automated Testing Methodology to 'Minecraft'": https://www.gdcvault.com/play/1027345/Lessons-Learned-in-Adapting-the https://www.gdcvault.com/play/1027345/Lessons-Learned-in-Ada... * "Automated Testing of Shader Code" (GDC 2024): https://schedule.gdconf.com/session/automated-testing-of-shader-code/899160 https://schedule.gdconf.com/session/automated-testing-of-sha...
frou_dh 2y ago
This makes me think of the claim you sometimes see that memory-safety is not that relevant for game development because in many cases games aren't security-sensitive software. But even putting security vulnerabilities aside completely, plain old memory corruption can be a major drag when it rears its head (and can even kill projects if the game can't be wrangled into being crash-free by the deadline). This particularly applies to games with huge codebases and numbers of programmers.
larsrc 2y ago
Disclaimer: never was a game dev You're conflating unit tests and functional/integration tests there. A unit test should test that a single function/method/class does what it's expected to do. The game design changes should change how they are put together, but not often what they do. If your setThingOnFire() method suddenly also flips things upside down, you're going to have a bad day. Instead your callers should add calls to flipThingUpsideDown().
bentt 2y ago
I don't see why design needs to have so much impact on whether there are Unit Tests. Most unit tests should be much lower level than anything which would be balanced or user tested out of relevance. You want stuff like "can the player character jump and clear these sets of obstacles which we are building all of our levels with?", and then a nice script that takes prerecorded input and sees if the outcome is determinant. So, this way, if someone inadvertantly changes gravity, or friction, or whatever in the basic systems that determine locomotion, you'll catch it early. Now, do most game makers take the time to do this? No, because they will likely have a lot else to do and make an excuse not to do it. However, for the most vital tech foundations, it is a good idea. What gamedev does tend to do more often is smoke testing. Just load up each level and see if it runs out of memory or not. Automated testing on build to see if something broke. It's less granular than unit testing, but when you're building over and over in the heat of a closeout on a project, this type of thing can tease out a breaking bug early as well. Overall, I like the title of the OP article, but not much that's said within.
ElectricBoogie 2y ago
Unit testing is something large organizations do to hire more devs than they need, and make them spend more time on any given project.

44 points by jgalecki 2y ago

Bugs are kind of the fun part of games. If every subroutine worked perfectly, you wouldn't have the chaos of real life. Some of players favorite mechanics are just bugs. (Overwatch example: Mercy's super jump, now a legitimate predictable mechanic that everyone can do, not just people that read the forums and watch YouTube videos about bugs. It started out as a bug, and it was so cool and skill-ceiling increasing that now it's just part of the game.) Having said that, sometimes you need unit tests. Overwatch had this bug where there is an ultimate ability called "amplification matrix" that is a window that you shoot through and the bullets do twice as much damage. One patch, that stopped working. This kind of issue is pretty easy to miss in play testing; if you're hitting headshots, then the bullets are doing the 2x damage they would if they were body shots that got properly amplified. If is very hard to tell damage numbers while play testing (as evidenced by how many patches are "we made character X do 1 more damage per bullet", and it smooths things out over the scale of millions of matches, but isn't really that noticeable to players unless breakpoints change). So for this reason, write an integration test where you set up this window thingie, put an enemy behind it, fire a bullet at a known point, and require.Equals(damage, 200). Ya just do it, so you don't ship the bug, make real people lose real MMR, and then have to "git stash" that cool thing you're working on today, check out the release branch, and uncomment the code that makes the amp matrix actually work. Games are just software engineering. Fun software engineering. But it's the same shit that your business logic brothers and sisters are working on. (Overwatch also had a really neat bug, that the community believes was due to a x == 0 check instead of an x < 0 check. If you pressed the right buttons while using Bastion's ultimate, you had infinite ammo. Normally it fires 3 air strikes, but if you got that counter to decrement twice and skip the == 0 check, then you got to do it an infinite number of times. (Well, actually 2^32 or 2^64 times. Eventually you'd integer overflow and have the chance to hit 0 again. Anyway, this was absolutely hilarious whenever it happened in game. The entire map would turn into a bunch of targets for artillery shells, the noise to alert you of incoming missiles would play 100 times more than normal, and it was total chaos as everyone on your team died. And not even that gamebreaking; both teams have the option to run the same characters, so you could just do it back to your opponent. Very fun, but they fixed the bug quickly. Follow up follow up: all of these silly bugs are in ultimates, which come up the least often of all abilities in the games. That's what happens with playtesting. You don't get test coverage where you need it. A test you write covers the stuff you're most scared about. A careful engineer that likes testing would have never shipped these.)

AndyPa32 2y ago
> Games are just software engineering. Fun software engineering. I do question the "fun" part. Midnight crunches, unpaid overtime and - as far as I have read - some of the worst working conditions in all of software engineering. I pass.
- jrockway 2y ago
  That is probably true, but you know you suffer for your art and all that. People don't really like software, but they love games. We know that games are just software, but it's so fun, that people forget that. It's pretty cool. Though to me, I kind of like getting 8 hours of sleep a night and playing other people's games. While getting paid more :/
- cableshaft 2y ago
  I've done both gamedev and webdev professionally. I don't miss the long hours or the low pay of gamedev, but I do miss what I worked on. Enough that I'm still working on my own games in my spare time. I still play the games I used to work on professionally from time to time, and like to show them off to people, even though none were financially successful (a few I worked on solo were at least popular, but we made some 'bad decisions in hindsight' or got really unlucky with release timing for all the games I worked on professionally). I don't care hardly at all about all the webdev projects I worked on professionally. They each had some intresting problem solving or coding challenges, and some of them were used WAY more than the games I made were played, but I'm still more into the games I made, and those are most likely to still be around after I'm dead, in a Flash game or ROM archive or torrent somewhere. Also, the worst long hours I ever had was actually in webdev. I once worked in software for a call center, and whenever the phone systems went down, I was expected to be on a call to help fix it (as there was 100+ call center employees that couldn't make calls and we were losing lots of money every minute they were down), sometimes for 16 hours or more sitting on a Zoom call, mostly waiting for people on the server teams to figure things out. And there was a data center migration that had some unexpected problems and required three of us to work for almost three days straight (I literally worked a 24 hour shift, then had 4 hours of sleep, then worked an 18 hour shift right after it). But that's not the norm in webdev at least. My current webdev job I only worked >40 hours a couple weeks to rework some issues with some junior dev's code for a feature before a big demo.

shaftway 2y ago

I don't buy this argument. Most game developers I know have said that unit tests are a waste of time so they never use them, but they're struggling with making changes to utility code and making sure that it doesn't do the wrong thing. Y'know, what unit tests are for. I think the key here is that the perceived cost / benefit ratio is too high. It's the perception that drives their behavior though. I'm in a company now that has zero unit tests, because they just don't see the value in it (and in their case they may be right for a whole slew of reasons). Also, remember that games are not very long-lived pieces of software. You build it, release it, maybe patch it, and move on. If the game moves to version 2 then you're probably going to re-write most of the game from scratch. When you support software for a decade then the code is what's valuable, and unit tests keep institutional knowledge about the code. But with disposable software like games, the mechanics of the game and IP are what's valuable. Why would you write a unit test for something you know you're going to throw away in 6 months?

Atotalnoob 2y ago
I am curious as to why your current company does not have unit tests. Do you mind sharing?
- shaftway 2y ago
  We produce a library that gets included in software made by our clients, and we have several thousand clients. The uptake on new releases is low (most of the clients believe in "if it ain't broke, don't fix it"). So every release has the potential to live in the wild and need support for a long time. We're also in an industry with a ton of competitors. On top of that, the company was founded by some very junior engineers. for most of them this was their first or second job out of college. Literally every anti-pattern is in our codebase, and a lot of them are considered best practices by them. Unit tests were perceived as a cost with little benefit, so none were written. New engineers were almost always new grads to save on money. These facts combined make for an interesting environment. For starters, leadership is afraid to ship new code, or even refactor existing code. Partially because nobody knows how it works, partially because they don't have unit tests to verify that things are going well. All new code has to be gated by feature flags (there's an experiment right now to switch from try-finally to try-with-resources). If there isn't a business reason to add code, it gets rejected (I had a rejected PR that removed a "synchronized" block from around "return boolValue;"). And it's hard to say they're wrong. If we push out a bad release, there's a very real chance that our customers will pack up and migrate to one of our competitors. Why risk it? And the team's experience level plays a role too. With so many junior engineers and so much coding skill in-breeding, "best practices" have become pretty painful. Code is written without an eye towards future maintainability, and the classes are a gas factory mixed with a god object. It's not uncommon to trace a series of calls through a dozen classes, looping back to classes that you've already looked at. And trying to isolate chunks of the code is difficult. I recently tried to isolate 6 classes and I ended up with an interface that used 67 methods from the god object, ranging from logging, to thread management, to http calls, to state manipulation. And because nobody else on the team has significant experience elsewhere, nobody else really sees the value of unit tests. They've all been brought up in this environment where unit test are not mentioned, and so it has ingrained this idea that they're useless. So the question is how do you fix this and move forward? Ideally we'd start by refactoring a couple of these classes so that they could be isolated and tested. While management doesn't see significant value in unit tests, they're not strictly against them, but they are against refactoring code. So we can't really add unit tests on the risky code. The only places that you can really add them without pushback would be in the simplest utility classes, which would benefit from them the least, and in doing so prove to management that unit tests aren't really valuable. And I mean the SIMPLEST utility classes. Most of our utility classes require the god object so that we can log and get feature flags. I say we take off and nuke the entire site from orbit (start over from scratch with stronger principles). It's the only way to be sure. But there's no way I'm convincing management to let the entire dev team have the year they'd need to do that with feature parity, and leadership would only see it as a massive number of bugs to fix. In the meantime developer velocity is slowing, but management seems to see that as a good thing. Slower development translates into more stable code in their minds. And the company makes enough that it pays well and can't figure out what to do with the excess money. So nobody really sees a problem. Our recruiters actually make this a selling point, making fun of other companies that say their code is "well organized".

whatasaas 2y ago

Seems like an excuse that might be fine for small indie teams for a while. The blog certainly blurs the lines between unit and functional tests. In the end, even modest code coverage can pay off. Tests help with code review, understanding the codebase, and can provide an easy map for debugging. But if you’re in an environment where everyone is constantly demanding changes and only testing the happy path, then good luck.

fwlr 2y ago

Arguments against testing tend to fall prey to the von Neumann Objection: they insist there is something tests can’t catch, and then they tell you precisely what it is that tests can’t catch… so you can always imagine writing tests for that specific thing. E.g. this article uses an example of removing the number 5, causing the developer to have to implement a base-9 numbering system. Unit tests that confirm this custom base number system is working as expected would be extremely reassuring to have. Alternatively, you could keep the base-10 system everyone is familiar with, and just have logic to eliminate or transform any 5s. This would normally be far too risky, but high coverage testing could provide strong enough assurance to trust that your “patched base-10” isn’t letting any 5s through. The same is true for the other examples - unit testing feels like the first thing I’d reach for when told about flaming numbers.

magoghm 2y ago
Tests can't catch race conditions in multithreaded code. Now that I told you what the tests can't catch, can you imagine writing tests for that specific thing?
- a_t48 2y ago
  I've written tests around multithreaded code, but they typically catch them in a statistical manner - either running a bit of code many times over to try and catch an edge condition, or by overloading the system to persuade rarer orderings to occur. There's also https://clang.llvm.org/docs/ThreadSafetyAnalysis.html https://clang.llvm.org/docs/ThreadSafetyAnalysis.html which can statically catch some threading issues, though I've not used it much myself.
- jonex 2y ago
  tsan will catch a bunch of potential race conditions for you, under the condition that you run it somehow. How to make sure it's run? Well, add a test for the relevant code and add it to your tsan run in your CI and you'll certainly catch a bunch of race conditions over time. This has saved me a bunch of times when I've be doing work in code with proneness to those kind of issues. Sometimes it will just lead to a flaky test, but the investigation of the flake will usually find the root cause in the end.

kelseydh 2y ago

Most video game bugs are subtle and not things that are easy to catch with unit testing because they are dynamic systems with many interacting parts. The interaction is where the bugs come from. QA processes do a good job catching the rest.

lionkor 2y ago
I would bet that a lot of those bugs come from utility code that is testable
- Aerroon 2y ago
  Perhaps in development, but the stuff that tends to make it into the release of games seems to be gameplay related. Npc behaviors not lining up, the developer literally not implementing certain stats in the game (looking at you, Diablo 4), graphical bugs caused by something not loading or loading too slowly, performance issues from something loading 1000 copies of itself etc.
  - hahn-kev 2y ago
    And where do you think many of those bugs come from? Usually utility code that isn't working in some edge case, those types of things can be easy to pull out and test in their own independent of the game
    - sfn42 2y ago
      I don't think I've ever seen someone argue against tests without giving the clear impression that they've never written a test in their life.

riffraff 2y ago

I am not convinced of the argument that games change a lot. I do buy the argument that the trade off between effort and value is different, but that's because it's harder to unit test user interactions than it is to unit test a physics engine. It's more or less the reason in the early life of the web few did end to end testing involving browsers, or unit tested iOS apps in the first releases of the iPhone.

mistercow 2y ago

I’ve found that the one thing you can always count on engineers to do is to dismiss sensible tools from adjacent domains using flimsy, post hoc justifications. All product development involves poorly defined boundaries where the product meets the user, where requirements shift frequently, and where the burdens of test maintenance have to be weighed against the benefits. You don’t throw out all of unit testing because it doesn’t work well for a subset of your code. You throw out all of unit testing because writing tests is annoying, none of your coworkers have set it up, and the rest of your industry doesn’t do it, so you feel justified in not doing it either.

stouset 2y ago
Right. And because the rest of the industry isn’t doing it, there’s no institutional knowledge of how to do it well. So someone tries it, they do a crap job of it out of understandable ignorance, and rather than taking forward any lessons learned the effort is discarded as a waste of time.
jayd16 2y ago
So wait, who's being dismissive of who's practices here?
- mistercow 2y ago
  I didn’t say engineers are dismissive of other engineers’ practices. The general pattern is “that makes sense for your field, but we can’t use it because…” followed by silly reasons. I was guilty of this myself back when I was an indie dev. It took me an embarrassingly long time, for example, to admit that git wasn’t just something teams needed to coordinate, and that I should be using it as the sole developer of a project.
  - eviks 2y ago
    Interesting, was the impetus for change some big issue you've run into? Or just a gradual accumulation of knowledge about other people experiences made you reconsider? Or something else?

rkachowski 2y ago

It's an interesting idea, but here you have the game designer taking the place of the product manager stereotype - coming up with bizarre unfeasible ideas and the programmer is to make it happen. In any games company I've worked for the designer is responsible for mapping and balancing the rules and mechanics of the game, they would provide a specification of what "red vs blue numbers" would look like and a balanced idea of how to remove the number 5 from the game (balancing and changing the rules like this being entirely within the domain of game design). incidentally any game company I've worked at has had an extensive set of test suites.

epgui 2y ago

What nonsense that is… The idea that intentional mathematical design / correct, well-specified behaviour doesn’t apply to games is absurd.

SillyUsername 2y ago

Ok I know which AAA game studio this might be because I interviewed with them and had to sign an NDA. In their case their flagship game is full of bugs, and they had to ship their product asap pre-aquistion when they were a startup. Because of the mentality of the managers, and weak minded devs, they don't write unit tests, and instead spend the vast majority of their days fighting bugs, so much so they have to hire dedicated staff for their (single game) backlog as they were struggling to keep up "with its success". This is BS of course, I saw their backlog and it was a shit show, with Devs expected to work overtime free of charge to get actual features out (funny how this works isn't it, never affects the business execs' time/life who make the demands of no tests). I was asked what I would bring to the company to help them support their now AAA game, and I stated up front "more unit tests" and indirectly criticised their lack of them. I got a call later that day that (the manager thought) "I would not be a good fit". I got a lead job elsewhere that has the company's highest performing team, literally because of testing practices being well balanced between time and effectiveness (i.e. don't bother with low value tests, add tests if you find a bug etc, if an integration test takes too long leave it and use unit tests). I think back to that interview every time I interview at games studios now, and wonder if I shouldn't push unit tests if they're missing. I'd still do it. The managers at that job were assholes to their developers, and I now recognise the trait in a company.

Joel_Mckay 2y ago

1. Most game engines have the horrible compatibility layers abstracted away, and already fully tested under previous mass deployments 2. Anything primarily visual, audio, and control input based is extremely hard to reliably automate testing. Thus, if the clipping glitches are improbable and hardly noticeable... no one cares. Some people did get around the finer game-play issues by simply allowing their AI character to cheat. Mortal Kombat II was famous for the impossible moves and combos the AI would inflict on players... yet the release was still super popular, as people just assumed they needed more practice with the game. Have fun out there, =)

throwaway115 2y ago

Having written a 3d game engine from scratch, I had automated tests, but they were more comparable to "golden" tests, which are popular in the UI test world. Basically, my renderer needed to produce a pixel-perfect frame. If a pixel didn't match, an image diff was produced. This saved my butt numerous times when I broke subtle parts of the renderer.

vmaurin 2y ago

> why game devs don't unit test Sources ?

PlunderBunny 2y ago

I wrote a game using a 'bottom up' design (i.e. IO and business logic first), and I wrote unit tests for the business logic as I went. With no UI, I effectively tested and stepped through my code with unit tests. I had the luxury of working by myself at my own pace. I have a reasonably clean separation between the UI and the rest of the code, but I don't have any unit tests for the UI (I think - correct me if I'm wrong here - that would require integration tests rather than unit tests?) What I'm trying to say is that, if you don't do it this way around, and/or you have multiple programmers writing the game at once, and/or you _really_ optimise for performance, I can imagine that would make it much harder to write unit tests.

smokel 2y ago

"It's never good to be dogmatic." In some situations unit tests can be very effective and useful, such as in testing complex algorithms, or in code bases where some serious refactoring is required, and where one don't want to break existing behavior. In backend development, where user facing output is limited, there is typically no other practical way to check that things are working properly. However, in games, and typical front-end development, especially in its early stages, it can be beneficial to be as flexible as possible. And however way you put it, unit tests simply make your code more rigid. In the latter situation, some people prefer guard rails and find that they are more flexible with unit tests in place. Others prefer not to care about unit tests and attain higher productivity without them. Only when an application grows to a certain size where a developer does not naturally inspect typical behavior all day, and if quality is important, it starts to make sense to put in automated testing, because it is simply more cost effective. Similar reasoning goes for dynamic vs static typing. It seems that some people think that everyone should always use the same approach for any kind of software development, because it worked for them at some point in time. Over time I have grown a preference to avoid working with such people.

follower 2y ago

For an alternative perspective on testing & game development, here's a video I've seen from few years ago: * "Automated Testing of Gameplay Features in 'Sea of Thieves'": https://www.youtube.com/embed/X673tOi8pU8?si=uj_lcMEC9nvMpa6C https://www.youtube.com/embed/X673tOi8pU8?si=uj_lcMEC9nvMpa6... ~via https://www.gdcvault.com/play/1026366/Automated-Testing-of-Gameplay-Features https://www.gdcvault.com/play/1026366/Automated-Testing-of-G... : "Automated testing of gameplay features has traditionally not been embraced by the industry, due to the perceived time required and difficulty in creating reliable tests. Sea of Thieves however was created with automated testing for gameplay from the start. This session explains why automated testing was the right choice for Sea of Thieves and how it could benefit your game. It shows the framework that was built by Rare to let team members create automated tests quickly and easily, the different test types we built, and what level of test coverage was found to be appropriate. The session also contains best practices for making tests work reliably and efficiently, using clear worked through examples." Looks like there's also related talks in later years (which may or may not be currently available as free-to-view--I've not watched these ones): * "Lessons Learned in Adapting the 'Sea of Thieves' Automated Testing Methodology to 'Minecraft'": https://www.gdcvault.com/play/1027345/Lessons-Learned-in-Adapting-the https://www.gdcvault.com/play/1027345/Lessons-Learned-in-Ada... * "Automated Testing of Shader Code" (GDC 2024): https://schedule.gdconf.com/session/automated-testing-of-shader-code/899160 https://schedule.gdconf.com/session/automated-testing-of-sha...

frou_dh 2y ago

This makes me think of the claim you sometimes see that memory-safety is not that relevant for game development because in many cases games aren't security-sensitive software. But even putting security vulnerabilities aside completely, plain old memory corruption can be a major drag when it rears its head (and can even kill projects if the game can't be wrangled into being crash-free by the deadline). This particularly applies to games with huge codebases and numbers of programmers.

larsrc 2y ago

Disclaimer: never was a game dev You're conflating unit tests and functional/integration tests there. A unit test should test that a single function/method/class does what it's expected to do. The game design changes should change how they are put together, but not often what they do. If your setThingOnFire() method suddenly also flips things upside down, you're going to have a bad day. Instead your callers should add calls to flipThingUpsideDown().

bentt 2y ago

I don't see why design needs to have so much impact on whether there are Unit Tests. Most unit tests should be much lower level than anything which would be balanced or user tested out of relevance. You want stuff like "can the player character jump and clear these sets of obstacles which we are building all of our levels with?", and then a nice script that takes prerecorded input and sees if the outcome is determinant. So, this way, if someone inadvertantly changes gravity, or friction, or whatever in the basic systems that determine locomotion, you'll catch it early. Now, do most game makers take the time to do this? No, because they will likely have a lot else to do and make an excuse not to do it. However, for the most vital tech foundations, it is a good idea. What gamedev does tend to do more often is smoke testing. Just load up each level and see if it runs out of memory or not. Automated testing on build to see if something broke. It's less granular than unit testing, but when you're building over and over in the heat of a closeout on a project, this type of thing can tease out a breaking bug early as well. Overall, I like the title of the OP article, but not much that's said within.

ElectricBoogie 2y ago

Unit testing is something large organizations do to hire more devs than they need, and make them spend more time on any given project.