6 ms·

I don't buy this argument. Most game developers I know have said that unit tests are a waste of time so they never use them, but they're struggling with making

I don't buy this argument. Most game developers I know have said that unit tests are a waste of time so they never use them, but they're struggling with making changes to utility code and making sure that it doesn't do the wrong thing. Y'know, what unit tests are for. I think the key here is that the perceived cost / benefit ratio is too high. It's the perception that drives their behavior though. I'm in a company now that has zero unit tests, because they just don't see the value in it (and in their case they may be right for a whole slew of reasons). Also, remember that games are not very long-lived pieces of software. You build it, release it, maybe patch it, and move on. If the game moves to version 2 then you're probably going to re-write most of the game from scratch. When you support software for a decade then the code is what's valuable, and unit tests keep institutional knowledge about the code. But with disposable software like games, the mechanics of the game and IP are what's valuable. Why would you write a unit test for something you know you're going to throw away in 6 months?

Atotalnoob 2y ago
I am curious as to why your current company does not have unit tests. Do you mind sharing?
- shaftway 2y ago
  We produce a library that gets included in software made by our clients, and we have several thousand clients. The uptake on new releases is low (most of the clients believe in "if it ain't broke, don't fix it"). So every release has the potential to live in the wild and need support for a long time. We're also in an industry with a ton of competitors. On top of that, the company was founded by some very junior engineers. for most of them this was their first or second job out of college. Literally every anti-pattern is in our codebase, and a lot of them are considered best practices by them. Unit tests were perceived as a cost with little benefit, so none were written. New engineers were almost always new grads to save on money. These facts combined make for an interesting environment. For starters, leadership is afraid to ship new code, or even refactor existing code. Partially because nobody knows how it works, partially because they don't have unit tests to verify that things are going well. All new code has to be gated by feature flags (there's an experiment right now to switch from try-finally to try-with-resources). If there isn't a business reason to add code, it gets rejected (I had a rejected PR that removed a "synchronized" block from around "return boolValue;"). And it's hard to say they're wrong. If we push out a bad release, there's a very real chance that our customers will pack up and migrate to one of our competitors. Why risk it? And the team's experience level plays a role too. With so many junior engineers and so much coding skill in-breeding, "best practices" have become pretty painful. Code is written without an eye towards future maintainability, and the classes are a gas factory mixed with a god object. It's not uncommon to trace a series of calls through a dozen classes, looping back to classes that you've already looked at. And trying to isolate chunks of the code is difficult. I recently tried to isolate 6 classes and I ended up with an interface that used 67 methods from the god object, ranging from logging, to thread management, to http calls, to state manipulation. And because nobody else on the team has significant experience elsewhere, nobody else really sees the value of unit tests. They've all been brought up in this environment where unit test are not mentioned, and so it has ingrained this idea that they're useless. So the question is how do you fix this and move forward? Ideally we'd start by refactoring a couple of these classes so that they could be isolated and tested. While management doesn't see significant value in unit tests, they're not strictly against them, but they are against refactoring code. So we can't really add unit tests on the risky code. The only places that you can really add them without pushback would be in the simplest utility classes, which would benefit from them the least, and in doing so prove to management that unit tests aren't really valuable. And I mean the SIMPLEST utility classes. Most of our utility classes require the god object so that we can log and get feature flags. I say we take off and nuke the entire site from orbit (start over from scratch with stronger principles). It's the only way to be sure. But there's no way I'm convincing management to let the entire dev team have the year they'd need to do that with feature parity, and leadership would only see it as a massive number of bugs to fix. In the meantime developer velocity is slowing, but management seems to see that as a good thing. Slower development translates into more stable code in their minds. And the company makes enough that it pays well and can't figure out what to do with the excess money. So nobody really sees a problem. Our recruiters actually make this a selling point, making fun of other companies that say their code is "well organized".
  - Atotalnoob 2y ago
    Thank you for the write up. That seems like a bad scenario with bad technical management. I am wondering if you have considered not trying to implement unit tests and think about end to end tests. This might be easier for antitesting people to buy into because it’s directly ensuring your end users get the desired outcomes. It doesn’t matter what bad terrible practices you have inside your library if the output is correct… If you input 1+1, and it outputs 5, it will be obvious how this can be an issue. What this will enable you to do is get some quick wins and make refactoring safer. If management still says no, I see 3 major choices. 1. Quit 2. Write your tests and keep them to yourself 3. Mind control
    - other_herbert 2y ago
      As a corollary to 2, management tends to love graphs… whatever your using to build should have a plugin that could show unit test success counts and generate even a simple line graph… that alone might be enough incentive to add more testing
      - Atotalnoob 2y ago
        I wouldn’t use the term “unit test” if they are negative on the concept. Edit; in fact, don’t say test at all. Talk about verification of the output
    - shaftway 2y ago
      We do have an integration test that runs just before releases. I've never seen it fail, even when something was obviously broken, so I question the utility of it. There's a specific person in charge of maintaining it. I've opted for option 4: continue to write code the way they want it written and keep cashing my paychecks. In the meantime there are tons of other improvements that I'm working on, some of which have a more direct impact on business revenue (which has a direct impact on my personal revenue).
  - phito 2y ago
    How do you stay sane working with clowns?
feoren 2y ago
> Also, remember that games are not very long-lived pieces of software. You build it, release it, maybe patch it, and move on. This was true a couple decades ago. Nowadays many games are cash cows for decades. Path of Exile was released in 2013, Minecraft in 2011, and World of Warcraft in 2004, and all of those continue to receive regular updates (and have over the course of their lives) and still make plenty of money today. Dwarf Fortress has been in continual development since 2002! (Although probably not your ideal cash-flow model.) Or you have the EA Sports model where you use the same "engine" and just re-skin some things and re-release the same game over and over. There has been a new "Football Manager" game every year since 2005 -- do you really think they throw out all their code and start over every year?
- shaftway 2y ago
  I maintain that the majority of games are still disposable, despite the occasional subscription model or long-lived hit that pops up. Remember that most games aren't made by AAA studios. Wasn't Minecraft completely rewritten from scratch in Java after a few years? And the EA one, like you said, it's just model updates. Very few gameplay mechanics get more than a simple tweak. Just recompile with the new models. You don't need unit tests if the code never changes.
  - mark38848 2y ago
    I think Minecraft was originally written in Java and rewritten in a good programming language (i.e. not Java).
    - grumpyprole 2y ago
      Whether or not one thinks C++ is a "good" language, I always thought that (original) Minecraft busted the myth that blockbuster games had to be written in C++.
    - CuriousSkeptic 2y ago
      Being written in Java was probably instrumental in enabling the huge modding community around Minecraft. Which in turn was probably in large part responsible for its success.
abdullahkhalids 2y ago
Valve became serious about software quality in Dota 2 around 2017 - about 7 years after launch. Before that game updates were accompanied with lots of bugs that would take weeks to fix. These days, there are still tons of bugs, but much better than before. They just released one of the biggest updates in the game's history this week, and there are hardly any bugs being reported. I am pretty sure there is some sort of automated testing happening that is catching these bugs before release.
- zubspace 2y ago
  Reminds me of an article about the testing infrastructure of League and Legends [1] back in 2016. 5500 tests per build in 1 to 2 hours. Games are extremely hard to test. For me it falls into the same category like GUI testing frameworks which imho are extremely annoying and brittle. Except that games are comparable to a user interface consisting of many buttons which you can short and long press and drag around while at the same time other bots are pressing the same buttons, sharing the same state influenced by a physics engine. How do you test such a ball of mud which also constantly changes by devs trying to follow the fun? Yes you can unittest individual, reusable parts. But integration tests, which require large, time sensitive modules, all strapped together and running at the same time? It's mindboggling hard. Moreover if you're in a conceptual phase of development and prototyping and idea, tests make no sense. The requirements change all the time and complex tests hold you back. But the funny thing is, that game development stays in that phase most of the time. And when the game is done, you start a new one with a completely different set of requirements. There are exceptions, like League of Legends. The game left the conceptual phase many years ago and its rules are set in stone. And a game which runs successfully for that long is super rare. [1] https://technology.riotgames.com/news/automated-testing-league-legends https://technology.riotgames.com/news/automated-testing-leag...
  - abdullahkhalids 2y ago
    I doubt Dota 2 devs are writing code like this to test. The game is far too complicated, even more so than league, and changes a lot over the years, for this to be viable. Dota 2 and openai had a collaboration in 2018ish, and during this time the Dota 2 bots system was reworked completely. They already can generate videos of every spell in action [1], and I would assume this is done by asking AI bots to demonstrate the spell. My guess is that before pushing out an update, a human looks at these videos and other more complex interaction videos for every major change, along with relevant numbers (damage, healing, movement speed), and see if everything makes sense. I think this, because a lot of times recently, changes in one hero often cause an un-updated hero to break, because they had some backend similarity. And the patch is released with the bug. Then again, there is no public info, so all the above are wild speculations. [1] example https://www.dota2.com/hero/treantprotector https://www.dota2.com/hero/treantprotector
treflop 2y ago
I’ve seen people slog through untested code where they fear to make a change but I’ve also seen people slog through code with too much test coverage where the tests go through constant churn. I don’t understand why people don’t just add one test even if the codebase otherwise has zero tests if they’re so scared of one area and I don’t get why people keep adding excessive coverage if it’s wasting their time. It’s like people pick a stance and then stick with it forever when I couldn’t care less how I’ve been doing something for 10 years if today you showed me a better way.
- smrq 2y ago
  This is the way. My work codebase has probably 5% unit test coverage -- it's frontend and a lot of it isn't sensible to unit test -- but I'm quite happy to have the tests we do. If it's nontrivial logic, just test it. If it isn't (it's trivial, it's aesthetic, whatever your reason)... just don't.
- wesselbindt 2y ago
  >too much test coverage where the tests go through constant churn This doesn't sound so much as too much coverage but rather like having your automated tests be coupled to implementation details. This has a multitude of possible causes, for example too the tests being too granular (prefer testing at the boundary of your system). I've worked in codebases where test-implementation detail coupling was taken seriously, and in those I've rarely had to write a commit message like "fix tests", and all that without losing coverage.
  - kbolino 2y ago
    It feels like there are two levels of test writing proficiency. The first is writing the tests that have high benefit and low cost: e.g. pure functions with comprehensive tabular tests, simple method chains that have well defined sequential behavior and few dependencies, high value regression tests against detailed bug reports, etc. IMO it's harder to argue against writing these tests than to argue for writing them. Then there's the second level of proficiency, related to what you're discussing with "test-implementation detail coupling". This is the domain of high test coverage, repeatable end-to-end tests, automated QA, etc. I've always struggled with this next level and I've yet to work in any environment where it was done effectively (if at all). It's also harder to argue for this kind of testing because the tests often end up brittle and false negatives drown out the benefits. Moreover, most of the discourse centers around the first level of proficiency only and it's much harder to find digestible advice for achieving the second.
jrockway 2y ago
Testing is a continuum. I don't write a test for every change. Sometimes I spend a week writing tests for a simple change. I will say that I've never said "I wish I didn't write a test for that". I have also never said, "your PR is fine, but please delete that test, it's useless". I throw away a lot of code. I still test stuff I expect to throw away. That's because it probably needs to run once before I throw it away, and I can't start throwing it away until it works :/ What it comes down to is what else you have to spend your time on. Sometimes you need to experiment with a feature; get it out to customers, and if it's buggy and rough around the edges, it's OK, because you were just trying out the idea. But sometimes that's not what you want; whatever time you spend on support back and forth finding a bug would have been better spent not doing that. The customer needed something rock solid, not an experiment. Test that so they don't have to. There are no rules. "Write a test for every change" is just as invalid and unworkable as "Never write any tests". It's a spectrum, and each change is going to land somewhere different. If you're unsure, ask a coworker. I have been testing stuff for 20+ years, and I usually guess OK (that is when I take a shortcut and don't test as much as I should, it's rarely the thing that caused the production outage), but a guess is just that, a guess. Solicit opinions.
withinboredom 2y ago
Also, non-testable code is often faster (as in cpu time).
lithos 2y ago
I've set unit tests for functions that are mostly math (base cases of collision, parts of bot predictive logic, and similar). Though I'm sitting at a hobbyist with electrical and commissioning background.

Atotalnoob 2y ago

I am curious as to why your current company does not have unit tests. Do you mind sharing?

shaftway 2y ago
We produce a library that gets included in software made by our clients, and we have several thousand clients. The uptake on new releases is low (most of the clients believe in "if it ain't broke, don't fix it"). So every release has the potential to live in the wild and need support for a long time. We're also in an industry with a ton of competitors. On top of that, the company was founded by some very junior engineers. for most of them this was their first or second job out of college. Literally every anti-pattern is in our codebase, and a lot of them are considered best practices by them. Unit tests were perceived as a cost with little benefit, so none were written. New engineers were almost always new grads to save on money. These facts combined make for an interesting environment. For starters, leadership is afraid to ship new code, or even refactor existing code. Partially because nobody knows how it works, partially because they don't have unit tests to verify that things are going well. All new code has to be gated by feature flags (there's an experiment right now to switch from try-finally to try-with-resources). If there isn't a business reason to add code, it gets rejected (I had a rejected PR that removed a "synchronized" block from around "return boolValue;"). And it's hard to say they're wrong. If we push out a bad release, there's a very real chance that our customers will pack up and migrate to one of our competitors. Why risk it? And the team's experience level plays a role too. With so many junior engineers and so much coding skill in-breeding, "best practices" have become pretty painful. Code is written without an eye towards future maintainability, and the classes are a gas factory mixed with a god object. It's not uncommon to trace a series of calls through a dozen classes, looping back to classes that you've already looked at. And trying to isolate chunks of the code is difficult. I recently tried to isolate 6 classes and I ended up with an interface that used 67 methods from the god object, ranging from logging, to thread management, to http calls, to state manipulation. And because nobody else on the team has significant experience elsewhere, nobody else really sees the value of unit tests. They've all been brought up in this environment where unit test are not mentioned, and so it has ingrained this idea that they're useless. So the question is how do you fix this and move forward? Ideally we'd start by refactoring a couple of these classes so that they could be isolated and tested. While management doesn't see significant value in unit tests, they're not strictly against them, but they are against refactoring code. So we can't really add unit tests on the risky code. The only places that you can really add them without pushback would be in the simplest utility classes, which would benefit from them the least, and in doing so prove to management that unit tests aren't really valuable. And I mean the SIMPLEST utility classes. Most of our utility classes require the god object so that we can log and get feature flags. I say we take off and nuke the entire site from orbit (start over from scratch with stronger principles). It's the only way to be sure. But there's no way I'm convincing management to let the entire dev team have the year they'd need to do that with feature parity, and leadership would only see it as a massive number of bugs to fix. In the meantime developer velocity is slowing, but management seems to see that as a good thing. Slower development translates into more stable code in their minds. And the company makes enough that it pays well and can't figure out what to do with the excess money. So nobody really sees a problem. Our recruiters actually make this a selling point, making fun of other companies that say their code is "well organized".
- Atotalnoob 2y ago
  Thank you for the write up. That seems like a bad scenario with bad technical management. I am wondering if you have considered not trying to implement unit tests and think about end to end tests. This might be easier for antitesting people to buy into because it’s directly ensuring your end users get the desired outcomes. It doesn’t matter what bad terrible practices you have inside your library if the output is correct… If you input 1+1, and it outputs 5, it will be obvious how this can be an issue. What this will enable you to do is get some quick wins and make refactoring safer. If management still says no, I see 3 major choices. 1. Quit 2. Write your tests and keep them to yourself 3. Mind control
  - other_herbert 2y ago
    As a corollary to 2, management tends to love graphs… whatever your using to build should have a plugin that could show unit test success counts and generate even a simple line graph… that alone might be enough incentive to add more testing
    - Atotalnoob 2y ago
      I wouldn’t use the term “unit test” if they are negative on the concept. Edit; in fact, don’t say test at all. Talk about verification of the output
  - shaftway 2y ago
    We do have an integration test that runs just before releases. I've never seen it fail, even when something was obviously broken, so I question the utility of it. There's a specific person in charge of maintaining it. I've opted for option 4: continue to write code the way they want it written and keep cashing my paychecks. In the meantime there are tons of other improvements that I'm working on, some of which have a more direct impact on business revenue (which has a direct impact on my personal revenue).
- phito 2y ago
  How do you stay sane working with clowns?

feoren 2y ago

> Also, remember that games are not very long-lived pieces of software. You build it, release it, maybe patch it, and move on. This was true a couple decades ago. Nowadays many games are cash cows for decades. Path of Exile was released in 2013, Minecraft in 2011, and World of Warcraft in 2004, and all of those continue to receive regular updates (and have over the course of their lives) and still make plenty of money today. Dwarf Fortress has been in continual development since 2002! (Although probably not your ideal cash-flow model.) Or you have the EA Sports model where you use the same "engine" and just re-skin some things and re-release the same game over and over. There has been a new "Football Manager" game every year since 2005 -- do you really think they throw out all their code and start over every year?

shaftway 2y ago
I maintain that the majority of games are still disposable, despite the occasional subscription model or long-lived hit that pops up. Remember that most games aren't made by AAA studios. Wasn't Minecraft completely rewritten from scratch in Java after a few years? And the EA one, like you said, it's just model updates. Very few gameplay mechanics get more than a simple tweak. Just recompile with the new models. You don't need unit tests if the code never changes.
- mark38848 2y ago
  I think Minecraft was originally written in Java and rewritten in a good programming language (i.e. not Java).
  - grumpyprole 2y ago
    Whether or not one thinks C++ is a "good" language, I always thought that (original) Minecraft busted the myth that blockbuster games had to be written in C++.
  - CuriousSkeptic 2y ago
    Being written in Java was probably instrumental in enabling the huge modding community around Minecraft. Which in turn was probably in large part responsible for its success.

abdullahkhalids 2y ago

Valve became serious about software quality in Dota 2 around 2017 - about 7 years after launch. Before that game updates were accompanied with lots of bugs that would take weeks to fix. These days, there are still tons of bugs, but much better than before. They just released one of the biggest updates in the game's history this week, and there are hardly any bugs being reported. I am pretty sure there is some sort of automated testing happening that is catching these bugs before release.

zubspace 2y ago
Reminds me of an article about the testing infrastructure of League and Legends [1] back in 2016. 5500 tests per build in 1 to 2 hours. Games are extremely hard to test. For me it falls into the same category like GUI testing frameworks which imho are extremely annoying and brittle. Except that games are comparable to a user interface consisting of many buttons which you can short and long press and drag around while at the same time other bots are pressing the same buttons, sharing the same state influenced by a physics engine. How do you test such a ball of mud which also constantly changes by devs trying to follow the fun? Yes you can unittest individual, reusable parts. But integration tests, which require large, time sensitive modules, all strapped together and running at the same time? It's mindboggling hard. Moreover if you're in a conceptual phase of development and prototyping and idea, tests make no sense. The requirements change all the time and complex tests hold you back. But the funny thing is, that game development stays in that phase most of the time. And when the game is done, you start a new one with a completely different set of requirements. There are exceptions, like League of Legends. The game left the conceptual phase many years ago and its rules are set in stone. And a game which runs successfully for that long is super rare. [1] https://technology.riotgames.com/news/automated-testing-league-legends https://technology.riotgames.com/news/automated-testing-leag...
- abdullahkhalids 2y ago
  I doubt Dota 2 devs are writing code like this to test. The game is far too complicated, even more so than league, and changes a lot over the years, for this to be viable. Dota 2 and openai had a collaboration in 2018ish, and during this time the Dota 2 bots system was reworked completely. They already can generate videos of every spell in action [1], and I would assume this is done by asking AI bots to demonstrate the spell. My guess is that before pushing out an update, a human looks at these videos and other more complex interaction videos for every major change, along with relevant numbers (damage, healing, movement speed), and see if everything makes sense. I think this, because a lot of times recently, changes in one hero often cause an un-updated hero to break, because they had some backend similarity. And the patch is released with the bug. Then again, there is no public info, so all the above are wild speculations. [1] example https://www.dota2.com/hero/treantprotector https://www.dota2.com/hero/treantprotector

treflop 2y ago

I’ve seen people slog through untested code where they fear to make a change but I’ve also seen people slog through code with too much test coverage where the tests go through constant churn. I don’t understand why people don’t just add one test even if the codebase otherwise has zero tests if they’re so scared of one area and I don’t get why people keep adding excessive coverage if it’s wasting their time. It’s like people pick a stance and then stick with it forever when I couldn’t care less how I’ve been doing something for 10 years if today you showed me a better way.

smrq 2y ago
This is the way. My work codebase has probably 5% unit test coverage -- it's frontend and a lot of it isn't sensible to unit test -- but I'm quite happy to have the tests we do. If it's nontrivial logic, just test it. If it isn't (it's trivial, it's aesthetic, whatever your reason)... just don't.
wesselbindt 2y ago
>too much test coverage where the tests go through constant churn This doesn't sound so much as too much coverage but rather like having your automated tests be coupled to implementation details. This has a multitude of possible causes, for example too the tests being too granular (prefer testing at the boundary of your system). I've worked in codebases where test-implementation detail coupling was taken seriously, and in those I've rarely had to write a commit message like "fix tests", and all that without losing coverage.
- kbolino 2y ago
  It feels like there are two levels of test writing proficiency. The first is writing the tests that have high benefit and low cost: e.g. pure functions with comprehensive tabular tests, simple method chains that have well defined sequential behavior and few dependencies, high value regression tests against detailed bug reports, etc. IMO it's harder to argue against writing these tests than to argue for writing them. Then there's the second level of proficiency, related to what you're discussing with "test-implementation detail coupling". This is the domain of high test coverage, repeatable end-to-end tests, automated QA, etc. I've always struggled with this next level and I've yet to work in any environment where it was done effectively (if at all). It's also harder to argue for this kind of testing because the tests often end up brittle and false negatives drown out the benefits. Moreover, most of the discourse centers around the first level of proficiency only and it's much harder to find digestible advice for achieving the second.

jrockway 2y ago

Testing is a continuum. I don't write a test for every change. Sometimes I spend a week writing tests for a simple change. I will say that I've never said "I wish I didn't write a test for that". I have also never said, "your PR is fine, but please delete that test, it's useless". I throw away a lot of code. I still test stuff I expect to throw away. That's because it probably needs to run once before I throw it away, and I can't start throwing it away until it works :/ What it comes down to is what else you have to spend your time on. Sometimes you need to experiment with a feature; get it out to customers, and if it's buggy and rough around the edges, it's OK, because you were just trying out the idea. But sometimes that's not what you want; whatever time you spend on support back and forth finding a bug would have been better spent not doing that. The customer needed something rock solid, not an experiment. Test that so they don't have to. There are no rules. "Write a test for every change" is just as invalid and unworkable as "Never write any tests". It's a spectrum, and each change is going to land somewhere different. If you're unsure, ask a coworker. I have been testing stuff for 20+ years, and I usually guess OK (that is when I take a shortcut and don't test as much as I should, it's rarely the thing that caused the production outage), but a guess is just that, a guess. Solicit opinions.

withinboredom 2y ago

Also, non-testable code is often faster (as in cpu time).

lithos 2y ago

I've set unit tests for functions that are mostly math (base cases of collision, parts of bot predictive logic, and similar). Though I'm sitting at a hobbyist with electrical and commissioning background.