3 ms·

Video games are disposable products, other software is usually mission critical and requires long-term support. So for games you can get away without automation

by Flankk 5y ago

Video games are disposable products, other software is usually mission critical and requires long-term support. So for games you can get away without automation but with other software it would destroy you. Testing is not free, it is a tradeoff where the benefits outweigh the costs by some margin. Video game testing is difficult to automate so in many circumstances the costs of automation may outweigh the benefits. Automated tests are just a subset of quality assurance, which can be accomplished in a variety of ways.

sovietmudkipz 5y ago
I appreciate the response! I’m not as satisfied with this answer. I’m not satisfied because a legion of QA engage in manual testing to ensure quality in game artifacts in the games industry. It’s like the process is begging for automation, at least that’s how I interpret the situation. Testing is challenging; as an industry we have learned that challenging things are a signal to do it more, not less.
- detaro 5y ago
  Just because something is "begging for automation" means automating it is easy or even just viable. You'll always need human QA, so at this point the question is merely "how can tests reduce the load for manual QA while keeping manual QA good at the parts we can't automate". Game testing is a lot like UI testing in other apps that way. And then in reverse, are the automatic tests worth the maintenance effort if a human is going to try anyways and notice quickly? IMHO that's the part where many people underestimate them, but getting the balance right is still hard. EDIT: of course its biased, but its interesting to think through bugs you encounter in games and how you'd catch them with tests. I feel often its more on the level of fuzzing than traditional tests that could catch these things, or it is visual issues that are very hard to automatically find.
mattbrewsbytes 5y ago
I know there are elements of a game engine and user interaction that would be hard to test but there are a lot of areas internal to a large online game could use automated tests. These could be actual REST types of API's that the game clients interact with as well as internal API's either in the client or server. There should be a large enough surface area of these API's to test things and I think that would make changing/tuning aspects of games easier on the game developers. I wonder if this is more of a culture within the gaming industry where getting humans to test games is cheaper (i.e. 15-20/hr) than hiring QA engineers to write code that tests things. I remember hearing anecdotes of people wanting to work for game companies and being a tester is a "foot in the door". Also sometimes they do open beta's where they just let players in and do the testing.
- sovietmudkipz 5y ago
  I have a similar hypothesis; I speculate that the reason automated testing isn’t a big thing in the gaming industry is because there are legions of humans who are willing to manually test the software. I’m glad to see that you have identified areas of game software that wouldn’t be useful to test, and other areas where it would be. Excellent to see because it does highlight that there ARE places where automated tests can provide huge value.
- salawat 5y ago
  To really do QA for a gaming company, in a way you could leverage automation effectively, the key would be process, and structuring things in a way where you are using design patterns that let your tools cross-cut the portfolio of product. You could totally automate tests around the web service client layers of the game, but what most people think of in gaming QA is actually playing the game. That would slot in as a System test, which is your most expensive, least scalable, latest in the process form of testing, and nigh impossible to make substantive groundreshaping changes from the input of, as that experience is reliant on everything being the way it is. If you were embedded with the engine design group, you might be able to get some wins in terms of influencing architecture choices so as to make downstream automation testing more likely/doable, but from my understanding that's like being on the team that develops the crypto libraries everyone else just adopts and uses in the manner you prescribe. Plus, as a longtime QA, if game developers are as resistant to feedback as corpo ones are, and with management structures almost guaranteeing you'll be underneath the Engineering group, cutting you off at the knees the first time the words "crunch time" are implied, given the rates most get in that vertical, I'd stay away from it unless your one purpose in life is to get confronted with disappointment after disappointment as economic/fiscal concerns destroy and deface what could have been over and over again. Then again, you wouldn't be in QA if you didn't already have a tolerance to that sort of thing. Now where'd I put that bottle?
- burntoutfire 5y ago
  The "disposable" aspect of games mentioned by parent is crucial. Any successful business software is going to be maintained and extended for years, possibly decades. This will be done by constantly churning team, where sooner or later no one will have a very good idea about the system's internal or requirements. In such cases, automated test cases protect against regression bugs AND also serve as a documentation of sorts. Whereas in games, after the game is released, the team will maybe patch it for a while and after that, chances are no one will ever touch that code ever again.

by Flankk 5y ago

sovietmudkipz 5y ago

I appreciate the response! I’m not as satisfied with this answer. I’m not satisfied because a legion of QA engage in manual testing to ensure quality in game artifacts in the games industry. It’s like the process is begging for automation, at least that’s how I interpret the situation. Testing is challenging; as an industry we have learned that challenging things are a signal to do it more, not less.

detaro 5y ago
Just because something is "begging for automation" means automating it is easy or even just viable. You'll always need human QA, so at this point the question is merely "how can tests reduce the load for manual QA while keeping manual QA good at the parts we can't automate". Game testing is a lot like UI testing in other apps that way. And then in reverse, are the automatic tests worth the maintenance effort if a human is going to try anyways and notice quickly? IMHO that's the part where many people underestimate them, but getting the balance right is still hard. EDIT: of course its biased, but its interesting to think through bugs you encounter in games and how you'd catch them with tests. I feel often its more on the level of fuzzing than traditional tests that could catch these things, or it is visual issues that are very hard to automatically find.

mattbrewsbytes 5y ago

I know there are elements of a game engine and user interaction that would be hard to test but there are a lot of areas internal to a large online game could use automated tests. These could be actual REST types of API's that the game clients interact with as well as internal API's either in the client or server. There should be a large enough surface area of these API's to test things and I think that would make changing/tuning aspects of games easier on the game developers. I wonder if this is more of a culture within the gaming industry where getting humans to test games is cheaper (i.e. 15-20/hr) than hiring QA engineers to write code that tests things. I remember hearing anecdotes of people wanting to work for game companies and being a tester is a "foot in the door". Also sometimes they do open beta's where they just let players in and do the testing.

sovietmudkipz 5y ago
I have a similar hypothesis; I speculate that the reason automated testing isn’t a big thing in the gaming industry is because there are legions of humans who are willing to manually test the software. I’m glad to see that you have identified areas of game software that wouldn’t be useful to test, and other areas where it would be. Excellent to see because it does highlight that there ARE places where automated tests can provide huge value.
salawat 5y ago
To really do QA for a gaming company, in a way you could leverage automation effectively, the key would be process, and structuring things in a way where you are using design patterns that let your tools cross-cut the portfolio of product. You could totally automate tests around the web service client layers of the game, but what most people think of in gaming QA is actually playing the game. That would slot in as a System test, which is your most expensive, least scalable, latest in the process form of testing, and nigh impossible to make substantive groundreshaping changes from the input of, as that experience is reliant on everything being the way it is. If you were embedded with the engine design group, you might be able to get some wins in terms of influencing architecture choices so as to make downstream automation testing more likely/doable, but from my understanding that's like being on the team that develops the crypto libraries everyone else just adopts and uses in the manner you prescribe. Plus, as a longtime QA, if game developers are as resistant to feedback as corpo ones are, and with management structures almost guaranteeing you'll be underneath the Engineering group, cutting you off at the knees the first time the words "crunch time" are implied, given the rates most get in that vertical, I'd stay away from it unless your one purpose in life is to get confronted with disappointment after disappointment as economic/fiscal concerns destroy and deface what could have been over and over again. Then again, you wouldn't be in QA if you didn't already have a tolerance to that sort of thing. Now where'd I put that bottle?
burntoutfire 5y ago
The "disposable" aspect of games mentioned by parent is crucial. Any successful business software is going to be maintained and extended for years, possibly decades. This will be done by constantly churning team, where sooner or later no one will have a very good idea about the system's internal or requirements. In such cases, automated test cases protect against regression bugs AND also serve as a documentation of sorts. Whereas in games, after the game is released, the team will maybe patch it for a while and after that, chances are no one will ever touch that code ever again.