6 ms·

You're misunderstanding what the parent poster was saying: when working on a large AAA game, content management (art, game design, etc.) is as much a bottleneck

by jaaron 6y ago

You're misunderstanding what the parent poster was saying: when working on a large AAA game, content management (art, game design, etc.) is as much a bottleneck as engineering efforts. And a number of those bugs you're concerned about are rooted in content too, not lower level engine bugs. I've worked about half of my career in the game industry. I've practiced TDD and written automated tests (and frameworks) for desktop, web and mobile apps. Some of those have been in the medical industry where the testing is crucial. I say this to make it clear that I'm familiar with solid software engineering practices. With that in mind, games are the hardest software I've encountered for writing automated tests. It's just notoriously difficult to do in an effective manner. It's not impossible, but it's incredibly difficult.

reader_mode 6y ago
>I've worked about half of my career in the game industry. I've practiced TDD and written automated tests (and frameworks) for desktop, web and mobile apps. Some of those have been in the medical industry where the testing is crucial. I say this to make it clear that I'm familiar with solid software engineering practices. Did you work in the game industry before the other stuff or after ? Because I went that way game dev -> application development - and frankly a lot of the SW engineering practices from game dev were terrible in the transition - not because I couldn't apply them in game dev - but because I didn't know about them - and nobody around me told me - and I haven't seen it in code from others. >With that in mind, games are the hardest software I've encountered for writing automated tests. It's just notoriously difficult to do in an effective manner. It's not impossible, but it's incredibly difficult. There's a ton of low hanging fruit - running recorded controllers, partial scenario tests, gold copy rendering tests, smoke tests, regression testing - frankly it's not that hard to raise the bar from 0. I'm not up to date in the industry so maybe they aren't at 0 anymore but from occasionally keeping tabs and playing games occasionally I would say it hasn't moved far. Just the number of regressions in MMOs for example where you could easily code tests for the stuff that was fixed is an obvious example that nobody was doing regression testing or adding tests after fixes. And this is for MMOs that have an incentive to keep a healthy codebase (not just ship and forget)
- SeanBoocock 6y ago
  Everything you have outlined was standard practice at studios I have worked at for more than five years. The lower in the stack you go, the easier it is to do things like unit tests. In my experience the section of the game industry that is the easiest to test in an automated way is AAA mobile. Where I work in “HD” AAA, it is considerably harder to test in the same ways but we do where it is effective. Don’t mistake the failures of some games or studios as a valid indictment of the industry.
  - WorldMaker 6y ago
    > Don’t mistake the failures of some games or studios as a valid indictment of the industry. It's an industry focused intentionally on as minimal as possible code and best practice sharing for fear of talent poaching/project poaching. It's an industry where the "best practice" is still reinvent as many wheels as possible every other game and open source little to nothing. The worst mistakes of the worst games/studios should likely remain a valid indictment of the industry as a whole when the industry itself is so focused on making sure the tide rises as few boats as possible.
marta_weber 6y ago
It's easy if the game engine you are using would display an in-memory model with deterministic algorithms. I have only briefly worked in the game industry and it's been more than enough for me :D. There were like 4-5 people with "10 years+" experience working on a Candy Crush clone. It was literally riddled with bugs and they used Unity's physics engine to make the stones fall (I mean, it doesn't get any more ridiculous than this). So I came in, throw it all away and built within 2 days a deterministic, fully tested and testable stone engine that would implement all the effects management wanted to see, some of which would have been close to impossible to implement with Unity's physics engine. The idea is basically to use the smallest time unit you want to support, and then base all algorithms on that. You don't work on milli-seconds, because that is for the engine to fill out. You work on seconds, for CandyCrush at least. Each time step is basically a second. The time in-between is filled out by potentially non-deteministic animations and particle effects and whatnot. But every second, the whole scenery will synchronize with the deterministic engine that drives it all. While I didn't have the chance to try this on shooters or MMORPGs, I think especially for MMORPGs, this would be a perfect fit. It solves sooo many problems, I would probably need a day listing all the benefits. Are there drawbacks? Probably, I can't think of one right now, except that it goes against anything normal game developers believe in. I think at some point, the gaming industry forked away from solid software engineering practises.
- oppositelock 6y ago
  It's a lot more difficult than you make it out to be. I'm a former game engine lead from an Activision studio. The whole engine architecture was my responsibility, testing included. You can easily write tests for all the deterministic stuff; math, physics, save games, utility code, filesystem code, etc. However, where it gets difficult is testing the content. Every platform has some quirks which force you to create art custom made for that platform (your other choice is to use the minimal subset of what they all support). If you want a good looking game, your content has several versions, so now, things like intersection and pathing code are slightly different between platforms. GPU behavior is different enough that using GPU's for anything results in multiple test code paths. AI isn't always fully deterministic, by design, making it even more difficult to test. So, what you have in the end are some deterministic tests to make sure your foundation is sound, and "fuzzy" tests to catch many problems in the higher level constructs, and your final line of defense are play testers, which is an awful job! Now, we engineers know how to make the games stable and reliable, and our time estimates are frequently at odds with the PM's. You MUST make Christmas, which in the days that I worked, where CD's must be mastered, meant code freeze and asset freeze in October sometime. So, in this mad scramble to make a hard deadline, quality suffered. It was much more important in the old days (PS1, PS2, GameCube, N64, etc), to get it right the first time, as you couldn't ship patches, but the notion of patching games allowed release versions to be more buggy, since people kicked the can down the road. Granted, the earlier consoles only supported simpler games, and the diffiulty was dealing with the particulars of the system, not the game itself. The PS1 had no Z-Buffer, the PS2 had a CPU/GPU combo built by a crazy person with an PS1 as its IO controller, and the PS3 required you to write DMA management code, while the GameCube had a memory model that was nuts and matrices were only 4x3 so you couldn't do projections. So, you spent most of your time dealing with these quirks and the game ended up simpler due to the effort being spent elsewhere. Now, it's much easier, consoles are effectively PC's with powerful general purpose CPU's and GPU's, so it should be possible to write high quality, reusable engines.

by jaaron 6y ago

reader_mode 6y ago

>I've worked about half of my career in the game industry. I've practiced TDD and written automated tests (and frameworks) for desktop, web and mobile apps. Some of those have been in the medical industry where the testing is crucial. I say this to make it clear that I'm familiar with solid software engineering practices. Did you work in the game industry before the other stuff or after ? Because I went that way game dev -> application development - and frankly a lot of the SW engineering practices from game dev were terrible in the transition - not because I couldn't apply them in game dev - but because I didn't know about them - and nobody around me told me - and I haven't seen it in code from others. >With that in mind, games are the hardest software I've encountered for writing automated tests. It's just notoriously difficult to do in an effective manner. It's not impossible, but it's incredibly difficult. There's a ton of low hanging fruit - running recorded controllers, partial scenario tests, gold copy rendering tests, smoke tests, regression testing - frankly it's not that hard to raise the bar from 0. I'm not up to date in the industry so maybe they aren't at 0 anymore but from occasionally keeping tabs and playing games occasionally I would say it hasn't moved far. Just the number of regressions in MMOs for example where you could easily code tests for the stuff that was fixed is an obvious example that nobody was doing regression testing or adding tests after fixes. And this is for MMOs that have an incentive to keep a healthy codebase (not just ship and forget)

SeanBoocock 6y ago
Everything you have outlined was standard practice at studios I have worked at for more than five years. The lower in the stack you go, the easier it is to do things like unit tests. In my experience the section of the game industry that is the easiest to test in an automated way is AAA mobile. Where I work in “HD” AAA, it is considerably harder to test in the same ways but we do where it is effective. Don’t mistake the failures of some games or studios as a valid indictment of the industry.
- WorldMaker 6y ago
  > Don’t mistake the failures of some games or studios as a valid indictment of the industry. It's an industry focused intentionally on as minimal as possible code and best practice sharing for fear of talent poaching/project poaching. It's an industry where the "best practice" is still reinvent as many wheels as possible every other game and open source little to nothing. The worst mistakes of the worst games/studios should likely remain a valid indictment of the industry as a whole when the industry itself is so focused on making sure the tide rises as few boats as possible.

marta_weber 6y ago

It's easy if the game engine you are using would display an in-memory model with deterministic algorithms. I have only briefly worked in the game industry and it's been more than enough for me :D. There were like 4-5 people with "10 years+" experience working on a Candy Crush clone. It was literally riddled with bugs and they used Unity's physics engine to make the stones fall (I mean, it doesn't get any more ridiculous than this). So I came in, throw it all away and built within 2 days a deterministic, fully tested and testable stone engine that would implement all the effects management wanted to see, some of which would have been close to impossible to implement with Unity's physics engine. The idea is basically to use the smallest time unit you want to support, and then base all algorithms on that. You don't work on milli-seconds, because that is for the engine to fill out. You work on seconds, for CandyCrush at least. Each time step is basically a second. The time in-between is filled out by potentially non-deteministic animations and particle effects and whatnot. But every second, the whole scenery will synchronize with the deterministic engine that drives it all. While I didn't have the chance to try this on shooters or MMORPGs, I think especially for MMORPGs, this would be a perfect fit. It solves sooo many problems, I would probably need a day listing all the benefits. Are there drawbacks? Probably, I can't think of one right now, except that it goes against anything normal game developers believe in. I think at some point, the gaming industry forked away from solid software engineering practises.

oppositelock 6y ago
It's a lot more difficult than you make it out to be. I'm a former game engine lead from an Activision studio. The whole engine architecture was my responsibility, testing included. You can easily write tests for all the deterministic stuff; math, physics, save games, utility code, filesystem code, etc. However, where it gets difficult is testing the content. Every platform has some quirks which force you to create art custom made for that platform (your other choice is to use the minimal subset of what they all support). If you want a good looking game, your content has several versions, so now, things like intersection and pathing code are slightly different between platforms. GPU behavior is different enough that using GPU's for anything results in multiple test code paths. AI isn't always fully deterministic, by design, making it even more difficult to test. So, what you have in the end are some deterministic tests to make sure your foundation is sound, and "fuzzy" tests to catch many problems in the higher level constructs, and your final line of defense are play testers, which is an awful job! Now, we engineers know how to make the games stable and reliable, and our time estimates are frequently at odds with the PM's. You MUST make Christmas, which in the days that I worked, where CD's must be mastered, meant code freeze and asset freeze in October sometime. So, in this mad scramble to make a hard deadline, quality suffered. It was much more important in the old days (PS1, PS2, GameCube, N64, etc), to get it right the first time, as you couldn't ship patches, but the notion of patching games allowed release versions to be more buggy, since people kicked the can down the road. Granted, the earlier consoles only supported simpler games, and the diffiulty was dealing with the particulars of the system, not the game itself. The PS1 had no Z-Buffer, the PS2 had a CPU/GPU combo built by a crazy person with an PS1 as its IO controller, and the PS3 required you to write DMA management code, while the GameCube had a memory model that was nuts and matrices were only 4x3 so you couldn't do projections. So, you spent most of your time dealing with these quirks and the game ended up simpler due to the effort being spent elsewhere. Now, it's much easier, consoles are effectively PC's with powerful general purpose CPU's and GPU's, so it should be possible to write high quality, reusable engines.