11 ms·

> There's a really important point to grok: AAA titles are more about asset management and art than they are about coding If that was true there wouldn't be so

> There's a really important point to grok: AAA titles are more about asset management and art than they are about coding If that was true there wouldn't be so much logic and code bugs in AAA titles. From what I've seen game industry has terrible software engineering practices - why have automated testing when your model is crunch to release and then leave a skeleton crew fixing the bugs after you shipped. Also being stuck in C++ doesn't help either, an ecosystem with bizarrely the most complicated frameworks I've ever seen (eg. boost) and yet the worst tooling out of anything I've used (with comparable adoption rate).

meheleventyone 6y ago
> If that was true there wouldn't be so much logic and code bugs in AAA titles. Those are pretty different concerns tackled by different groups of people on AAA projects. The art requirements can be an order of magnitude larger than the gameplay side of things. That doesn’t make the gameplay side of things easy.
- reader_mode 6y ago
  My impression is that asset creation is more scalable (even if labor intensive) and they figured out a way to manage it - I haven't seen a AAA title with asset issues, and from what I understand hiring a bunch of asset creators is cheaper than hiring developers to automate their work.
jaaron 6y ago
You're misunderstanding what the parent poster was saying: when working on a large AAA game, content management (art, game design, etc.) is as much a bottleneck as engineering efforts. And a number of those bugs you're concerned about are rooted in content too, not lower level engine bugs. I've worked about half of my career in the game industry. I've practiced TDD and written automated tests (and frameworks) for desktop, web and mobile apps. Some of those have been in the medical industry where the testing is crucial. I say this to make it clear that I'm familiar with solid software engineering practices. With that in mind, games are the hardest software I've encountered for writing automated tests. It's just notoriously difficult to do in an effective manner. It's not impossible, but it's incredibly difficult.
- reader_mode 6y ago
  >I've worked about half of my career in the game industry. I've practiced TDD and written automated tests (and frameworks) for desktop, web and mobile apps. Some of those have been in the medical industry where the testing is crucial. I say this to make it clear that I'm familiar with solid software engineering practices. Did you work in the game industry before the other stuff or after ? Because I went that way game dev -> application development - and frankly a lot of the SW engineering practices from game dev were terrible in the transition - not because I couldn't apply them in game dev - but because I didn't know about them - and nobody around me told me - and I haven't seen it in code from others. >With that in mind, games are the hardest software I've encountered for writing automated tests. It's just notoriously difficult to do in an effective manner. It's not impossible, but it's incredibly difficult. There's a ton of low hanging fruit - running recorded controllers, partial scenario tests, gold copy rendering tests, smoke tests, regression testing - frankly it's not that hard to raise the bar from 0. I'm not up to date in the industry so maybe they aren't at 0 anymore but from occasionally keeping tabs and playing games occasionally I would say it hasn't moved far. Just the number of regressions in MMOs for example where you could easily code tests for the stuff that was fixed is an obvious example that nobody was doing regression testing or adding tests after fixes. And this is for MMOs that have an incentive to keep a healthy codebase (not just ship and forget)
  - SeanBoocock 6y ago
    Everything you have outlined was standard practice at studios I have worked at for more than five years. The lower in the stack you go, the easier it is to do things like unit tests. In my experience the section of the game industry that is the easiest to test in an automated way is AAA mobile. Where I work in “HD” AAA, it is considerably harder to test in the same ways but we do where it is effective. Don’t mistake the failures of some games or studios as a valid indictment of the industry.
    - WorldMaker 6y ago
      > Don’t mistake the failures of some games or studios as a valid indictment of the industry. It's an industry focused intentionally on as minimal as possible code and best practice sharing for fear of talent poaching/project poaching. It's an industry where the "best practice" is still reinvent as many wheels as possible every other game and open source little to nothing. The worst mistakes of the worst games/studios should likely remain a valid indictment of the industry as a whole when the industry itself is so focused on making sure the tide rises as few boats as possible.
flohofwoe 6y ago
You'll have to search very long to find a C++ game code-base that uses boost, game devs are not that stupid ;) Also, Unity games are usually written in C# (I think it's quite safe to say that - overall - most games are not written in C++ but in C#), yet I've seen no data so far which would indicate that Unity games have any less problems than games written in C++ during production and after release (if anything, the opposite seems to be true, not for technological reasons, but because Unity is so much more beginner-friendly). I'm no fan of C++ either, but blaming a programming language for bugs and quality problems without any counter examples at hand is a bit ridiculous.
- makuto 6y ago
  I'm not sure why you say it's a "quite safe" assumption that most games are written in C#. Unity's beginner-friendliness gives it a disproportionate presence online, while the vast majority of AAA games are still solidly C++. If we eliminate all games with less than 1000 sales or something, I think it would be a very low confidence estimate.
  - kbenson 6y ago
    I think perhaps they meant "most Unity games", where Unity was supposed to be implied by the context. It probably is safe to say most Unity games are C#, if we ignore the portion of the engine runtime that is written in C++, which makes sense for some metrics and not others.
- reader_mode 6y ago
  I'm blaming C++ because : - there is very little information in the community on how to do this kind of engineering efficiently (at least I haven't encountered it nearly as much as I have when I transitioned to application development in higher level languages) - there is very little code sharing in the community and zero standardisation - everyone reinvents shit from standard library, coding conventions, what subset of the language is "allowed", etc. etc. - this means developing good tooling, practices and patterns across large projects is hard Unity is written in C++, C# is scripting layer and more importantly I doubt Unity developers doing C++ are C# engineers with C# application development background where stuff like automated testing is pretty standard.
senko 6y ago
> From what I've seen game industry has terrible software engineering practices. You'd be surprised. Here's a talk from Croteam on how they test their games and engine: https://m.youtube.com/watch?v=YGIvWT-NBHk https://m.youtube.com/watch?v=YGIvWT-NBHk I'd wager all major engines are exhaustively tested. Trouble is, there's combinatorial explosion of game state, user input, assets, scripted behaviour and engine, so there's a huge area to cover.
- reader_mode 6y ago
  I mean he's basically saying what I am - they are the exception, very few teams are doing it, no public information on how to do it or best practices, everyone reinvents everything on their own from scratch. I'd wager the popular engines are well tested because of the number of titles shipped on them not because they have good testing automation - but TBH I haven't worked in this industry for almost 10 years so maybe things changed.
  - megameter 6y ago
    10 years ago was right when "TDD" became hugely hyped. Before that, test automation was patchy throughout the software world, not just in games. I believe the same is true of "best practice" today: game studios aren't actually behind the curve, you just don't hear much about how things are progressing on this end because most of the conference talks aren't about broad concerns like testing, they're about the myriad specialities of the field. And there absolutely is a legacy-code thing that hinders AAA in many cases. When the engine is old, that's good, because it's shipped something, but it's bad, because it's using older practices and Things Have Moved On.
    - reader_mode 6y ago
      That's a fair take too - maybe TDD wasn't as widespread in general so I just got onboard when everyone else did. Although I should note I'm not a fan of TDD and it's not something I would recommend for games or anything similar - it's a very narrow tool - I think you agree because you put "TDD" in quotes I just don't want to make it sound like I'm recommending it - I'm a fan of automated testing.
Kapura 6y ago
Extremely bad take, from somebody who has clearly not worked in AAA games, but believes all the things that gamers post on reddit. A few points: 1) you can only test-driven-develop so much in games, and that line usually stops at the engine level because the game itself is in flux so much. Automated testing is confined to making sure that checkins build on every platform. Game dev engineering is fundamentally different than programming in other fields because the goal posts move constantly. 2) If an engineer writes code expecting there to be no more than 1024 physics objects in the system, tells the designers and artists this, but then they turn around put in 2000 colliding pieces of silverware on a dinner table "because it needs to feel like a big feast" is that an engineering problem, or an art and asset management problem? Because something like 80% of my bugs are shit like this. 3) Professional game codebases use their own styles (i hesitate to say dialects) of C++ that do the things we need them to do the ways we need to do them. We don't use anybody else's framework; all of the bonus stuff we're doing lives in macros that can be inspected if an issue arises. But, please don't push your language purism on anybody else. What a tired argument to have.
- tstrimple 6y ago
  None of your points are unique to game development. Moving goal posts and requirements changing constantly is a challenge at literally every single development job I've had. There is an eternal back and forth between what the sales team says the software can do, and what the programmers desperately attempt to account for late in the development cycle. Plenty of teams have idiosyncrasies around their tool chain, refuse to reflect on it and maintain that their project is a special case to justify all the mess.
  - Guvante 6y ago
    Let me first say that game developers should do a better job about testing, especially when it comes to developing isolated systems to support unit tests. But it certainly seems like you misunderstand how much change there is in the vast majority of game development. > Moving goal posts and requirements changing constantly is a challenge at literally every single development job I've had. It is hard to describe because it sounds the same. But game development is really different because there are no fundamentals of things you care to test. Even doing a combat sequence requires a herculean effort. You need basically the whole game running because otherwise what is the point. You fake out most of the data so your test doesn't fail when the designers decide to make combat harder. But now it ends up that pressing A defends instead of attacks because reasons and all of your combat tests now fail. This is all fixable but it makes the cost per test of anything but the tiniest things hard enough that broad test coverage can sometimes be a determent as you end up testing what the game is now which means it will all be thrown away if your assumptions change. Sure that can always be the case but "the sum of the lines equals the total" kind of tests are much less likely to backfire in this way. > Plenty of teams have idiosyncrasies around their tool chain C++ is chosen because the tooling doesn't exist outside of C++. Full stop. You have to build all the tooling in language X which when talking about 3D graphics is a huge amount of tooling. Rust is starting to have some cool stuff but if you compare you will see there is a world of difference. Thus if you choose not C++ you get to write C wrappers around your API and deal with all that nonsense since C++ interop is the worst in most languages. At some point Rust will get proper C++ interop and then the gap will be smaller but for now you are giving up a ton for a slightly safer language by not choosing C++. Also note that nearly everybody writes a huge amount of non-C++ code, they just call it a scripting language instead.

meheleventyone 6y ago

> If that was true there wouldn't be so much logic and code bugs in AAA titles. Those are pretty different concerns tackled by different groups of people on AAA projects. The art requirements can be an order of magnitude larger than the gameplay side of things. That doesn’t make the gameplay side of things easy.

reader_mode 6y ago
My impression is that asset creation is more scalable (even if labor intensive) and they figured out a way to manage it - I haven't seen a AAA title with asset issues, and from what I understand hiring a bunch of asset creators is cheaper than hiring developers to automate their work.

jaaron 6y ago

You're misunderstanding what the parent poster was saying: when working on a large AAA game, content management (art, game design, etc.) is as much a bottleneck as engineering efforts. And a number of those bugs you're concerned about are rooted in content too, not lower level engine bugs. I've worked about half of my career in the game industry. I've practiced TDD and written automated tests (and frameworks) for desktop, web and mobile apps. Some of those have been in the medical industry where the testing is crucial. I say this to make it clear that I'm familiar with solid software engineering practices. With that in mind, games are the hardest software I've encountered for writing automated tests. It's just notoriously difficult to do in an effective manner. It's not impossible, but it's incredibly difficult.

reader_mode 6y ago
>I've worked about half of my career in the game industry. I've practiced TDD and written automated tests (and frameworks) for desktop, web and mobile apps. Some of those have been in the medical industry where the testing is crucial. I say this to make it clear that I'm familiar with solid software engineering practices. Did you work in the game industry before the other stuff or after ? Because I went that way game dev -> application development - and frankly a lot of the SW engineering practices from game dev were terrible in the transition - not because I couldn't apply them in game dev - but because I didn't know about them - and nobody around me told me - and I haven't seen it in code from others. >With that in mind, games are the hardest software I've encountered for writing automated tests. It's just notoriously difficult to do in an effective manner. It's not impossible, but it's incredibly difficult. There's a ton of low hanging fruit - running recorded controllers, partial scenario tests, gold copy rendering tests, smoke tests, regression testing - frankly it's not that hard to raise the bar from 0. I'm not up to date in the industry so maybe they aren't at 0 anymore but from occasionally keeping tabs and playing games occasionally I would say it hasn't moved far. Just the number of regressions in MMOs for example where you could easily code tests for the stuff that was fixed is an obvious example that nobody was doing regression testing or adding tests after fixes. And this is for MMOs that have an incentive to keep a healthy codebase (not just ship and forget)
- SeanBoocock 6y ago
  Everything you have outlined was standard practice at studios I have worked at for more than five years. The lower in the stack you go, the easier it is to do things like unit tests. In my experience the section of the game industry that is the easiest to test in an automated way is AAA mobile. Where I work in “HD” AAA, it is considerably harder to test in the same ways but we do where it is effective. Don’t mistake the failures of some games or studios as a valid indictment of the industry.
  - WorldMaker 6y ago
    > Don’t mistake the failures of some games or studios as a valid indictment of the industry. It's an industry focused intentionally on as minimal as possible code and best practice sharing for fear of talent poaching/project poaching. It's an industry where the "best practice" is still reinvent as many wheels as possible every other game and open source little to nothing. The worst mistakes of the worst games/studios should likely remain a valid indictment of the industry as a whole when the industry itself is so focused on making sure the tide rises as few boats as possible.

flohofwoe 6y ago

You'll have to search very long to find a C++ game code-base that uses boost, game devs are not that stupid ;) Also, Unity games are usually written in C# (I think it's quite safe to say that - overall - most games are not written in C++ but in C#), yet I've seen no data so far which would indicate that Unity games have any less problems than games written in C++ during production and after release (if anything, the opposite seems to be true, not for technological reasons, but because Unity is so much more beginner-friendly). I'm no fan of C++ either, but blaming a programming language for bugs and quality problems without any counter examples at hand is a bit ridiculous.

makuto 6y ago
I'm not sure why you say it's a "quite safe" assumption that most games are written in C#. Unity's beginner-friendliness gives it a disproportionate presence online, while the vast majority of AAA games are still solidly C++. If we eliminate all games with less than 1000 sales or something, I think it would be a very low confidence estimate.
- kbenson 6y ago
  I think perhaps they meant "most Unity games", where Unity was supposed to be implied by the context. It probably is safe to say most Unity games are C#, if we ignore the portion of the engine runtime that is written in C++, which makes sense for some metrics and not others.
reader_mode 6y ago
I'm blaming C++ because : - there is very little information in the community on how to do this kind of engineering efficiently (at least I haven't encountered it nearly as much as I have when I transitioned to application development in higher level languages) - there is very little code sharing in the community and zero standardisation - everyone reinvents shit from standard library, coding conventions, what subset of the language is "allowed", etc. etc. - this means developing good tooling, practices and patterns across large projects is hard Unity is written in C++, C# is scripting layer and more importantly I doubt Unity developers doing C++ are C# engineers with C# application development background where stuff like automated testing is pretty standard.

senko 6y ago

> From what I've seen game industry has terrible software engineering practices. You'd be surprised. Here's a talk from Croteam on how they test their games and engine: https://m.youtube.com/watch?v=YGIvWT-NBHk https://m.youtube.com/watch?v=YGIvWT-NBHk I'd wager all major engines are exhaustively tested. Trouble is, there's combinatorial explosion of game state, user input, assets, scripted behaviour and engine, so there's a huge area to cover.

reader_mode 6y ago
I mean he's basically saying what I am - they are the exception, very few teams are doing it, no public information on how to do it or best practices, everyone reinvents everything on their own from scratch. I'd wager the popular engines are well tested because of the number of titles shipped on them not because they have good testing automation - but TBH I haven't worked in this industry for almost 10 years so maybe things changed.
- megameter 6y ago
  10 years ago was right when "TDD" became hugely hyped. Before that, test automation was patchy throughout the software world, not just in games. I believe the same is true of "best practice" today: game studios aren't actually behind the curve, you just don't hear much about how things are progressing on this end because most of the conference talks aren't about broad concerns like testing, they're about the myriad specialities of the field. And there absolutely is a legacy-code thing that hinders AAA in many cases. When the engine is old, that's good, because it's shipped something, but it's bad, because it's using older practices and Things Have Moved On.
  - reader_mode 6y ago
    That's a fair take too - maybe TDD wasn't as widespread in general so I just got onboard when everyone else did. Although I should note I'm not a fan of TDD and it's not something I would recommend for games or anything similar - it's a very narrow tool - I think you agree because you put "TDD" in quotes I just don't want to make it sound like I'm recommending it - I'm a fan of automated testing.

Kapura 6y ago

Extremely bad take, from somebody who has clearly not worked in AAA games, but believes all the things that gamers post on reddit. A few points: 1) you can only test-driven-develop so much in games, and that line usually stops at the engine level because the game itself is in flux so much. Automated testing is confined to making sure that checkins build on every platform. Game dev engineering is fundamentally different than programming in other fields because the goal posts move constantly. 2) If an engineer writes code expecting there to be no more than 1024 physics objects in the system, tells the designers and artists this, but then they turn around put in 2000 colliding pieces of silverware on a dinner table "because it needs to feel like a big feast" is that an engineering problem, or an art and asset management problem? Because something like 80% of my bugs are shit like this. 3) Professional game codebases use their own styles (i hesitate to say dialects) of C++ that do the things we need them to do the ways we need to do them. We don't use anybody else's framework; all of the bonus stuff we're doing lives in macros that can be inspected if an issue arises. But, please don't push your language purism on anybody else. What a tired argument to have.

tstrimple 6y ago
None of your points are unique to game development. Moving goal posts and requirements changing constantly is a challenge at literally every single development job I've had. There is an eternal back and forth between what the sales team says the software can do, and what the programmers desperately attempt to account for late in the development cycle. Plenty of teams have idiosyncrasies around their tool chain, refuse to reflect on it and maintain that their project is a special case to justify all the mess.
- Guvante 6y ago
  Let me first say that game developers should do a better job about testing, especially when it comes to developing isolated systems to support unit tests. But it certainly seems like you misunderstand how much change there is in the vast majority of game development. > Moving goal posts and requirements changing constantly is a challenge at literally every single development job I've had. It is hard to describe because it sounds the same. But game development is really different because there are no fundamentals of things you care to test. Even doing a combat sequence requires a herculean effort. You need basically the whole game running because otherwise what is the point. You fake out most of the data so your test doesn't fail when the designers decide to make combat harder. But now it ends up that pressing A defends instead of attacks because reasons and all of your combat tests now fail. This is all fixable but it makes the cost per test of anything but the tiniest things hard enough that broad test coverage can sometimes be a determent as you end up testing what the game is now which means it will all be thrown away if your assumptions change. Sure that can always be the case but "the sum of the lines equals the total" kind of tests are much less likely to backfire in this way. > Plenty of teams have idiosyncrasies around their tool chain C++ is chosen because the tooling doesn't exist outside of C++. Full stop. You have to build all the tooling in language X which when talking about 3D graphics is a huge amount of tooling. Rust is starting to have some cool stuff but if you compare you will see there is a world of difference. Thus if you choose not C++ you get to write C wrappers around your API and deal with all that nonsense since C++ interop is the worst in most languages. At some point Rust will get proper C++ interop and then the gap will be smaller but for now you are giving up a ton for a slightly safer language by not choosing C++. Also note that nearly everybody writes a huge amount of non-C++ code, they just call it a scripting language instead.