9 ms·

Automated testing in Bevy

107 points by ChadNauseam 4y ago

Jensson 4y ago
> Bevy uses the ECS pattern to organize game code, and it occurred to me that ECS substantially alleviates the spaghetti-code problem that makes testing in Unity and Unreal so terrible. Both Unreal and Unity uses ECS.
- guzik 4y ago
  Seems like the author never truly worked with these engines? Also, I never had problem with unit testing on Unity in pure C#. Even when it comes to complicated scenario there is always a way to separate your GameObjects from game logic.
  - harles 4y ago
    I think this comment and its parent have misunderstood what ECS is.
  - ChadNauseam 4y ago
    I have spent way too much time in Unity. (I was a Unity developer for pay for several years.) I suppose I didn't do a good job explaining the differences between what Unity does and an ECS approach in my post, but if you'd like to get a feel for what an ECS you could try out DOTS. (Although DOTS is kindn of unpleasant to use compared to Bevy.)
- tarsiel 4y ago
  Unity at least isn’t a true ECS architecture, more just Entity-Component. You can write game logic in an entity with Unity, whereas in ECS that would have to live in a System
  - runevault 4y ago
    Unity is working on a full ECS system as part of a larger DOTS initiative, but it is still pre-alpha last I knew.
- metadat 4y ago
  What is ECS? Assuming it's not AWS Elastic Container Service :>
  - worble 4y ago
    Entity component system
  - keyle 4y ago
    Around 2007 I believe the game industry came up with a semi standard way of building games that somehow alleviated a lot of issues in producing complex games... aka spaghetti loops of hell. There are a few ECS approach and to this day it's still vague who does it best (depends on your goals). One I found interesting is the "sparse entity component system".
    - cobalt 4y ago
      it's more to avoid god classes and inheritance messes.
  - deleted 4y ago
    [deleted]
- keyle 4y ago
  Thanks, I rolled my eyes and had an issue with the author's statement. My guess is what he meant was that the Bevy approach is cleaner to test upon.
  - harles 4y ago
    Can you point me to what part of Unreal uses an ECS approach? I’ve never seen this. I know Unity DOTS added ECS features, but that still lives in a hybrid world.
    - lfowles 4y ago
      This surely isn't what was meant in previous conversation, but Unreal Engine 5 added MassEntity which is uh... an Entity Fragment Processor approach. (Because Components are already an object that exists, it probably would have been an extra special kind of confusing if it adopted typical ECS naming.) https://docs.unrealengine.com/5.0/en-US/overview-of-mass-entity-in-unreal-engine/ https://docs.unrealengine.com/5.0/en-US/overview-of-mass-ent...
    - dimgl 4y ago
      Based on the definitions in this thread, it wouldn't be a stretch to pair Unreal's `AActor` as the entities and `UActorComponent` as the components. You could technically go a step further and pair `UWorldSubsystem` as the system. I realize that there's the whole "grouping" and "batching" concept in ECS paradigms, but I'm failing to see how that's much different from using the functions in `UWorld` that can query both components and actors of a certain class.
- Jasper_ 4y ago
  This is where the unfortunately-chosen name "Entity-Component-Systems" (ECS) comes back to bite us. Unity's GameObjects are Entity-Component (EC): you have an Entity (GameObject), onto which you attach Components (MonoBehavior). Unreal is similar: you have an Entity (UActor), onto which you attach Components (UActorComponent), though you can also have plenty of code living outside the UActorComponent system in Unreal. Entities are objects, and Components are code/data. Entity-Compontity-System (ECS) uses these nouns differently; an Entity ends up being little more than an identifier for a large number of Components, which are data structures, and Systems are the code that modify groups of Components. "System" here is a noun, not a descriptor of the practice. The usual analogy is similar to a relational database (in fact, the point of the system is to achieve wide-scale performance like RDSes do): Entities are rows, identified by a primary key, Components are columns, and Systems are the SELECT/INSERT/UPDATE/DELETE queries which acts on the data. Unity has a new-ish ECS framework under the DOTS initiative; Unreal has no parallel. Very few games are using these.
  - Jensson 4y ago
    In unity the methods "Update", "Start" and so on are the system part of ECS. That you write their implementations alongside the data doesn't really matter.
    - forrestthewoods 4y ago
      Sorry, you're wrong on this one. ECS has come to mean a very particular style of architecture. It's not a great name, but it's what the industry has converged on. Naming things is hard. Unity's MonoBehavior framework uses GameObjects and Components. You don't have to squint hard to see that it contains Entities, Components, and arguably Systems. However ECS has come to mean an architecture that is quite different from MonoBehavior. So different infact that Unity is writing an ECS framework called DOTS! Both DOTS and MonoBehaviors containing entities, components, and functions that operate on them. However the two Unity systems are extremely different. MonoBehaviors are not an ECS.
      - Jensson 4y ago
stuckinhell 4y ago
I've been looking into Unity and game development in my spare time. It seems writing tests for game code is incredibly dependent on the style used to develop the game. If you are doing everything by hand in C#, its super easy. The moment you introduce editor workflows or third party libraries, I have no idea how anybody can write sensible tests. Using Unity is like a weird mix of no code tools and code.
- guzik 4y ago
  > Using Unity is like a weird mix of no code tools and code. Haha, great summary of my 5 years experience in Unity. Whenever I am too lazy I am switching to ECS/MonoBehaviour, then I regret of not having properly testable architecture.
  - Jaxkr 4y ago
    ECS is a competing system to supersede MonoBehavior
andrewallbright 4y ago
Writing automated tests for video games is some of the most fun programming I do.
Goz3rr 4y ago
The author mentions Sea of Thieves and Mindustry, but I think Factorio is worth a mention too (seemingly as usual when something game related comes up on HN). They do TDD and talk about their automatic tests in several blogs: https://factorio.com/blog/post/fff-366 https://factorio.com/blog/post/fff-366 https://factorio.com/blog/post/fff-288 https://factorio.com/blog/post/fff-288 https://factorio.com/blog/post/fff-186 https://factorio.com/blog/post/fff-186 https://factorio.com/blog/post/fff-62 https://factorio.com/blog/post/fff-62
- Jasper_ 4y ago
  Riot Games also has an automated testing framework: https://technology.riotgames.com/news/automated-testing-league-legends https://technology.riotgames.com/news/automated-testing-leag... I still think the best combo is a mix of automated and QA. Software in general went through a large decline in quality after QA was laid off across the industry, and in my experience, automated testing just tends to find different kinds of issues. Also, playtesting is also not the same thing as QA, but I digress :)
  - tmpz22 4y ago
    What a lot of people don’t realize about QA is that it’s also a point of Empathy to your end users. Some might think it’s the PMs job to think as an end user but in my experience PMs are so bogged down with other sales adjacent duties that they rarely if ever can provide that pillar. This is a core issue with for example Google products where MANUAL PROCESS BAD seems to be a religious bastion.
  - bongobingo1 4y ago
    Wonder what their coverage is in this - or if they even still use it. There are many occurrences of "how does this ship" bugs in League where damage numbers don't add up or status's have no actual effect, etc - sometimes hanging around for many patches. Not that I don't understand how hard it is to test the big ball of mud that (10+ years old!) game engines end up being.
  - treeman79 4y ago
    Recently Inherited a code base with almost 20000 tests. Many very detailed. Most of them were so mocked out that at end of day it was just ensuring the mock library was well tested.
0xCMP 4y ago
This kind of reminds me of making UIs in SwiftUI which requires a way to render in the editor so you need to provide test data and I thought it was neat because it sort of acts like a test.
jasonjmcghee 4y ago
Really enjoyable read! A small note, the code samples are a rather large font size on mobile. Setting the size to 0.9em for mobile widths works well for me!
- ChadNauseam 4y ago
  Thank you! It doesn't look like that when you make the window small in Chrome so I didn't catch it.
  - jasonjmcghee 4y ago
    You can use dev tools to simulate various mobile devices! https://developer.chrome.com/docs/devtools/device-mode/ https://developer.chrome.com/docs/devtools/device-mode/
harles 4y ago
Based on many of the comments here, it seems like there’s a lot of misunderstanding of what an ECS is - unfortunately the name does this no favors as it should probably be written as entity/component/system paradigm or renamed entirely. It’s not just a system that contains entities and components (that’s not what “system” refers to in the ECS name). ECS is a paradigm where entities are the data-free IDs, components contain data associated with entities, and the systems operate on entities and components in batch, usually with database-like queries. I have yet to find a good ECS reference, but Unity’s intro[0] gives some decent flavor for it. Note that only the DOTS subset of Unity (part of it anyways) uses an ECS. The default GameObject/MonoBehavior way of writing code in Unity isn’t part of an ECS. [0]: https://learn.unity.com/tutorial/entity-component-system https://learn.unity.com/tutorial/entity-component-system
- Buttons840 4y ago
  So your saying everything is a entity, and that entities have components, and the systems operate on entities (possibly based on their components). Do I understand that correctly? That understanding is definitely worth the time it took to read.
  - pmichaud 4y ago
    This is essentially correct, yes.
  - reidjs 4y ago
    That’s my understanding too. Think of super Mario. Entity is Mario, the koopa troopers, the coins, the clouds, the floor tiles, the spikes, the mushrooms. All the proper nouns. Component is the XY location in the world, ability to jump, sprite image, the height, are they affected by gravity. All the adjectives. System is gravity - all entities with weight drop at the same rate until they hit ground. Another system may be the collision mechanic - if a mario entity lands on a koopa entity, the Koopa dies and some sound effects plays.
    - forrestthewoods 4y ago
charcircuit 4y ago
>the benefit of automated testing is that it scales in O(1) with respect to number of code changes, while playtesting is O(n) This isn't true. Running more tests takes more time even with automated tests.
- ChadNauseam 4y ago
  True, although the constant factor can be pretty low depending on how many cores you have :P
evgpbfhnr 4y ago
I never understand why people always seem to think 'test flakiness' should/can be fixed by saying "it's OK if test works 1/10 times" -- if it's so dependent on the RNG then something is bound to fail for users as well, so either the test is not representative of the game or something is rotten. I guess time to fix tests is expensive...
- jamesfinlayson 4y ago
  Agreed - at a previous job lots of time was invested in setting up test retries, and splitting test runs out so that you could retrigger only the subsets that failed instead of the whole test suite.
- ChadNauseam 4y ago
  That's a really good point. It would be interesting to run the tests 10 times and have the CI fail if it doesn't pass all 10 times.
- pydry 4y ago
  It depends entirely on the reason for that 1/10 times. E.g. I had a test that hit a 3rd party sandbox that would go down occasionally. I could have mocked it to avoid this but I had way bigger fish to fry. The flakiness triggered a very specific error that we felt safe ignoring (other kinds of flakiness we would not ignore). The corresponding prod API never had issues. Sometimes it's a select statement without an order by in which case even if it's not strictly a bug but just quickly adding an order by solves the flakiness forever so you might as well. Sometimes it's a pretty terrifying race condition in the code that will absolutely crop up in prod.
Operyl 4y ago
> The only big-budget one I know of that has publicly put a large effort into it is Sea of Thieves. Another big big title that makes use of automated testing is Riot’s League of Legends. https://technology.riotgames.com/news/automated-testing-league-legends https://technology.riotgames.com/news/automated-testing-leag...
jerf 4y ago
I have often thought that GUIs, including browsers, are hard to test because the monstrous frameworks that they are make no accommodation for testing, so... of course they're hard to test. Not only are you writing a test harness around your own code, you also must rather laboriously and unreliably harness up their code too, and from the "outside" not designed to be used the way you're trying to use it to boot. Now, I don't necessarily fault GUI designers for not thinking about how to expose testability easily to their users, they're big enough projects as it is, enough to choke even a major multibillion dollar corporation, but it doesn't mean I'm any less frustrated while using them. Game engines strike me as much the same thing for similar reasons. Yes, it would be really hard to not only have Unity as it stands today but also have all the requisite changes it would take to make it really testable, but without that you're really going uphill. It's an uphill battle at best and a sheer cliff face at worst trying to test these things. And of course when that's the case, every step your local engine programmer takes only takes you farther in the direction of being untestable because why would they bother structuring their code to be testable when the framework it is embedded in is untestable? I have no solution. Only the observation that both of these systems (and also the browser, if you think of "the browser" as something other than a really funky large GUI) have problems from the very, very beginning, and until the foundation gets its act together w.r.t. testability it is always going to be a major uphill battle with anything built on that foundation. (I'm not so idealistic that I imagine any world where a AAA game has 100% automated test coverage or anything, but even in my own limited experience in game dev and less limited, but still limited, experience in GUI programming, both places where I tried to leverage my extensive years of writing automated tests even in hostile legacy codebases, I found myself not merely stymied a bit, not merely inconvenienced a bit, but comprehensively blocked at every turn. An simple example: A table cell in a table widget that had a .SetColor, but no way whatsoever to fetch the color. Meaning my code to highlight rows matching a certain criterion could not be tested; in this case, for that reason alone. What a ^#&*$@'ing stupid reason to be unable to test some non-trivial code, at least from my perspective gained over years of writing a lot of other test code... and discovering plenty of errors in code of similar complexity in places where I could test it.)
jamesfinlayson 4y ago
For the last few years I've been trying to add unit tests to all of my personal projects. I do a bit of game modding in my spare time and those are the last projects that ended up getting unit tests, but they were also the ones I most wanted to add tests for because they're the most complicated and also the most time-consuming to test manually. The biggest hurdle to start with was that the Visual Studio projects are all set up in such a way that integrating Microsoft's C++ unit test framework either requires you to (probably) recreate the project the right way, or to hack around with scripts to create a working test runner. Setting up Boost tests might be less painful. But once that is done, my next hurdle was mocking out the game engine. Not overly difficult as the engine exposes a nice API but still time consuming. Now that that's done it's been good start adding tests, though I haven't added any tests for anything too complicated yet.
syntheweave 4y ago
The post contains a common fallacy that you will know of if you have tested substantially sized games: What ECS architecture does is offload the testing burden into data instead of code. The code itself can do the right thing in all circumstances, but as you componentize your higher-level behaviors, more business logic becomes bindings of data to other data. So after a 100% green test suite, you still have bugs and all of them are of the form of "mysterious runtime behavior". The engine code doesn't know how to automatically check for a "good data shape" or flag it as a bug; you can add checks, but it's project specific and asset-specific compiler development: a missing animation is a different kind of bug from a typo in a text prompt, or a hitbox that's slightly misplaced. Thus, for a lot of categories of assets, your cheapest test is still to periodically eyeball things and say "that's in spec" or "that's fishy". ML AI might actually be able to change this scenario by acting more like human players and deriving more second-order metrics.
- echelon 4y ago
  > ML AI might actually be able to change this scenario by acting more like human players and deriving more second-order metrics. New startup idea. Probably applies to any "hard to test" thing.
- yladiz 4y ago
  I think you're reading something that isn't being argued in the post, though. The author is not arguing that you will have no bugs in your game if you use ECS and add tests, it's that testing in this way can make it easier for development because you reduce the possibility of introducing a bug inadvertently in one of your systems and then have to rely on QA to find it later (components too I guess, but I would guess component "tests" can be covered by types since they're just data and the systems are the ones that interact with the data). The kinds of bugs you mention are of course different and some of those kinds of bugs aren't worth the trouble of even trying to test, but if your game is shaped in a way that it is basically a collection of code modifying data that represents the game, you can eliminate the need for a lot of manual QA testing by just automating it.
- xwolfi 4y ago
  And it's not just games, this trend of moving code into data. I've been in several companies where their fundamental logic was a configuration interpreter, very complex, tied to a gigantic set of xml or json "configurations" which themselves had interdependent logic written as structure. So when maybe the first iteration 20 years ago was all fun and giggles "look a monkey could edit a few knobs and it all moves around fine", today it's a tangled mess that an ever rotating set of new people build on top of with no validation tool, no formalized syntax, no ability to test nor explain the holistic logic of. So, as you say, we end up running the thing and looking if the output makes sense, the code working more out of luck than engineering.
nextlevelwizard 4y ago
>But in videogames, correctness matters barely at all. This is pretty weird blanket statement or rather it very much depends on what kind of game you are making. If you are making a casual single player experience then having glitches isn't that big of a deal. However if you want to make the next e-sports game then glitches matter a lot more especially if they can be abused to gain power - be it actually powering up your character in some way or having (near) invincible position. Another aspect many game devs ignore is speed running. Having glitches in a single player game might not be that big of a deal for normal playthroughs, but many games have had long lives after their initial release as people want to speed run them and there glitches matter. If your game is broken it can create situations where speed running isn't fun.

107 points by ChadNauseam 4y ago

> Bevy uses the ECS pattern to organize game code, and it occurred to me that ECS substantially alleviates the spaghetti-code problem that makes testing in Unity and Unreal so terrible. Both Unreal and Unity uses ECS.

guzik 4y ago
Seems like the author never truly worked with these engines? Also, I never had problem with unit testing on Unity in pure C#. Even when it comes to complicated scenario there is always a way to separate your GameObjects from game logic.
- harles 4y ago
  I think this comment and its parent have misunderstood what ECS is.
- ChadNauseam 4y ago
  I have spent way too much time in Unity. (I was a Unity developer for pay for several years.) I suppose I didn't do a good job explaining the differences between what Unity does and an ECS approach in my post, but if you'd like to get a feel for what an ECS you could try out DOTS. (Although DOTS is kindn of unpleasant to use compared to Bevy.)
tarsiel 4y ago
Unity at least isn’t a true ECS architecture, more just Entity-Component. You can write game logic in an entity with Unity, whereas in ECS that would have to live in a System
- runevault 4y ago
  Unity is working on a full ECS system as part of a larger DOTS initiative, but it is still pre-alpha last I knew.
metadat 4y ago
What is ECS? Assuming it's not AWS Elastic Container Service :>
- worble 4y ago
  Entity component system
- keyle 4y ago
  Around 2007 I believe the game industry came up with a semi standard way of building games that somehow alleviated a lot of issues in producing complex games... aka spaghetti loops of hell. There are a few ECS approach and to this day it's still vague who does it best (depends on your goals). One I found interesting is the "sparse entity component system".
  - cobalt 4y ago
    it's more to avoid god classes and inheritance messes.
- deleted 4y ago
  [deleted]
keyle 4y ago
Thanks, I rolled my eyes and had an issue with the author's statement. My guess is what he meant was that the Bevy approach is cleaner to test upon.
- harles 4y ago
  Can you point me to what part of Unreal uses an ECS approach? I’ve never seen this. I know Unity DOTS added ECS features, but that still lives in a hybrid world.
  - lfowles 4y ago
    This surely isn't what was meant in previous conversation, but Unreal Engine 5 added MassEntity which is uh... an Entity Fragment Processor approach. (Because Components are already an object that exists, it probably would have been an extra special kind of confusing if it adopted typical ECS naming.) https://docs.unrealengine.com/5.0/en-US/overview-of-mass-entity-in-unreal-engine/ https://docs.unrealengine.com/5.0/en-US/overview-of-mass-ent...
  - dimgl 4y ago
    Based on the definitions in this thread, it wouldn't be a stretch to pair Unreal's `AActor` as the entities and `UActorComponent` as the components. You could technically go a step further and pair `UWorldSubsystem` as the system. I realize that there's the whole "grouping" and "batching" concept in ECS paradigms, but I'm failing to see how that's much different from using the functions in `UWorld` that can query both components and actors of a certain class.
Jasper_ 4y ago
This is where the unfortunately-chosen name "Entity-Component-Systems" (ECS) comes back to bite us. Unity's GameObjects are Entity-Component (EC): you have an Entity (GameObject), onto which you attach Components (MonoBehavior). Unreal is similar: you have an Entity (UActor), onto which you attach Components (UActorComponent), though you can also have plenty of code living outside the UActorComponent system in Unreal. Entities are objects, and Components are code/data. Entity-Compontity-System (ECS) uses these nouns differently; an Entity ends up being little more than an identifier for a large number of Components, which are data structures, and Systems are the code that modify groups of Components. "System" here is a noun, not a descriptor of the practice. The usual analogy is similar to a relational database (in fact, the point of the system is to achieve wide-scale performance like RDSes do): Entities are rows, identified by a primary key, Components are columns, and Systems are the SELECT/INSERT/UPDATE/DELETE queries which acts on the data. Unity has a new-ish ECS framework under the DOTS initiative; Unreal has no parallel. Very few games are using these.
- Jensson 4y ago
  In unity the methods "Update", "Start" and so on are the system part of ECS. That you write their implementations alongside the data doesn't really matter.
  - forrestthewoods 4y ago
    Sorry, you're wrong on this one. ECS has come to mean a very particular style of architecture. It's not a great name, but it's what the industry has converged on. Naming things is hard. Unity's MonoBehavior framework uses GameObjects and Components. You don't have to squint hard to see that it contains Entities, Components, and arguably Systems. However ECS has come to mean an architecture that is quite different from MonoBehavior. So different infact that Unity is writing an ECS framework called DOTS! Both DOTS and MonoBehaviors containing entities, components, and functions that operate on them. However the two Unity systems are extremely different. MonoBehaviors are not an ECS.
    - Jensson 4y ago

stuckinhell 4y ago

I've been looking into Unity and game development in my spare time. It seems writing tests for game code is incredibly dependent on the style used to develop the game. If you are doing everything by hand in C#, its super easy. The moment you introduce editor workflows or third party libraries, I have no idea how anybody can write sensible tests. Using Unity is like a weird mix of no code tools and code.

guzik 4y ago
> Using Unity is like a weird mix of no code tools and code. Haha, great summary of my 5 years experience in Unity. Whenever I am too lazy I am switching to ECS/MonoBehaviour, then I regret of not having properly testable architecture.
- Jaxkr 4y ago
  ECS is a competing system to supersede MonoBehavior

andrewallbright 4y ago

Writing automated tests for video games is some of the most fun programming I do.

Goz3rr 4y ago

The author mentions Sea of Thieves and Mindustry, but I think Factorio is worth a mention too (seemingly as usual when something game related comes up on HN). They do TDD and talk about their automatic tests in several blogs: https://factorio.com/blog/post/fff-366 https://factorio.com/blog/post/fff-366 https://factorio.com/blog/post/fff-288 https://factorio.com/blog/post/fff-288 https://factorio.com/blog/post/fff-186 https://factorio.com/blog/post/fff-186 https://factorio.com/blog/post/fff-62 https://factorio.com/blog/post/fff-62

Jasper_ 4y ago
Riot Games also has an automated testing framework: https://technology.riotgames.com/news/automated-testing-league-legends https://technology.riotgames.com/news/automated-testing-leag... I still think the best combo is a mix of automated and QA. Software in general went through a large decline in quality after QA was laid off across the industry, and in my experience, automated testing just tends to find different kinds of issues. Also, playtesting is also not the same thing as QA, but I digress :)
- tmpz22 4y ago
  What a lot of people don’t realize about QA is that it’s also a point of Empathy to your end users. Some might think it’s the PMs job to think as an end user but in my experience PMs are so bogged down with other sales adjacent duties that they rarely if ever can provide that pillar. This is a core issue with for example Google products where MANUAL PROCESS BAD seems to be a religious bastion.
- bongobingo1 4y ago
  Wonder what their coverage is in this - or if they even still use it. There are many occurrences of "how does this ship" bugs in League where damage numbers don't add up or status's have no actual effect, etc - sometimes hanging around for many patches. Not that I don't understand how hard it is to test the big ball of mud that (10+ years old!) game engines end up being.
- treeman79 4y ago
  Recently Inherited a code base with almost 20000 tests. Many very detailed. Most of them were so mocked out that at end of day it was just ensuring the mock library was well tested.

0xCMP 4y ago

This kind of reminds me of making UIs in SwiftUI which requires a way to render in the editor so you need to provide test data and I thought it was neat because it sort of acts like a test.

jasonjmcghee 4y ago

Really enjoyable read! A small note, the code samples are a rather large font size on mobile. Setting the size to 0.9em for mobile widths works well for me!

ChadNauseam 4y ago
Thank you! It doesn't look like that when you make the window small in Chrome so I didn't catch it.
- jasonjmcghee 4y ago
  You can use dev tools to simulate various mobile devices! https://developer.chrome.com/docs/devtools/device-mode/ https://developer.chrome.com/docs/devtools/device-mode/

harles 4y ago

Based on many of the comments here, it seems like there’s a lot of misunderstanding of what an ECS is - unfortunately the name does this no favors as it should probably be written as entity/component/system paradigm or renamed entirely. It’s not just a system that contains entities and components (that’s not what “system” refers to in the ECS name). ECS is a paradigm where entities are the data-free IDs, components contain data associated with entities, and the systems operate on entities and components in batch, usually with database-like queries. I have yet to find a good ECS reference, but Unity’s intro[0] gives some decent flavor for it. Note that only the DOTS subset of Unity (part of it anyways) uses an ECS. The default GameObject/MonoBehavior way of writing code in Unity isn’t part of an ECS. [0]: https://learn.unity.com/tutorial/entity-component-system https://learn.unity.com/tutorial/entity-component-system

Buttons840 4y ago
So your saying everything is a entity, and that entities have components, and the systems operate on entities (possibly based on their components). Do I understand that correctly? That understanding is definitely worth the time it took to read.
- pmichaud 4y ago
  This is essentially correct, yes.
- reidjs 4y ago
  That’s my understanding too. Think of super Mario. Entity is Mario, the koopa troopers, the coins, the clouds, the floor tiles, the spikes, the mushrooms. All the proper nouns. Component is the XY location in the world, ability to jump, sprite image, the height, are they affected by gravity. All the adjectives. System is gravity - all entities with weight drop at the same rate until they hit ground. Another system may be the collision mechanic - if a mario entity lands on a koopa entity, the Koopa dies and some sound effects plays.
  - forrestthewoods 4y ago

charcircuit 4y ago

>the benefit of automated testing is that it scales in O(1) with respect to number of code changes, while playtesting is O(n) This isn't true. Running more tests takes more time even with automated tests.

ChadNauseam 4y ago
True, although the constant factor can be pretty low depending on how many cores you have :P

evgpbfhnr 4y ago

I never understand why people always seem to think 'test flakiness' should/can be fixed by saying "it's OK if test works 1/10 times" -- if it's so dependent on the RNG then something is bound to fail for users as well, so either the test is not representative of the game or something is rotten. I guess time to fix tests is expensive...

jamesfinlayson 4y ago
Agreed - at a previous job lots of time was invested in setting up test retries, and splitting test runs out so that you could retrigger only the subsets that failed instead of the whole test suite.
ChadNauseam 4y ago
That's a really good point. It would be interesting to run the tests 10 times and have the CI fail if it doesn't pass all 10 times.
pydry 4y ago
It depends entirely on the reason for that 1/10 times. E.g. I had a test that hit a 3rd party sandbox that would go down occasionally. I could have mocked it to avoid this but I had way bigger fish to fry. The flakiness triggered a very specific error that we felt safe ignoring (other kinds of flakiness we would not ignore). The corresponding prod API never had issues. Sometimes it's a select statement without an order by in which case even if it's not strictly a bug but just quickly adding an order by solves the flakiness forever so you might as well. Sometimes it's a pretty terrifying race condition in the code that will absolutely crop up in prod.

Operyl 4y ago

> The only big-budget one I know of that has publicly put a large effort into it is Sea of Thieves. Another big big title that makes use of automated testing is Riot’s League of Legends. https://technology.riotgames.com/news/automated-testing-league-legends https://technology.riotgames.com/news/automated-testing-leag...

jerf 4y ago

I have often thought that GUIs, including browsers, are hard to test because the monstrous frameworks that they are make no accommodation for testing, so... of course they're hard to test. Not only are you writing a test harness around your own code, you also must rather laboriously and unreliably harness up their code too, and from the "outside" not designed to be used the way you're trying to use it to boot. Now, I don't necessarily fault GUI designers for not thinking about how to expose testability easily to their users, they're big enough projects as it is, enough to choke even a major multibillion dollar corporation, but it doesn't mean I'm any less frustrated while using them. Game engines strike me as much the same thing for similar reasons. Yes, it would be really hard to not only have Unity as it stands today but also have all the requisite changes it would take to make it really testable, but without that you're really going uphill. It's an uphill battle at best and a sheer cliff face at worst trying to test these things. And of course when that's the case, every step your local engine programmer takes only takes you farther in the direction of being untestable because why would they bother structuring their code to be testable when the framework it is embedded in is untestable? I have no solution. Only the observation that both of these systems (and also the browser, if you think of "the browser" as something other than a really funky large GUI) have problems from the very, very beginning, and until the foundation gets its act together w.r.t. testability it is always going to be a major uphill battle with anything built on that foundation. (I'm not so idealistic that I imagine any world where a AAA game has 100% automated test coverage or anything, but even in my own limited experience in game dev and less limited, but still limited, experience in GUI programming, both places where I tried to leverage my extensive years of writing automated tests even in hostile legacy codebases, I found myself not merely stymied a bit, not merely inconvenienced a bit, but comprehensively blocked at every turn. An simple example: A table cell in a table widget that had a .SetColor, but no way whatsoever to fetch the color. Meaning my code to highlight rows matching a certain criterion could not be tested; in this case, for that reason alone. What a ^#&*$@'ing stupid reason to be unable to test some non-trivial code, at least from my perspective gained over years of writing a lot of other test code... and discovering plenty of errors in code of similar complexity in places where I could test it.)

jamesfinlayson 4y ago

For the last few years I've been trying to add unit tests to all of my personal projects. I do a bit of game modding in my spare time and those are the last projects that ended up getting unit tests, but they were also the ones I most wanted to add tests for because they're the most complicated and also the most time-consuming to test manually. The biggest hurdle to start with was that the Visual Studio projects are all set up in such a way that integrating Microsoft's C++ unit test framework either requires you to (probably) recreate the project the right way, or to hack around with scripts to create a working test runner. Setting up Boost tests might be less painful. But once that is done, my next hurdle was mocking out the game engine. Not overly difficult as the engine exposes a nice API but still time consuming. Now that that's done it's been good start adding tests, though I haven't added any tests for anything too complicated yet.

syntheweave 4y ago

The post contains a common fallacy that you will know of if you have tested substantially sized games: What ECS architecture does is offload the testing burden into data instead of code. The code itself can do the right thing in all circumstances, but as you componentize your higher-level behaviors, more business logic becomes bindings of data to other data. So after a 100% green test suite, you still have bugs and all of them are of the form of "mysterious runtime behavior". The engine code doesn't know how to automatically check for a "good data shape" or flag it as a bug; you can add checks, but it's project specific and asset-specific compiler development: a missing animation is a different kind of bug from a typo in a text prompt, or a hitbox that's slightly misplaced. Thus, for a lot of categories of assets, your cheapest test is still to periodically eyeball things and say "that's in spec" or "that's fishy". ML AI might actually be able to change this scenario by acting more like human players and deriving more second-order metrics.

echelon 4y ago
> ML AI might actually be able to change this scenario by acting more like human players and deriving more second-order metrics. New startup idea. Probably applies to any "hard to test" thing.
yladiz 4y ago
I think you're reading something that isn't being argued in the post, though. The author is not arguing that you will have no bugs in your game if you use ECS and add tests, it's that testing in this way can make it easier for development because you reduce the possibility of introducing a bug inadvertently in one of your systems and then have to rely on QA to find it later (components too I guess, but I would guess component "tests" can be covered by types since they're just data and the systems are the ones that interact with the data). The kinds of bugs you mention are of course different and some of those kinds of bugs aren't worth the trouble of even trying to test, but if your game is shaped in a way that it is basically a collection of code modifying data that represents the game, you can eliminate the need for a lot of manual QA testing by just automating it.
xwolfi 4y ago
And it's not just games, this trend of moving code into data. I've been in several companies where their fundamental logic was a configuration interpreter, very complex, tied to a gigantic set of xml or json "configurations" which themselves had interdependent logic written as structure. So when maybe the first iteration 20 years ago was all fun and giggles "look a monkey could edit a few knobs and it all moves around fine", today it's a tangled mess that an ever rotating set of new people build on top of with no validation tool, no formalized syntax, no ability to test nor explain the holistic logic of. So, as you say, we end up running the thing and looking if the output makes sense, the code working more out of luck than engineering.

nextlevelwizard 4y ago

>But in videogames, correctness matters barely at all. This is pretty weird blanket statement or rather it very much depends on what kind of game you are making. If you are making a casual single player experience then having glitches isn't that big of a deal. However if you want to make the next e-sports game then glitches matter a lot more especially if they can be abused to gain power - be it actually powering up your character in some way or having (near) invincible position. Another aspect many game devs ignore is speed running. Having glitches in a single player game might not be that big of a deal for normal playthroughs, but many games have had long lives after their initial release as people want to speed run them and there glitches matter. If your game is broken it can create situations where speed running isn't fun.