9 ms·

I think the focus on ECS when talking about data-oriented design largely misses the point of what data-oriented design is all about. Focusing on a _code_ design

I think the focus on ECS when talking about data-oriented design largely misses the point of what data-oriented design is all about. Focusing on a _code_ design pattern is the antithesis of _data_-oriented design. Data-oriented is about breaking away from the obsession with taxonomy, abstraction and world-modeling and moving towards the understanding that all software problems are data transformation problems. It's that all games essentially (and most software in general) boil down to: transform(input, current_state) -> output, new_state Then, for some finite set of platforms and hardware there will be an optimal transform to accomplish this and it is our job as engineers to make "the code" approach this optimal transform.

BobbyJo 5y ago
> Focusing on a _code_ design pattern is the antithesis of _data_-oriented design Doesn't the former enable the latter? Ideally, language (both human and machine) would have the semantics needed to represent all transforms, but that's not the case. Code you rely on, since none of it is written in isolation, needs to enable you to implement data-oriented design should you so choose. Also, I don't think pointing out that 'all games are essentially...' is particularly useful. It's true, no question, but that doesn't mean it's the most useful mental model for people to use when developing software. Our job as engineers is to make software that functions according to some set of desires, and those desires may directly conflict with approaching an optimal transform.
- jeremycw 5y ago
  > Doesn't the former enable the latter? Not necessarily. ECS is a local maxima when developing a general purpose game engine. Since it's general purpose it can do nothing more than provide a lowest common denominator interface that can be used to make any game. If you are building a game from scratch why would you limit yourself to a lowest common denominator interface when there's no need? Just write the exact concrete code that needs to be there to solve the problem. > Our job as engineers is to make software that functions according to some set of desires, and those desires may directly conflict with approaching an optimal transform. All runtime desires of the software must be encoded in the transform. So no software functionality should get in the way of approaching the optimal transform. What does get in the way of approaching the optimal transform is code organization, architecture and abstraction that is non-essential to performing the transform.
  - BrS96bVxXBLzf5B 5y ago
    > Just write the exact concrete code that needs to be there to solve the problem. Good luck with that when the exact code to solve the problem is not the exact code the next week, because the problem has changed or evolved. Not to suggest an ECS is the answer, but this line of thinking is reductive to the realities of creating a piece of art. It's not a spec you can draw a diagram for and trust will be basically the same. It's a creature you discover, revealing more of itself over time. The popularity of the ECS is because it provides accessible composition. It's not the only way of composing data but being able to say "AddX", "RemoveX" without the implementation details of what struct holds what data and what groupings might matter is what makes it appealing.
    - meheleventyone 5y ago
      I think there’s two orthogonal things being conflated by you. Flexibility of a solution and how general the solution is. What you’re basically saying is a solution should be flexible to change because making a game requires trial and error. I totally agree with that. Using a general solution is one path to flexibility but it does come with a cost associated. It’s flexibility built on a tower of complexity and if you look at a modern ECS implementation that is performant it’s actually quite a lot of complexity. You’re also reducing flexibility in the sense that these sort of solutions generally have preferred patterns you need to fit your game design into. So you end up introducing a learning, maintainance and conceptual burden into the project you might not need. OTOH if you have a specific problem you can write a specific solution for you will end up with less code, hopefully in a conceptually coherent form. That in itself offers flexibility. Simple code you can easily replace is often more flexible than complex code you need to coax into a new form. The key is to recognise whether your problem is specific or general you need flexibility. These architectural patterns are fun to argue over and obsessed over by armchair game developers but are a trap if you’re trying to make a game rather than a general purpose game engine. Which isn’t to say you don’t want some framework underlying things for all sorts of mundane reasons. But most games could get away with that being an entity type that gets specialised rather than anything more complex.
      - 5y ago
WorldMaker 5y ago
This fits my gut feeling every time I see an ECS system that videogame design has gotten stuck in a local maxima abstraction/pattern. Often what they really want is a Monadic abstraction of a data/state transformation process, but they are often stuck in languages ("for performance reasons") that make it hard to impossible to get good Monadic abstractions. So instead they use the hammers that to make nails of the abstractions that they can get. ECS feels to me like a strange attempt to build Self-like OO dynamic prototypes in a class-based OO language, and that's almost exactly what you would expect for an industry only just now taking baby steps outside of C/C++. C# has some good tools to head towards that direction (async/await is a powerful Monadic transformer, for instance; it's not a generic enough transformer on its own of course, but an interesting start), but as this article points out most videogames work in C# today still has to keep the back foot in C/C++ land at all times and C/C++ mentalities are still going to clip the wings of abstraction work. (ETA: Local maxima are still useful of course! Just that I'd like to point out that they can also be a trap.)
- learc83 5y ago
  >"for performance reasons" The quotes imply that this is a bad reason, but in soft realtime systems you often want complete control of memory allocation. Even in the case of something like Unity--in order to give developers the performance they want--they've designed subset of C# they call high performance C# where memory is manually allocated. In most cases if you're using an ECS, it's because you care so much about performance that you want to organize most of your data around cache locality. If you don't care about performance, something like the classic Unity Game Object component architecture is a lot easier to work with.
  - unknownOrigin 5y ago
    Yea, you're right, I think the previous poster seriouspy underestimates videogames as performance critical (and performance consistent!) apps. In the modern days of desktop Java and C# (and even more in web dev) the vast majority of coders just don't come across the need to "do everything you need to do" in 33ms or less, consistently.
bob1029 5y ago
> Data-oriented is about breaking away from the obsession with taxonomy, abstraction and world-modeling Something about this does not sit well with me. Data is absolutely worthless if it generated on top of a garbage schema. Having poor modeling is catastrophic to any complex software project, and will be the root of all evil downstream. In my view, the principal reason people hate SQL is because no one took the time to "build the world" and consult with the business experts to verify if their model was well-aligned with reality (i.e. the schema is a dumpster fire). As a consequence, recursive queries and other abominations are required to obtain meaningful business insights. If you took the time to listen to the business explain the complex journey that - for instance - user email addresses went down, you may have decided to model them in their own table rather than as a dumb string fact on the Customers table with zero historization potential. Imagine if you could go back in time and undo all those little fuck ups in your schemas. With the power of experience and planning ahead, you can do the 2nd best thing.
- typon 5y ago
  Schema design is THE problem data oriented programming is focused on. It's saying, let's design our data structures in memory and on disk such that they exist to solve the problem at hand. I think youre talking about the same thing
  - Keyframe 5y ago
    Or to circle.back again to Fred Brooks, time and time again: "Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowcharts; they'll be obvious." - Fred Brooks
- jeremycw 5y ago
  You're right, when I mentioned "taxonomy, abstraction and world-modeling" I meant as it pertains to code organization in the tradition OOP/OOD sense where it's generally about naming classes, creating inheritance hierarchies, etc. Data-oriented design is _absolutely_ concerned with the data schema. I would, however, disagree that the focus should be on "building the world" with your schema. To me this means creating the schema based off of some gut/fuzzy feeling you get when the names of things all end up being real world nouns. To me creating a good schema is less about world building than it is about having the exact data that you need, well normalized and in a format that works well with the algorithm you want to apply to it.
typon 5y ago
It's really annoying how many people misunderstand the term 'data oriented design'. Usually to mean something like 'not object oriented programming'. If your data was inherently hierarchical and talking about animals that meow or moo, go ahead and implement the textbook OO modeling. This Mike Acton post describes it accurately: http://www.macton.ninja/home/onwhydodisntamodellingapproachatall http://www.macton.ninja/home/onwhydodisntamodellingapproacha...
- nikki93 5y ago
  I do think that in practice a hierarchical ontology like that is still best not modeled as a language-level hierarchy because the language inheritance / hierarchy concepts are often not the exact semantic you want, IME. Esp. if it then ties to static types, since you then can't change the hierarchy at runtime. I think even with a data-oriented approach to a hierarchy -- your code isn't necessarily hierarchically organized, it just handles data that happens to express a hierarchy. And you want to be in control of the semantics of said hierarchy with more freedom (and explicitness) than the language-level hierarchy gives you -- so you want your own code that interprets the hierarchy expressed in the data and performs your desired semantics. This also allows artists and narrative or gameplay / level designers to go see and edit the hierarchy and add elements to it. An example is the prefab hierarchy you get in Unity, which is expressed through the data (prefabs and their relationships). (Note: I mean specifically the prefab inheritance hierarchy, not the transform spatial hierarchy -- the former has more overlap with the "is a" relationships). The code processing this hierarchy could've just been plain C code that parses the files and maintains an in-memory set of structures about them, even. You then get to define how properties inherit, what overriding means, etc. yourself.
  - typon 5y ago
    Totally agreed. The fact that some of these concepts are embedded into the language design (say for C++) are minor conveniences at best - when they almost perfectly line up with the data you have - but just get in the way most of the time.
dragonwriter 5y ago
> moving towards the understanding that all software problems are data transformation problems. But this understanding is fundamentally, deeply wrong, in the same way that civil engineering based approaches to software engineering are wrong for most software applications. That is: yes, all software systems are data transformation systems, but most software problems are not “how do I produce the system most narrowly tailored to the present requirements” but more often “how to engineer a system for success with the pace and kind of change that we can expect over time in this space”. (Now, games, particularly, are both pushing the limits of hardware and fairly static, so making them narrowly-tailored, poorly adaptable static works is often not wrong. But that doesn't generalize to all, or even most, software.)
- sitkack 5y ago
  That is how you think about the software you write. System evolution is only one aspect. Most patterned OO codebases I have come across were *not* engineered for evolution. Sure there were some classes you could implement or replace, but the complexity was not paid back later. Design principles can be applied to all implementation mechanisms.

BobbyJo 5y ago

> Focusing on a _code_ design pattern is the antithesis of _data_-oriented design Doesn't the former enable the latter? Ideally, language (both human and machine) would have the semantics needed to represent all transforms, but that's not the case. Code you rely on, since none of it is written in isolation, needs to enable you to implement data-oriented design should you so choose. Also, I don't think pointing out that 'all games are essentially...' is particularly useful. It's true, no question, but that doesn't mean it's the most useful mental model for people to use when developing software. Our job as engineers is to make software that functions according to some set of desires, and those desires may directly conflict with approaching an optimal transform.

jeremycw 5y ago
> Doesn't the former enable the latter? Not necessarily. ECS is a local maxima when developing a general purpose game engine. Since it's general purpose it can do nothing more than provide a lowest common denominator interface that can be used to make any game. If you are building a game from scratch why would you limit yourself to a lowest common denominator interface when there's no need? Just write the exact concrete code that needs to be there to solve the problem. > Our job as engineers is to make software that functions according to some set of desires, and those desires may directly conflict with approaching an optimal transform. All runtime desires of the software must be encoded in the transform. So no software functionality should get in the way of approaching the optimal transform. What does get in the way of approaching the optimal transform is code organization, architecture and abstraction that is non-essential to performing the transform.
- BrS96bVxXBLzf5B 5y ago
  > Just write the exact concrete code that needs to be there to solve the problem. Good luck with that when the exact code to solve the problem is not the exact code the next week, because the problem has changed or evolved. Not to suggest an ECS is the answer, but this line of thinking is reductive to the realities of creating a piece of art. It's not a spec you can draw a diagram for and trust will be basically the same. It's a creature you discover, revealing more of itself over time. The popularity of the ECS is because it provides accessible composition. It's not the only way of composing data but being able to say "AddX", "RemoveX" without the implementation details of what struct holds what data and what groupings might matter is what makes it appealing.
  - meheleventyone 5y ago
    I think there’s two orthogonal things being conflated by you. Flexibility of a solution and how general the solution is. What you’re basically saying is a solution should be flexible to change because making a game requires trial and error. I totally agree with that. Using a general solution is one path to flexibility but it does come with a cost associated. It’s flexibility built on a tower of complexity and if you look at a modern ECS implementation that is performant it’s actually quite a lot of complexity. You’re also reducing flexibility in the sense that these sort of solutions generally have preferred patterns you need to fit your game design into. So you end up introducing a learning, maintainance and conceptual burden into the project you might not need. OTOH if you have a specific problem you can write a specific solution for you will end up with less code, hopefully in a conceptually coherent form. That in itself offers flexibility. Simple code you can easily replace is often more flexible than complex code you need to coax into a new form. The key is to recognise whether your problem is specific or general you need flexibility. These architectural patterns are fun to argue over and obsessed over by armchair game developers but are a trap if you’re trying to make a game rather than a general purpose game engine. Which isn’t to say you don’t want some framework underlying things for all sorts of mundane reasons. But most games could get away with that being an entity type that gets specialised rather than anything more complex.
    - 5y ago

WorldMaker 5y ago

This fits my gut feeling every time I see an ECS system that videogame design has gotten stuck in a local maxima abstraction/pattern. Often what they really want is a Monadic abstraction of a data/state transformation process, but they are often stuck in languages ("for performance reasons") that make it hard to impossible to get good Monadic abstractions. So instead they use the hammers that to make nails of the abstractions that they can get. ECS feels to me like a strange attempt to build Self-like OO dynamic prototypes in a class-based OO language, and that's almost exactly what you would expect for an industry only just now taking baby steps outside of C/C++. C# has some good tools to head towards that direction (async/await is a powerful Monadic transformer, for instance; it's not a generic enough transformer on its own of course, but an interesting start), but as this article points out most videogames work in C# today still has to keep the back foot in C/C++ land at all times and C/C++ mentalities are still going to clip the wings of abstraction work. (ETA: Local maxima are still useful of course! Just that I'd like to point out that they can also be a trap.)

learc83 5y ago
>"for performance reasons" The quotes imply that this is a bad reason, but in soft realtime systems you often want complete control of memory allocation. Even in the case of something like Unity--in order to give developers the performance they want--they've designed subset of C# they call high performance C# where memory is manually allocated. In most cases if you're using an ECS, it's because you care so much about performance that you want to organize most of your data around cache locality. If you don't care about performance, something like the classic Unity Game Object component architecture is a lot easier to work with.
- unknownOrigin 5y ago
  Yea, you're right, I think the previous poster seriouspy underestimates videogames as performance critical (and performance consistent!) apps. In the modern days of desktop Java and C# (and even more in web dev) the vast majority of coders just don't come across the need to "do everything you need to do" in 33ms or less, consistently.

bob1029 5y ago

> Data-oriented is about breaking away from the obsession with taxonomy, abstraction and world-modeling Something about this does not sit well with me. Data is absolutely worthless if it generated on top of a garbage schema. Having poor modeling is catastrophic to any complex software project, and will be the root of all evil downstream. In my view, the principal reason people hate SQL is because no one took the time to "build the world" and consult with the business experts to verify if their model was well-aligned with reality (i.e. the schema is a dumpster fire). As a consequence, recursive queries and other abominations are required to obtain meaningful business insights. If you took the time to listen to the business explain the complex journey that - for instance - user email addresses went down, you may have decided to model them in their own table rather than as a dumb string fact on the Customers table with zero historization potential. Imagine if you could go back in time and undo all those little fuck ups in your schemas. With the power of experience and planning ahead, you can do the 2nd best thing.

typon 5y ago
Schema design is THE problem data oriented programming is focused on. It's saying, let's design our data structures in memory and on disk such that they exist to solve the problem at hand. I think youre talking about the same thing
- Keyframe 5y ago
  Or to circle.back again to Fred Brooks, time and time again: "Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowcharts; they'll be obvious." - Fred Brooks
jeremycw 5y ago
You're right, when I mentioned "taxonomy, abstraction and world-modeling" I meant as it pertains to code organization in the tradition OOP/OOD sense where it's generally about naming classes, creating inheritance hierarchies, etc. Data-oriented design is _absolutely_ concerned with the data schema. I would, however, disagree that the focus should be on "building the world" with your schema. To me this means creating the schema based off of some gut/fuzzy feeling you get when the names of things all end up being real world nouns. To me creating a good schema is less about world building than it is about having the exact data that you need, well normalized and in a format that works well with the algorithm you want to apply to it.

typon 5y ago

It's really annoying how many people misunderstand the term 'data oriented design'. Usually to mean something like 'not object oriented programming'. If your data was inherently hierarchical and talking about animals that meow or moo, go ahead and implement the textbook OO modeling. This Mike Acton post describes it accurately: http://www.macton.ninja/home/onwhydodisntamodellingapproachatall http://www.macton.ninja/home/onwhydodisntamodellingapproacha...

nikki93 5y ago
I do think that in practice a hierarchical ontology like that is still best not modeled as a language-level hierarchy because the language inheritance / hierarchy concepts are often not the exact semantic you want, IME. Esp. if it then ties to static types, since you then can't change the hierarchy at runtime. I think even with a data-oriented approach to a hierarchy -- your code isn't necessarily hierarchically organized, it just handles data that happens to express a hierarchy. And you want to be in control of the semantics of said hierarchy with more freedom (and explicitness) than the language-level hierarchy gives you -- so you want your own code that interprets the hierarchy expressed in the data and performs your desired semantics. This also allows artists and narrative or gameplay / level designers to go see and edit the hierarchy and add elements to it. An example is the prefab hierarchy you get in Unity, which is expressed through the data (prefabs and their relationships). (Note: I mean specifically the prefab inheritance hierarchy, not the transform spatial hierarchy -- the former has more overlap with the "is a" relationships). The code processing this hierarchy could've just been plain C code that parses the files and maintains an in-memory set of structures about them, even. You then get to define how properties inherit, what overriding means, etc. yourself.
- typon 5y ago
  Totally agreed. The fact that some of these concepts are embedded into the language design (say for C++) are minor conveniences at best - when they almost perfectly line up with the data you have - but just get in the way most of the time.

dragonwriter 5y ago

> moving towards the understanding that all software problems are data transformation problems. But this understanding is fundamentally, deeply wrong, in the same way that civil engineering based approaches to software engineering are wrong for most software applications. That is: yes, all software systems are data transformation systems, but most software problems are not “how do I produce the system most narrowly tailored to the present requirements” but more often “how to engineer a system for success with the pace and kind of change that we can expect over time in this space”. (Now, games, particularly, are both pushing the limits of hardware and fairly static, so making them narrowly-tailored, poorly adaptable static works is often not wrong. But that doesn't generalize to all, or even most, software.)

sitkack 5y ago
That is how you think about the software you write. System evolution is only one aspect. Most patterned OO codebases I have come across were *not* engineered for evolution. Sure there were some classes you could implement or replace, but the complexity was not paid back later. Design principles can be applied to all implementation mechanisms.