10 ms·

Data-Oriented Design (2018)

389 points by DeathArrow 3y ago

eska 3y ago
Looked at it for 10 seconds and immediately found random 0s at the end of the document or unreplaced text like “Noel Llopis in his September 2009 article[#!NoelDOD!#]”
- jerrygenser 3y ago
  The intro describes that it's a free version of a book that contains most of the content except for a few sections. The book was converted using an automated format from it's native design into html so a large portion could be made available for free and this automated conversion process might have some minor issues -- like the one you described.
dosshell 3y ago
One key concept when I use DoD is to not abstract away the data. Less is more. But when quickly reading the intro text, I found it doing the opposite, it talks too much and abstracts away the key concepts. Only me who found it a bit ironic of not drinking the wine?
Ciantic 3y ago
Everytime I hear about Data-Driven/Oriented Design I remember a paper from OOP course I had to read in University, it used Data-Driven Design as an example how to not to do things. The paper in question is this from 1989: https://dl.acm.org/doi/pdf/10.1145/74877.74885 https://dl.acm.org/doi/pdf/10.1145/74877.74885 It highlights that: "Even though the goal of data-driven design is to encapsulate data and algorithms, it inherently violates that encapsulation by making the structure of an object part of the definition of the object. This in turn leads to the definition of operations that reflect that structure (because they were designed with the structure in mind). Attempts to change the structure of an object transparently are destined to fail because other classes rely on that structure. This is the antithesis of encapsulation." Then goes to show that Responsibility-Driven Design has better approach. What we mean with Data-Driven Design have come a long way though, and isn't comparable to those days. I find it a bit amusing that Data-Driven Design used to be an insult of sorts, like you didn't know how to do things OOP the right way.
- dosshell 3y ago
  You do realize that DoD has nothing to do with data-driven, right? Data oriented design is about structure the data in regard of how it is processed and the hardware. This is opposite of object oriented design which models the data around your mental model. For example in DoD you could design a map with keys in one array together and one array of values. This would make it much faster to iterate over keys while searching because of cache memory. While in object oriented you would store an array of pairs. Both approaches can use data-driven. Edit: would -> could
  - morelisp 3y ago
    There are also multiple definitions of "data-driven" floating around depending on qualifier/context. An OLAP RDBMS, for example, will certainly be written in a data-oriented way, and also have almost fully data-driven behavior, but will certainly not have data-driven (in the sense of that paper) design.
  - drainyard 3y ago
    While I agree with your point, the map example is not entirely well represented in your comment. The idea is not to just store keys and values in separate arrays, the idea is to look at your use case and model your data after the transformation you need. So if you have a case where storing keys and values as pairs because of your access pattern, then do that, if you have a case where you do a lot of searches through keys, then store them in separate arrays. The point of DoD is to look at the data you have and the data you need it transformed into and then structure your data after.
- dgb23 3y ago
  The example given as "data driven design" in this paper neither resembles modern data driven programming nor data oriented design. I don't quite understand why the authors thought of this as a good example of "data driven design". I know that functions in a data driven program tend to be very generic so the conclusion you cite is very much a mismatch from my experience.
gjadi 3y ago
Found an online review of the book: https://gist.github.com/seece/25ed1b2108cf5782718b026382f2c52c https://gist.github.com/seece/25ed1b2108cf5782718b026382f2c5...
- victor106 3y ago
  Thanks for the link. Found this interesting and against common advice. "The bane of many projects, and the cause of their lateness, has been the insistence on not doing optimisation prematurely. The reason optimisation at late stages is so difficult is that many pieces of software are built up with instances of objects everywhere, even when not needed." There are definitely applications where performance is of primary concern (maybe only a few) and others where they are not. In apps where it is, this gives me thought that maybe premature optimization is okay? Am I reading that right? There's also this called Data-Oriented Programming https://www.manning.com/books/data-oriented-programming https://www.manning.com/books/data-oriented-programming. Are both these concepts the same?
  - crabmusket 3y ago
    No, DOP is basically just functional programming. There are some overlaps (e.g. separate code from data) but they're not related.
bob1029 3y ago
> Is your data layout defined by a single interpretation from a single point of view? I think this might be the most important question at technology selection and architecture time. Answering it usually requires talking to the business and customers. If you are certain there is exactly 1 valid "view" of the data that will be used throughout, then perhaps enshrining it in code makes sense. If you are even a tiny bit uncertain of this, a relational-style model probably works better. SQL is the end game for most businesses once they realize the game theory around this one... I am curious what HN thinks as major reasons for why everyone seems to have moved away from 1 big SQL database. From my perspective, yeah we have "web scale" edge cases that threaten vertical scalability on writes, but most businesses will never touch this, including members of the F100.
- jwestbury 3y ago
  At a previous F100 company -- a tech company whose products are widely used, we'll say -- we received guidance that RDBMS was verboten except with explicit approval. This had nothing to do with the best ways to model a given dataset, or achieving the best performance, and everything to do with schema flexibility and a history of outages caused by fucking up schema migrations. These problems weren't occurring in our NoSQL designs, and whatever benefits SQL databases offered didn't counter the huge benefits we gained from NoSQL's lack of rigid schema. Of course, bad uses of key-value stores can have massive performance impacts, and huge monetary costs when leveraging cloud platforms like DynamoDB -- I've seen a lot of cases where people didn't properly structure their data for DDB, and ended up performing loads of scans and sending costs through the roof.
  - bob1029 3y ago
    > schema flexibility If the business is of the notion that the schema is "flexible", then it is probably time to bring all of the MBAs into a conference room and have a come-to-jesus conversation about the limitations of information theory and human suffering. At a certain point, when someone says "Widget", everyone in the organization needs to be on the same page. This goes well beyond any specific technology.
  - 3y ago
inopinatus 3y ago
Some of the best advice I ever got for writing composable, high-performance code was “work on structs of arrays, not arrays of structs”. I hear many echoes of that advice in this text. Turns out that entity-component architectures work well in line-of-business applications too, not just games. Alas, many developers in enterprise are rusted onto a record-keeping CRUD model and struggle to think in columns rather than rows. The idea of inserting an entity id into a “published” table, instead of setting a boolean “published” field to true, doesn’t always come naturally. Yet once you realise how readily polymorphic this is, you may start wanting to use such approaches to data for everything. Rich new opportunities then arise from cross-pollinating component data. Some may question why it is structurally permissible that, say, a network interface can have a birthday, or why an invoice has an IPv6 address, why my cat is in the DHCP pool, whilst limegreen is deleted and $5 on Tuesdays. This of course is half the fun. I don’t accept that it’s wholly incompatible with OO, though, a thesis you’ll see dotted around the place. I’ve even taken this approach with Ruby using Active Record for persistence; not normally a domain where the words “high performance” are bandied about. That worked particularly because Ruby’s object system, being more Smalltalk-ish than C++/Java-ish, strongly favours composition over inheritance.
- m_mueller 3y ago
  > I don’t accept that it’s wholly incompatible with OO I don't think it is, not even in Java. If you use its primitive array and value types, you can easily wrap this in a data wrapper class with functional interfaces, which then can be used quite elegantly in the rest of your code in an OO way - you just don't box all the records into objects.
  - inopinatus 3y ago
    Agreed. Honestly I think the hardest part here is programmer mindset - when you suggest that identity is fluid and records don’t have to correspond to a class hierarchy some folks get a panicked look going on, like you just claimed to eat their pets
    - m_mueller 3y ago
camjw 3y ago
Best SWE book I've ever read and I don't even work on something video game adjacent any more
htk 3y ago
Did anyone here by any chance create an epub out of this? If not, any recommendations on utilities to convert several linked html files into a single epub?
- asimpletune 3y ago
  I think standardebook's has a command line utility that they use for producing their ebooks, which you might be able to use to produce an ebook from a bunch of html files. Ultimately an epub is a zip of html anyways, I think.
  - htk 3y ago
    I'll give this a try, thank you.
oaiey 3y ago
Data Oriented Design is more beginner friendly. Because it does not deal with people and business but only with purity of data modelling. When I was young, my first step in a new project was painting the Entity Relationship Model. That gave me the foundation for everything else. Nowadays, I try to understand the problems and the domain, work on capabilities and how to group/box them before I start doing data models.
- chrisgd 3y ago
  I like the content of your comment. Everyone who has experience recognizes that our love of data and programming often gets sideswiped by business needs. I think though this article tries to say that if we focus on gathering data needs from the beginning, it might make the business needs conversation moot
  - oaiey 3y ago
    I completely agree with that. I have problems that after a DDD phase we have a suitable entity model and afterwards conversations with business and UX are very often very interesting.
BonitaPersona 3y ago
I love this book and have been very influenced by it. However, it should definitely be called: Data-Oriented Design FOR GAME DEVELOPMENT.
- loup-vaillant 3y ago
  Are you suggesting the advice in there isn't applicable in other domains? Or just that it uses games as an example?
  - BonitaPersona 3y ago
    The second, more or less. The advice is certainly applicable in other domains, however the explanations and examples used in the book are heavily focused on software that will be executing repetitive code in a frequency-controlled loop, that loops over arrays of arrays. This also applies to simulation software. There are some key characteristics on this kind of software that may or may not be present in other domains. Once deeply inside game development, these characteristics are like the oxygen you breath, and become intrinsically ingrained in how you think. This is the case with this book and author: I have the impression he thought gamedev-focused examples were general and not specific. As I said, I was really positively influenced by this book, and I tend to go back to it from time to time. Always worth it. Just be aware of what it is focused on.
synergy20 3y ago
How useful and practical for DoD in ML and AI field to do parallel (matrix) computing at scale? where performance is crucial as well, but not as crucial as gaming(milliseconds matter). DoD is a new paradigm for me.
deneas 3y ago
I like Data-Oriented Design, but beware of one thing: You organise your data like a database? You'll eventually be writing a database management system, unless you can use a framework like one of the many Entity-Component-System ones.
marcosdumay 3y ago
The entire advice is context-dependent. Games just happen to have a lot of operations that need column-based access; but that's not true for all domains. When you go and blindly push the game best practices into other domain, you are just making everybody's life hard and most systems worse.
- deleted 3y ago
  [deleted]
- GartzenDeHaes 3y ago
  It's for high-performance computing with current CPU designs that are dependent on data locality for performance. I agree that it's a harmful design for business data. Programmers want to push their runtime data model into the database and they have no interest in the operational, maintenance, and performance problems this causes. When someone suggests this kind of thing, I'll ask them "how do we diagnose performance problems with this technology when there are 100,000 concurrent users and millions of data elements?" The rows-and-columns people can answer this question.
  - feoren 3y ago
    > When someone suggests this kind of thing, I'll ask them "how do we diagnose performance problems with this technology when there are 100,000 concurrent users and millions of data elements?" I don't understand; the exact same performance diagnostics work in both cases. Why is this different? There's nothing intrinsically less performant about this approach. You really think your checkerboard tables and long lists of columns with names like "VALUE12" and "VALUE13" and multiple different kinds of key/value pairs you jammed in there for different clients -- you think those are better performance!? > 100,000 concurrent users Do you actually have 100,000 concurrent users? Really? You don't, do you? You just kinda hope you will eventually. And again: this approach is not worse for that. > millions of data elements This is absolute peanuts for any modern database system. It's weird that this is your extreme example.
- flohofwoe 3y ago
  > Games just happen to have a lot of operations that need column-based access. And that's not even true for many code areas in typical games, only where there's at least a few thousands 'things' to process (e.g. particle systems or navigation/collision systems). DOD makes a lot of sense within specific subsystems, but not necessarily in high level gameplay code (outside specific genres at least).
Acumen321 3y ago
Mike Acton's talk "Data-Oriented Design and C++" from CppCon 2014 is the best programming talk ever given in my opinion. A must watch: https://youtu.be/rX0ItVEVjHc https://youtu.be/rX0ItVEVjHc
- wrapperup 3y ago
  It's fantastic, and also my favorite. And for those who might not know, he was the one who really mainstreamed Data-oriented design and ECS architecture in my eyes. He previously was also leading the charge on Unity DOTS, though unfortunately it seems Unity is having a tailspin at the moment. The work on DOTS is solid, if incomplete.
- jrvarela56 3y ago
  Beat me to it, was about to post this talk. For those reading, check the video out if you want to get a gist how to world-class performance is implemented. Most of my career has been writing web apps and this talk showed me 'why would someone use C?'
lincpa 3y ago
[dead]
debanjan16 3y ago
Even beginners can learn to program in a data oriented way from the beginning. Two books that teach this style of programming to beginners are: 1. How to Design Programs - https://htdp.org/ https://htdp.org/ 2. A Data-Centric Introduction to Computing - https://dcic-world.org/ https://dcic-world.org/
- morelisp 3y ago
  Those are not this.
  - debanjan16 3y ago
    Can you elaborate a bit more on the reason?
    - morelisp 3y ago
      The books you presented are, roughly speaking, introductions to programming with a focus on data science, functional programming, and common structures/ideas used in those which are, in other texts, not usually considered introductory material. "Data" in this sense means, like, collected facts about the world and how to model them. Data-oriented design is a particular way of designing your programs where you focus on efficiently laying out your "data" - in a different sense, meaning "whatever it is I've got in storage" - within that storage - to compute with it as fast as possible. The industry-standard tools used for the first thing are often using techniques developed in the second of the second thing, but that's not relevant for the pedagogical framing. The tools they are teaching (Scheme and Pyret) actually make it very hard to play with low-level data layout details. And the emphasis in these texts on "real [as in, world] data" is in direct contradiction to the DOD axiom that "data is not the problem domain... The data-oriented design approach doesn't build the real-world problem into the code." A rule of thumb: Is anyone talking about GPUs, SIMD, or CPU cache sizes? If not, you're looking at something about data modeling or data science, not data orientation. And this, sorry, is all super fucking obvious if you actually read the intro to all three things.
    - jrvarela56 3y ago
cempaka 3y ago
Andrew Kelley gave an informative and entertaining talk on how DOD had inspired a lot of his work on the Zig compiler: https://vimeo.com/649009599 https://vimeo.com/649009599
- GuestHNUser 3y ago
  Thanks for sharing this! Hands down the most practical talk I have seen about DOD. That made so many things click for me.
dkarl 3y ago
> [Abstraction heavy paradigms] structure the code around the description of the problem domain My experience of Domain-Driven Design has been that it is extremely effective for driving conversations about the domain throughout the product life-cycle, but it produces frustrating codebases that are poorly attuned to the world outside the running process. Domain-Driven Design codebases want to be self-contained universes and treat external systems as details. OO design paradigms in general seem to have little respect for messages exchanged with external systems. This wasn't true back when JavaBeans and object databases other distributed object systems were expected to take over the world, but the failure of those technologies shrank the OO world from distributed systems to isolated programs. These days, the messages exchanged with external systems are just data, without behavior. The marriage of data and behavior can only exist inside a single process. So object-oriented design turned inward and concentrated on the creation of rich inner worlds. I think this is backwards, or at least incomplete. As Rich Hickey says, effective programs need to be situated in the real world. They are not ethereal abstract models. They have concrete functions, inputs and outputs. Having rich internal abstractions that mimic some aspects of reality is a means to an end, entirely subordinate to the purpose of executing interactions with other computing systems. Data-driven design embraces this reality. By treating data as essential, and behavior as something to be added when it is needed, it allows inputs and outputs to be first-class citizens. > The data-oriented design approach doesn't build the real-world problem into the code. This could be seen as a failing of the data-oriented approach by veteran object-oriented developers, as examples of the success of object-oriented design come from being able to bring the human concepts to the machine I think this perfectly sums up the confusion that OO modeling creates. You use code to write programs, or services, or cloud functions, things like that. The real-world problem that a program, service, or cloud function solves is interacting in a certain way with other programs, services, cloud functions. The real-world problems that OO modeling paradigms want you to focus on are the domain problems. This is vitally important for design products, systems of programs that solve real human problems. If you are designing a system for managing medical records in a hospital, you need to model doctors, nurses, patients, labs, radiology images, patient stays, all those real-world things. However, when you are designing a piece of software to do one thing within that medical records system, the "real-world problem" your code is solving is limited to its role within the larger system. Data-oriented design is a natural mental fit for writing programs or services that play a limited part in larger systems, which is always what you are doing when you are writing code. Object-oriented design wants to take on the complexity of the whole real world, which is the right perspective for product design and architecture, not for writing code.
kaycebasques 3y ago
Pretty great intro paragraph. The eloquent writing and interesting ideas motivate me to keep reading: > Data is all we have. Data is what we need to transform in order to create a user experience. Data is what we load when we open a document. Data is the graphics on the screen, the pulses from the buttons on your gamepad, the cause of your speakers producing waves in the air, the method by which you level up and how the bad guy knew where you were so as to shoot at you. Data is how long the dynamite took to explode and how many rings you dropped when you fell on the spikes. It is the current position and velocity of every particle in the beautiful scene that ended the game which was loaded off the disc and into your life via transformations by machinery driven by decoded instructions themselves ordered by assemblers instructed by compilers fed with source-code.
- crabmusket 3y ago
  I see these ideas in a lot of data-oriented-design literature and it always struck me as needlessly reductive. Maybe it's useful to set the scene and provide a "cold shower" to get you out of an OOP-abstraction state of mind. But aside from that it seems about as useful as an engineer saying "look around! everything is made of atoms! engineering is fundamentally about moving atoms!" Which is not wrong, just not really going to help actually do any engineering.
hyperthesis 3y ago
https://dataorienteddesign.com/dodbook.pdf https://dataorienteddesign.com/dodbook.pdf
esjeon 3y ago
It was a good read, a really good one. However, to my liking, the book is too centered around game engine design and a little bit too practical, but, hey, the author did explore it all the way down to the bottom. I almost envy him at this moment for being able to explore it to the depth. It would be such a great joy to have firsthand experience in topics like this. Personally, I'm more interested in generalized architecture, which can be applied to applications. The priority is certainly NOT performance, but the goal is still not very clear. I just vaguely guess it's possibly a way for better composable software.

389 points by DeathArrow 3y ago

eska 3y ago

Looked at it for 10 seconds and immediately found random 0s at the end of the document or unreplaced text like “Noel Llopis in his September 2009 article[#!NoelDOD!#]”

jerrygenser 3y ago
The intro describes that it's a free version of a book that contains most of the content except for a few sections. The book was converted using an automated format from it's native design into html so a large portion could be made available for free and this automated conversion process might have some minor issues -- like the one you described.

dosshell 3y ago

One key concept when I use DoD is to not abstract away the data. Less is more. But when quickly reading the intro text, I found it doing the opposite, it talks too much and abstracts away the key concepts. Only me who found it a bit ironic of not drinking the wine?

Ciantic 3y ago

Everytime I hear about Data-Driven/Oriented Design I remember a paper from OOP course I had to read in University, it used Data-Driven Design as an example how to not to do things. The paper in question is this from 1989: https://dl.acm.org/doi/pdf/10.1145/74877.74885 https://dl.acm.org/doi/pdf/10.1145/74877.74885 It highlights that: "Even though the goal of data-driven design is to encapsulate data and algorithms, it inherently violates that encapsulation by making the structure of an object part of the definition of the object. This in turn leads to the definition of operations that reflect that structure (because they were designed with the structure in mind). Attempts to change the structure of an object transparently are destined to fail because other classes rely on that structure. This is the antithesis of encapsulation." Then goes to show that Responsibility-Driven Design has better approach. What we mean with Data-Driven Design have come a long way though, and isn't comparable to those days. I find it a bit amusing that Data-Driven Design used to be an insult of sorts, like you didn't know how to do things OOP the right way.

dosshell 3y ago
You do realize that DoD has nothing to do with data-driven, right? Data oriented design is about structure the data in regard of how it is processed and the hardware. This is opposite of object oriented design which models the data around your mental model. For example in DoD you could design a map with keys in one array together and one array of values. This would make it much faster to iterate over keys while searching because of cache memory. While in object oriented you would store an array of pairs. Both approaches can use data-driven. Edit: would -> could
- morelisp 3y ago
  There are also multiple definitions of "data-driven" floating around depending on qualifier/context. An OLAP RDBMS, for example, will certainly be written in a data-oriented way, and also have almost fully data-driven behavior, but will certainly not have data-driven (in the sense of that paper) design.
- drainyard 3y ago
  While I agree with your point, the map example is not entirely well represented in your comment. The idea is not to just store keys and values in separate arrays, the idea is to look at your use case and model your data after the transformation you need. So if you have a case where storing keys and values as pairs because of your access pattern, then do that, if you have a case where you do a lot of searches through keys, then store them in separate arrays. The point of DoD is to look at the data you have and the data you need it transformed into and then structure your data after.
dgb23 3y ago
The example given as "data driven design" in this paper neither resembles modern data driven programming nor data oriented design. I don't quite understand why the authors thought of this as a good example of "data driven design". I know that functions in a data driven program tend to be very generic so the conclusion you cite is very much a mismatch from my experience.

gjadi 3y ago

Found an online review of the book: https://gist.github.com/seece/25ed1b2108cf5782718b026382f2c52c https://gist.github.com/seece/25ed1b2108cf5782718b026382f2c5...

victor106 3y ago
Thanks for the link. Found this interesting and against common advice. "The bane of many projects, and the cause of their lateness, has been the insistence on not doing optimisation prematurely. The reason optimisation at late stages is so difficult is that many pieces of software are built up with instances of objects everywhere, even when not needed." There are definitely applications where performance is of primary concern (maybe only a few) and others where they are not. In apps where it is, this gives me thought that maybe premature optimization is okay? Am I reading that right? There's also this called Data-Oriented Programming https://www.manning.com/books/data-oriented-programming https://www.manning.com/books/data-oriented-programming. Are both these concepts the same?
- crabmusket 3y ago
  No, DOP is basically just functional programming. There are some overlaps (e.g. separate code from data) but they're not related.

bob1029 3y ago

> Is your data layout defined by a single interpretation from a single point of view? I think this might be the most important question at technology selection and architecture time. Answering it usually requires talking to the business and customers. If you are certain there is exactly 1 valid "view" of the data that will be used throughout, then perhaps enshrining it in code makes sense. If you are even a tiny bit uncertain of this, a relational-style model probably works better. SQL is the end game for most businesses once they realize the game theory around this one... I am curious what HN thinks as major reasons for why everyone seems to have moved away from 1 big SQL database. From my perspective, yeah we have "web scale" edge cases that threaten vertical scalability on writes, but most businesses will never touch this, including members of the F100.

jwestbury 3y ago
At a previous F100 company -- a tech company whose products are widely used, we'll say -- we received guidance that RDBMS was verboten except with explicit approval. This had nothing to do with the best ways to model a given dataset, or achieving the best performance, and everything to do with schema flexibility and a history of outages caused by fucking up schema migrations. These problems weren't occurring in our NoSQL designs, and whatever benefits SQL databases offered didn't counter the huge benefits we gained from NoSQL's lack of rigid schema. Of course, bad uses of key-value stores can have massive performance impacts, and huge monetary costs when leveraging cloud platforms like DynamoDB -- I've seen a lot of cases where people didn't properly structure their data for DDB, and ended up performing loads of scans and sending costs through the roof.
- bob1029 3y ago
  > schema flexibility If the business is of the notion that the schema is "flexible", then it is probably time to bring all of the MBAs into a conference room and have a come-to-jesus conversation about the limitations of information theory and human suffering. At a certain point, when someone says "Widget", everyone in the organization needs to be on the same page. This goes well beyond any specific technology.
- 3y ago

inopinatus 3y ago

Some of the best advice I ever got for writing composable, high-performance code was “work on structs of arrays, not arrays of structs”. I hear many echoes of that advice in this text. Turns out that entity-component architectures work well in line-of-business applications too, not just games. Alas, many developers in enterprise are rusted onto a record-keeping CRUD model and struggle to think in columns rather than rows. The idea of inserting an entity id into a “published” table, instead of setting a boolean “published” field to true, doesn’t always come naturally. Yet once you realise how readily polymorphic this is, you may start wanting to use such approaches to data for everything. Rich new opportunities then arise from cross-pollinating component data. Some may question why it is structurally permissible that, say, a network interface can have a birthday, or why an invoice has an IPv6 address, why my cat is in the DHCP pool, whilst limegreen is deleted and $5 on Tuesdays. This of course is half the fun. I don’t accept that it’s wholly incompatible with OO, though, a thesis you’ll see dotted around the place. I’ve even taken this approach with Ruby using Active Record for persistence; not normally a domain where the words “high performance” are bandied about. That worked particularly because Ruby’s object system, being more Smalltalk-ish than C++/Java-ish, strongly favours composition over inheritance.

m_mueller 3y ago
> I don’t accept that it’s wholly incompatible with OO I don't think it is, not even in Java. If you use its primitive array and value types, you can easily wrap this in a data wrapper class with functional interfaces, which then can be used quite elegantly in the rest of your code in an OO way - you just don't box all the records into objects.
- inopinatus 3y ago
  Agreed. Honestly I think the hardest part here is programmer mindset - when you suggest that identity is fluid and records don’t have to correspond to a class hierarchy some folks get a panicked look going on, like you just claimed to eat their pets
  - m_mueller 3y ago

camjw 3y ago

Best SWE book I've ever read and I don't even work on something video game adjacent any more

htk 3y ago

Did anyone here by any chance create an epub out of this? If not, any recommendations on utilities to convert several linked html files into a single epub?

asimpletune 3y ago
I think standardebook's has a command line utility that they use for producing their ebooks, which you might be able to use to produce an ebook from a bunch of html files. Ultimately an epub is a zip of html anyways, I think.
- htk 3y ago
  I'll give this a try, thank you.

oaiey 3y ago

Data Oriented Design is more beginner friendly. Because it does not deal with people and business but only with purity of data modelling. When I was young, my first step in a new project was painting the Entity Relationship Model. That gave me the foundation for everything else. Nowadays, I try to understand the problems and the domain, work on capabilities and how to group/box them before I start doing data models.

chrisgd 3y ago
I like the content of your comment. Everyone who has experience recognizes that our love of data and programming often gets sideswiped by business needs. I think though this article tries to say that if we focus on gathering data needs from the beginning, it might make the business needs conversation moot
- oaiey 3y ago
  I completely agree with that. I have problems that after a DDD phase we have a suitable entity model and afterwards conversations with business and UX are very often very interesting.

BonitaPersona 3y ago

I love this book and have been very influenced by it. However, it should definitely be called: Data-Oriented Design FOR GAME DEVELOPMENT.

loup-vaillant 3y ago
Are you suggesting the advice in there isn't applicable in other domains? Or just that it uses games as an example?
- BonitaPersona 3y ago
  The second, more or less. The advice is certainly applicable in other domains, however the explanations and examples used in the book are heavily focused on software that will be executing repetitive code in a frequency-controlled loop, that loops over arrays of arrays. This also applies to simulation software. There are some key characteristics on this kind of software that may or may not be present in other domains. Once deeply inside game development, these characteristics are like the oxygen you breath, and become intrinsically ingrained in how you think. This is the case with this book and author: I have the impression he thought gamedev-focused examples were general and not specific. As I said, I was really positively influenced by this book, and I tend to go back to it from time to time. Always worth it. Just be aware of what it is focused on.

synergy20 3y ago

How useful and practical for DoD in ML and AI field to do parallel (matrix) computing at scale? where performance is crucial as well, but not as crucial as gaming(milliseconds matter). DoD is a new paradigm for me.

deneas 3y ago

I like Data-Oriented Design, but beware of one thing: You organise your data like a database? You'll eventually be writing a database management system, unless you can use a framework like one of the many Entity-Component-System ones.

marcosdumay 3y ago

The entire advice is context-dependent. Games just happen to have a lot of operations that need column-based access; but that's not true for all domains. When you go and blindly push the game best practices into other domain, you are just making everybody's life hard and most systems worse.

deleted 3y ago
[deleted]
GartzenDeHaes 3y ago
It's for high-performance computing with current CPU designs that are dependent on data locality for performance. I agree that it's a harmful design for business data. Programmers want to push their runtime data model into the database and they have no interest in the operational, maintenance, and performance problems this causes. When someone suggests this kind of thing, I'll ask them "how do we diagnose performance problems with this technology when there are 100,000 concurrent users and millions of data elements?" The rows-and-columns people can answer this question.
- feoren 3y ago
  > When someone suggests this kind of thing, I'll ask them "how do we diagnose performance problems with this technology when there are 100,000 concurrent users and millions of data elements?" I don't understand; the exact same performance diagnostics work in both cases. Why is this different? There's nothing intrinsically less performant about this approach. You really think your checkerboard tables and long lists of columns with names like "VALUE12" and "VALUE13" and multiple different kinds of key/value pairs you jammed in there for different clients -- you think those are better performance!? > 100,000 concurrent users Do you actually have 100,000 concurrent users? Really? You don't, do you? You just kinda hope you will eventually. And again: this approach is not worse for that. > millions of data elements This is absolute peanuts for any modern database system. It's weird that this is your extreme example.
flohofwoe 3y ago
> Games just happen to have a lot of operations that need column-based access. And that's not even true for many code areas in typical games, only where there's at least a few thousands 'things' to process (e.g. particle systems or navigation/collision systems). DOD makes a lot of sense within specific subsystems, but not necessarily in high level gameplay code (outside specific genres at least).

Acumen321 3y ago

Mike Acton's talk "Data-Oriented Design and C++" from CppCon 2014 is the best programming talk ever given in my opinion. A must watch: https://youtu.be/rX0ItVEVjHc https://youtu.be/rX0ItVEVjHc

wrapperup 3y ago
It's fantastic, and also my favorite. And for those who might not know, he was the one who really mainstreamed Data-oriented design and ECS architecture in my eyes. He previously was also leading the charge on Unity DOTS, though unfortunately it seems Unity is having a tailspin at the moment. The work on DOTS is solid, if incomplete.
jrvarela56 3y ago
Beat me to it, was about to post this talk. For those reading, check the video out if you want to get a gist how to world-class performance is implemented. Most of my career has been writing web apps and this talk showed me 'why would someone use C?'

lincpa 3y ago

[dead]

debanjan16 3y ago

Even beginners can learn to program in a data oriented way from the beginning. Two books that teach this style of programming to beginners are: 1. How to Design Programs - https://htdp.org/ https://htdp.org/ 2. A Data-Centric Introduction to Computing - https://dcic-world.org/ https://dcic-world.org/

morelisp 3y ago
Those are not this.
- debanjan16 3y ago
  Can you elaborate a bit more on the reason?
  - morelisp 3y ago
    The books you presented are, roughly speaking, introductions to programming with a focus on data science, functional programming, and common structures/ideas used in those which are, in other texts, not usually considered introductory material. "Data" in this sense means, like, collected facts about the world and how to model them. Data-oriented design is a particular way of designing your programs where you focus on efficiently laying out your "data" - in a different sense, meaning "whatever it is I've got in storage" - within that storage - to compute with it as fast as possible. The industry-standard tools used for the first thing are often using techniques developed in the second of the second thing, but that's not relevant for the pedagogical framing. The tools they are teaching (Scheme and Pyret) actually make it very hard to play with low-level data layout details. And the emphasis in these texts on "real [as in, world] data" is in direct contradiction to the DOD axiom that "data is not the problem domain... The data-oriented design approach doesn't build the real-world problem into the code." A rule of thumb: Is anyone talking about GPUs, SIMD, or CPU cache sizes? If not, you're looking at something about data modeling or data science, not data orientation. And this, sorry, is all super fucking obvious if you actually read the intro to all three things.
  - jrvarela56 3y ago

cempaka 3y ago

Andrew Kelley gave an informative and entertaining talk on how DOD had inspired a lot of his work on the Zig compiler: https://vimeo.com/649009599 https://vimeo.com/649009599

GuestHNUser 3y ago
Thanks for sharing this! Hands down the most practical talk I have seen about DOD. That made so many things click for me.

dkarl 3y ago

> [Abstraction heavy paradigms] structure the code around the description of the problem domain My experience of Domain-Driven Design has been that it is extremely effective for driving conversations about the domain throughout the product life-cycle, but it produces frustrating codebases that are poorly attuned to the world outside the running process. Domain-Driven Design codebases want to be self-contained universes and treat external systems as details. OO design paradigms in general seem to have little respect for messages exchanged with external systems. This wasn't true back when JavaBeans and object databases other distributed object systems were expected to take over the world, but the failure of those technologies shrank the OO world from distributed systems to isolated programs. These days, the messages exchanged with external systems are just data, without behavior. The marriage of data and behavior can only exist inside a single process. So object-oriented design turned inward and concentrated on the creation of rich inner worlds. I think this is backwards, or at least incomplete. As Rich Hickey says, effective programs need to be situated in the real world. They are not ethereal abstract models. They have concrete functions, inputs and outputs. Having rich internal abstractions that mimic some aspects of reality is a means to an end, entirely subordinate to the purpose of executing interactions with other computing systems. Data-driven design embraces this reality. By treating data as essential, and behavior as something to be added when it is needed, it allows inputs and outputs to be first-class citizens. > The data-oriented design approach doesn't build the real-world problem into the code. This could be seen as a failing of the data-oriented approach by veteran object-oriented developers, as examples of the success of object-oriented design come from being able to bring the human concepts to the machine I think this perfectly sums up the confusion that OO modeling creates. You use code to write programs, or services, or cloud functions, things like that. The real-world problem that a program, service, or cloud function solves is interacting in a certain way with other programs, services, cloud functions. The real-world problems that OO modeling paradigms want you to focus on are the domain problems. This is vitally important for design products, systems of programs that solve real human problems. If you are designing a system for managing medical records in a hospital, you need to model doctors, nurses, patients, labs, radiology images, patient stays, all those real-world things. However, when you are designing a piece of software to do one thing within that medical records system, the "real-world problem" your code is solving is limited to its role within the larger system. Data-oriented design is a natural mental fit for writing programs or services that play a limited part in larger systems, which is always what you are doing when you are writing code. Object-oriented design wants to take on the complexity of the whole real world, which is the right perspective for product design and architecture, not for writing code.

kaycebasques 3y ago

Pretty great intro paragraph. The eloquent writing and interesting ideas motivate me to keep reading: > Data is all we have. Data is what we need to transform in order to create a user experience. Data is what we load when we open a document. Data is the graphics on the screen, the pulses from the buttons on your gamepad, the cause of your speakers producing waves in the air, the method by which you level up and how the bad guy knew where you were so as to shoot at you. Data is how long the dynamite took to explode and how many rings you dropped when you fell on the spikes. It is the current position and velocity of every particle in the beautiful scene that ended the game which was loaded off the disc and into your life via transformations by machinery driven by decoded instructions themselves ordered by assemblers instructed by compilers fed with source-code.

crabmusket 3y ago
I see these ideas in a lot of data-oriented-design literature and it always struck me as needlessly reductive. Maybe it's useful to set the scene and provide a "cold shower" to get you out of an OOP-abstraction state of mind. But aside from that it seems about as useful as an engineer saying "look around! everything is made of atoms! engineering is fundamentally about moving atoms!" Which is not wrong, just not really going to help actually do any engineering.

hyperthesis 3y ago

https://dataorienteddesign.com/dodbook.pdf https://dataorienteddesign.com/dodbook.pdf

esjeon 3y ago

It was a good read, a really good one. However, to my liking, the book is too centered around game engine design and a little bit too practical, but, hey, the author did explore it all the way down to the bottom. I almost envy him at this moment for being able to explore it to the depth. It would be such a great joy to have firsthand experience in topics like this. Personally, I'm more interested in generalized architecture, which can be applied to applications. The priority is certainly NOT performance, but the goal is still not very clear. I just vaguely guess it's possibly a way for better composable software.