6 ms·

I don't think these principles add up to something useful. It's not complete and I think some of the principles don't align that well to the problem space. The

by jmull 6y ago

I don't think these principles add up to something useful. It's not complete and I think some of the principles don't align that well to the problem space. The big one: "Data is immutable". The problem here is that data isn't actually immutable (generally) and mutability isn't actually the problem. The problem is unmanaged references or other dependencies on the mutable data. The "source of truth" becomes muddled which creates the problem of how to keep the various instances in sync (or otherwise handle cases where they are/become out-of-sync). Immutability is a very useful tool since you have have any kind of dependency -- direct, indirect, implied, etc) -- and there's no worry. But you still need a mechanism to manage mutating data. Maybe it comes out somewhere, but the principles of DO don't cover it, which is a rather serious omission. Also, principle 2 isn't really a principle. I think what it's getting at is that you don't really know the precise type of your data, over time, in a distributed system, so it's good to include the flexibility to handle that. That makes sense to me. but generic data structures aren't necessarily always the right way to handle that.

rellekio 6y ago
Your right about the sync and dependency issue. Big reason why in JS land you treat each mutation of the data as immutable destruction and refresh of an object is to eliminate any old references that might not of been cleaned up by the garbage collector. Data Oriented Programmming is both old and new. In that it does not have the same amount of programming patterns that OOP has. As it is a more bare metal means of programming without a ton of abstraction to ease most programmers into it. Where I find the idea interesting is concurrent and parallel processes are more natural in the data oriented. And that is through immutablility and ownership as first principles.
wiremine 6y ago
> Immutability is a very useful tool since you have have any kind of dependency -- direct, indirect, implied, etc) -- and there's no worry. But you still need a mechanism to manage mutating data. I agree. For me it feels like immutable data is a tool, not a universal principle. For example, in low-level C programming or direct control of a register in an embedded context, immutable data isn't a principle, it's just another approach. What I mean by "universal" is something like SOLID. [1] (Most people only use SOLID for OOP, but Uncle Bob makes it clear in Clean Architecture that he thinks SOLID is universal, and not limited to OOP.) That said: it does feel like immutable data should be the _default_ approach in many situations. But that's really hard to do in most languages. [1] https://www.amazon.com/Clean-Architecture-Craftsmans-Software-Structure/dp/0134494164 https://www.amazon.com/Clean-Architecture-Craftsmans-Softwar...
hjntmp 6y ago
Immutability is not the fact that something can not change in this case. It has more to do with the identity of a value. Every time you change anything inside the data structure you get a new reference, that is it!
- hjntmp 6y ago
  Think of it as versioning for your data. So instead of referring to some data structure by its general concept with immutable data structures we are a lot more specific with respect to the identity represented by the name, here the name represents a version. This is useful because, for example, you stop having the "unmanaged references" you were talking about because now since you are pointing to a version of the data and not the data itself you can be sure of what you are talking about.
  - hjntmp 6y ago
    Also remember here the promise is that the data you are talking about wont suddenly change underneath not that the reference 'is up to date". It is not a solution for change in time it is a solution for taking change in time out of the equation when we don't need to talk about that. With immutable ds when we talk about data we are just talking about exactly that and time is taken out of the picture. Its called immutable because now you are talking about facts and not the representation in time of those facts. Because now we are talking in versions so it does not make sense. The thing is this data is a snapshot so you are not guaranteed to have the most up to date snapshot. And that's ok because precisely here we take change in time out of the question in order to be able to talk more precisely about the data. Tracking change in time is another story. So for example you could have things like react. There you have snapshots of the world updating. When you talk about the data it is immutable but then you change it and update the mutable variable where you are keeping change in time.
dnautics 6y ago
I think "data are immutable" should be qualified - usually it means "data are immutable through your codepaths" and if you are mutating data, you need explicit checkouts and checkins of the data. This is the basic principle behind how sane database transactions work. ORMs in some languages can be especially dangerous if they overload the getter/setters of the object in such a way that the checkins and checkouts are obscured; you could be passing your object to a function or method that expects to mutate a polymorphic class[0] that is usually a traditional "shared memory" form of objects, well hopefully you can imagine the chaos, redundant database transactions, consistency problems, failure modes, uncaught exceptions, etc. that are going to be a nightmare to debug. [0] worse yet, imagine if it's someone else's code and they change the api from not mutating to mutating for performance reasons. Will you notice the documentation change or the changelog? It's bad enough in the case when it's not an ORM and just a mutable object.
ajuc 6y ago
You realize data is immutable when you first try to implement history. Mutability is just a hack to save some memory.
- talaketu 6y ago
  > just a hack to save some memory .. and time. If we needed to compute an account balance by summing all the debits and credits since the account was opened...
  - simongray 6y ago
    Persistent data structures already solved this issue.
  - pyrale 6y ago
    You don't need mutability to save on time, you just need to use referential transparency effectively when calculating your projections.
- zmmmmm 6y ago
  I think when you start pulling on that string you eventually end up at event sourcing because once you are keeping comprehensive immutable history of all your changes keeping the mutable record anywhere just starts to look like a liability.
  - pyrale 6y ago
    Yes, and it is incredibly comfortable to work in a properly tooled event-sourced codebase.
phendrenad2 6y ago
This seems to be describing a style of programming, and you'll have to take these principles in the spirit that they were intended, i.e. in the context of that programming style. I recognize this style as a common style in OOP/FP languages such as Scala, where it's common to pass around immutable Plain Old Scala Objects (made from lists and hashmaps).
adamkl 6y ago
Honestly, I think this series of blog posts could have had a more complete title: "Principals of Data Oriented Programming (aka, Idiomatic Clojure)" All of these ideas (and I think they are good ones) are inspired heavily by Rich Hickey's talks and rational behind developing the Clojure language (the author of the post states as much). And while you can use these techniques in other languages/paradigms/problem domains, they are really intended to work well inside the constructs of Clojure, and when applied to "information-driven situated programs" [0] (read business applications with dynamic requirements). As for some of the short-comings you mentioned: "But you still need a mechanism to manage mutating data" Clojure supports this through the use of locking constructs like atoms. [1] "I think what it's getting at is that you don't really know the precise type of your data, over time, in a distributed system, so it's good to include the flexibility to handle that. That makes sense to me. but generic data structures aren't necessarily always the right way to handle that." Clojure attempts to bridge the gap between generic data-structures and strongly-typed constructs using run-time specifications. [2] I mean, the ideas presented here can be generally useful, but your mileage may vary if the principals take you too far out of the idiomatic for your particular language/paradigm/problem domain. If that's the case, you could find yourself wasting energy swimming up stream. [0] - https://www.youtube.com/watch?v=2V1FtfBDsLU https://www.youtube.com/watch?v=2V1FtfBDsLU [1] - https://clojure.org/reference/atoms https://clojure.org/reference/atoms [2] - https://clojure.org/about/spec https://clojure.org/about/spec
- deleted 6y ago
  [deleted]
ummonk 6y ago
I think the principles do basically summarize data oriented programming. Particularly, principle 1 is important since it is the exact opposite of OOP's insistence of encapsulation, and is possibly the biggest reason DOP works so much better than OOP.
- joe_the_user 6y ago
  Evidence that "DOP works so much better than OOP" is scant imo. The rise and fall of paradigms that present themselves as panaceas is instructive. You have "structured programming", "object oriented programming", "functional programming" and now "data oriented programming". What I'd like to see is paradigms paired with "where this works well" rather than paradigms sold based on "this will solve the software crisis", "this works better(unqualified by when)" and "if you're not doing this, you're doing it wrong". The later two claims leave a bad taste in the mouth of the casual users/observers, who gradually morph into active critics and sink the paradigm, wasting the good and useful parts of each (OOP has a vast universe of haters because it's most successful of the panacea-paradigms so far and as a panacea, there's much to hate here but still).
  - valenterry 6y ago
    It's not so much a rise and fall rather than refinement. You still use if/then/else and repetition and subfunctions in OOP. But you restrict and replace some of their previous usage. Same for functional programming: you take OOP but remove all the sideeffects. You still have objects which encapsulate data which can only be accessed in a certain way and can behave polymorphic. Mind that inheritance was never really part of OOP in general. Now about DOP - not sure what exactly it adds on top of FP or if it is even an established and well defined term. Doesn't look to me like it, looks more of a bundle of recommendations right now.
  - vvanders 6y ago
    That's because data oriented programming has already succeed on a large scale, you just might not recognize it. SQL is about as data-oriented as it gets, you have a programming model that's constrained and focused on data layout, structure and performance over being a general purpose language. There's a good article a while back about ECS that I can't find which talks about how most performant ECS start to mirror SQL and other row based data engines.
lacrimacida 6y ago
> I don't think these principles add up to something useful. It's not complete and I think some of the principles don't align that well to the problem space. These are are part of a book that is being written at the moment as far as I can tell: "This article is an excerpt from my upcoming book about Data Oriented Programming. The book will be published by Manning, once it is completed (hopefully in 2021). "
namelosw 6y ago
I was a self-taught programmer and when I first learned to program, I found the concept of 'reference' is very counter-intuitive to me. I assume people have no CS background may find the same - why is this 5 and another 5 is the same, but not people with the same name and social security numbers? Traditionally people use mutable struct or objects for non-primitive data structures, which implicitly allocate addresses in computer memories - that is a very implementation-wised approach, more specifically, a very Von Neumann approach. For example, there are many systems having entities in the databases, like Person { id, name }. But if there's no database, in many programming languages people would write Person { name } only. If people write the in-memory version first, they have to re-write their models. Systems with mutable references inconsistently force the referencing system on people by default, instead of just giving people only they ask for. In modern days, a lot of computing goes beyond a single computer, there are many distributed systems, a lot of serialization is going on, and the reference in one computer's memory doesn't mean anything to another. If we try to loosen the definition of the computer, we can think of many systems as bizarre computers. If we take a look at data processing systems like Spark or Flink, conceptually they also look like computers - bunches of hardware and 'operating systems' sit on top of them, but in the form of clusters instead of single computers. In such cases, there are no shared memories, and the memories are implementation details - users won't aware of them like they have to realize there are underlying memory systems in Von Neumann computers. In such kind of systems, people only care about computing which is more substitution based instead of mutable references. While at most, they only need to aware of nodes as a whole in the cluster. It's close to the 'substitution model' vs the 'environment model', or the 'functional' vs the 'object-oriented' described in the SICP. Where I found conceptually the 'substitution model' is more fundamental, and simpler - it's basically elementary or middle school maths. > The problem here is that data isn't actually immutable (generally) and mutability isn't actually the problem. The problem is unmanaged references or other dependencies on the mutable data. So another novel way to look at it, is if you don't have mutable references, you don't have to manage it, then the problem is eliminated as a whole. Like Haskell, Erlang, or Spark as we just mentioned. At the end of the day, you can have references in outer worlds, be it ST monads, processes, database, or just our real world, sometimes the processing part of the program don't have to care about them.

by jmull 6y ago

rellekio 6y ago

Your right about the sync and dependency issue. Big reason why in JS land you treat each mutation of the data as immutable destruction and refresh of an object is to eliminate any old references that might not of been cleaned up by the garbage collector. Data Oriented Programmming is both old and new. In that it does not have the same amount of programming patterns that OOP has. As it is a more bare metal means of programming without a ton of abstraction to ease most programmers into it. Where I find the idea interesting is concurrent and parallel processes are more natural in the data oriented. And that is through immutablility and ownership as first principles.

wiremine 6y ago

> Immutability is a very useful tool since you have have any kind of dependency -- direct, indirect, implied, etc) -- and there's no worry. But you still need a mechanism to manage mutating data. I agree. For me it feels like immutable data is a tool, not a universal principle. For example, in low-level C programming or direct control of a register in an embedded context, immutable data isn't a principle, it's just another approach. What I mean by "universal" is something like SOLID. [1] (Most people only use SOLID for OOP, but Uncle Bob makes it clear in Clean Architecture that he thinks SOLID is universal, and not limited to OOP.) That said: it does feel like immutable data should be the _default_ approach in many situations. But that's really hard to do in most languages. [1] https://www.amazon.com/Clean-Architecture-Craftsmans-Software-Structure/dp/0134494164 https://www.amazon.com/Clean-Architecture-Craftsmans-Softwar...

hjntmp 6y ago

Immutability is not the fact that something can not change in this case. It has more to do with the identity of a value. Every time you change anything inside the data structure you get a new reference, that is it!

hjntmp 6y ago
Think of it as versioning for your data. So instead of referring to some data structure by its general concept with immutable data structures we are a lot more specific with respect to the identity represented by the name, here the name represents a version. This is useful because, for example, you stop having the "unmanaged references" you were talking about because now since you are pointing to a version of the data and not the data itself you can be sure of what you are talking about.
- hjntmp 6y ago
  Also remember here the promise is that the data you are talking about wont suddenly change underneath not that the reference 'is up to date". It is not a solution for change in time it is a solution for taking change in time out of the equation when we don't need to talk about that. With immutable ds when we talk about data we are just talking about exactly that and time is taken out of the picture. Its called immutable because now you are talking about facts and not the representation in time of those facts. Because now we are talking in versions so it does not make sense. The thing is this data is a snapshot so you are not guaranteed to have the most up to date snapshot. And that's ok because precisely here we take change in time out of the question in order to be able to talk more precisely about the data. Tracking change in time is another story. So for example you could have things like react. There you have snapshots of the world updating. When you talk about the data it is immutable but then you change it and update the mutable variable where you are keeping change in time.

dnautics 6y ago

I think "data are immutable" should be qualified - usually it means "data are immutable through your codepaths" and if you are mutating data, you need explicit checkouts and checkins of the data. This is the basic principle behind how sane database transactions work. ORMs in some languages can be especially dangerous if they overload the getter/setters of the object in such a way that the checkins and checkouts are obscured; you could be passing your object to a function or method that expects to mutate a polymorphic class[0] that is usually a traditional "shared memory" form of objects, well hopefully you can imagine the chaos, redundant database transactions, consistency problems, failure modes, uncaught exceptions, etc. that are going to be a nightmare to debug. [0] worse yet, imagine if it's someone else's code and they change the api from not mutating to mutating for performance reasons. Will you notice the documentation change or the changelog? It's bad enough in the case when it's not an ORM and just a mutable object.

ajuc 6y ago

You realize data is immutable when you first try to implement history. Mutability is just a hack to save some memory.

talaketu 6y ago
> just a hack to save some memory .. and time. If we needed to compute an account balance by summing all the debits and credits since the account was opened...
- simongray 6y ago
  Persistent data structures already solved this issue.
- pyrale 6y ago
  You don't need mutability to save on time, you just need to use referential transparency effectively when calculating your projections.
zmmmmm 6y ago
I think when you start pulling on that string you eventually end up at event sourcing because once you are keeping comprehensive immutable history of all your changes keeping the mutable record anywhere just starts to look like a liability.
- pyrale 6y ago
  Yes, and it is incredibly comfortable to work in a properly tooled event-sourced codebase.

phendrenad2 6y ago

This seems to be describing a style of programming, and you'll have to take these principles in the spirit that they were intended, i.e. in the context of that programming style. I recognize this style as a common style in OOP/FP languages such as Scala, where it's common to pass around immutable Plain Old Scala Objects (made from lists and hashmaps).

adamkl 6y ago

Honestly, I think this series of blog posts could have had a more complete title: "Principals of Data Oriented Programming (aka, Idiomatic Clojure)" All of these ideas (and I think they are good ones) are inspired heavily by Rich Hickey's talks and rational behind developing the Clojure language (the author of the post states as much). And while you can use these techniques in other languages/paradigms/problem domains, they are really intended to work well inside the constructs of Clojure, and when applied to "information-driven situated programs" [0] (read business applications with dynamic requirements). As for some of the short-comings you mentioned: "But you still need a mechanism to manage mutating data" Clojure supports this through the use of locking constructs like atoms. [1] "I think what it's getting at is that you don't really know the precise type of your data, over time, in a distributed system, so it's good to include the flexibility to handle that. That makes sense to me. but generic data structures aren't necessarily always the right way to handle that." Clojure attempts to bridge the gap between generic data-structures and strongly-typed constructs using run-time specifications. [2] I mean, the ideas presented here can be generally useful, but your mileage may vary if the principals take you too far out of the idiomatic for your particular language/paradigm/problem domain. If that's the case, you could find yourself wasting energy swimming up stream. [0] - https://www.youtube.com/watch?v=2V1FtfBDsLU https://www.youtube.com/watch?v=2V1FtfBDsLU [1] - https://clojure.org/reference/atoms https://clojure.org/reference/atoms [2] - https://clojure.org/about/spec https://clojure.org/about/spec

deleted 6y ago
[deleted]

ummonk 6y ago

I think the principles do basically summarize data oriented programming. Particularly, principle 1 is important since it is the exact opposite of OOP's insistence of encapsulation, and is possibly the biggest reason DOP works so much better than OOP.

joe_the_user 6y ago
Evidence that "DOP works so much better than OOP" is scant imo. The rise and fall of paradigms that present themselves as panaceas is instructive. You have "structured programming", "object oriented programming", "functional programming" and now "data oriented programming". What I'd like to see is paradigms paired with "where this works well" rather than paradigms sold based on "this will solve the software crisis", "this works better(unqualified by when)" and "if you're not doing this, you're doing it wrong". The later two claims leave a bad taste in the mouth of the casual users/observers, who gradually morph into active critics and sink the paradigm, wasting the good and useful parts of each (OOP has a vast universe of haters because it's most successful of the panacea-paradigms so far and as a panacea, there's much to hate here but still).
- valenterry 6y ago
  It's not so much a rise and fall rather than refinement. You still use if/then/else and repetition and subfunctions in OOP. But you restrict and replace some of their previous usage. Same for functional programming: you take OOP but remove all the sideeffects. You still have objects which encapsulate data which can only be accessed in a certain way and can behave polymorphic. Mind that inheritance was never really part of OOP in general. Now about DOP - not sure what exactly it adds on top of FP or if it is even an established and well defined term. Doesn't look to me like it, looks more of a bundle of recommendations right now.
- vvanders 6y ago
  That's because data oriented programming has already succeed on a large scale, you just might not recognize it. SQL is about as data-oriented as it gets, you have a programming model that's constrained and focused on data layout, structure and performance over being a general purpose language. There's a good article a while back about ECS that I can't find which talks about how most performant ECS start to mirror SQL and other row based data engines.

lacrimacida 6y ago

> I don't think these principles add up to something useful. It's not complete and I think some of the principles don't align that well to the problem space. These are are part of a book that is being written at the moment as far as I can tell: "This article is an excerpt from my upcoming book about Data Oriented Programming. The book will be published by Manning, once it is completed (hopefully in 2021). "

namelosw 6y ago

I was a self-taught programmer and when I first learned to program, I found the concept of 'reference' is very counter-intuitive to me. I assume people have no CS background may find the same - why is this 5 and another 5 is the same, but not people with the same name and social security numbers? Traditionally people use mutable struct or objects for non-primitive data structures, which implicitly allocate addresses in computer memories - that is a very implementation-wised approach, more specifically, a very Von Neumann approach. For example, there are many systems having entities in the databases, like Person { id, name }. But if there's no database, in many programming languages people would write Person { name } only. If people write the in-memory version first, they have to re-write their models. Systems with mutable references inconsistently force the referencing system on people by default, instead of just giving people only they ask for. In modern days, a lot of computing goes beyond a single computer, there are many distributed systems, a lot of serialization is going on, and the reference in one computer's memory doesn't mean anything to another. If we try to loosen the definition of the computer, we can think of many systems as bizarre computers. If we take a look at data processing systems like Spark or Flink, conceptually they also look like computers - bunches of hardware and 'operating systems' sit on top of them, but in the form of clusters instead of single computers. In such cases, there are no shared memories, and the memories are implementation details - users won't aware of them like they have to realize there are underlying memory systems in Von Neumann computers. In such kind of systems, people only care about computing which is more substitution based instead of mutable references. While at most, they only need to aware of nodes as a whole in the cluster. It's close to the 'substitution model' vs the 'environment model', or the 'functional' vs the 'object-oriented' described in the SICP. Where I found conceptually the 'substitution model' is more fundamental, and simpler - it's basically elementary or middle school maths. > The problem here is that data isn't actually immutable (generally) and mutability isn't actually the problem. The problem is unmanaged references or other dependencies on the mutable data. So another novel way to look at it, is if you don't have mutable references, you don't have to manage it, then the problem is eliminated as a whole. Like Haskell, Erlang, or Spark as we just mentioned. At the end of the day, you can have references in outer worlds, be it ST monads, processes, database, or just our real world, sometimes the processing part of the program don't have to care about them.