9 ms·

Patterns of Distributed Systems (2020)

313 points by sbmthakur 6y ago

edoceo 6y ago
Oh man, patterns of enterprise, it's like 20 years old now! I really like Martin's work. Worth every penny
- mpfundstein 6y ago
  id rather argue that his book was one of the major reasons why we ended up in over engineering hell. i was affected by that as a junior and even medior as well and i caused a lot of harm :-) luckily i learned to step away from it.
  - edoceo 6y ago
    Oh, see, I saw that book after watching dozens of dot-com companies re-implemting common patterns (poorly). Basically under-engineering heck. So, once we had the book we could shift the conversation and use common terminology. The discipline is, of course, knowing which spells to cast and when. And it's super useful to see the scroll of common spells.
    - zerkten 6y ago
      A problem in a lot of cases is that for some people, the discipline is about casting spells as opposed to solving problems. It doesn't take a lot of these people to damage projects, but they exist independent of pattern books. If I was going to complain about engineering hell, it'd be about how shallow we investigate these areas. We stop at the immediate technical events, or inappropriate technology choices, instead of getting to the motivations behind it. This gets into a qualitative area that HN readers tend to want to avoid, but these flaws are with the humans and their circumstances.
  - nivertech 6y ago
    Can you provide some concrete examples? I.e. pattern X was recommended for a scenario Y, but ended up in over-engineering hell for Z.
    - withinboredom 6y ago
      IoC is one that comes to mind. It greatly simplifies testing, but the engineering (before we had libraries for it) was ridiculous to implement.
      - mmcdermott 6y ago
        I think this is a good one. Most recent Java/C# systems would have a full IoC container, but have no dynamically selected components (which is how frameworks like Dagger -https://dagger.dev/ https://dagger.dev/ - can exist). A lot of runtime reflection/calculation gets done for something that can be known at compile time.
      - ivanche 6y ago
        Usable IoC actually can be implemented in just 4-5 classes. Yes, it won't be as powerful as Guice but for some (or a lot?) of applications it's probably enough. [0] http://misko.hevery.com/2010/05/26/do-it-yourself-dependency-injection/ http://misko.hevery.com/2010/05/26/do-it-yourself-dependency... [1] http://blacksheep.parry.org/wp-content/uploads/2010/03/DIY-DI.pdf http://blacksheep.parry.org/wp-content/uploads/2010/03/DIY-D...
        withinboredom 6y ago
        Compared to just creating your own dependencies inside a class and not writing interfaces, IOC adds a great deal of complexity.
        ivanche 6y ago
        You don't have to write interfaces to have dependency injection. You can and should declare all your dependencies as constructor parameters. If you create them inside a class you'll permanently lock the two together, thus making it a testability nightmare.
  - kitd 6y ago
    I'd argue that the over-engineering hell preceded this book, with every tiny J2EE project using a full 3-tier stack, incl (but not limited to) stateful EJBs, session EJBs, countless JSPs, DAO layers, god only know what else. It was this book that actually revealed, in patterns, the bigger picture, where it was all going wrong, and the alternatives. This book and Spring v1.0 go along together in my mind. BTW (and OT), another underrated Fowler book in a similar vein is "Analysis Patterns", another 10k-ft view of common business problems.
  - zwieback 6y ago
    I think I know what you're saying. I remember that time, around the birth of Java. But I also remember that we were trying to solve real problems that aren't such big problems anymore: modularization, dependency management, large codebases with crude editing tools, SW delivery and distribution in the age of diskettes! It turns out that developing large systems in static languages with heavily constrained libraries is difficult. Rapid zero-cost distribution and decentralized storage AND GUIs have really changed the field. Does anyone even call themselves a SW Engineer anymore?
  - mmcdermott 6y ago
    I'm a little torn on this one. On the one hand I see it, because it feels like patterns books made the limitations of C++/Java/C# seem respectable, even desirable. Norvig observed this with his talk on patterns in dynamic languages (p. 10 of https://norvig.com/design-patterns/ https://norvig.com/design-patterns/ summarizes this). On the other hand, I have found his work to be useful in selling approaches to others. There were times where I did a design, then browsed the patterns books to find names for what I was doing to "make it respectable."
    - blandflakes 6y ago
      I've done the opposite, disguising something that is a named pattern as anything but a factory to prevent structure-avoidant zealots from knee-jerk reacting to the word in a PR.
  - GordonS 6y ago
    I was exactly the same in my earlier dev days! I'd learn about a pattern, think it was the greatest thing since sliced bread, and see uses for it everywhere... where it really wasn't suitable at all. I was like a builder with only a hammer - everything looked like a nail! I ended up making a lot of things a lot more complex than they needed to be, or/and a lot less performant than they could be. At some point, lot's of people seemed to come to the same realisation - worshipping design patterns is a bad idea. Around this time I first heard about "cargo culting". These days my motto is "no dogma" (well, except about not being dogmatic ;), and I think it serves me well.
  - amrx431 6y ago
    I never had the courage of say this...
- AtlasBarfed 6y ago
  I'll plant a midpost flag and say, while it shouldn't be revered as a bible of eternal truths, it did document and progress the discussion on many things. I think Fowler does a good job of identifying and classifying things, but that hasn't necessarily made IT / Enterprise all that simpler. What has made "progress" in IT has fundamentally been programmers treating more and more things like code and automating the hell out of everything.
vishnugupta 6y ago
> What does it mean for a system to be distributed? There are two aspects: > 1. They run on multiple servers. The number of servers in a cluster can vary from as few as three servers to a few thousand servers. > 2. They manage data. So these are inherently 'stateful' systems. It's a pity that they don't get to the crux of distributed systems because it's very well defined and described for ~40 years now. Instead they describe the key characteristics of a distributed system in a very hand-wavy manner. The two fundamental ways in which distributed computing differs from single-server/machine computing are. 1. No shared memory. 2. No shared clock. Almost every problem faced in distributed systems could be traced to one of these aspects. Because there's no shared memory it's impossible for any one server to know the global state. And so you need consensus algorithms. And due to lack of shared clock it's impossible to order the events. To overcome this software logical clock has to be overlaid on top of distributed systems. Added to this is the failure modes that are peculiar to distributed systems, be it transient/permanent link failures and transient/permaent server failures. This[1] is a decent description of what I've just described here. I also recommend to read up on some key impossibility results in distributed system. The most famous one being the impossibility of achiving common knowledge. I'm surprised that someone as reputed as Thoughtworks don't describe the topic in more precise terms. [1] https://www.geeksforgeeks.org/limitation-of-distributed-system/ https://www.geeksforgeeks.org/limitation-of-distributed-syst...
- zvrba 6y ago
  > 1. No shared memory. I'd rather say "no reliable message delivery". The only difference between completely reliable messaging and shared memory is performance. > Because there's no shared memory it's impossible for any one server to know the global state. Even _with_ shared memory it's impossible to _know_ the global state. Just after you've loaded some data from memory, it can be immediately changed by another thread.
  - jayd16 6y ago
    I don't agree with this. Reliability and transactional/atomic reads and writes are different things. "Reliable" is also a vague value judgement. "Shared memory" implies data coherency.
davydog187 6y ago
Curious that there is not a single mention of Erlang or RabbitMQ, which follow patterns of distributed systems quite nicely.
- whycombagator 6y ago
  Are you referring to the table which presents categories & then examples of technology that falls under said category? If so, the examples are indeed not in any way exhaustive. But I don’t believe they are intended to be, nor could/should be.
  - davydog187 6y ago
    Sure, but arguably RabbitMQ has a much wider adoption and success story as a message broker. Also showing Akka as an actor system, but not mentioning Erlang
    - whycombagator 6y ago
      Okay...to restate what I already have: I'm honestly not sure why not having RabbitMQ/Erlang or X tech listed as an example of Y is such a big deal - especially when the other examples are fine examples.
- nickserv 6y ago
  The target audience seems to be enterprise developers, running JVM applications and needing an introduction to distributed computing. Very unlikely these sorts will jump on the BEAM-wagon. But completely agree that it should be mentioned, in my view Erlang is probably the most elegant approach to the problem.
beckman466 6y ago
holochain > blockchain

313 points by sbmthakur 6y ago

edoceo 6y ago

Oh man, patterns of enterprise, it's like 20 years old now! I really like Martin's work. Worth every penny

mpfundstein 6y ago
id rather argue that his book was one of the major reasons why we ended up in over engineering hell. i was affected by that as a junior and even medior as well and i caused a lot of harm :-) luckily i learned to step away from it.
- edoceo 6y ago
  Oh, see, I saw that book after watching dozens of dot-com companies re-implemting common patterns (poorly). Basically under-engineering heck. So, once we had the book we could shift the conversation and use common terminology. The discipline is, of course, knowing which spells to cast and when. And it's super useful to see the scroll of common spells.
  - zerkten 6y ago
    A problem in a lot of cases is that for some people, the discipline is about casting spells as opposed to solving problems. It doesn't take a lot of these people to damage projects, but they exist independent of pattern books. If I was going to complain about engineering hell, it'd be about how shallow we investigate these areas. We stop at the immediate technical events, or inappropriate technology choices, instead of getting to the motivations behind it. This gets into a qualitative area that HN readers tend to want to avoid, but these flaws are with the humans and their circumstances.
- nivertech 6y ago
  Can you provide some concrete examples? I.e. pattern X was recommended for a scenario Y, but ended up in over-engineering hell for Z.
  - withinboredom 6y ago
    IoC is one that comes to mind. It greatly simplifies testing, but the engineering (before we had libraries for it) was ridiculous to implement.
    - mmcdermott 6y ago
      I think this is a good one. Most recent Java/C# systems would have a full IoC container, but have no dynamically selected components (which is how frameworks like Dagger -https://dagger.dev/ https://dagger.dev/ - can exist). A lot of runtime reflection/calculation gets done for something that can be known at compile time.
    - ivanche 6y ago
      Usable IoC actually can be implemented in just 4-5 classes. Yes, it won't be as powerful as Guice but for some (or a lot?) of applications it's probably enough. [0] http://misko.hevery.com/2010/05/26/do-it-yourself-dependency-injection/ http://misko.hevery.com/2010/05/26/do-it-yourself-dependency... [1] http://blacksheep.parry.org/wp-content/uploads/2010/03/DIY-DI.pdf http://blacksheep.parry.org/wp-content/uploads/2010/03/DIY-D...
      - withinboredom 6y ago
        Compared to just creating your own dependencies inside a class and not writing interfaces, IOC adds a great deal of complexity.
        ivanche 6y ago
        You don't have to write interfaces to have dependency injection. You can and should declare all your dependencies as constructor parameters. If you create them inside a class you'll permanently lock the two together, thus making it a testability nightmare.
- kitd 6y ago
  I'd argue that the over-engineering hell preceded this book, with every tiny J2EE project using a full 3-tier stack, incl (but not limited to) stateful EJBs, session EJBs, countless JSPs, DAO layers, god only know what else. It was this book that actually revealed, in patterns, the bigger picture, where it was all going wrong, and the alternatives. This book and Spring v1.0 go along together in my mind. BTW (and OT), another underrated Fowler book in a similar vein is "Analysis Patterns", another 10k-ft view of common business problems.
- zwieback 6y ago
  I think I know what you're saying. I remember that time, around the birth of Java. But I also remember that we were trying to solve real problems that aren't such big problems anymore: modularization, dependency management, large codebases with crude editing tools, SW delivery and distribution in the age of diskettes! It turns out that developing large systems in static languages with heavily constrained libraries is difficult. Rapid zero-cost distribution and decentralized storage AND GUIs have really changed the field. Does anyone even call themselves a SW Engineer anymore?
- mmcdermott 6y ago
  I'm a little torn on this one. On the one hand I see it, because it feels like patterns books made the limitations of C++/Java/C# seem respectable, even desirable. Norvig observed this with his talk on patterns in dynamic languages (p. 10 of https://norvig.com/design-patterns/ https://norvig.com/design-patterns/ summarizes this). On the other hand, I have found his work to be useful in selling approaches to others. There were times where I did a design, then browsed the patterns books to find names for what I was doing to "make it respectable."
  - blandflakes 6y ago
    I've done the opposite, disguising something that is a named pattern as anything but a factory to prevent structure-avoidant zealots from knee-jerk reacting to the word in a PR.
- GordonS 6y ago
  I was exactly the same in my earlier dev days! I'd learn about a pattern, think it was the greatest thing since sliced bread, and see uses for it everywhere... where it really wasn't suitable at all. I was like a builder with only a hammer - everything looked like a nail! I ended up making a lot of things a lot more complex than they needed to be, or/and a lot less performant than they could be. At some point, lot's of people seemed to come to the same realisation - worshipping design patterns is a bad idea. Around this time I first heard about "cargo culting". These days my motto is "no dogma" (well, except about not being dogmatic ;), and I think it serves me well.
- amrx431 6y ago
  I never had the courage of say this...
AtlasBarfed 6y ago
I'll plant a midpost flag and say, while it shouldn't be revered as a bible of eternal truths, it did document and progress the discussion on many things. I think Fowler does a good job of identifying and classifying things, but that hasn't necessarily made IT / Enterprise all that simpler. What has made "progress" in IT has fundamentally been programmers treating more and more things like code and automating the hell out of everything.

vishnugupta 6y ago

> What does it mean for a system to be distributed? There are two aspects: > 1. They run on multiple servers. The number of servers in a cluster can vary from as few as three servers to a few thousand servers. > 2. They manage data. So these are inherently 'stateful' systems. It's a pity that they don't get to the crux of distributed systems because it's very well defined and described for ~40 years now. Instead they describe the key characteristics of a distributed system in a very hand-wavy manner. The two fundamental ways in which distributed computing differs from single-server/machine computing are. 1. No shared memory. 2. No shared clock. Almost every problem faced in distributed systems could be traced to one of these aspects. Because there's no shared memory it's impossible for any one server to know the global state. And so you need consensus algorithms. And due to lack of shared clock it's impossible to order the events. To overcome this software logical clock has to be overlaid on top of distributed systems. Added to this is the failure modes that are peculiar to distributed systems, be it transient/permanent link failures and transient/permaent server failures. This[1] is a decent description of what I've just described here. I also recommend to read up on some key impossibility results in distributed system. The most famous one being the impossibility of achiving common knowledge. I'm surprised that someone as reputed as Thoughtworks don't describe the topic in more precise terms. [1] https://www.geeksforgeeks.org/limitation-of-distributed-system/ https://www.geeksforgeeks.org/limitation-of-distributed-syst...

zvrba 6y ago
> 1. No shared memory. I'd rather say "no reliable message delivery". The only difference between completely reliable messaging and shared memory is performance. > Because there's no shared memory it's impossible for any one server to know the global state. Even _with_ shared memory it's impossible to _know_ the global state. Just after you've loaded some data from memory, it can be immediately changed by another thread.
- jayd16 6y ago
  I don't agree with this. Reliability and transactional/atomic reads and writes are different things. "Reliable" is also a vague value judgement. "Shared memory" implies data coherency.

davydog187 6y ago

Curious that there is not a single mention of Erlang or RabbitMQ, which follow patterns of distributed systems quite nicely.

whycombagator 6y ago
Are you referring to the table which presents categories & then examples of technology that falls under said category? If so, the examples are indeed not in any way exhaustive. But I don’t believe they are intended to be, nor could/should be.
- davydog187 6y ago
  Sure, but arguably RabbitMQ has a much wider adoption and success story as a message broker. Also showing Akka as an actor system, but not mentioning Erlang
  - whycombagator 6y ago
    Okay...to restate what I already have: I'm honestly not sure why not having RabbitMQ/Erlang or X tech listed as an example of Y is such a big deal - especially when the other examples are fine examples.
nickserv 6y ago
The target audience seems to be enterprise developers, running JVM applications and needing an introduction to distributed computing. Very unlikely these sorts will jump on the BEAM-wagon. But completely agree that it should be mentioned, in my view Erlang is probably the most elegant approach to the problem.

beckman466 6y ago

holochain > blockchain