9 ms·

> What does it mean for a system to be distributed? There are two aspects: > 1. They run on multiple servers. The number of servers in a cluster can vary from a

> What does it mean for a system to be distributed? There are two aspects: > 1. They run on multiple servers. The number of servers in a cluster can vary from as few as three servers to a few thousand servers. > 2. They manage data. So these are inherently 'stateful' systems. It's a pity that they don't get to the crux of distributed systems because it's very well defined and described for ~40 years now. Instead they describe the key characteristics of a distributed system in a very hand-wavy manner. The two fundamental ways in which distributed computing differs from single-server/machine computing are. 1. No shared memory. 2. No shared clock. Almost every problem faced in distributed systems could be traced to one of these aspects. Because there's no shared memory it's impossible for any one server to know the global state. And so you need consensus algorithms. And due to lack of shared clock it's impossible to order the events. To overcome this software logical clock has to be overlaid on top of distributed systems. Added to this is the failure modes that are peculiar to distributed systems, be it transient/permanent link failures and transient/permaent server failures. This[1] is a decent description of what I've just described here. I also recommend to read up on some key impossibility results in distributed system. The most famous one being the impossibility of achiving common knowledge. I'm surprised that someone as reputed as Thoughtworks don't describe the topic in more precise terms. [1] https://www.geeksforgeeks.org/limitation-of-distributed-system/ https://www.geeksforgeeks.org/limitation-of-distributed-syst...

zvrba 6y ago
> 1. No shared memory. I'd rather say "no reliable message delivery". The only difference between completely reliable messaging and shared memory is performance. > Because there's no shared memory it's impossible for any one server to know the global state. Even _with_ shared memory it's impossible to _know_ the global state. Just after you've loaded some data from memory, it can be immediately changed by another thread.
- jayd16 6y ago
  I don't agree with this. Reliability and transactional/atomic reads and writes are different things. "Reliable" is also a vague value judgement. "Shared memory" implies data coherency.
  - DSingularity 6y ago
    I think the distinction he is trying to raise is that messages can be lost in distributed systems. Building distributed shared memory is possible but expensive (readers must write, writers must broadcast). That is why he is raising that distinction and I think it is a good one to raise.
  - zvrba 6y ago
    > "Reliable" is also a vague value judgement It is not. Reliable: message delivery is guaranteed, i.e., the message _will_ eventually reach its intended destination. > "Shared memory" implies data coherency. It does not, re. CPUs with weak memory models. Yes, you can insert barriers, but what happens underneath is that HW exchanges a bunch of messages (reliably!) between nodes to bring them to a consistent view of the global state.
- AtlasBarfed 6y ago
  The discussion is kind of walking around the CAP triangle at this point.
  - inopinatus 6y ago
    Distributed system, n. Any computing structure that produces CAP-theoretic bickering when discussed. See also: halting problem.
    - rualca 6y ago
      That's pretty funny and clever. Thanks for the chuckle.
- mav3rick 6y ago
  Your second point is moot. Even in a multi threaded single machine program you can load state and have it changed by another thread. That's bad design and not a distributed system characteristic.
  - omginternets 6y ago
    >Even in a multi threaded single machine Multiprocessors are distributed systems. (See my other comment: https://news.ycombinator.com/item?id=26093556 https://news.ycombinator.com/item?id=26093556)
    - mav3rick 6y ago
      2 threads can run on 1 core (without HT) sequentially and this problem can still exist. More commonly if you don't have the right locking.
      - omginternets 6y ago
        Yes, and from this we can conclude that single-core processors are distributed systems too! It's counter-intuitive from an architectural (i.e. teleological) perspective, but it makes perfect sense when you approach it from a suitable level of analysis (intentionality).
        mav3rick 6y ago
        It doesn't because memory is shared, clock is shared. Some of the tougher challenges in DS don't reside in this situation.
        omginternets 6y ago
        >It doesn't because memory is shared, clock is shared. You're arguing your point using the definition that I argue is insufficient. We're running around in circles. >Some of the tougher challenges in DS don't reside in this situation. Yes, and others still do. We can definitely agree on the fact that it's an easier variant of DS.
- joshuak 6y ago
  No reliable message delivery, does not help define distributed systems because it can be broken down into a more fundamental elements that are required to guarantee that reliability. Shared memory works well for me because in my mind shared memory means concurrent processes and memory have a common clock and are connected to the same power supply. This avoids confusion around how one handles some parts running while other parts are not. Since we are attempting to define distributed systems not "concurrent processing", concurrent processing can be used in the definition and memory safety can be assumed. We can say that shared memory represents a known global state between concurrent processes because that problem is solved by the common clock and the tools that concurrent processes use to synchronize.
  - zvrba 6y ago
    You're not coherent. > in my mind shared memory means concurrent processes and memory have a common clock You're aware that modern CPUs clock their cores independently? (E.g., AVX512 throttling.) Hence, no common clock. > and are connected to the same power supply That's irrelevant. Unused cores get powered down all the time. On x86, the OS can execute HLT instruction when there's nothing to do and the CPU gets woken up by an interrupt. > This avoids confusion around how one handles some parts running while other parts are not. What is a "part"? The system's scheduler can suspend a thread for an indefinite amount of time. > memory safety can be assumed What does that even mean? > We can say that shared memory represents a known global state [...] Except that it's false, every single point can be countered with a single modern multicore cpu. On an abstract level, distributed computation is a computation where it's possible that a message does not reach its intended destination within a fixed time bound, for any reason. If you can guarantee message deliver within an arbitrary, but fixed, time bound, you can simulate shared memory. Yes, performance will suck if the time bound is large, but you have an equivalent system. That's how HW works underneath anyway. Tip: reading Lamport's papers is illuminating. You can also read about cache coherence protocols (e.g., MESI or MOESI) and you'll see that the illusion of shared memory is fundamentally implemented by message passing on HW level. The illusion is pretty much solid precisely because HW has time bounds on message delivery.
gcblkjaidfj 6y ago
Martin Fowler is in the certification for box checkers business. 99% of the people that read them work at places where they must "move to X" to justify some department. They will likely implement a simulacrum of X (usually by importing some java library someone wrote as homework), adding all the pitfalls and future problems of X with zero of the benefits of X.
- thewarrior 6y ago
  This is too dismissive. Most of Fowler’s work is written after interviewing lots of real world practitioners.
  - zwieback 6y ago
    Agreed, Fowler, Martin, etc. are often criticized not because of their work but because of their audience, or more specifically their paying customers. Makes little sense to me, I got a lot out of their writing, especially in the early days of OO.
    - deleted 6y ago
      [deleted]
    - disgruntledphd2 6y ago
      Fowler and Beck in particular have been massively useful to me recently. Refactoring and TDD by example are wonderful books, and completely changed my approach to software. I also love Feathers and Working Effectively with Legacy Code, but that might be more of a niche taste ;)
      - Jtsummers 6y ago
        Working Effectively is a bit niche, but unless a team is willing to rewrite from scratch or stumbled on a perfect design, refactoring and dealing with untested code will hit everyone eventually. Hell, test suites may themselves not be sufficiently preserved or remain fully functional after enough time has passed. One of my best time investments was reading that book.
ctvo 6y ago
Agreed. Giving this summary and then going into details would benefit readers more. AWS's BuilderLibrary, while containing excellent content, also gives an overview of distributed systems that sorts of point this out: https://aws.amazon.com/builders-library/challenges-with-distributed-systems https://aws.amazon.com/builders-library/challenges-with-dist...
jstimpfle 6y ago
> 1. No shared memory. Isn't it rather "no synchronized data access"? Remote memory isn't a problem if you can read it in a synchronized fashion (taking locks and so on). And actually "no synchronized information retrieval" is the default even on multithreaded, shared memory systems, which is why they're a lot like distributed systems. You can use mutexes and other synchronization primitives though, to solve some of the problems that just aren't solvable on a computer network, due to much higher latency of synchronization.
- waynesonfire 6y ago
  You can devise all sorts of distributed system architectures. You could for example have a synchronous system system composed of nodes organized in a ring. There is not "one definition" of what a distributed system is. You have to define that. There are some common distributed system architectures that perhaps most of us are familiar with--asynchronous networked system, e.g. no shared memory with point-to-point communication. There are other dichotomies; though I'm not an expert in the field and am unable to succinctly define them. As you add more "stuff" into your distributed system--people talkig about adding a memcached or whatever in other comments, you've introduce a completely different system. Maybe some sort of hybrid. And if you're interested, you can formally reason about its behavior. Regardless, you have to define what you're talking about. It's an interesting question to ask what is the most fundamental component of a distributed system? Could it be multiple processing nodes?
jedberg 6y ago
> No shared memory I'm not sure that is entirely accurate. If you have a memcache cluster and all data is stored in there, you have shared memory. Albeit slow and subject to atomicity problems, it's still shared state. It's also a bad idea to rely on that to run your app, so there is that. But it's possible to have shared memory if you're willing to accept the hit to reliability.
- svieira 6y ago
  Remote memory does not count as shared because of the atomicity problems. The same reason _local_ memory doesn't count as shared the minute you spin up two writer threads with access to the same mutable memory space. (And why Rust is popular for distributed problems that share the same clock) If you replaced Memcache with a single Redis instance where all operations were managed by Lua scripts (e. g. you introduced atomicity to your operations) you wouldn't have a distributed system, just one with slow, sometimes rather faulty memory.
g9yuayon 6y ago
Maybe it's because Fowler's target readers are developers of enterprise software who are not familiar with distributed systems at all, and Fowler's background is not in distributed system either. Therefore, he chose to use colloquial terms.
omginternets 6y ago
>It's a pity that they don't get to the crux of distributed systems because it's very well defined and described for ~40 years now. Really? I'm incidentally in the midst of a lit-review on the subject and it seems quite apparent that no standard definitions have emerged. >The two fundamental ways in which distributed computing differs from single-server/machine computing are. >1. No shared memory. 2. No shared clock. The typical multiprocessor is, in fact, a distributed system under the hood. Most of the time the programmer is unaware of this thanks to cache coherence algorithms, which in turn benefit from a reliable communication layer between individual cores. And yet, we can still observe consistency failures when we operate the chip outside of the parameters for which it guarantees a single-system image (namely: when using something like OS threads). I think the problem is that we're using the wrong kind of definition. Your definitions -- an indeed most definitions encountered in literature, with some exceptions -- appeal to design. They are teleological definitions, and as such they can't define "distributed" in the sense of "distributed programming" or "distributed computation". A more useful kind of definition is intentional [0]. It is constructed at a higher level of analysis that assumes the design serves the purpose of representing the world, among others. Thus, you get a definition like this: Distributed computing is a computational paradigm in which local action taken by processes on the basis of locally-available information has the potential to alter some global state. Returning to multiprocessor initial example, the more useful question is often not whether computation is distributed, but when it makes sense to regard it as such. There are three typical cases in which an engineer is engaged in the practice of distributed computing: 1. He is designing or developing a distributed computing system. 2. The system is operating outside of specified parameters, such that design invariants no longer hold. 3. The system is malfunctioning, which is to say it violates its specification despite operating within specified parameters. The second case is the most relevant to our prototypical multiprocessor. The use of OS threads, for example, can be understood as operating outside of the range of parameters for which the SSI can fulfill its guarantees. It is important to note that the system can still be made to function correctly (contrary to case #3), provided the programmer shoulders the burden of distributed control. It's definitely possible -- and I would argue, correct -- to reframe "no shared memory" and "no shared clock" in intentional terms, but as we've seen with the multiprocessor example, those two conditions alone do not define "distributed system" in general; they are not fundamental properties. I will however grant that they are the most common manifestations of distribution in practice. To summarize: the literature has not -- to my knowledge -- arrived at a good definition ~40 years ago. If I've missed something, please point it out, though. I'd hate to publish something incorrect. :) [0] https://en.wikipedia.org/wiki/Intentional_stance#Dennett's_three_levels https://en.wikipedia.org/wiki/Intentional_stance#Dennett's_t...
throwaway894345 6y ago
How does "ephemeral computing" fit into your notion of distributed systems? Perhaps this is a concern not shared by all distributed systems, but it's a practical reality that we in the cloud space have to deal with pervasively and it drives profound architectural differences.

zvrba 6y ago

> 1. No shared memory. I'd rather say "no reliable message delivery". The only difference between completely reliable messaging and shared memory is performance. > Because there's no shared memory it's impossible for any one server to know the global state. Even _with_ shared memory it's impossible to _know_ the global state. Just after you've loaded some data from memory, it can be immediately changed by another thread.

jayd16 6y ago
I don't agree with this. Reliability and transactional/atomic reads and writes are different things. "Reliable" is also a vague value judgement. "Shared memory" implies data coherency.
- DSingularity 6y ago
  I think the distinction he is trying to raise is that messages can be lost in distributed systems. Building distributed shared memory is possible but expensive (readers must write, writers must broadcast). That is why he is raising that distinction and I think it is a good one to raise.
- zvrba 6y ago
  > "Reliable" is also a vague value judgement It is not. Reliable: message delivery is guaranteed, i.e., the message _will_ eventually reach its intended destination. > "Shared memory" implies data coherency. It does not, re. CPUs with weak memory models. Yes, you can insert barriers, but what happens underneath is that HW exchanges a bunch of messages (reliably!) between nodes to bring them to a consistent view of the global state.
AtlasBarfed 6y ago
The discussion is kind of walking around the CAP triangle at this point.
- inopinatus 6y ago
  Distributed system, n. Any computing structure that produces CAP-theoretic bickering when discussed. See also: halting problem.
  - rualca 6y ago
    That's pretty funny and clever. Thanks for the chuckle.
mav3rick 6y ago
Your second point is moot. Even in a multi threaded single machine program you can load state and have it changed by another thread. That's bad design and not a distributed system characteristic.
- omginternets 6y ago
  >Even in a multi threaded single machine Multiprocessors are distributed systems. (See my other comment: https://news.ycombinator.com/item?id=26093556 https://news.ycombinator.com/item?id=26093556)
  - mav3rick 6y ago
    2 threads can run on 1 core (without HT) sequentially and this problem can still exist. More commonly if you don't have the right locking.
    - omginternets 6y ago
      Yes, and from this we can conclude that single-core processors are distributed systems too! It's counter-intuitive from an architectural (i.e. teleological) perspective, but it makes perfect sense when you approach it from a suitable level of analysis (intentionality).
      - mav3rick 6y ago
        It doesn't because memory is shared, clock is shared. Some of the tougher challenges in DS don't reside in this situation.
        omginternets 6y ago
        >It doesn't because memory is shared, clock is shared. You're arguing your point using the definition that I argue is insufficient. We're running around in circles. >Some of the tougher challenges in DS don't reside in this situation. Yes, and others still do. We can definitely agree on the fact that it's an easier variant of DS.
joshuak 6y ago
No reliable message delivery, does not help define distributed systems because it can be broken down into a more fundamental elements that are required to guarantee that reliability. Shared memory works well for me because in my mind shared memory means concurrent processes and memory have a common clock and are connected to the same power supply. This avoids confusion around how one handles some parts running while other parts are not. Since we are attempting to define distributed systems not "concurrent processing", concurrent processing can be used in the definition and memory safety can be assumed. We can say that shared memory represents a known global state between concurrent processes because that problem is solved by the common clock and the tools that concurrent processes use to synchronize.
- zvrba 6y ago
  You're not coherent. > in my mind shared memory means concurrent processes and memory have a common clock You're aware that modern CPUs clock their cores independently? (E.g., AVX512 throttling.) Hence, no common clock. > and are connected to the same power supply That's irrelevant. Unused cores get powered down all the time. On x86, the OS can execute HLT instruction when there's nothing to do and the CPU gets woken up by an interrupt. > This avoids confusion around how one handles some parts running while other parts are not. What is a "part"? The system's scheduler can suspend a thread for an indefinite amount of time. > memory safety can be assumed What does that even mean? > We can say that shared memory represents a known global state [...] Except that it's false, every single point can be countered with a single modern multicore cpu. On an abstract level, distributed computation is a computation where it's possible that a message does not reach its intended destination within a fixed time bound, for any reason. If you can guarantee message deliver within an arbitrary, but fixed, time bound, you can simulate shared memory. Yes, performance will suck if the time bound is large, but you have an equivalent system. That's how HW works underneath anyway. Tip: reading Lamport's papers is illuminating. You can also read about cache coherence protocols (e.g., MESI or MOESI) and you'll see that the illusion of shared memory is fundamentally implemented by message passing on HW level. The illusion is pretty much solid precisely because HW has time bounds on message delivery.

gcblkjaidfj 6y ago

Martin Fowler is in the certification for box checkers business. 99% of the people that read them work at places where they must "move to X" to justify some department. They will likely implement a simulacrum of X (usually by importing some java library someone wrote as homework), adding all the pitfalls and future problems of X with zero of the benefits of X.

thewarrior 6y ago
This is too dismissive. Most of Fowler’s work is written after interviewing lots of real world practitioners.
- zwieback 6y ago
  Agreed, Fowler, Martin, etc. are often criticized not because of their work but because of their audience, or more specifically their paying customers. Makes little sense to me, I got a lot out of their writing, especially in the early days of OO.
  - deleted 6y ago
    [deleted]
  - disgruntledphd2 6y ago
    Fowler and Beck in particular have been massively useful to me recently. Refactoring and TDD by example are wonderful books, and completely changed my approach to software. I also love Feathers and Working Effectively with Legacy Code, but that might be more of a niche taste ;)
    - Jtsummers 6y ago
      Working Effectively is a bit niche, but unless a team is willing to rewrite from scratch or stumbled on a perfect design, refactoring and dealing with untested code will hit everyone eventually. Hell, test suites may themselves not be sufficiently preserved or remain fully functional after enough time has passed. One of my best time investments was reading that book.

ctvo 6y ago

Agreed. Giving this summary and then going into details would benefit readers more. AWS's BuilderLibrary, while containing excellent content, also gives an overview of distributed systems that sorts of point this out: https://aws.amazon.com/builders-library/challenges-with-distributed-systems https://aws.amazon.com/builders-library/challenges-with-dist...

jstimpfle 6y ago

> 1. No shared memory. Isn't it rather "no synchronized data access"? Remote memory isn't a problem if you can read it in a synchronized fashion (taking locks and so on). And actually "no synchronized information retrieval" is the default even on multithreaded, shared memory systems, which is why they're a lot like distributed systems. You can use mutexes and other synchronization primitives though, to solve some of the problems that just aren't solvable on a computer network, due to much higher latency of synchronization.

waynesonfire 6y ago
You can devise all sorts of distributed system architectures. You could for example have a synchronous system system composed of nodes organized in a ring. There is not "one definition" of what a distributed system is. You have to define that. There are some common distributed system architectures that perhaps most of us are familiar with--asynchronous networked system, e.g. no shared memory with point-to-point communication. There are other dichotomies; though I'm not an expert in the field and am unable to succinctly define them. As you add more "stuff" into your distributed system--people talkig about adding a memcached or whatever in other comments, you've introduce a completely different system. Maybe some sort of hybrid. And if you're interested, you can formally reason about its behavior. Regardless, you have to define what you're talking about. It's an interesting question to ask what is the most fundamental component of a distributed system? Could it be multiple processing nodes?

jedberg 6y ago

> No shared memory I'm not sure that is entirely accurate. If you have a memcache cluster and all data is stored in there, you have shared memory. Albeit slow and subject to atomicity problems, it's still shared state. It's also a bad idea to rely on that to run your app, so there is that. But it's possible to have shared memory if you're willing to accept the hit to reliability.

svieira 6y ago
Remote memory does not count as shared because of the atomicity problems. The same reason _local_ memory doesn't count as shared the minute you spin up two writer threads with access to the same mutable memory space. (And why Rust is popular for distributed problems that share the same clock) If you replaced Memcache with a single Redis instance where all operations were managed by Lua scripts (e. g. you introduced atomicity to your operations) you wouldn't have a distributed system, just one with slow, sometimes rather faulty memory.

g9yuayon 6y ago

Maybe it's because Fowler's target readers are developers of enterprise software who are not familiar with distributed systems at all, and Fowler's background is not in distributed system either. Therefore, he chose to use colloquial terms.

omginternets 6y ago

>It's a pity that they don't get to the crux of distributed systems because it's very well defined and described for ~40 years now. Really? I'm incidentally in the midst of a lit-review on the subject and it seems quite apparent that no standard definitions have emerged. >The two fundamental ways in which distributed computing differs from single-server/machine computing are. >1. No shared memory. 2. No shared clock. The typical multiprocessor is, in fact, a distributed system under the hood. Most of the time the programmer is unaware of this thanks to cache coherence algorithms, which in turn benefit from a reliable communication layer between individual cores. And yet, we can still observe consistency failures when we operate the chip outside of the parameters for which it guarantees a single-system image (namely: when using something like OS threads). I think the problem is that we're using the wrong kind of definition. Your definitions -- an indeed most definitions encountered in literature, with some exceptions -- appeal to design. They are teleological definitions, and as such they can't define "distributed" in the sense of "distributed programming" or "distributed computation". A more useful kind of definition is intentional [0]. It is constructed at a higher level of analysis that assumes the design serves the purpose of representing the world, among others. Thus, you get a definition like this: Distributed computing is a computational paradigm in which local action taken by processes on the basis of locally-available information has the potential to alter some global state. Returning to multiprocessor initial example, the more useful question is often not whether computation is distributed, but when it makes sense to regard it as such. There are three typical cases in which an engineer is engaged in the practice of distributed computing: 1. He is designing or developing a distributed computing system. 2. The system is operating outside of specified parameters, such that design invariants no longer hold. 3. The system is malfunctioning, which is to say it violates its specification despite operating within specified parameters. The second case is the most relevant to our prototypical multiprocessor. The use of OS threads, for example, can be understood as operating outside of the range of parameters for which the SSI can fulfill its guarantees. It is important to note that the system can still be made to function correctly (contrary to case #3), provided the programmer shoulders the burden of distributed control. It's definitely possible -- and I would argue, correct -- to reframe "no shared memory" and "no shared clock" in intentional terms, but as we've seen with the multiprocessor example, those two conditions alone do not define "distributed system" in general; they are not fundamental properties. I will however grant that they are the most common manifestations of distribution in practice. To summarize: the literature has not -- to my knowledge -- arrived at a good definition ~40 years ago. If I've missed something, please point it out, though. I'd hate to publish something incorrect. :) [0] https://en.wikipedia.org/wiki/Intentional_stance#Dennett's_three_levels https://en.wikipedia.org/wiki/Intentional_stance#Dennett's_t...

throwaway894345 6y ago

How does "ephemeral computing" fit into your notion of distributed systems? Perhaps this is a concern not shared by all distributed systems, but it's a practical reality that we in the cloud space have to deal with pervasively and it drives profound architectural differences.