4 ms·

> Every non-trivial production system is a distributed system. In the HN bubble, yes. From the perspective of the average business, definitely not. Or at least

> Every non-trivial production system is a distributed system. In the HN bubble, yes. From the perspective of the average business, definitely not. Or at least, doesn't have to be.

eatonphil 3y ago
Not to get reductionist but if you interact with more than one (e.g. Linux) process, it's already a distributed system. > A distributed system is a system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another. https://en.wikipedia.org/wiki/Distributed_computing https://en.wikipedia.org/wiki/Distributed_computing
- nrr 3y ago
  I'm unsure whether it's worth splitting hairs over the definition of what a distributed system is. If you have to linearize concurrent access to shared resources, that's where the bulk of your pain will come from. In the distributed system case, at least as far as that term is commonly understood by technologists, that linearization also has to take into account odd behaviors in interconnects that you don't typically see within a single machine.
  - vlovich123 3y ago
    You wouldn’t classify storage as an odd behaving interconnect?
    - nrr 3y ago
      No. I classify storage (read: DASDs, not main memory) as universally cursed.
      - vlovich123 3y ago
        Nicely said. Personally I view it all as a sliding scale with varying techniques coming into play depending on the parameters. Eg networked computers have physical separation and typically independent operational failure domains which is a + (a failing component doesn’t take out the system) and - (requires complex algorithms like distributed concensus and has unique performance issues due to the latencies involved between components). But fundamentally even a single computer can still be viewed as a distributed system and I don’t see that as splitting hairs. Your memory controller, storage controller have non trivial pieces of software running with their own failure modes. The kernel running on the CPU is another component. Sure you don’t need distributed consensus per se but for example some RAID configurations look an awful lot like what you’d do in a distributed storage system (erasure codes, monitoring the state of individual disks to know if recovery is needed when a component fails etc). There’s a lot of work to hide that distributed system complexity behind APIs and standards but that’s also true for web services (eg S3 exposes a decentish distributed storage interface and you don’t have to reason about the consensus problems it’s managing for you under the hood). For example, take the Pixel Buds 2. That has 5 microprocessors communicating with each other and links can go up and down between them. There’s no consensus done and we did a lot to hide the messaging complexity but I would still count that as a distributed system (even more so once you factor in that you also need a host machine to pair with). And all the distributed systems problems came about (eg if you think a connection is there but then it disappears and you missed that signal and didn’t update your state).
        nrr 3y ago
        I don't disagree, but for the sake of having productive conversations, it's easiest to acknowledge that this is a matter of one's frame of reference. Are you technically correct? Yes. How much does that matter at the end of the day? For most people, myself included, I'd argue not a lot. As someone who straddles the system call boundary, I generally consider myself as not running in a distributed system if the layers of the stack below me will fall out from underneath me when some invariant of their abstraction around the distributed nature of my execution context fails to hold. For PCIe-interconnected devices (except for HBAs and disk because that entire stack outright lies to you), the chances are pretty good that the kernel will bugcheck if it can't successfully remediate the problem. For something connected over Bluetooth? Not so.
- mekoka 3y ago
  > Not to get reductionist but [...] Then let's just avoid the pedantry altogether. When we're quizzed on DS, it's understood that we're not talking about simple IPC. It's not about your garden variety backend application and its dedicated postgres instance. We're talking Kafka, and all the accompanying jazz. Is it necessary now to pretend that we don't know what we're talking about?
  - eatonphil 3y ago
    No, I think multiple processes talking to each other even on a single machine is sufficiently complex to qualify as a distributed system. The same mistakes or broad distsys design decisions can happen there or with Kafka. Even if it's architecturally simpler. Specifically, take a look at linearizability. It is a measurement of consistency in the CAP theorem. It's a property quantifiable in almost any system: > A concurrent system consists of a collection of processes communicating through shared data structures or objects. Linearizability is important in these concurrent systems where objects may be accessed by multiple processes at the same time and a programmer needs to be able to reason about the expected results. https://en.m.wikipedia.org/wiki/Linearizability https://en.m.wikipedia.org/wiki/Linearizability
golergka 3y ago
Even the P and M of the classic LAMP stack already form a distributed system.
- kag0 3y ago
  I agree, but in the context of this conversation I think we're just talking about databases/state. So "is M a distributed system on its own?"

> Every non-trivial production system is a distributed system. In the HN bubble, yes. From the perspective of the average business, definitely not. Or at least, doesn't have to be.

eatonphil 3y ago

Not to get reductionist but if you interact with more than one (e.g. Linux) process, it's already a distributed system. > A distributed system is a system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another. https://en.wikipedia.org/wiki/Distributed_computing https://en.wikipedia.org/wiki/Distributed_computing

nrr 3y ago
I'm unsure whether it's worth splitting hairs over the definition of what a distributed system is. If you have to linearize concurrent access to shared resources, that's where the bulk of your pain will come from. In the distributed system case, at least as far as that term is commonly understood by technologists, that linearization also has to take into account odd behaviors in interconnects that you don't typically see within a single machine.
- vlovich123 3y ago
  You wouldn’t classify storage as an odd behaving interconnect?
  - nrr 3y ago
    No. I classify storage (read: DASDs, not main memory) as universally cursed.
    - vlovich123 3y ago
      Nicely said. Personally I view it all as a sliding scale with varying techniques coming into play depending on the parameters. Eg networked computers have physical separation and typically independent operational failure domains which is a + (a failing component doesn’t take out the system) and - (requires complex algorithms like distributed concensus and has unique performance issues due to the latencies involved between components). But fundamentally even a single computer can still be viewed as a distributed system and I don’t see that as splitting hairs. Your memory controller, storage controller have non trivial pieces of software running with their own failure modes. The kernel running on the CPU is another component. Sure you don’t need distributed consensus per se but for example some RAID configurations look an awful lot like what you’d do in a distributed storage system (erasure codes, monitoring the state of individual disks to know if recovery is needed when a component fails etc). There’s a lot of work to hide that distributed system complexity behind APIs and standards but that’s also true for web services (eg S3 exposes a decentish distributed storage interface and you don’t have to reason about the consensus problems it’s managing for you under the hood). For example, take the Pixel Buds 2. That has 5 microprocessors communicating with each other and links can go up and down between them. There’s no consensus done and we did a lot to hide the messaging complexity but I would still count that as a distributed system (even more so once you factor in that you also need a host machine to pair with). And all the distributed systems problems came about (eg if you think a connection is there but then it disappears and you missed that signal and didn’t update your state).
      - nrr 3y ago
        I don't disagree, but for the sake of having productive conversations, it's easiest to acknowledge that this is a matter of one's frame of reference. Are you technically correct? Yes. How much does that matter at the end of the day? For most people, myself included, I'd argue not a lot. As someone who straddles the system call boundary, I generally consider myself as not running in a distributed system if the layers of the stack below me will fall out from underneath me when some invariant of their abstraction around the distributed nature of my execution context fails to hold. For PCIe-interconnected devices (except for HBAs and disk because that entire stack outright lies to you), the chances are pretty good that the kernel will bugcheck if it can't successfully remediate the problem. For something connected over Bluetooth? Not so.
mekoka 3y ago
> Not to get reductionist but [...] Then let's just avoid the pedantry altogether. When we're quizzed on DS, it's understood that we're not talking about simple IPC. It's not about your garden variety backend application and its dedicated postgres instance. We're talking Kafka, and all the accompanying jazz. Is it necessary now to pretend that we don't know what we're talking about?
- eatonphil 3y ago
  No, I think multiple processes talking to each other even on a single machine is sufficiently complex to qualify as a distributed system. The same mistakes or broad distsys design decisions can happen there or with Kafka. Even if it's architecturally simpler. Specifically, take a look at linearizability. It is a measurement of consistency in the CAP theorem. It's a property quantifiable in almost any system: > A concurrent system consists of a collection of processes communicating through shared data structures or objects. Linearizability is important in these concurrent systems where objects may be accessed by multiple processes at the same time and a programmer needs to be able to reason about the expected results. https://en.m.wikipedia.org/wiki/Linearizability https://en.m.wikipedia.org/wiki/Linearizability

golergka 3y ago

Even the P and M of the classic LAMP stack already form a distributed system.

kag0 3y ago
I agree, but in the context of this conversation I think we're just talking about databases/state. So "is M a distributed system on its own?"