4 ms·

> Please avoid using distributed systems when non distributed solutions suffice. I would give the opposite advice. Every non-trivial production system is a di

> Please avoid using distributed systems when non distributed solutions suffice. I would give the opposite advice. Every non-trivial production system is a distributed system. At the very least your database is a replica set and therefore a distributed system. So avoid learning distributed systems at your own peril. Check out https://jepsen.io/ https://jepsen.io/ and https://raft.github.io/ https://raft.github.io/

emerongi 3y ago
> Every non-trivial production system is a distributed system. In the HN bubble, yes. From the perspective of the average business, definitely not. Or at least, doesn't have to be.
- eatonphil 3y ago
  Not to get reductionist but if you interact with more than one (e.g. Linux) process, it's already a distributed system. > A distributed system is a system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another. https://en.wikipedia.org/wiki/Distributed_computing https://en.wikipedia.org/wiki/Distributed_computing
  - nrr 3y ago
    I'm unsure whether it's worth splitting hairs over the definition of what a distributed system is. If you have to linearize concurrent access to shared resources, that's where the bulk of your pain will come from. In the distributed system case, at least as far as that term is commonly understood by technologists, that linearization also has to take into account odd behaviors in interconnects that you don't typically see within a single machine.
    - vlovich123 3y ago
      You wouldn’t classify storage as an odd behaving interconnect?
      - nrr 3y ago
        No. I classify storage (read: DASDs, not main memory) as universally cursed.
        vlovich123 3y ago
        Nicely said. Personally I view it all as a sliding scale with varying techniques coming into play depending on the parameters. Eg networked computers have physical separation and typically independent operational failure domains which is a + (a failing component doesn’t take out the system) and - (requires complex algorithms like distributed concensus and has unique performance issues due to the latencies involved between components). But fundamentally even a single computer can still be viewed as a distributed system and I don’t see that as splitting hairs. Your memory controller, storage controller have non trivial pieces of software running with their own failure modes. The kernel running on the CPU is another component. Sure you don’t need distributed consensus per se but for example some RAID configurations look an awful lot like what you’d do in a distributed storage system (erasure codes, monitoring the state of individual disks to know if recovery is needed when a component fails etc). There’s a lot of work to hide that distributed system complexity behind APIs and standards but that’s also true for web services (eg S3 exposes a decentish distributed storage interface and you don’t have to reason about the consensus problems it’s managing for you under the hood). For example, take the Pixel Buds 2. That has 5 microprocessors communicating with each other and links can go up and down between them. There’s no consensus done and we did a lot to hide the messaging complexity but I would still count that as a distributed system (even more so once you factor in that you also need a host machine to pair with). And all the distributed systems problems came about (eg if you think a connection is there but then it disappears and you missed that signal and didn’t update your state).
        nrr 3y ago
        I don't disagree, but for the sake of having productive conversations, it's easiest to acknowledge that this is a matter of one's frame of reference. Are you technically correct? Yes. How much does that matter at the end of the day? For most people, myself included, I'd argue not a lot. As someone who straddles the system call boundary, I generally consider myself as not running in a distributed system if the layers of the stack below me will fall out from underneath me when some invariant of their abstraction around the distributed nature of my execution context fails to hold. For PCIe-interconnected devices (except for HBAs and disk because that entire stack outright lies to you), the chances are pretty good that the kernel will bugcheck if it can't successfully remediate the problem. For something connected over Bluetooth? Not so.
  - mekoka 3y ago
    > Not to get reductionist but [...] Then let's just avoid the pedantry altogether. When we're quizzed on DS, it's understood that we're not talking about simple IPC. It's not about your garden variety backend application and its dedicated postgres instance. We're talking Kafka, and all the accompanying jazz. Is it necessary now to pretend that we don't know what we're talking about?
    - eatonphil 3y ago
      No, I think multiple processes talking to each other even on a single machine is sufficiently complex to qualify as a distributed system. The same mistakes or broad distsys design decisions can happen there or with Kafka. Even if it's architecturally simpler. Specifically, take a look at linearizability. It is a measurement of consistency in the CAP theorem. It's a property quantifiable in almost any system: > A concurrent system consists of a collection of processes communicating through shared data structures or objects. Linearizability is important in these concurrent systems where objects may be accessed by multiple processes at the same time and a programmer needs to be able to reason about the expected results. https://en.m.wikipedia.org/wiki/Linearizability https://en.m.wikipedia.org/wiki/Linearizability
- golergka 3y ago
  Even the P and M of the classic LAMP stack already form a distributed system.
  - kag0 3y ago
    I agree, but in the context of this conversation I think we're just talking about databases/state. So "is M a distributed system on its own?"
PeterCorless 3y ago
If you are running shard-per-core on more than one core, you're a distributed system.
majormajor 3y ago
Some parts of your system may not be able to avoid network calls or distributed aspects. That doesn't mean introducing them everywhere doesn't cause a lot more complexity than you'd otherwise have.
mekoka 3y ago
> Every non-trivial production system is a distributed system. And now you have to define non-trivial. Just thrown like this it does not invalidate the suggestion to avoid needless complexity, since it's really what it's about. Not what is "technically" distributed. Learning and using distributed systems are different things. Once you learn, can you exert some restraint to only apply what fits? Lately, there seems to be considerable effort deployed to transition simple and perfectly working systems to a more heavily distributed model, as if it's, at worst, zero cost. But then you look at the problem being solved, the scale of the solution, and it's obvious that the single postgres instance speaking to a monolith was just fine. I think that's what the author's advice is about.
ElectricalUnion 3y ago
Distributed/redundant system is not a backup. I would still give the "go with the simple solution" advice. Systems are often unable to correctly save, backup and restore persistent state even for the "trivial simple storage" systems, never mind attempting to restoring state for a distributed storage solution in a disaster recovery situation. Once you have a working backup solution in place, then you can adopt a distributed solution.
rqtwteye 3y ago
It depends on your definition of "trivial". Most systems I have seen so far would have been perfectly fine on a big server.
crazygringo 3y ago
I'd hardly call a replica failover a distributed system. And even a master with read-only replicas is not what people usually mean by "distributed", because writes are not. In practice, distributed usually means that data is sharded. And that's what you absolutely want to avoid unless you truly need it.
- pyrolistical 3y ago
  So then you wouldn’t call raft algorithm a solution to consensus for distributed systems? It is strictly single writer at a time

emerongi 3y ago

> Every non-trivial production system is a distributed system. In the HN bubble, yes. From the perspective of the average business, definitely not. Or at least, doesn't have to be.

eatonphil 3y ago
Not to get reductionist but if you interact with more than one (e.g. Linux) process, it's already a distributed system. > A distributed system is a system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another. https://en.wikipedia.org/wiki/Distributed_computing https://en.wikipedia.org/wiki/Distributed_computing
- nrr 3y ago
  I'm unsure whether it's worth splitting hairs over the definition of what a distributed system is. If you have to linearize concurrent access to shared resources, that's where the bulk of your pain will come from. In the distributed system case, at least as far as that term is commonly understood by technologists, that linearization also has to take into account odd behaviors in interconnects that you don't typically see within a single machine.
  - vlovich123 3y ago
    You wouldn’t classify storage as an odd behaving interconnect?
    - nrr 3y ago
      No. I classify storage (read: DASDs, not main memory) as universally cursed.
      - vlovich123 3y ago
        Nicely said. Personally I view it all as a sliding scale with varying techniques coming into play depending on the parameters. Eg networked computers have physical separation and typically independent operational failure domains which is a + (a failing component doesn’t take out the system) and - (requires complex algorithms like distributed concensus and has unique performance issues due to the latencies involved between components). But fundamentally even a single computer can still be viewed as a distributed system and I don’t see that as splitting hairs. Your memory controller, storage controller have non trivial pieces of software running with their own failure modes. The kernel running on the CPU is another component. Sure you don’t need distributed consensus per se but for example some RAID configurations look an awful lot like what you’d do in a distributed storage system (erasure codes, monitoring the state of individual disks to know if recovery is needed when a component fails etc). There’s a lot of work to hide that distributed system complexity behind APIs and standards but that’s also true for web services (eg S3 exposes a decentish distributed storage interface and you don’t have to reason about the consensus problems it’s managing for you under the hood). For example, take the Pixel Buds 2. That has 5 microprocessors communicating with each other and links can go up and down between them. There’s no consensus done and we did a lot to hide the messaging complexity but I would still count that as a distributed system (even more so once you factor in that you also need a host machine to pair with). And all the distributed systems problems came about (eg if you think a connection is there but then it disappears and you missed that signal and didn’t update your state).
        nrr 3y ago
        I don't disagree, but for the sake of having productive conversations, it's easiest to acknowledge that this is a matter of one's frame of reference. Are you technically correct? Yes. How much does that matter at the end of the day? For most people, myself included, I'd argue not a lot. As someone who straddles the system call boundary, I generally consider myself as not running in a distributed system if the layers of the stack below me will fall out from underneath me when some invariant of their abstraction around the distributed nature of my execution context fails to hold. For PCIe-interconnected devices (except for HBAs and disk because that entire stack outright lies to you), the chances are pretty good that the kernel will bugcheck if it can't successfully remediate the problem. For something connected over Bluetooth? Not so.
- mekoka 3y ago
  > Not to get reductionist but [...] Then let's just avoid the pedantry altogether. When we're quizzed on DS, it's understood that we're not talking about simple IPC. It's not about your garden variety backend application and its dedicated postgres instance. We're talking Kafka, and all the accompanying jazz. Is it necessary now to pretend that we don't know what we're talking about?
  - eatonphil 3y ago
    No, I think multiple processes talking to each other even on a single machine is sufficiently complex to qualify as a distributed system. The same mistakes or broad distsys design decisions can happen there or with Kafka. Even if it's architecturally simpler. Specifically, take a look at linearizability. It is a measurement of consistency in the CAP theorem. It's a property quantifiable in almost any system: > A concurrent system consists of a collection of processes communicating through shared data structures or objects. Linearizability is important in these concurrent systems where objects may be accessed by multiple processes at the same time and a programmer needs to be able to reason about the expected results. https://en.m.wikipedia.org/wiki/Linearizability https://en.m.wikipedia.org/wiki/Linearizability
golergka 3y ago
Even the P and M of the classic LAMP stack already form a distributed system.
- kag0 3y ago
  I agree, but in the context of this conversation I think we're just talking about databases/state. So "is M a distributed system on its own?"

PeterCorless 3y ago

If you are running shard-per-core on more than one core, you're a distributed system.

majormajor 3y ago

Some parts of your system may not be able to avoid network calls or distributed aspects. That doesn't mean introducing them everywhere doesn't cause a lot more complexity than you'd otherwise have.

mekoka 3y ago

> Every non-trivial production system is a distributed system. And now you have to define non-trivial. Just thrown like this it does not invalidate the suggestion to avoid needless complexity, since it's really what it's about. Not what is "technically" distributed. Learning and using distributed systems are different things. Once you learn, can you exert some restraint to only apply what fits? Lately, there seems to be considerable effort deployed to transition simple and perfectly working systems to a more heavily distributed model, as if it's, at worst, zero cost. But then you look at the problem being solved, the scale of the solution, and it's obvious that the single postgres instance speaking to a monolith was just fine. I think that's what the author's advice is about.

ElectricalUnion 3y ago

Distributed/redundant system is not a backup. I would still give the "go with the simple solution" advice. Systems are often unable to correctly save, backup and restore persistent state even for the "trivial simple storage" systems, never mind attempting to restoring state for a distributed storage solution in a disaster recovery situation. Once you have a working backup solution in place, then you can adopt a distributed solution.

rqtwteye 3y ago

It depends on your definition of "trivial". Most systems I have seen so far would have been perfectly fine on a big server.

crazygringo 3y ago

I'd hardly call a replica failover a distributed system. And even a master with read-only replicas is not what people usually mean by "distributed", because writes are not. In practice, distributed usually means that data is sharded. And that's what you absolutely want to avoid unless you truly need it.

pyrolistical 3y ago
So then you wouldn’t call raft algorithm a solution to consensus for distributed systems? It is strictly single writer at a time