4 ms·

But in a way you are doing distributed systems; that's what the "with replication" part of your architecture is saying. Now you can choose to never ever test th

But in a way you are doing distributed systems; that's what the "with replication" part of your architecture is saying. Now you can choose to never ever test that your replication setup is working, but I think that "distributed systems testing is for big tech" is bad advice. Something that would work pretty well in your setup: take down the Postgres leader, and see if your website is still up. That's already Chaos Engineering.

yakshaving_jgt 5y ago
In a way, yes. But that’s not what most people are referring to when they talk about “distributed systems”, so I think you’ve constructed a straw man.
- eklavya 5y ago
  What do those people mean by "distributed systems"? Are they running a single node server with web app and database on the same node? Is there a load balancer/cache/replica/cluster? If it's 1+ machine, it is distributed. In your opinion what would qualify as a "distributed system"?
  - urthor 5y ago
    Distributed system probably isn't precise enough in of itself. But I was very much thinking of multiple leader distributed systems. Anecdotal. But dealing with clock errors and multi-leader issues are of far greater complexity than caching data. Replicas, load balancers, and all the vagaries of single leader systems are not absurdly difficult technical problems. They're problems that can be solved by a team with a single senior engineer who knows what they're doing.
  - yakshaving_jgt 5y ago
    I'm not taking a position on what would qualify as a distributed system, and neither was that the point of my comment. I believe that when most people talk about "distributed systems", they're talking about some form of microservices architecture, where components of a single system are arbitrarily separated by network requests for nebulous reasons.
urthor 5y ago
Well yes exactly. That's kind of why I brought up the idea of clock errors. Single leader distributed systems are honestly an entirely different paradigm to multi-leader. You have a main node, the main node controls routing and which data goes where. There can only be one main node at one time, hence no latency/clock errors/multi leader fail-over/PAXOS/voting/stale keys/exotic application specific conflict resolution logic. Done, simple. (Secretly not so simple, all of the above problems are actually hugely important to resolve in any monolithic database. However, Postgres sorts it out for you.) Really, what I am calling "distributed systems problems" are the subset with multiple simultaneous, geographically separate leaders.
- dikei 5y ago
  >> There can only be one main node at one time This is how most distributed system solve the consensus problem: the nodes automatically run leader election algorithm so that eventually only one leader remains. >> (Secretly not so simple, all of the above problems are actually hugely important to resolve in any monolithic database. However, Postgres sorts it out for you.) Except that in case of network partition of the primary node, Postgres can't promote a new primary node from the replicas automatically, unlike consensus systems such as ETCD/Zookeeper.
- _vvhw 5y ago
  > There can only be one main node at one time, hence no latency/clock errors/multi leader fail-over/PAXOS/voting/stale keys/exotic application specific conflict resolution logic. This is not accurate. You can certainly have clock errors in "single leader distributed systems", simply by virtue of the fact that the primary rotates throughout the cluster. And, perhaps even more surprisingly, you can certainly have clock errors in "single node non-distributed systems like Postgres", i.e. clock errors are completely orthogonal to distributed consensus protocols (unless you're doing something dangerous like leader leases, which I don't recommend). The problem is simple: what if the system time strobes or gets stuck in the future? How does this affect things like recording financial transactions? What if NTP stops working? Distributed systems aside, it's pretty tricky. I've worked on this problem of clock errors specifically for TigerBeetle [1], a financial accounting database that can process a million transactions a second. In our domain, we can't afford to timestamp transactions with the wrong timestamp, because this could mean that financial transactions (e.g. credit card payments) don't rollback if the other party to the transaction times out, so liquidity could get stuck and tied up, for the duration of the clock error. [1] https://www.tigerbeetle.com/post/three-clocks-are-better-than-one https://www.tigerbeetle.com/post/three-clocks-are-better-tha...

yakshaving_jgt 5y ago

In a way, yes. But that’s not what most people are referring to when they talk about “distributed systems”, so I think you’ve constructed a straw man.

eklavya 5y ago
What do those people mean by "distributed systems"? Are they running a single node server with web app and database on the same node? Is there a load balancer/cache/replica/cluster? If it's 1+ machine, it is distributed. In your opinion what would qualify as a "distributed system"?
- urthor 5y ago
  Distributed system probably isn't precise enough in of itself. But I was very much thinking of multiple leader distributed systems. Anecdotal. But dealing with clock errors and multi-leader issues are of far greater complexity than caching data. Replicas, load balancers, and all the vagaries of single leader systems are not absurdly difficult technical problems. They're problems that can be solved by a team with a single senior engineer who knows what they're doing.
- yakshaving_jgt 5y ago
  I'm not taking a position on what would qualify as a distributed system, and neither was that the point of my comment. I believe that when most people talk about "distributed systems", they're talking about some form of microservices architecture, where components of a single system are arbitrarily separated by network requests for nebulous reasons.

urthor 5y ago

Well yes exactly. That's kind of why I brought up the idea of clock errors. Single leader distributed systems are honestly an entirely different paradigm to multi-leader. You have a main node, the main node controls routing and which data goes where. There can only be one main node at one time, hence no latency/clock errors/multi leader fail-over/PAXOS/voting/stale keys/exotic application specific conflict resolution logic. Done, simple. (Secretly not so simple, all of the above problems are actually hugely important to resolve in any monolithic database. However, Postgres sorts it out for you.) Really, what I am calling "distributed systems problems" are the subset with multiple simultaneous, geographically separate leaders.

dikei 5y ago
>> There can only be one main node at one time This is how most distributed system solve the consensus problem: the nodes automatically run leader election algorithm so that eventually only one leader remains. >> (Secretly not so simple, all of the above problems are actually hugely important to resolve in any monolithic database. However, Postgres sorts it out for you.) Except that in case of network partition of the primary node, Postgres can't promote a new primary node from the replicas automatically, unlike consensus systems such as ETCD/Zookeeper.
_vvhw 5y ago
> There can only be one main node at one time, hence no latency/clock errors/multi leader fail-over/PAXOS/voting/stale keys/exotic application specific conflict resolution logic. This is not accurate. You can certainly have clock errors in "single leader distributed systems", simply by virtue of the fact that the primary rotates throughout the cluster. And, perhaps even more surprisingly, you can certainly have clock errors in "single node non-distributed systems like Postgres", i.e. clock errors are completely orthogonal to distributed consensus protocols (unless you're doing something dangerous like leader leases, which I don't recommend). The problem is simple: what if the system time strobes or gets stuck in the future? How does this affect things like recording financial transactions? What if NTP stops working? Distributed systems aside, it's pretty tricky. I've worked on this problem of clock errors specifically for TigerBeetle [1], a financial accounting database that can process a million transactions a second. In our domain, we can't afford to timestamp transactions with the wrong timestamp, because this could mean that financial transactions (e.g. credit card payments) don't rollback if the other party to the transaction times out, so liquidity could get stuck and tied up, for the duration of the clock error. [1] https://www.tigerbeetle.com/post/three-clocks-are-better-than-one https://www.tigerbeetle.com/post/three-clocks-are-better-tha...