4 ms·

Well yes exactly. That's kind of why I brought up the idea of clock errors. Single leader distributed systems are honestly an entirely different paradigm to m

by urthor 5y ago

Well yes exactly. That's kind of why I brought up the idea of clock errors. Single leader distributed systems are honestly an entirely different paradigm to multi-leader. You have a main node, the main node controls routing and which data goes where. There can only be one main node at one time, hence no latency/clock errors/multi leader fail-over/PAXOS/voting/stale keys/exotic application specific conflict resolution logic. Done, simple. (Secretly not so simple, all of the above problems are actually hugely important to resolve in any monolithic database. However, Postgres sorts it out for you.) Really, what I am calling "distributed systems problems" are the subset with multiple simultaneous, geographically separate leaders.

dikei 5y ago
>> There can only be one main node at one time This is how most distributed system solve the consensus problem: the nodes automatically run leader election algorithm so that eventually only one leader remains. >> (Secretly not so simple, all of the above problems are actually hugely important to resolve in any monolithic database. However, Postgres sorts it out for you.) Except that in case of network partition of the primary node, Postgres can't promote a new primary node from the replicas automatically, unlike consensus systems such as ETCD/Zookeeper.
_vvhw 5y ago
> There can only be one main node at one time, hence no latency/clock errors/multi leader fail-over/PAXOS/voting/stale keys/exotic application specific conflict resolution logic. This is not accurate. You can certainly have clock errors in "single leader distributed systems", simply by virtue of the fact that the primary rotates throughout the cluster. And, perhaps even more surprisingly, you can certainly have clock errors in "single node non-distributed systems like Postgres", i.e. clock errors are completely orthogonal to distributed consensus protocols (unless you're doing something dangerous like leader leases, which I don't recommend). The problem is simple: what if the system time strobes or gets stuck in the future? How does this affect things like recording financial transactions? What if NTP stops working? Distributed systems aside, it's pretty tricky. I've worked on this problem of clock errors specifically for TigerBeetle [1], a financial accounting database that can process a million transactions a second. In our domain, we can't afford to timestamp transactions with the wrong timestamp, because this could mean that financial transactions (e.g. credit card payments) don't rollback if the other party to the transaction times out, so liquidity could get stuck and tied up, for the duration of the clock error. [1] https://www.tigerbeetle.com/post/three-clocks-are-better-than-one https://www.tigerbeetle.com/post/three-clocks-are-better-tha...
- urthor 5y ago
  Fair point.

by urthor 5y ago

dikei 5y ago

>> There can only be one main node at one time This is how most distributed system solve the consensus problem: the nodes automatically run leader election algorithm so that eventually only one leader remains. >> (Secretly not so simple, all of the above problems are actually hugely important to resolve in any monolithic database. However, Postgres sorts it out for you.) Except that in case of network partition of the primary node, Postgres can't promote a new primary node from the replicas automatically, unlike consensus systems such as ETCD/Zookeeper.

_vvhw 5y ago

> There can only be one main node at one time, hence no latency/clock errors/multi leader fail-over/PAXOS/voting/stale keys/exotic application specific conflict resolution logic. This is not accurate. You can certainly have clock errors in "single leader distributed systems", simply by virtue of the fact that the primary rotates throughout the cluster. And, perhaps even more surprisingly, you can certainly have clock errors in "single node non-distributed systems like Postgres", i.e. clock errors are completely orthogonal to distributed consensus protocols (unless you're doing something dangerous like leader leases, which I don't recommend). The problem is simple: what if the system time strobes or gets stuck in the future? How does this affect things like recording financial transactions? What if NTP stops working? Distributed systems aside, it's pretty tricky. I've worked on this problem of clock errors specifically for TigerBeetle [1], a financial accounting database that can process a million transactions a second. In our domain, we can't afford to timestamp transactions with the wrong timestamp, because this could mean that financial transactions (e.g. credit card payments) don't rollback if the other party to the transaction times out, so liquidity could get stuck and tied up, for the duration of the clock error. [1] https://www.tigerbeetle.com/post/three-clocks-are-better-than-one https://www.tigerbeetle.com/post/three-clocks-are-better-tha...

urthor 5y ago
Fair point.