3 ms·

I'm not a big fan of conflict resolution, and LWW is a pretty bad choice in most cases. Conflict resolution means data loss (it violates the D of ACID, as an us

I'm not a big fan of conflict resolution, and LWW is a pretty bad choice in most cases. Conflict resolution means data loss (it violates the D of ACID, as an user expects that a transaction that was committed would endure, but it is deleted or overwritten by another one), and most likely C. The effects of this are hard to predict and can hit pretty hard your business logic. This is not the kind of expectation you get from Postgres. This would fail Jepsen at second 1. So I'd advise not to use it. The only use case I see for bi-directional replication is where you ensure that all the master nodes have disjoint sets of data, and the logical replication just aggregates them, without a possibility for conflicts. If you really want multi-master you need a system that works on top of a consensus protocol. And there are definitely tools out there for this. But not LWW.

bonesmoses 7y ago
LWW / LUW systems are a fine first approximation, but generally do not fulfill the strictest guarantees and desirable outcomes for a database without accounting for a lot of edge cases. A good conflict management system will have that only as a default. Better ones will also provide mechanisms for overriding LWW in specific scenarios on a per table or even row basis depending on the in/out values from both nodes. The trick is building these conflict management rules so they're independently deterministic, or you'll end up with node divergence. Normally we just do what you said, and strongly recommend interacting with sticky sessions, geographically disparate segments, etc., to prevent the problem before it happens. It's far easier to avoid a conflict than to correct it after the fact. While a consensus model "fixes" this, it drastically cuts your throughput to a function of the latency of the most remote node in the quorum, essentially defeating the purpose of having a local Master node. Ideally you would reserve this for specific transactions that are sensitive in that they require absolute consistency regardless of incurred latency (See PACELC). And even this requires a distributed deadlock detector unless more than just the transactions are managed through the consensus layer. BDR for example provides both of these models for the above reasons. Multi-Master is not an easy problem to solve, and requires a multi-faceted approach to even come close to something usable in a generalized scenario.
- ahachete 7y ago
  I don't agree with the "fine" in the "fine first approximation". I believe it is indeed a bad solution. LWW/LUW basically leads to data loss. Something unexpected for Postgres users. Which potentially undermines its heritage of caring of data to the extreme. From a practical perspective, if you run an inventory or sales of physical goods, a counter over a multi-master with said conflict resolution may lead to double bookings, lost bookings or negative inventory stocks. None of these events are fine to me. I'd prefer not to have conflict resolution and have the replication system explicitly check that data sets are disjoints. In this case, these multi-master systems would be very useful (for a particular set of applications). That consensus is "slow" is a problem also well solved in the real world. Sure, it's not easy, but when you partition and parallelize, you overcome many of the problems. Look at Spanner, AzureDB or the Postgres-compatible APIs of YugabyteDB or CockroachDB. They all have implemented multi-master with consensus.
  - bonesmoses 7y ago
    Right tool for the right job. Users in Tokyo won't be editing accounts for users in Austrailia. There's often a natural partition there where you can leverage the data locality with an eventual consistency basis safely. Conflict-free Replicated Data Types (CRDTs) and column-level conflict resolution also largely address the issue with data merges and coarse counter systems. I'll just ignore the general design flaw there, considering these should be an insert-only ledger system rather than counters, because bad or naive implementations abound. I get that the application layer shouldn't necessarily have to coddle the database, given absolute ACID would resist non-ideal usage patterns. I haven't played with the others, but I set up an 8-node CockroachDB and managed to get pgbench to run on it. I was clearly doing something wrong, because I wasn't impressed at the throughput even with very high client counts. From what I could see there, I was clearly being throttled by some mix of RTT and locking contention between the 3-node default consensus group. Supposing this is not problematic in non-overlapping shard locality, you still hit a hard performance ceiling when multiple clients interact with the same pool fragment. I'll have to look at the others now.
    - 7y ago
jinhualuo 7y ago
Yes, if multi-masters change the same rows of same table, it would cause conflict. Consensus protocol like raft, has limitation: it requires low delay among the nodes, so normally, it would be deployed in the same data-center. Moreover, if all changes (conflict or not), handled by consensus protocol, the performance would be bad. In fact, data-centers are normally loose coupling (the application normally choose the same data-center as the data source/target). So data-centers just need somehow light-weighted conflict resolution. pgcat lww, just like Cassandra, provides counter column concept, which is used to retain each change among data-centers, and simple aggregation helper function there. pgcat lww is done by pg trigger, so if you need customization of your business, you could modify the trigger function to fit it.
jinhualuo 7y ago
Note that LWW is optional, pgcat implement LWW in trigger way, see the pgcat_setup_lww (https://github.com/kingluo/pgcat/tree/master/cmd/pgcat_setup_lww https://github.com/kingluo/pgcat/tree/master/cmd/pgcat_setup...).

bonesmoses 7y ago

LWW / LUW systems are a fine first approximation, but generally do not fulfill the strictest guarantees and desirable outcomes for a database without accounting for a lot of edge cases. A good conflict management system will have that only as a default. Better ones will also provide mechanisms for overriding LWW in specific scenarios on a per table or even row basis depending on the in/out values from both nodes. The trick is building these conflict management rules so they're independently deterministic, or you'll end up with node divergence. Normally we just do what you said, and strongly recommend interacting with sticky sessions, geographically disparate segments, etc., to prevent the problem before it happens. It's far easier to avoid a conflict than to correct it after the fact. While a consensus model "fixes" this, it drastically cuts your throughput to a function of the latency of the most remote node in the quorum, essentially defeating the purpose of having a local Master node. Ideally you would reserve this for specific transactions that are sensitive in that they require absolute consistency regardless of incurred latency (See PACELC). And even this requires a distributed deadlock detector unless more than just the transactions are managed through the consensus layer. BDR for example provides both of these models for the above reasons. Multi-Master is not an easy problem to solve, and requires a multi-faceted approach to even come close to something usable in a generalized scenario.

ahachete 7y ago
I don't agree with the "fine" in the "fine first approximation". I believe it is indeed a bad solution. LWW/LUW basically leads to data loss. Something unexpected for Postgres users. Which potentially undermines its heritage of caring of data to the extreme. From a practical perspective, if you run an inventory or sales of physical goods, a counter over a multi-master with said conflict resolution may lead to double bookings, lost bookings or negative inventory stocks. None of these events are fine to me. I'd prefer not to have conflict resolution and have the replication system explicitly check that data sets are disjoints. In this case, these multi-master systems would be very useful (for a particular set of applications). That consensus is "slow" is a problem also well solved in the real world. Sure, it's not easy, but when you partition and parallelize, you overcome many of the problems. Look at Spanner, AzureDB or the Postgres-compatible APIs of YugabyteDB or CockroachDB. They all have implemented multi-master with consensus.
- bonesmoses 7y ago
  Right tool for the right job. Users in Tokyo won't be editing accounts for users in Austrailia. There's often a natural partition there where you can leverage the data locality with an eventual consistency basis safely. Conflict-free Replicated Data Types (CRDTs) and column-level conflict resolution also largely address the issue with data merges and coarse counter systems. I'll just ignore the general design flaw there, considering these should be an insert-only ledger system rather than counters, because bad or naive implementations abound. I get that the application layer shouldn't necessarily have to coddle the database, given absolute ACID would resist non-ideal usage patterns. I haven't played with the others, but I set up an 8-node CockroachDB and managed to get pgbench to run on it. I was clearly doing something wrong, because I wasn't impressed at the throughput even with very high client counts. From what I could see there, I was clearly being throttled by some mix of RTT and locking contention between the 3-node default consensus group. Supposing this is not problematic in non-overlapping shard locality, you still hit a hard performance ceiling when multiple clients interact with the same pool fragment. I'll have to look at the others now.
  - 7y ago

jinhualuo 7y ago

Yes, if multi-masters change the same rows of same table, it would cause conflict. Consensus protocol like raft, has limitation: it requires low delay among the nodes, so normally, it would be deployed in the same data-center. Moreover, if all changes (conflict or not), handled by consensus protocol, the performance would be bad. In fact, data-centers are normally loose coupling (the application normally choose the same data-center as the data source/target). So data-centers just need somehow light-weighted conflict resolution. pgcat lww, just like Cassandra, provides counter column concept, which is used to retain each change among data-centers, and simple aggregation helper function there. pgcat lww is done by pg trigger, so if you need customization of your business, you could modify the trigger function to fit it.

jinhualuo 7y ago

Note that LWW is optional, pgcat implement LWW in trigger way, see the pgcat_setup_lww (https://github.com/kingluo/pgcat/tree/master/cmd/pgcat_setup_lww https://github.com/kingluo/pgcat/tree/master/cmd/pgcat_setup...).