4 ms·

I don't agree with the "fine" in the "fine first approximation". I believe it is indeed a bad solution. LWW/LUW basically leads to data loss. Something unexpec

I don't agree with the "fine" in the "fine first approximation". I believe it is indeed a bad solution. LWW/LUW basically leads to data loss. Something unexpected for Postgres users. Which potentially undermines its heritage of caring of data to the extreme. From a practical perspective, if you run an inventory or sales of physical goods, a counter over a multi-master with said conflict resolution may lead to double bookings, lost bookings or negative inventory stocks. None of these events are fine to me. I'd prefer not to have conflict resolution and have the replication system explicitly check that data sets are disjoints. In this case, these multi-master systems would be very useful (for a particular set of applications). That consensus is "slow" is a problem also well solved in the real world. Sure, it's not easy, but when you partition and parallelize, you overcome many of the problems. Look at Spanner, AzureDB or the Postgres-compatible APIs of YugabyteDB or CockroachDB. They all have implemented multi-master with consensus.

bonesmoses 7y ago
Right tool for the right job. Users in Tokyo won't be editing accounts for users in Austrailia. There's often a natural partition there where you can leverage the data locality with an eventual consistency basis safely. Conflict-free Replicated Data Types (CRDTs) and column-level conflict resolution also largely address the issue with data merges and coarse counter systems. I'll just ignore the general design flaw there, considering these should be an insert-only ledger system rather than counters, because bad or naive implementations abound. I get that the application layer shouldn't necessarily have to coddle the database, given absolute ACID would resist non-ideal usage patterns. I haven't played with the others, but I set up an 8-node CockroachDB and managed to get pgbench to run on it. I was clearly doing something wrong, because I wasn't impressed at the throughput even with very high client counts. From what I could see there, I was clearly being throttled by some mix of RTT and locking contention between the 3-node default consensus group. Supposing this is not problematic in non-overlapping shard locality, you still hit a hard performance ceiling when multiple clients interact with the same pool fragment. I'll have to look at the others now.
- ahachete 7y ago
  > Right tool for the right job. Users in Tokyo won't be editing accounts for users in Austrailia. There's often a natural partition [...] Exactly. That's why I'd say for tools that support multi-master to provide and enhance (enforce) support for this use case and avoid (don't support) the case where data durability and consistency are lost. > Conflict-free Replicated Data Types (CRDTs) [...] Yes, that's another solution for the problem... but I don't see them provided by BDR or Pgcat. > I'll just ignore the general design flaw there, considering these should be an insert-only ledger system rather than counters [...] It's not that I advocate for that system, but many users may choose this implementation. That you say "hey, you need to do all this and implement this best practices (and pass the Jepsen test) just to be sure you don't screw up data consistency and durability... it's putting a heavy burden in the user. Databases, if anything, are abstracting users away from complex issues with data and providing solid primitives to work on them. Again: Postgres is known as a very well trusted data store with given ACID properties. Conflict resolution breaks away with this. > I haven't played with the others, but I set up an 8-node CockroachDB and managed to get pgbench to run on it. I was clearly doing something wrong, because I wasn't impressed at the throughput even with very high client counts [...] I haven't benchmarked it myself, but I heard stories of low performance. Actually, I always believed that the internal key-value store that they use would never scale to represent table workloads. But give a shot to the other solutions. Spanner, for one, is known to process millions of transactions per second, with global (planet scale) replication (of course, with notable lag, but total consistency). That's pretty cool ;)
  - irfansharif 7y ago
    > Actually, I always believed that the internal key-value store that they use would never scale to represent table workloads. Care to elaborate here? As someone working at that layer of the system, our RocksDB usage is but a blip in any execution trace (as it should be, any network overhead you have given it's a distributed system would dominate single-node key-value perf). That aside, plenty of RDBMS systems are designed such that they sit atop internal key-value stores. See MySQL+InnoDB[0], or MySQL+RocksDB[1] used at facebook. [1]: https://en.wikipedia.org/wiki/InnoDB https://en.wikipedia.org/wiki/InnoDB [0]: https://myrocks.io/ https://myrocks.io/

bonesmoses 7y ago

Right tool for the right job. Users in Tokyo won't be editing accounts for users in Austrailia. There's often a natural partition there where you can leverage the data locality with an eventual consistency basis safely. Conflict-free Replicated Data Types (CRDTs) and column-level conflict resolution also largely address the issue with data merges and coarse counter systems. I'll just ignore the general design flaw there, considering these should be an insert-only ledger system rather than counters, because bad or naive implementations abound. I get that the application layer shouldn't necessarily have to coddle the database, given absolute ACID would resist non-ideal usage patterns. I haven't played with the others, but I set up an 8-node CockroachDB and managed to get pgbench to run on it. I was clearly doing something wrong, because I wasn't impressed at the throughput even with very high client counts. From what I could see there, I was clearly being throttled by some mix of RTT and locking contention between the 3-node default consensus group. Supposing this is not problematic in non-overlapping shard locality, you still hit a hard performance ceiling when multiple clients interact with the same pool fragment. I'll have to look at the others now.

ahachete 7y ago
> Right tool for the right job. Users in Tokyo won't be editing accounts for users in Austrailia. There's often a natural partition [...] Exactly. That's why I'd say for tools that support multi-master to provide and enhance (enforce) support for this use case and avoid (don't support) the case where data durability and consistency are lost. > Conflict-free Replicated Data Types (CRDTs) [...] Yes, that's another solution for the problem... but I don't see them provided by BDR or Pgcat. > I'll just ignore the general design flaw there, considering these should be an insert-only ledger system rather than counters [...] It's not that I advocate for that system, but many users may choose this implementation. That you say "hey, you need to do all this and implement this best practices (and pass the Jepsen test) just to be sure you don't screw up data consistency and durability... it's putting a heavy burden in the user. Databases, if anything, are abstracting users away from complex issues with data and providing solid primitives to work on them. Again: Postgres is known as a very well trusted data store with given ACID properties. Conflict resolution breaks away with this. > I haven't played with the others, but I set up an 8-node CockroachDB and managed to get pgbench to run on it. I was clearly doing something wrong, because I wasn't impressed at the throughput even with very high client counts [...] I haven't benchmarked it myself, but I heard stories of low performance. Actually, I always believed that the internal key-value store that they use would never scale to represent table workloads. But give a shot to the other solutions. Spanner, for one, is known to process millions of transactions per second, with global (planet scale) replication (of course, with notable lag, but total consistency). That's pretty cool ;)
- irfansharif 7y ago
  > Actually, I always believed that the internal key-value store that they use would never scale to represent table workloads. Care to elaborate here? As someone working at that layer of the system, our RocksDB usage is but a blip in any execution trace (as it should be, any network overhead you have given it's a distributed system would dominate single-node key-value perf). That aside, plenty of RDBMS systems are designed such that they sit atop internal key-value stores. See MySQL+InnoDB[0], or MySQL+RocksDB[1] used at facebook. [1]: https://en.wikipedia.org/wiki/InnoDB https://en.wikipedia.org/wiki/InnoDB [0]: https://myrocks.io/ https://myrocks.io/