4 ms·

Since they all lack schemas, constraints, strict consistency and ACID transactions, their replication logic can be much simpler. Why would an RDBMS txlog be so

by moe 12y ago

Since they all lack schemas, constraints, strict consistency and ACID transactions, their replication logic can be much simpler. Why would an RDBMS txlog be somehow harder to ship than any other db log? And how is that an excuse for crappy usability in any case?

lobster_johnson 12y ago
Conflict management and consistency in general. The products you mention have no interdependencies within relations or between relations, such as an RDBMS has, which means that they can just push streams of individual K/V pairs without having to maintain consistency between them. Most of those projects only offer temporal consistency (last write wins) and the CRDTs offered (eg., Riak) are exceedingly simple compared to the complexities of managing uniqueness constraints or foreign key constraints across transactions. Hinted handoff -- a common technique that mitigates split-brain data loss and increases availability in a shared scenario -- simply wouldn't work in a relational scenario because nobody would have any idea if the changes would apply when the target shard came back. Projects like Postgres-XC (ActorDB also looks promising) have shown that multimaster replication is clearly possible in a relational database, but also that it's clearly a difficult problem to crack, because as has been discovered the last decade or so, consistency is the enemy of availability and partition tolerance. Witness the very slow progress made by XC to retrofit multimaster replication into Postgres, and the many challenges (such as sequences) they have had to solve. Or the fact that it's pretty much alone in the open-source field in trying to solve this issue with a modern relational database. Even very mature, advanced, commercial solutions like Oracle RAC and Oracle Advanced Replication require a lot of configuration, careful design and monitoring to get right. Note that I'm explicitly using the projects you mentioned as the benchmark here, ie. multimaster-replicated stores with automatic failover based on consensus protocols. You can build a simpler system — single master, consensus-based election, strict consistency guarantee across shards (reducing either availability or partition tolerance) — but it's still harder than you seem to think it is.
- moe 12y ago
  Hinted handoff multimaster What does multi-master have to do with anything? The subject was making the postgres master/slave replication as easy to use as the aforementioned db's. Yes, some of them use more advanced clustering models (multi-master) than postgres and are still easier to use. That doesn't exactly make postgres look better in the comparison.
  - lobster_johnson 12y ago
    Are you being deliberately obtuse? You explicitly mentioned four multimaster key/value stores: Rethinkdb, Redis, Riak, ElasticSearch all do. (them not being relational is irrelevant to replication) I argued that those are not RDBMSes, and that their replication model was much simpler, something you still seem to be arguing against. I don't disagree that Postgres's replication could be much nicer, but you're not articulating your position well. For one, bringing up these completely apples-to-orange comparisons makes no sense.
    - moe 12y ago
      Are you being deliberately obtuse? You're the one being obtuse here. I introduced four databases, three of which use more complex replication models than PG, all of which are easier to admin than PG. And I'm asking PG to make their (simpler) replication as easy to use as these other databases. If that's too confusing then just use Redis for reference and ignore the other three. Redis is plain master/slave and spinning up a slave takes a single command ("SLAVEOF foo"). That's exactly how it should be in postgresql as well. Not more, not less.
      - lobster_johnson 12y ago
        They can and do use different replication models because their data model is explicitly wildly different from Postgres. For example, Postgres couldn't support eventual consistency even if it wanted to. You're barking up the wrong tree. As for Redis, sure, it has a simple master/slave system and Postgres could offer a similarly simple command UI that would let you issue a command instead of doing the current, trivial amount of setup. I don't disagree, but I think it's an inconsequential problem that affects a sysadmin/devops person about five minutes each year, and an issue that ultimately only deters dilettantes. The real problem to solve is dynamic promotion/demotion of masters and slave. It's not entirely trivial [1]. [1] http://redis.io/topics/sentinel http://redis.io/topics/sentinel

by moe 12y ago

lobster_johnson 12y ago

Conflict management and consistency in general. The products you mention have no interdependencies within relations or between relations, such as an RDBMS has, which means that they can just push streams of individual K/V pairs without having to maintain consistency between them. Most of those projects only offer temporal consistency (last write wins) and the CRDTs offered (eg., Riak) are exceedingly simple compared to the complexities of managing uniqueness constraints or foreign key constraints across transactions. Hinted handoff -- a common technique that mitigates split-brain data loss and increases availability in a shared scenario -- simply wouldn't work in a relational scenario because nobody would have any idea if the changes would apply when the target shard came back. Projects like Postgres-XC (ActorDB also looks promising) have shown that multimaster replication is clearly possible in a relational database, but also that it's clearly a difficult problem to crack, because as has been discovered the last decade or so, consistency is the enemy of availability and partition tolerance. Witness the very slow progress made by XC to retrofit multimaster replication into Postgres, and the many challenges (such as sequences) they have had to solve. Or the fact that it's pretty much alone in the open-source field in trying to solve this issue with a modern relational database. Even very mature, advanced, commercial solutions like Oracle RAC and Oracle Advanced Replication require a lot of configuration, careful design and monitoring to get right. Note that I'm explicitly using the projects you mentioned as the benchmark here, ie. multimaster-replicated stores with automatic failover based on consensus protocols. You can build a simpler system — single master, consensus-based election, strict consistency guarantee across shards (reducing either availability or partition tolerance) — but it's still harder than you seem to think it is.

moe 12y ago
Hinted handoff multimaster What does multi-master have to do with anything? The subject was making the postgres master/slave replication as easy to use as the aforementioned db's. Yes, some of them use more advanced clustering models (multi-master) than postgres and are still easier to use. That doesn't exactly make postgres look better in the comparison.
- lobster_johnson 12y ago
  Are you being deliberately obtuse? You explicitly mentioned four multimaster key/value stores: Rethinkdb, Redis, Riak, ElasticSearch all do. (them not being relational is irrelevant to replication) I argued that those are not RDBMSes, and that their replication model was much simpler, something you still seem to be arguing against. I don't disagree that Postgres's replication could be much nicer, but you're not articulating your position well. For one, bringing up these completely apples-to-orange comparisons makes no sense.
  - moe 12y ago
    Are you being deliberately obtuse? You're the one being obtuse here. I introduced four databases, three of which use more complex replication models than PG, all of which are easier to admin than PG. And I'm asking PG to make their (simpler) replication as easy to use as these other databases. If that's too confusing then just use Redis for reference and ignore the other three. Redis is plain master/slave and spinning up a slave takes a single command ("SLAVEOF foo"). That's exactly how it should be in postgresql as well. Not more, not less.
    - lobster_johnson 12y ago
      They can and do use different replication models because their data model is explicitly wildly different from Postgres. For example, Postgres couldn't support eventual consistency even if it wanted to. You're barking up the wrong tree. As for Redis, sure, it has a simple master/slave system and Postgres could offer a similarly simple command UI that would let you issue a command instead of doing the current, trivial amount of setup. I don't disagree, but I think it's an inconsequential problem that affects a sysadmin/devops person about five minutes each year, and an issue that ultimately only deters dilettantes. The real problem to solve is dynamic promotion/demotion of masters and slave. It's not entirely trivial [1]. [1] http://redis.io/topics/sentinel http://redis.io/topics/sentinel