8 ms·

Can we just start posting "Jepsen or it didn't happen" on these? Clustered relational DBs are really remarkably hard to get right. A shocking percentage of pro

Can we just start posting "Jepsen or it didn't happen" on these? Clustered relational DBs are really remarkably hard to get right. A shocking percentage of projects and vendors don't really even make a serious try (cf GaleraDB.) This project may or may not have done a reasonable job (no clue,) but the lack of information about how they tested it is a bad sign.

merb 11y ago
Jepsen isn't a real world test either. Also PostgreSQL HA has two options, either you have 1 Master and 1 Sync or 1 Async, if you have a Async Secondary it will of course loose data. Also you will have a short downtime on switchover.
EwanToo 11y ago
I'm pretty sure this is simply active/standby clustering with some level of log shipping, not the active/active clustering that Jepsen is relevant for?
- teraflop 11y ago
  Jepsen is completely relevant to this kind of system, because it doesn't care what words you use to describe your algorithm -- just the patterns of communication it exhibits, and the consistency properties it satisfies. (What Jepsen isn't useful for is databases that don't even try to be consistent -- for example, Cassandra in its default modes. All Jepsen can do for you in that case is confirm what you should have been able to tell from the documentation.) And arguably, active/standby with automatic failover is just a special case of active/active, with particar tradeoffs about throughput and failover latency.
  - EwanToo 11y ago
    Well you can run Jepsen tests against single nodes and use it to validate consistency on that single node if you want, but it's not really what it's made for, the github tagline for it is: "Call Me Maybe: simulating network partitions in DBs" Running Jepsen against a log shipping automation script is, to my mind, largely redundant. Running it against this system would be testing Postgres [1], not yoke. [1] - https://aphyr.com/posts/282-call-me-maybe-postgres https://aphyr.com/posts/282-call-me-maybe-postgres
    - icebraining 11y ago
      Can't an extra layer of inconsistencies arise if there's a network partition, and each yoke process thinks it should be the primary, so both accept writes?
      - EwanToo 11y ago
        I guess so yeah, I hadn't considered that :)
      - dan353hehe 11y ago
        Thats true, however in yokes case, each cluster needs a third member called a monitor who only serves to resolve network partitions. a secondary cannot self promote to primary without the monitor agreeing. two node systems can never work as a primary/secondary failover solution in the presence of a network partition. disclosure: I helped write yoke.
        tveita 11y ago
        Can the old master still accept requests if it loses contact with the monitor or secondary? What happens if the monitor goes down? Can there be data loss on failover? It would be nice if the readme listed what failure scenarios are handled and what guarantees you intend the system to have.
deleted 11y ago
[deleted]
antirez 11y ago
Most (but not all) MySQL / PostgreSQL HA systems have similar semantics of Redis Sentinel. They don't try to achieve any linearizable semantics, but to perform failure detection and fail over to some slave, with heuristics about what slave to pick. All this kind of solutions will fail to pass Jepsen tests since they are not designed to achieve the kind of strong properties Jepsen will test for. The behavior of those systems is acceptable in many production situations (it's up to the product requirements to understand if it is ok that a given set of failure modes produce data loss and/or inconsistencies) but the important bit is that the semantics is very well documented so that users know what tradeoffs they are making. EDIT: Shameless plug: I believe it was a shame that Sentinel was initially badly received because of the Jepsen test (now tons of users are using Sentinel with success, fortunately), since for what it does, it is very advanced compared to * SQL failover solutions, so a porting of Redis Sentinel to failover those systems could make sense IMHO. A few things Sentinel has that are desirable in * SQL failover systems: 1. It's distributed so Sentinel itself is not a single point of failure. 2. When the partition heals Sentinel is able to have a coherent view of the configuration. 3. It is able to automatically reconfigure the old master and the other slaves to replicate from the new master. 4. It is able to work as a configuration provider for clients, with a well defined handshake in order to trigger the reconfiguration of all the clients.
- contingencies 11y ago
  Generic solutions like corosync/pacemaker also deserve a mention. At some point you need inter-service dependencies, floating IPs, hot service spares and ordered recovery, plus tolerance for network fabric faults. Not every daemon needs to implement this stuff!
- lucian1900 11y ago
  A large reason to use Postgres in the first place is because it's a good CP system. Any clustered version is only worthwhile if it preserves consistency. It's already possible to use more than one Postgres database while not maintaining consistency, but that isn't desirable for most users.
  - 11y ago
wobbleblob 11y ago
> Clustered relational DBs are really remarkably hard to get right. That is probably be true, but I'd like to point out that clustered non-relational DBs are also really remarkably hard to get right, it's just a hard problem to solve. The tradeoffs you make with a non relational DB to make this easier are usually also available in relational DBs. It's not like Galera is your only choice.
- angelbob 11y ago
  Many of those choices to make it easier only work if you're willing to disallow a lot of SQL semantics. Like, tradeoffs in consistency? No transactions? That requires not implementing some important bits of SQL.
  - wobbleblob 11y ago
    You're not forced to use joins and most DBs don't force you to use transactions if you don't need them. Sharding is supported by postgres, oracle, sqlserver and mysql, even if it's not the recommended solution in all cases. A database doesn't magically start to scale by not supporting SQL nor does SQL support magically prevent it from scaling.
  - jeffdavis 11y ago
    Interestly, those tradeoffs have nothing to do with relational or non-relational. They are a part of SQL.
vidarh 11y ago
Yes, my first reaction was "auto-failover? what are the odds this will lose data for me".
- angelbob 11y ago
  There are systems that get this right. Just apparently none for Postgres or MySQL :-(
  - Jweb_Guru 11y ago
    There are remarkably few systems that get this right, and most of the few that remain (like Zookeeper) are key-value stores that don't support complex transactional semantics.
deleted 11y ago
[deleted]

merb 11y ago

Jepsen isn't a real world test either. Also PostgreSQL HA has two options, either you have 1 Master and 1 Sync or 1 Async, if you have a Async Secondary it will of course loose data. Also you will have a short downtime on switchover.

EwanToo 11y ago

I'm pretty sure this is simply active/standby clustering with some level of log shipping, not the active/active clustering that Jepsen is relevant for?

teraflop 11y ago
Jepsen is completely relevant to this kind of system, because it doesn't care what words you use to describe your algorithm -- just the patterns of communication it exhibits, and the consistency properties it satisfies. (What Jepsen isn't useful for is databases that don't even try to be consistent -- for example, Cassandra in its default modes. All Jepsen can do for you in that case is confirm what you should have been able to tell from the documentation.) And arguably, active/standby with automatic failover is just a special case of active/active, with particar tradeoffs about throughput and failover latency.
- EwanToo 11y ago
  Well you can run Jepsen tests against single nodes and use it to validate consistency on that single node if you want, but it's not really what it's made for, the github tagline for it is: "Call Me Maybe: simulating network partitions in DBs" Running Jepsen against a log shipping automation script is, to my mind, largely redundant. Running it against this system would be testing Postgres [1], not yoke. [1] - https://aphyr.com/posts/282-call-me-maybe-postgres https://aphyr.com/posts/282-call-me-maybe-postgres
  - icebraining 11y ago
    Can't an extra layer of inconsistencies arise if there's a network partition, and each yoke process thinks it should be the primary, so both accept writes?
    - EwanToo 11y ago
      I guess so yeah, I hadn't considered that :)
    - dan353hehe 11y ago
      Thats true, however in yokes case, each cluster needs a third member called a monitor who only serves to resolve network partitions. a secondary cannot self promote to primary without the monitor agreeing. two node systems can never work as a primary/secondary failover solution in the presence of a network partition. disclosure: I helped write yoke.
      - tveita 11y ago
        Can the old master still accept requests if it loses contact with the monitor or secondary? What happens if the monitor goes down? Can there be data loss on failover? It would be nice if the readme listed what failure scenarios are handled and what guarantees you intend the system to have.

deleted 11y ago

[deleted]

antirez 11y ago

Most (but not all) MySQL / PostgreSQL HA systems have similar semantics of Redis Sentinel. They don't try to achieve any linearizable semantics, but to perform failure detection and fail over to some slave, with heuristics about what slave to pick. All this kind of solutions will fail to pass Jepsen tests since they are not designed to achieve the kind of strong properties Jepsen will test for. The behavior of those systems is acceptable in many production situations (it's up to the product requirements to understand if it is ok that a given set of failure modes produce data loss and/or inconsistencies) but the important bit is that the semantics is very well documented so that users know what tradeoffs they are making. EDIT: Shameless plug: I believe it was a shame that Sentinel was initially badly received because of the Jepsen test (now tons of users are using Sentinel with success, fortunately), since for what it does, it is very advanced compared to * SQL failover solutions, so a porting of Redis Sentinel to failover those systems could make sense IMHO. A few things Sentinel has that are desirable in * SQL failover systems: 1. It's distributed so Sentinel itself is not a single point of failure. 2. When the partition heals Sentinel is able to have a coherent view of the configuration. 3. It is able to automatically reconfigure the old master and the other slaves to replicate from the new master. 4. It is able to work as a configuration provider for clients, with a well defined handshake in order to trigger the reconfiguration of all the clients.

contingencies 11y ago
Generic solutions like corosync/pacemaker also deserve a mention. At some point you need inter-service dependencies, floating IPs, hot service spares and ordered recovery, plus tolerance for network fabric faults. Not every daemon needs to implement this stuff!
lucian1900 11y ago
A large reason to use Postgres in the first place is because it's a good CP system. Any clustered version is only worthwhile if it preserves consistency. It's already possible to use more than one Postgres database while not maintaining consistency, but that isn't desirable for most users.
- 11y ago

wobbleblob 11y ago

> Clustered relational DBs are really remarkably hard to get right. That is probably be true, but I'd like to point out that clustered non-relational DBs are also really remarkably hard to get right, it's just a hard problem to solve. The tradeoffs you make with a non relational DB to make this easier are usually also available in relational DBs. It's not like Galera is your only choice.

angelbob 11y ago
Many of those choices to make it easier only work if you're willing to disallow a lot of SQL semantics. Like, tradeoffs in consistency? No transactions? That requires not implementing some important bits of SQL.
- wobbleblob 11y ago
  You're not forced to use joins and most DBs don't force you to use transactions if you don't need them. Sharding is supported by postgres, oracle, sqlserver and mysql, even if it's not the recommended solution in all cases. A database doesn't magically start to scale by not supporting SQL nor does SQL support magically prevent it from scaling.
- jeffdavis 11y ago
  Interestly, those tradeoffs have nothing to do with relational or non-relational. They are a part of SQL.

vidarh 11y ago

Yes, my first reaction was "auto-failover? what are the odds this will lose data for me".

angelbob 11y ago
There are systems that get this right. Just apparently none for Postgres or MySQL :-(
- Jweb_Guru 11y ago
  There are remarkably few systems that get this right, and most of the few that remain (like Zookeeper) are key-value stores that don't support complex transactional semantics.

deleted 11y ago

[deleted]