3 ms·

He's also pretty favorable about Postgresql: https://aphyr.com/posts/282-call-me-maybe-postgres https://aphyr.com/posts/282-call-me-maybe-postgres

by jxf 11y ago

rdtsc 11y ago
Postgres was a bit of a funny one. I don't know what to think of it. The partition case studies was with a single server and clients. The failure was found related to the server accepting the transaction but the ack never reaching the client (due to partition) so client thinks transaction hasn't completed.
- perlgeek 11y ago
  Yes, that surprised me too; I expected a cluster of several Pg servers instead.
  - justizin 11y ago
    Anyone who has run a number of postgres replicas can tell you that their ability to recover from a network partition is directly proportionate to their wal_keep_segments value and there isn't otherwise much to it. If the network recovers while the last WAL log your replica read still exists on the server, your replica probably get to catch up, as long as your replica can read the logs as fast as they're being rotated out. Postgres doesn't really try much else in the realm of distributed systems, so there's not much else you can test. Skytools would be interesting, though, I've never felt particularly comfortable with replication schemes that trust an external process to ship rows and other changes, compared to the binary replication that will absolutely fail if you don't have every bit.
    - ahachete 11y ago
      >Anyone who has run a number of postgres replicas can tell you that their ability to recover from a network partition is directly proportionate to their wal_keep_segments value and there isn't otherwise much to it. Rather than wal_keep_segments, you may (should) use replication slots (http://www.postgresql.org/docs/9.4/static/warm-standby.html#STREAMING-REPLICATION-SLOTS http://www.postgresql.org/docs/9.4/static/warm-standby.html#...), by setting the primary_slot_name parameter in recovery.conf. With replication slots, WAL files will be kept for every replica, regardless of how far the replica lags behind. So you can very directly control the ability to recover from network partitions (although in exchange you will need to monitor the disk usage for pg_xlog at the master and, if necessary, forcibly drop the slot).
      - justizin 11y ago
        Neat, I haven't had the pleasure of running postgres 9.4 in production, and I don't run postgres at all at my current gig, but that's pretty dope. In practice, the last time I ran pg, I was fond of wal-e which, esp if you're on EC2, is handy because you can store basically all of the WAL segments ever, with snapshots, in S3, and you can bring new replicas online without a read load on any of the existing nodes. It will also bring a replica online that's been shut off for days or weeks for service. This is really just an S3 version implementation of the wal archiver, which was originally designed afaict for storing infinite history on NFS. Come to think of it, the new place has NFS. rubs chin Anyway, thanks for the pointer! Def something I would look into in the future. Maybe it would be fun to write a jepsen test for recovering pg replicas.
- takeda 11y ago
  Well technically clients are also part of distributed systems, but you are right Postgres is different than all other systems he analyzed.
- ahachete 11y ago
  Well, PostgreSQL is not a distributed database. Sure, the study may have involved one of the HA solutions available for PostgreSQL, but it wouldn't have been a PostgreSQL study then. As some comment below, the reason for this observed behavior is due to the fact that the server, together with the client, form a kind of distributed system. It is not classified as a failure to perform a write and fail to ACK that to the client. The reverse (not performing an ACKed write) is a failure. However, this suggests that every non ACKed write operation should be retried with care, because it may have gone through. It's better if you can use idempotency semantics, like 9.5's upcoming http://www.postgresql.org/docs/9.5/static/sql-insert.html#SQL-ON-CONFLICT http://www.postgresql.org/docs/9.5/static/sql-insert.html#SQ... (ON CONFLICT, aka UPSERT).
  - anarazel 11y ago
    > It is not classified as a failure to perform a write and fail to ACK that to the client. It's worthwhile to note that this is a fairly general problem in many, many kinds of systems. If that's actually a problem, you can use 2PC. If you didn't get back an ack for the PREPARE TRANSACTION it failed (and you better abort it explicitly until you get an error that it doesnt exist!). Once the PREPARE succeeded you send COMMIT PREPARED until you suceed or get an error that the repared transaction doesn't exist anymore (in which case you suceeded but didn't get the memo about it).

He's also pretty favorable about Postgresql: https://aphyr.com/posts/282-call-me-maybe-postgres https://aphyr.com/posts/282-call-me-maybe-postgres

by jxf 11y ago

He's also pretty favorable about Postgresql: https://aphyr.com/posts/282-call-me-maybe-postgres https://aphyr.com/posts/282-call-me-maybe-postgres

rdtsc 11y ago

Postgres was a bit of a funny one. I don't know what to think of it. The partition case studies was with a single server and clients. The failure was found related to the server accepting the transaction but the ack never reaching the client (due to partition) so client thinks transaction hasn't completed.

perlgeek 11y ago
Yes, that surprised me too; I expected a cluster of several Pg servers instead.
- justizin 11y ago
  Anyone who has run a number of postgres replicas can tell you that their ability to recover from a network partition is directly proportionate to their wal_keep_segments value and there isn't otherwise much to it. If the network recovers while the last WAL log your replica read still exists on the server, your replica probably get to catch up, as long as your replica can read the logs as fast as they're being rotated out. Postgres doesn't really try much else in the realm of distributed systems, so there's not much else you can test. Skytools would be interesting, though, I've never felt particularly comfortable with replication schemes that trust an external process to ship rows and other changes, compared to the binary replication that will absolutely fail if you don't have every bit.
  - ahachete 11y ago
    >Anyone who has run a number of postgres replicas can tell you that their ability to recover from a network partition is directly proportionate to their wal_keep_segments value and there isn't otherwise much to it. Rather than wal_keep_segments, you may (should) use replication slots (http://www.postgresql.org/docs/9.4/static/warm-standby.html#STREAMING-REPLICATION-SLOTS http://www.postgresql.org/docs/9.4/static/warm-standby.html#...), by setting the primary_slot_name parameter in recovery.conf. With replication slots, WAL files will be kept for every replica, regardless of how far the replica lags behind. So you can very directly control the ability to recover from network partitions (although in exchange you will need to monitor the disk usage for pg_xlog at the master and, if necessary, forcibly drop the slot).
    - justizin 11y ago
      Neat, I haven't had the pleasure of running postgres 9.4 in production, and I don't run postgres at all at my current gig, but that's pretty dope. In practice, the last time I ran pg, I was fond of wal-e which, esp if you're on EC2, is handy because you can store basically all of the WAL segments ever, with snapshots, in S3, and you can bring new replicas online without a read load on any of the existing nodes. It will also bring a replica online that's been shut off for days or weeks for service. This is really just an S3 version implementation of the wal archiver, which was originally designed afaict for storing infinite history on NFS. Come to think of it, the new place has NFS. rubs chin Anyway, thanks for the pointer! Def something I would look into in the future. Maybe it would be fun to write a jepsen test for recovering pg replicas.
takeda 11y ago
Well technically clients are also part of distributed systems, but you are right Postgres is different than all other systems he analyzed.
ahachete 11y ago
Well, PostgreSQL is not a distributed database. Sure, the study may have involved one of the HA solutions available for PostgreSQL, but it wouldn't have been a PostgreSQL study then. As some comment below, the reason for this observed behavior is due to the fact that the server, together with the client, form a kind of distributed system. It is not classified as a failure to perform a write and fail to ACK that to the client. The reverse (not performing an ACKed write) is a failure. However, this suggests that every non ACKed write operation should be retried with care, because it may have gone through. It's better if you can use idempotency semantics, like 9.5's upcoming http://www.postgresql.org/docs/9.5/static/sql-insert.html#SQL-ON-CONFLICT http://www.postgresql.org/docs/9.5/static/sql-insert.html#SQ... (ON CONFLICT, aka UPSERT).
- anarazel 11y ago
  > It is not classified as a failure to perform a write and fail to ACK that to the client. It's worthwhile to note that this is a fairly general problem in many, many kinds of systems. If that's actually a problem, you can use 2PC. If you didn't get back an ack for the PREPARE TRANSACTION it failed (and you better abort it explicitly until you get an error that it doesnt exist!). Once the PREPARE succeeded you send COMMIT PREPARED until you suceed or get an error that the repared transaction doesn't exist anymore (in which case you suceeded but didn't get the memo about it).