5 ms·
Kyle series "Call Me Maybe" is great. The general take-away from most of them is -- do not believe the vendors of distributed systems. Even systems that were m
by rdtsc 11y ago
Kyle series "Call Me Maybe" is great.
The general take-away from most of them is -- do not believe the vendors of distributed systems. Even systems that were meant to solve consensus (etcd, consul) failed:
https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul
Elasticsearch didn't do well, MongoDB, Kafka, etc.
The only ones that did ok that I remember were: Zookeeper, Riak (with siblings), and maybe Cassandra.
- jxf 11y agoHe's also pretty favorable about Postgresql: https://aphyr.com/posts/282-call-me-maybe-postgres https://aphyr.com/posts/282-call-me-maybe-postgres
- rdtsc 11y agoPostgres was a bit of a funny one. I don't know what to think of it. The partition case studies was with a single server and clients. The failure was found related to the server accepting the transaction but the ack never reaching the client (due to partition) so client thinks transaction hasn't completed.
- perlgeek 11y agoYes, that surprised me too; I expected a cluster of several Pg servers instead.
- justizin 11y agoAnyone who has run a number of postgres replicas can tell you that their ability to recover from a network partition is directly proportionate to their wal_keep_segments value and there isn't otherwise much to it. If the network recovers while the last WAL log your replica read still exists on the server, your replica probably get to catch up, as long as your replica can read the logs as fast as they're being rotated out. Postgres doesn't really try much else in the realm of distributed systems, so there's not much else you can test. Skytools would be interesting, though, I've never felt particularly comfortable with replication schemes that trust an external process to ship rows and other changes, compared to the binary replication that will absolutely fail if you don't have every bit.
- ahachete 11y ago>Anyone who has run a number of postgres replicas can tell you that their ability to recover from a network partition is directly proportionate to their wal_keep_segments value and there isn't otherwise much to it. Rather than wal_keep_segments, you may (should) use replication slots (http://www.postgresql.org/docs/9.4/static/warm-standby.html#STREAMING-REPLICATION-SLOTS http://www.postgresql.org/docs/9.4/static/warm-standby.html#...), by setting the primary_slot_name parameter in recovery.conf. With replication slots, WAL files will be kept for every replica, regardless of how far the replica lags behind. So you can very directly control the ability to recover from network partitions (although in exchange you will need to monitor the disk usage for pg_xlog at the master and, if necessary, forcibly drop the slot).
- justizin 11y agoNeat, I haven't had the pleasure of running postgres 9.4 in production, and I don't run postgres at all at my current gig, but that's pretty dope. In practice, the last time I ran pg, I was fond of wal-e which, esp if you're on EC2, is handy because you can store basically all of the WAL segments ever, with snapshots, in S3, and you can bring new replicas online without a read load on any of the existing nodes. It will also bring a replica online that's been shut off for days or weeks for service. This is really just an S3 version implementation of the wal archiver, which was originally designed afaict for storing infinite history on NFS. Come to think of it, the new place has NFS. rubs chin Anyway, thanks for the pointer! Def something I would look into in the future. Maybe it would be fun to write a jepsen test for recovering pg replicas.
- takeda 11y agoWell technically clients are also part of distributed systems, but you are right Postgres is different than all other systems he analyzed.
- ahachete 11y agoWell, PostgreSQL is not a distributed database. Sure, the study may have involved one of the HA solutions available for PostgreSQL, but it wouldn't have been a PostgreSQL study then. As some comment below, the reason for this observed behavior is due to the fact that the server, together with the client, form a kind of distributed system. It is not classified as a failure to perform a write and fail to ACK that to the client. The reverse (not performing an ACKed write) is a failure. However, this suggests that every non ACKed write operation should be retried with care, because it may have gone through. It's better if you can use idempotency semantics, like 9.5's upcoming http://www.postgresql.org/docs/9.5/static/sql-insert.html#SQL-ON-CONFLICT http://www.postgresql.org/docs/9.5/static/sql-insert.html#SQ... (ON CONFLICT, aka UPSERT).
- anarazel 11y ago> It is not classified as a failure to perform a write and fail to ACK that to the client. It's worthwhile to note that this is a fairly general problem in many, many kinds of systems. If that's actually a problem, you can use 2PC. If you didn't get back an ack for the PREPARE TRANSACTION it failed (and you better abort it explicitly until you get an error that it doesnt exist!). Once the PREPARE succeeded you send COMMIT PREPARED until you suceed or get an error that the repared transaction doesn't exist anymore (in which case you suceeded but didn't get the memo about it).
- bbrazil 11y agoAs well it's interesting to see the different responses of teams when problems are raised. Some are very thankful for the analysis, others don't seem to appreciate how important correctness is in this type of distributed systems. I find it's a good indicator of the maturity of the teams working on them, and how much I'm willing to trust each system.
- rdtsc 11y agoExactly. Notably etcd and consul teams have quickly responded with fixes.
- dopamean 11y ago> I find it's a good indicator of the maturity of the teams working on them, and how much I'm willing to trust each system. I hate to nitpick but I'm generally curious. How do you know it's a good indicator? It can definitely be _an_ indicator but how do you know it was good? Did you have a follow up experience that validated what the indicator indicated?
- jerf 11y agoHow a project performs today tells you the zeroth derivative of its location. Looking at the commit log tells you about the first derivative. How people react to Call Me Maybe when its about their product gives you a lot of information about the second derivative.
- deleted 11y ago[deleted]
- _ak 11y agoIIRC, Riak only did well with the CRDT data types. That result and the rather well done presentation by Peter Bourgon about CRDT and the time series database they built with it at SoundCloud made me look into more, and it seems a quite compelling concept. And obviously proven robust.
- jamwt 11y agoNot true, Riak was completely, 100% correct with siblings enabled, which is Totally Expected behavior for a standard Dynamo-style system. The criticism is that last-write-wins is the default, and that's a fair criticism--that's not an honest default for a robust eventually consistent system like Riak. My guess is they default to LWW to ease onboarding of new users. Everyone is worried about rivaling MongoDB's out-of-the-box experience so you don't scare people away... TBH, though, if you're using a distributed, eventually consistent storage system without understanding the consequences, you're probably using the wrong tool.
- gshx 11y agoHBase also did well. They need to blog about it. There was a related jira ticket I read through: https://issues.apache.org/jira/browse/HBASE-9719 https://issues.apache.org/jira/browse/HBASE-9719 Also, an old blog entry from yammer engg. blog: http://old.eng.yammer.com/call-me-maybe-hbase/ http://old.eng.yammer.com/call-me-maybe-hbase/
- chatman 11y agoApache Solr did well.
- halestock 11y agoI remember reading that FoundationDB did very well, OH WELL...
- rdtsc 11y agoThey ran it themselves if I recall correctly.
- abrookewood 11y agoI spoke to him privately and in terms of how well message brokers handle fault tolerance, his response was that Kafka appears to be the best at the moment.