9 ms·

The most interesting lessons from the Jepsen series: * You should never trust, and always verify, the claims made by database manufacturers. * Especially when

by jxf 12y ago

The most interesting lessons from the Jepsen series: * You should never trust, and always verify, the claims made by database manufacturers. * Especially when those claims relate to data integrity. * Super-especially when every safety level provided by the manufacturer that includes the word "SAFE" is actually unsafe.

threeseed 12y ago
Actually the broader lesson would be to assume the worst in your application layer and try and remediate/verify wherever possible. If you look at his articles: Redis, PostgreSQL, Cassandra, ElasticSearch etc all had data consistency errors. And none of those have vendors making any claims. It's pretty sobering to say the least.
- chatman 12y ago
  Apache Solr's jepsen tests have had very good results.
- ak4g 12y ago
  Um, this is the postgres article: https://aphyr.com/posts/282-call-me-maybe-postgres https://aphyr.com/posts/282-call-me-maybe-postgres There were no acknowledged writes lost. The only unacked-but-successful writes resulted from a connection while a commit ack was in-flight. That doesn't qualify as a data-consistency error, it means the client has to check if the data is present after reconnecting. But in no cases would the client reconnect to find that there were acknowledged-as-committed records that were missing or stale. In no cases would the client find that responded-as-rolled-back data was actually committed. This is very, very different than what is seen with MongoDB.
  - hendzen 12y ago
    I don't think the results of Aphyr's MongoDB and postgres experiments are directly comparable. In the OP, MongoDB was run in a 5 node replicated configuration. In the post you reference, the experiment was run against a single postgres node. Furthermore, the postgres experiment only checked that no writes were lost. As Aphyr acknowledges, MongoDB did not lose any writes with "majority" write concern. The postgres experiment did not include the verification of a linearizable history of reads, which is what the bulk of the OP is about. I'd like to see a similar experiment run against a replicated postgres configuration with auto-failover.
  - tracker1 12y ago
    Now try it again with PostgreSQL's built in sharding or replication functionality... oh, wait.
    - zapov 12y ago
      The parent post correctly pointed out that including Postgres in that list is misleading at least.
  - threeseed 12y ago
    I was talking about data consistency more broadly. "However, two writes (215 and 218) succeeded, even though they threw an exception claiming that a failure occurred". This obviously isn't ideal behaviour from the perspective of a developer but isn't necessarily not correct. My broader point was that you need to assume the worst from your database in the application layer. Which I think you missed.
    - zapov 12y ago
      I read your post as FUD regarding Postgres. That article has various issues, for example, calling Postgres commit protocol as a special case of two phase commit is not really correct. Postgres has 2pc: http://www.postgresql.org/docs/9.2/static/sql-prepare-transaction.html http://www.postgresql.org/docs/9.2/static/sql-prepare-transa... but that was not tested. The described behavior is "expected" and "understood". Saying that "you should assume worst from your database" is not something I would ever use for describing DB with ACID semantics.
      - threeseed 12y ago
        I wasn't spreading FUD about anything. From the article there was issues with every databases expected behaviour. My point again was that you should expect and manage failure in your application layer. It's what sensible architecture looks like. And ACID does NOT guarentee that you will not lose data. It is a theory not an implementation. I have lost data with both Oracle and Teradata due to bugs.
        zapov 12y ago
        Well, you said that Postgres had data consistency errors (referencing Aphyr articles). This is not true (at least regarding that article). Aphyr article about Postgres could be renamed to call-me-maybe-acid-db-over-the-network and could remain the same.
- rdtsc 12y ago
  Actually, just use Riak. Those people know distributed bases. https://aphyr.com/posts/285-call-me-maybe-riak https://aphyr.com/posts/285-call-me-maybe-riak Now I hear they support consistency as well.
  - iconfluence 12y ago
    I think at one point the employees of Basho knew how to write distributed DBs - Riak is the most advanced AP DB from a distributed systems theory perspective. However, in recent months, their CEO, CTO and Chief Architect have left, as well as many of their prominent engineers. Worryingly, the new CTO seems content to make inane comments about "Data Gravity" [0]. [0] - http://www.kdnuggets.com/2015/03/interview-dave-mccrory-basho-data-gravity.html http://www.kdnuggets.com/2015/03/interview-dave-mccrory-bash...
    - ChuckMcM 12y ago
      oooh, that is scary.
    - scurvy 12y ago
      Indeed. We're evaluating Riak CS and Swift. In a vacuum, I'd choose Riak CS any day. Knowing all of the drama at Basho, we're in a holding pattern at best and leaning (reluctantly) towards Swift. Basho needs to sell damn fast or just call it a day and open source the enterprise version.
      - jodah 12y ago
        Drama aside, Riak and Swift are not in the same league.
        scurvy 12y ago
        Riak CS is. I said Riak CS, not Riak.
      - rdtsc 12y ago
        This is kind of surprising to hear. I had no idea. Well in that case have you heard about LeoFS? Wonder if it overlaps with any of the Riak CS features for you. http://leo-project.net/leofs/ http://leo-project.net/leofs/
- cpks 12y ago
  You misunderstand the results. PostgreSQL behaves as expected, and indeed, the only way it can behave. That's the Two General's Problem (http://en.wikipedia.org/wiki/Two_Generals%27_Problem http://en.wikipedia.org/wiki/Two_Generals%27_Problem). There is no way to solve it. PostgreSQL does as well as theoretically possible. I understand you lost data with Oracle and Teradata. 1. Most big corporate vendors do not have systems which are very well designed. (1) The sale is made at a business level. (2) Most Oracle customers are not very tech companies, and have mixed quality employees. As a result, there is little pressure on building a robust, correct product, rather than one which meets a feature checklist. In addition, Oracle doesn't really recruit smart people (I know people who work there). It's just not very robust compared to something like PostgreSQL, which was written by Stonebraker, a legendary computer science professor and entrepreneur. 2. Still, more likely, the reason you lost data is because you didn't know what you were doing. Words like Eventual Consistency, ACID, Two-General's Problem, etc. are not just abstract. They have strict, formal meanings, and you need to understand what they do and do not guarantee. Otherwise, you will lose data again.
  - threeseed 12y ago
    > the reason you lost data is because you didn't know what you were doing Missed this reply and thought it was funny. I work for one of the world's largest retailers and we are one of both Oracle's and Teradata's most loved customers. We have 4 Teradata DBAs provided BY Teradata amongst a team of 20 SQL Developers. We aren't messing around. What YOU don't seem to understand is that bugs in your database can cause data loss. ACID or Strong Consistency will not save you.
    - cpks 11y ago
      That's not exactly the sort of team where you'd expect MIT Ph.Ds to work. It's precisely big teams of mediocre people who run into issues based on not knowing exactly what the database is doing and how it's supposed to work that lead to data corruption due to misuse. It's almost always a boring problem (e.g. retail business software). It's almost always a clunky commercial "enterprise" solution (e.g. Oracle). It's almost always a big team. I'm not sure if I even need to go into application engineers -- you put your best and brightest into the core product, and solutions typically gets those that can't quite cut it there. Regardless, your comment was about PostgreSQL, not Oracle. Oracle is a giant piece of software written by a corporation with over 100,000 employees. Something like that is bound to have bugs, and it has bugs indeed. Data corruption with Oracle certainly happens. PostgreSQL is written by a small, ultra-elite team. It's a much smaller codebase, so an expert developer can understand how the whole system works. There's a big difference in robustness between the two. Of course database bugs can cause corruption. I've certainly had MonogDB eat my data. But the level of robustness of different databases is very different. There are many databases which are essentially bug-free. If you're losing data with PostgreSQL, odds are you're the one losing the data, not PostgreSQL.
DenisM 12y ago
It seems easier to me to just go with a reputable database vendor.
- jxf 12y ago
  That island of safety is a rapidly shrinking one. Jepsen covers more ground with each new post from Kyle. :)
  - aidanhs 12y ago
    Has there been any CP system passing a call-me-maybe test first time apart from Zookeeper [1]? [1] https://aphyr.com/posts/291-call-me-maybe-zookeeper https://aphyr.com/posts/291-call-me-maybe-zookeeper
    - ddlatham 12y ago
      HBase. With the caveats that it's piggy backing off the success of Zookeeper, and the tests were not run by Kyle himself: https://eng.yammer.com/call-me-maybe-hbase/ https://eng.yammer.com/call-me-maybe-hbase/
      - yobennett 12y ago
        Note that Robert Yokota's addendum[1] points out that HBase "cannot achieve both consistency and availability." His earlier results did not take into account that HBase clients continuously retry failed ops. (According to Nicolas Liochon, upon its death a server's regions are moved to another node.) Failures start rolling in once network partition(s) extend beyond the configured timeout. Kyle [2] was not impressed: "During the network partition, no requests are successful" is not the best result for a CP system, IMO." HBase should provide partial availability in the face partitions. [1] http://eng.yammer.com/call-me-maybe-hbase-addendum/ http://eng.yammer.com/call-me-maybe-hbase-addendum/ [2] https://twitter.com/aphyr/status/509841011816665088 https://twitter.com/aphyr/status/509841011816665088
        12y ago

by jxf 12y ago

threeseed 12y ago

Actually the broader lesson would be to assume the worst in your application layer and try and remediate/verify wherever possible. If you look at his articles: Redis, PostgreSQL, Cassandra, ElasticSearch etc all had data consistency errors. And none of those have vendors making any claims. It's pretty sobering to say the least.

chatman 12y ago
Apache Solr's jepsen tests have had very good results.
ak4g 12y ago
Um, this is the postgres article: https://aphyr.com/posts/282-call-me-maybe-postgres https://aphyr.com/posts/282-call-me-maybe-postgres There were no acknowledged writes lost. The only unacked-but-successful writes resulted from a connection while a commit ack was in-flight. That doesn't qualify as a data-consistency error, it means the client has to check if the data is present after reconnecting. But in no cases would the client reconnect to find that there were acknowledged-as-committed records that were missing or stale. In no cases would the client find that responded-as-rolled-back data was actually committed. This is very, very different than what is seen with MongoDB.
- hendzen 12y ago
  I don't think the results of Aphyr's MongoDB and postgres experiments are directly comparable. In the OP, MongoDB was run in a 5 node replicated configuration. In the post you reference, the experiment was run against a single postgres node. Furthermore, the postgres experiment only checked that no writes were lost. As Aphyr acknowledges, MongoDB did not lose any writes with "majority" write concern. The postgres experiment did not include the verification of a linearizable history of reads, which is what the bulk of the OP is about. I'd like to see a similar experiment run against a replicated postgres configuration with auto-failover.
- tracker1 12y ago
  Now try it again with PostgreSQL's built in sharding or replication functionality... oh, wait.
  - zapov 12y ago
    The parent post correctly pointed out that including Postgres in that list is misleading at least.
- threeseed 12y ago
  I was talking about data consistency more broadly. "However, two writes (215 and 218) succeeded, even though they threw an exception claiming that a failure occurred". This obviously isn't ideal behaviour from the perspective of a developer but isn't necessarily not correct. My broader point was that you need to assume the worst from your database in the application layer. Which I think you missed.
  - zapov 12y ago
    I read your post as FUD regarding Postgres. That article has various issues, for example, calling Postgres commit protocol as a special case of two phase commit is not really correct. Postgres has 2pc: http://www.postgresql.org/docs/9.2/static/sql-prepare-transaction.html http://www.postgresql.org/docs/9.2/static/sql-prepare-transa... but that was not tested. The described behavior is "expected" and "understood". Saying that "you should assume worst from your database" is not something I would ever use for describing DB with ACID semantics.
    - threeseed 12y ago
      I wasn't spreading FUD about anything. From the article there was issues with every databases expected behaviour. My point again was that you should expect and manage failure in your application layer. It's what sensible architecture looks like. And ACID does NOT guarentee that you will not lose data. It is a theory not an implementation. I have lost data with both Oracle and Teradata due to bugs.
      - zapov 12y ago
        Well, you said that Postgres had data consistency errors (referencing Aphyr articles). This is not true (at least regarding that article). Aphyr article about Postgres could be renamed to call-me-maybe-acid-db-over-the-network and could remain the same.
rdtsc 12y ago
Actually, just use Riak. Those people know distributed bases. https://aphyr.com/posts/285-call-me-maybe-riak https://aphyr.com/posts/285-call-me-maybe-riak Now I hear they support consistency as well.
- iconfluence 12y ago
  I think at one point the employees of Basho knew how to write distributed DBs - Riak is the most advanced AP DB from a distributed systems theory perspective. However, in recent months, their CEO, CTO and Chief Architect have left, as well as many of their prominent engineers. Worryingly, the new CTO seems content to make inane comments about "Data Gravity" [0]. [0] - http://www.kdnuggets.com/2015/03/interview-dave-mccrory-basho-data-gravity.html http://www.kdnuggets.com/2015/03/interview-dave-mccrory-bash...
  - ChuckMcM 12y ago
    oooh, that is scary.
  - scurvy 12y ago
    Indeed. We're evaluating Riak CS and Swift. In a vacuum, I'd choose Riak CS any day. Knowing all of the drama at Basho, we're in a holding pattern at best and leaning (reluctantly) towards Swift. Basho needs to sell damn fast or just call it a day and open source the enterprise version.
    - jodah 12y ago
      Drama aside, Riak and Swift are not in the same league.
      - scurvy 12y ago
        Riak CS is. I said Riak CS, not Riak.
    - rdtsc 12y ago
      This is kind of surprising to hear. I had no idea. Well in that case have you heard about LeoFS? Wonder if it overlaps with any of the Riak CS features for you. http://leo-project.net/leofs/ http://leo-project.net/leofs/
cpks 12y ago
You misunderstand the results. PostgreSQL behaves as expected, and indeed, the only way it can behave. That's the Two General's Problem (http://en.wikipedia.org/wiki/Two_Generals%27_Problem http://en.wikipedia.org/wiki/Two_Generals%27_Problem). There is no way to solve it. PostgreSQL does as well as theoretically possible. I understand you lost data with Oracle and Teradata. 1. Most big corporate vendors do not have systems which are very well designed. (1) The sale is made at a business level. (2) Most Oracle customers are not very tech companies, and have mixed quality employees. As a result, there is little pressure on building a robust, correct product, rather than one which meets a feature checklist. In addition, Oracle doesn't really recruit smart people (I know people who work there). It's just not very robust compared to something like PostgreSQL, which was written by Stonebraker, a legendary computer science professor and entrepreneur. 2. Still, more likely, the reason you lost data is because you didn't know what you were doing. Words like Eventual Consistency, ACID, Two-General's Problem, etc. are not just abstract. They have strict, formal meanings, and you need to understand what they do and do not guarantee. Otherwise, you will lose data again.
- threeseed 12y ago
  > the reason you lost data is because you didn't know what you were doing Missed this reply and thought it was funny. I work for one of the world's largest retailers and we are one of both Oracle's and Teradata's most loved customers. We have 4 Teradata DBAs provided BY Teradata amongst a team of 20 SQL Developers. We aren't messing around. What YOU don't seem to understand is that bugs in your database can cause data loss. ACID or Strong Consistency will not save you.
  - cpks 11y ago
    That's not exactly the sort of team where you'd expect MIT Ph.Ds to work. It's precisely big teams of mediocre people who run into issues based on not knowing exactly what the database is doing and how it's supposed to work that lead to data corruption due to misuse. It's almost always a boring problem (e.g. retail business software). It's almost always a clunky commercial "enterprise" solution (e.g. Oracle). It's almost always a big team. I'm not sure if I even need to go into application engineers -- you put your best and brightest into the core product, and solutions typically gets those that can't quite cut it there. Regardless, your comment was about PostgreSQL, not Oracle. Oracle is a giant piece of software written by a corporation with over 100,000 employees. Something like that is bound to have bugs, and it has bugs indeed. Data corruption with Oracle certainly happens. PostgreSQL is written by a small, ultra-elite team. It's a much smaller codebase, so an expert developer can understand how the whole system works. There's a big difference in robustness between the two. Of course database bugs can cause corruption. I've certainly had MonogDB eat my data. But the level of robustness of different databases is very different. There are many databases which are essentially bug-free. If you're losing data with PostgreSQL, odds are you're the one losing the data, not PostgreSQL.

DenisM 12y ago

It seems easier to me to just go with a reputable database vendor.

jxf 12y ago
That island of safety is a rapidly shrinking one. Jepsen covers more ground with each new post from Kyle. :)
- aidanhs 12y ago
  Has there been any CP system passing a call-me-maybe test first time apart from Zookeeper [1]? [1] https://aphyr.com/posts/291-call-me-maybe-zookeeper https://aphyr.com/posts/291-call-me-maybe-zookeeper
  - ddlatham 12y ago
    HBase. With the caveats that it's piggy backing off the success of Zookeeper, and the tests were not run by Kyle himself: https://eng.yammer.com/call-me-maybe-hbase/ https://eng.yammer.com/call-me-maybe-hbase/
    - yobennett 12y ago
      Note that Robert Yokota's addendum[1] points out that HBase "cannot achieve both consistency and availability." His earlier results did not take into account that HBase clients continuously retry failed ops. (According to Nicolas Liochon, upon its death a server's regions are moved to another node.) Failures start rolling in once network partition(s) extend beyond the configured timeout. Kyle [2] was not impressed: "During the network partition, no requests are successful" is not the best result for a CP system, IMO." HBase should provide partial availability in the face partitions. [1] http://eng.yammer.com/call-me-maybe-hbase-addendum/ http://eng.yammer.com/call-me-maybe-hbase-addendum/ [2] https://twitter.com/aphyr/status/509841011816665088 https://twitter.com/aphyr/status/509841011816665088
      - 12y ago