6 ms·

Jepsen: Distributed Systems Safety Analysis

285 points by luu 11y ago

MCRed 11y ago
I really would like to see Couchbase analyzed... and for aphyr to marry me.
- mbesto 11y ago
  I'm actually genuinely interested in that as well (Not the marrying part), especially considering aphyr and Damien Katz went at it about a year ago: https://twitter.com/damienkatz/status/527588456747499520 https://twitter.com/damienkatz/status/527588456747499520
  - willchen 11y ago
    Educational back-and-forth, although unnecessarily biting. TL;DR - Aphyr thinks Damien Katz never read this: https://en.wikipedia.org/wiki/Consensus_%28computer_science%29#Problem_description https://en.wikipedia.org/wiki/Consensus_%28computer_science%...
  - augustl 11y ago
    What I find really interesting is that while it appears to me that Katz has very sparse knowledge of databases, he has still managed to create one or two of them, and successfully run a company that sells one. I don't like the thought I had next, but: perhaps knowing about consensus algos and papers actually doesn't matter?
    - Saavedro 11y ago
      well, matter for what? For building software that seems to work, sure, no. But for being able to speak to its capabilities -- or even know what they are, especially in the case of distributed systems it's definitely important. this is why so many of the new DB offerings have docs that are either misleading or wrong -- often the doc writers or even the devs themselves don't know what invariants their system can guarantee and assume several that it can't.
    - DomreiRoam 11y ago
      Depends on the problem you want to solve. CouchDB is a very good document database when you have more read than write and when writer are mostly writing on different documents. If I understand correctly replication conflict resolution is up to the application. So knowing about consensus algo is unnecessary for CouchDB because he doesn't support it. I think it's smart to know a little about the literature; enough to know what battle you don't want to fight. It is not smart to try to develop an engine on the basis that it will need to beat the law of thermodynamic to be possible. But also, “They did not know it was impossible so they did it”, Mark Twain
      - krugster 11y ago
        FWIW, Couchbase != CouchDB: http://www.couchbase.com/couchbase-vs-couchdb http://www.couchbase.com/couchbase-vs-couchdb +1 For Jepson on Couchbase
    - Jweb_Guru 11y ago
      Kyle's point there is that it does matter, in that CouchDB can't achieve consensus, and would have failed Jepsen immediately.
- deleted 11y ago
  [deleted]
votemedown 11y ago
The only thing that passes Jespen: vim.
- deleted 11y ago
  [deleted]
JoelJacobson 11y ago
Why isn't PostgreSQL in the list on the first page? It is in the [blog posts] page though. I would love to see the multixact data corruption problems introduced in 9.3 analyzed, and see if he can verify them to be solved in the latest version.
- rdtsc 11y ago
  (Not author) but from what I remember PostreSQL was a bit of a funny case. It was about partitions between clients and a single database server. In most other cases, it was analysis of distributed back-end server instances. Not that those failures are not important but it is just harder to compare it seems with the others.
  - lomnakkus 11y ago
    Yeah, basically with PostgreSQL it boils down to the fact that even if a client receives an error on commit, its writes may in fact have succeeded -- and it cannot assume that they haven't. (Since the final "ack" that everything went well may not make it.) As an example: This means that idempotency is really important if you're doing any kind of retries on commit failure. EDIT: Another example: If you're doing application-level optimistic concurrency control (using e.g. an incrementing version number) you can end up in a situation where an update appears to conflict with itself. Usually, though, this isn't a big concern since client<->server paritions are usually catastrophic anyway. (Off the top of my head that's what I can recall -- there may be other things to be gleaned from the original article.) [1] At least for all practical purposes.
    - biokoda 11y ago
      > Yeah, basically with PostgreSQL it boils down to the fact that even if a client receives an error on commit, its writes may in fact have succeeded -- and it cannot assume that they haven't. This is true for almost every database. Otherwise they would require a distributed transaction between the client and server.
      - 11y ago
willchen 11y ago
I'd be very interested to see RethinkDB analyzed, particularly with the 2.1 release promising high availability through Raft. RethinkDB and Aphyr have talked about doing Jepsen tests for it, but I'm not sure where that's landed (https://github.com/rethinkdb/rethinkdb/issues/1493 https://github.com/rethinkdb/rethinkdb/issues/1493).
- coffeemug 11y ago
  Slava @ Rethink here. Talked to Kyle yesterday, he's doing RethinkDB analysis next. You should see something in a month or two. EDIT: don't mean to speak for him though; I hope this is ok.
  - Spiritus 11y ago
    Will that be on 2.1 or the upcoming 2.2?
    - coffeemug 11y ago
      Probably on the 2.1 branch, but it shouldn't matter too much -- the clustering codebase is very stable and there aren't any major changes to it between releases.
jhasse 11y ago
Please make the title more verbose.
- deleted 11y ago
  [deleted]
- HNLogInsSuckAss 11y ago
  Amen. These types of posts are annoying as shit, because they seem smug and presumptuous. Not reading this, because I'm not going to run around and do the poster's work for him. The attitude!
- 885895 11y ago
  Either OP or mods have corrected the title now.
striking 11y ago
A lot of people are noting their interest in seeing a certain database tested. Sorry to say so, but: >> Can you test X next? > Tests take about a month. I do take suggestions into consideration, but I can't promise you anything. Backlog is a few years long at this point. (from https://aphyr.com/about https://aphyr.com/about)
- willchen 11y ago
  Fair point. For RethinkDB, I think there's a good business case for getting it tested sooner than later (on both sides). For aphyr who's leaving Stripe, I think it would be reasonable for RethinkDB to compensate him for an unbiased study, given that they have $12M+ in funding. A positive report could generate a lot of attention for RethinkDB and give confidence to larger enterprises who are very hesitant of trying anything remotely "new" or "experimental" in the DB space.
  - rdtsc 11y ago
    > A positive report could generate a lot of attention for RethinkDB A negative report could generate the opposite though. Are they ready to bet the new $12M+ on that? Then also if they compensate him, not sure how unbiased others will think (and we are talking about perception here) the study will be.
    - coffeemug 11y ago
      > A negative report could generate the opposite though. Are they ready to bet the new $12M+ on that? Slava at RethinkDB here. Yes, we are.
      - rdtsc 11y ago
        Good answer! I want to see the results.
    - BinaryIdiot 11y ago
      > A negative report could generate the opposite though. Are they ready to bet the new $12M+ on that? I disagree; it all depends on how they act upon it. So worst case scenario he finds something deeply flawed. What does RethinkDB do? Work to fix it while communicating and being responsive to the development community. Show developers that you're humble and, more importantly, you listen and fix things. That would generate a good amount of good will, in my opinion.
headcanon 11y ago
I wonder if Carly Rae Jepsen is aware of the legacy of her song in the software world...
- TeMPOraL 11y ago
  Somebody call her maybe?
- annnnd 11y ago
  Especially as the product is named after her... This for sure doesn't hurt her image, but some lawyers might have a problem with this.
HNLogInsSuckAss 11y ago
Type a description, so people can see what you're posting about. Is this somehow made difficult by the site?
OMGWTF 11y ago
> Flash plugin missing > Get the latest Flash player to view this content No, I won't. This scares me too much: https://www.cvedetails.com/vulnerability-list/vendor_id-53/product_id-6761/Adobe-Flash-Player.html https://www.cvedetails.com/vulnerability-list/vendor_id-53/p...
- annnnd 11y ago
  +1. Not sure why anyone would want to play video (I guess it's a video? I don't do Flash either) through Flash nowadays... Isn't it easier and better to just use HTML video tag?
  - biokoda 11y ago
    With live video you don't really have a choice if you want compatibility with all browsers.
    - annnnd 11y ago
      Not really - there is no flash on (most?) mobile devices. You can also use JS detection. For instance, Youtube uses Flash unless you allow google.com & googlevideo.com, then it uses HTML video tag. It's not like it costs you anything to add HTML video tag.
      - biokoda 11y ago
        live video. There is no cross browser compatible format for live streaming.
        annnnd 11y ago
        Sorry, missed "live" word. I don't have any (recent) experience with that, so can't comment on that. But in case of OP's page the video is not live.
      - mdaniel 11y ago
        You can also submit your preference to YouTube, even if you _do_ have Flash: https://www.youtube.com/html5 https://www.youtube.com/html5
doomrobo 11y ago
Was Jespen developed at Stripe? If so, how can aphyr get the rights to use Jespen for individual contracting work?
- buryat 11y ago
  https://github.com/aphyr/jepsen/commits/master?page=14 https://github.com/aphyr/jepsen/commits/master?page=14
- masklinn 11y ago
  Jepsen was developed by aphyr/kyle. Stripe hired him early this year (January 2015[0]), I understand in part to support his Jepsen work, but he's been working on it since 2013[1]. So to answer the question, no Jepsen was not "developed at stripe" though part of its development probably happened there. [0] https://twitter.com/avibryant/status/561315494204952579 https://twitter.com/avibryant/status/561315494204952579 [1] https://aphyr.com/posts/281-call-me-maybe-carly-rae-jepsen-and-the-perils-of-network-partitions https://aphyr.com/posts/281-call-me-maybe-carly-rae-jepsen-a...
eddd 11y ago
I really dig the Riak test. One of the best comparison of LWW and CRDT. Also very good conclusion, tl; dr: If you don't use idempotent writes, you are going to have bad time.
- rspeer 11y ago
  Well, Last Write Wins is idempotent too. I would say the conclusion is that it's important that your writes are commutative and associative. (The problem, then, is that data structures with all these properties are hard to find.)
felixgallo 11y ago
On a human note: Kyle is brave as fuck for stepping out into the black unknown yawning abyss to try to do this under his own banner. Bravo, Kyle, and may you find incredible success.
- mdaniel 11y ago
  The other side to that coin is that these reports are always in the top of the front page of HN, and likely on Reddit, also. They are popular because they are great, and I don't want to live in a world where people can't be paid fairly to do great things.

285 points by luu 11y ago

MCRed 11y ago

I really would like to see Couchbase analyzed... and for aphyr to marry me.

mbesto 11y ago
I'm actually genuinely interested in that as well (Not the marrying part), especially considering aphyr and Damien Katz went at it about a year ago: https://twitter.com/damienkatz/status/527588456747499520 https://twitter.com/damienkatz/status/527588456747499520
- willchen 11y ago
  Educational back-and-forth, although unnecessarily biting. TL;DR - Aphyr thinks Damien Katz never read this: https://en.wikipedia.org/wiki/Consensus_%28computer_science%29#Problem_description https://en.wikipedia.org/wiki/Consensus_%28computer_science%...
- augustl 11y ago
  What I find really interesting is that while it appears to me that Katz has very sparse knowledge of databases, he has still managed to create one or two of them, and successfully run a company that sells one. I don't like the thought I had next, but: perhaps knowing about consensus algos and papers actually doesn't matter?
  - Saavedro 11y ago
    well, matter for what? For building software that seems to work, sure, no. But for being able to speak to its capabilities -- or even know what they are, especially in the case of distributed systems it's definitely important. this is why so many of the new DB offerings have docs that are either misleading or wrong -- often the doc writers or even the devs themselves don't know what invariants their system can guarantee and assume several that it can't.
  - DomreiRoam 11y ago
    Depends on the problem you want to solve. CouchDB is a very good document database when you have more read than write and when writer are mostly writing on different documents. If I understand correctly replication conflict resolution is up to the application. So knowing about consensus algo is unnecessary for CouchDB because he doesn't support it. I think it's smart to know a little about the literature; enough to know what battle you don't want to fight. It is not smart to try to develop an engine on the basis that it will need to beat the law of thermodynamic to be possible. But also, “They did not know it was impossible so they did it”, Mark Twain
    - krugster 11y ago
      FWIW, Couchbase != CouchDB: http://www.couchbase.com/couchbase-vs-couchdb http://www.couchbase.com/couchbase-vs-couchdb +1 For Jepson on Couchbase
  - Jweb_Guru 11y ago
    Kyle's point there is that it does matter, in that CouchDB can't achieve consensus, and would have failed Jepsen immediately.
deleted 11y ago
[deleted]

votemedown 11y ago

The only thing that passes Jespen: vim.

deleted 11y ago
[deleted]

JoelJacobson 11y ago

Why isn't PostgreSQL in the list on the first page? It is in the [blog posts] page though. I would love to see the multixact data corruption problems introduced in 9.3 analyzed, and see if he can verify them to be solved in the latest version.

rdtsc 11y ago
(Not author) but from what I remember PostreSQL was a bit of a funny case. It was about partitions between clients and a single database server. In most other cases, it was analysis of distributed back-end server instances. Not that those failures are not important but it is just harder to compare it seems with the others.
- lomnakkus 11y ago
  Yeah, basically with PostgreSQL it boils down to the fact that even if a client receives an error on commit, its writes may in fact have succeeded -- and it cannot assume that they haven't. (Since the final "ack" that everything went well may not make it.) As an example: This means that idempotency is really important if you're doing any kind of retries on commit failure. EDIT: Another example: If you're doing application-level optimistic concurrency control (using e.g. an incrementing version number) you can end up in a situation where an update appears to conflict with itself. Usually, though, this isn't a big concern since client<->server paritions are usually catastrophic anyway. (Off the top of my head that's what I can recall -- there may be other things to be gleaned from the original article.) [1] At least for all practical purposes.
  - biokoda 11y ago
    > Yeah, basically with PostgreSQL it boils down to the fact that even if a client receives an error on commit, its writes may in fact have succeeded -- and it cannot assume that they haven't. This is true for almost every database. Otherwise they would require a distributed transaction between the client and server.
    - 11y ago

willchen 11y ago

I'd be very interested to see RethinkDB analyzed, particularly with the 2.1 release promising high availability through Raft. RethinkDB and Aphyr have talked about doing Jepsen tests for it, but I'm not sure where that's landed (https://github.com/rethinkdb/rethinkdb/issues/1493 https://github.com/rethinkdb/rethinkdb/issues/1493).

coffeemug 11y ago
Slava @ Rethink here. Talked to Kyle yesterday, he's doing RethinkDB analysis next. You should see something in a month or two. EDIT: don't mean to speak for him though; I hope this is ok.
- Spiritus 11y ago
  Will that be on 2.1 or the upcoming 2.2?
  - coffeemug 11y ago
    Probably on the 2.1 branch, but it shouldn't matter too much -- the clustering codebase is very stable and there aren't any major changes to it between releases.

jhasse 11y ago

Please make the title more verbose.

deleted 11y ago
[deleted]
HNLogInsSuckAss 11y ago
Amen. These types of posts are annoying as shit, because they seem smug and presumptuous. Not reading this, because I'm not going to run around and do the poster's work for him. The attitude!
885895 11y ago
Either OP or mods have corrected the title now.

striking 11y ago

A lot of people are noting their interest in seeing a certain database tested. Sorry to say so, but: >> Can you test X next? > Tests take about a month. I do take suggestions into consideration, but I can't promise you anything. Backlog is a few years long at this point. (from https://aphyr.com/about https://aphyr.com/about)

willchen 11y ago
Fair point. For RethinkDB, I think there's a good business case for getting it tested sooner than later (on both sides). For aphyr who's leaving Stripe, I think it would be reasonable for RethinkDB to compensate him for an unbiased study, given that they have $12M+ in funding. A positive report could generate a lot of attention for RethinkDB and give confidence to larger enterprises who are very hesitant of trying anything remotely "new" or "experimental" in the DB space.
- rdtsc 11y ago
  > A positive report could generate a lot of attention for RethinkDB A negative report could generate the opposite though. Are they ready to bet the new $12M+ on that? Then also if they compensate him, not sure how unbiased others will think (and we are talking about perception here) the study will be.
  - coffeemug 11y ago
    > A negative report could generate the opposite though. Are they ready to bet the new $12M+ on that? Slava at RethinkDB here. Yes, we are.
    - rdtsc 11y ago
      Good answer! I want to see the results.
  - BinaryIdiot 11y ago
    > A negative report could generate the opposite though. Are they ready to bet the new $12M+ on that? I disagree; it all depends on how they act upon it. So worst case scenario he finds something deeply flawed. What does RethinkDB do? Work to fix it while communicating and being responsive to the development community. Show developers that you're humble and, more importantly, you listen and fix things. That would generate a good amount of good will, in my opinion.

headcanon 11y ago

I wonder if Carly Rae Jepsen is aware of the legacy of her song in the software world...

TeMPOraL 11y ago
Somebody call her maybe?
annnnd 11y ago
Especially as the product is named after her... This for sure doesn't hurt her image, but some lawyers might have a problem with this.

HNLogInsSuckAss 11y ago

Type a description, so people can see what you're posting about. Is this somehow made difficult by the site?

OMGWTF 11y ago

> Flash plugin missing > Get the latest Flash player to view this content No, I won't. This scares me too much: https://www.cvedetails.com/vulnerability-list/vendor_id-53/product_id-6761/Adobe-Flash-Player.html https://www.cvedetails.com/vulnerability-list/vendor_id-53/p...

annnnd 11y ago
+1. Not sure why anyone would want to play video (I guess it's a video? I don't do Flash either) through Flash nowadays... Isn't it easier and better to just use HTML video tag?
- biokoda 11y ago
  With live video you don't really have a choice if you want compatibility with all browsers.
  - annnnd 11y ago
    Not really - there is no flash on (most?) mobile devices. You can also use JS detection. For instance, Youtube uses Flash unless you allow google.com & googlevideo.com, then it uses HTML video tag. It's not like it costs you anything to add HTML video tag.
    - biokoda 11y ago
      live video. There is no cross browser compatible format for live streaming.
      - annnnd 11y ago
        Sorry, missed "live" word. I don't have any (recent) experience with that, so can't comment on that. But in case of OP's page the video is not live.
    - mdaniel 11y ago
      You can also submit your preference to YouTube, even if you _do_ have Flash: https://www.youtube.com/html5 https://www.youtube.com/html5

doomrobo 11y ago

Was Jespen developed at Stripe? If so, how can aphyr get the rights to use Jespen for individual contracting work?

buryat 11y ago
https://github.com/aphyr/jepsen/commits/master?page=14 https://github.com/aphyr/jepsen/commits/master?page=14
masklinn 11y ago
Jepsen was developed by aphyr/kyle. Stripe hired him early this year (January 2015[0]), I understand in part to support his Jepsen work, but he's been working on it since 2013[1]. So to answer the question, no Jepsen was not "developed at stripe" though part of its development probably happened there. [0] https://twitter.com/avibryant/status/561315494204952579 https://twitter.com/avibryant/status/561315494204952579 [1] https://aphyr.com/posts/281-call-me-maybe-carly-rae-jepsen-and-the-perils-of-network-partitions https://aphyr.com/posts/281-call-me-maybe-carly-rae-jepsen-a...

eddd 11y ago

I really dig the Riak test. One of the best comparison of LWW and CRDT. Also very good conclusion, tl; dr: If you don't use idempotent writes, you are going to have bad time.

rspeer 11y ago
Well, Last Write Wins is idempotent too. I would say the conclusion is that it's important that your writes are commutative and associative. (The problem, then, is that data structures with all these properties are hard to find.)

felixgallo 11y ago

On a human note: Kyle is brave as fuck for stepping out into the black unknown yawning abyss to try to do this under his own banner. Bravo, Kyle, and may you find incredible success.

mdaniel 11y ago
The other side to that coin is that these reports are always in the top of the front page of HN, and likely on Reddit, also. They are popular because they are great, and I don't want to live in a world where people can't be paid fairly to do great things.