3 ms·

I have a pretty much opposite perspective on Jepsen than most of the folks here. My feeling is that essentially no distributed system is perfect or without trad

I have a pretty much opposite perspective on Jepsen than most of the folks here. My feeling is that essentially no distributed system is perfect or without trade offs (and certainly no single node system is perfect and wothout tradeoffs) and Jepsen posts basically make that clear over and over again like some form of techie outrage porn... but the tone and implication is as if somehow there is some alternative panacea without tradeoffs... and I find that a little bit misleading. Basically an astute reader of Jepsen testing may deduce "I need to use a single node system" which is one option but without the availability characteristics modern users usually want

belter 5y ago
That is not what Jepsen demonstrates. Instead it shows systems making claims that are not supported by their implementation and/or algorithms. If you want to take on your claim/suggestion then those systems just have to stop making the claims he is testing for. PS: There is also another trend, that is System claiming they fixed the issues Jepsen finds, without submitting themselves again for analysis...but I digress now...
- redwood 5y ago
  I respectfully disagree. Human language and characteristics around distributed data stores in the real world have inherent ambiguity implicit in them that Jepsen in my view pretends isn't the case. Here's an analogy that may help me communicate how I feel since I realize my message is not landing: Let's say I'm buying a condo in the San Francisco Bay area. And let's say the building that I'm looking to buy in advertisers that it is historic but seismically retrofitted. Then say Jepsen-earthquake-test comes in and shakes the ground beneath the building and shows that the building indeed collapses with enough of an earthquake: would that or would that not be enough information for me to decide whether or not the seismically retrofitted building is good enough for my needs? There's a lot of ambiguity in answer that question.
  - belter 5y ago
    Although I agree with your point, on language and its ambiguity, I would argue that is a different claim of the one you made above, and that I replied to. When he demonstrated that Riak was dropping 30-70% of writes, even with the strongest consistency settings, or that Mongo had multiple scenarios of data loss, we are not talking about the subtleties of the English idiom
    - redwood 5y ago
      But the contrived nature of Jepsen tests is so different than the real world. In the real world no system behaves exactly the same as it was designed to behave; the real world has cosmic rays, earthquakes and everything in between. So no statement about a software system can ever be simply accepted as a fundamental fact and so to prove that software systems break is misleading for what most users need to understand about said systems. A Jepsen style test optimized not for bending to show where things break but instead for showing likely real world style situations with an eval of which are most likely to arise would be far more valuable for people
      - eklavya 5y ago
        We do know that some databases fare much better than others and that’s useful to many. In your analogy it would be many builders claiming their buildings to be earthquake proof with only some actually being it. Thanks to Jepsen, customers know, where to buy.
        redwood 5y ago
        I'm genuinely curious to hear which database is the winner from your perspective based on Jepsen's insight?
        aphyr 5y ago
        If you find out, let me know! Folks are always asking me "What database is best?" and I'm just shuffling my feet and going "Uhhhh, well, systems are really hard to build correctly, there's lots of tradeoffs, you probably want something with a proven replication model under the hood but so much depends on workload and operational characteristics beyond just safety... If you find a database you love please tell me about it because I like hearing people's stories..." FWIW, I think you're raising really good questions in this thread. Qualitative safety is highly contextual, depending on fault model, significance of individual operations, concurrency, throughput, latency demands, operational characteristics, data volume, etc etc., and I try to touch on that in the "Toward a Culture of Safety" section in this report. Hopefully that resonated for you.
cmiles74 5y ago
I disagree that the analysis articles from Jepsen, taken as whole, are arguing for a "perfect distributed system without trade offs". In my experience these articles are fairly even-handed, indeed, many products make some changes based on the analysis and then contract for a follow-up. That the "tone and implication is as if somehow there is some alternative panacea without tradeoffs..." is not something I observed in this analysis or the previous ones that I have read. In my opinion the Rethink DB 2.1.5 report went fairly well for that project. If the claims are aligned with the reality of the product, it's clear the Jepsen report will highlight that. https://jepsen.io/analyses/rethinkdb-2-1-5 https://jepsen.io/analyses/rethinkdb-2-1-5
TheDong 5y ago
If you want to look at a Jepsen test which shows a system mostly working as designed, the etcd 3.4.3 one (https://jepsen.io/analyses/etcd-3.4.3 https://jepsen.io/analyses/etcd-3.4.3) is a much nicer read. Quoting from the discussion section, "etcd 3.4.3 lived up to its claims for key-value operations: we observed nothing but strict-serializable consistency for reads, writes, and even multi-key transactions, during process pauses, crashes, clock skew, network partitions, and membership changes" That's what a successful test looks like. Sure, they found correctness issues with the lock api, but they were able to lend some confidence to etcd's core api. There might be bugs, but jepsen didn't observe any. Sure, that's not proof that systems don't have tradeoffs, but it's proof that jepsen doesn't literally always find consistency issues or other core problems. Said another way, Jepsen is testing for correctness issues in systems. They either find some or they don't. These results are interesting, even if they are not a panacea. There are other tradeoffs, but for most systems, correctness is important enough it merits consideration on its own right. I don't think readers of jepsen misunderstand what's being tested or what it means, nor do they misunderstand that there are other tradeoffs to consider, so I think your comment is off the mark. By analogy, if we were reading a post about "I load-tested this bridge which claims to support 50 tons of weight, and it broke at 5 tons", no one would be saying "yeah, but there are tradeoffs for bridges. This post just makes it clear that there's tradeoffs. If you make the bridge stronger, it would be more expensive, and to imply there's not tradeoffs is misleading. An astute reader might deduce that they should never drive on bridges again". I don't think that would be a reasonable interpretation of such a post, nor do I think the interpretation you espouse here portrays an accurate sentiment.

belter 5y ago

That is not what Jepsen demonstrates. Instead it shows systems making claims that are not supported by their implementation and/or algorithms. If you want to take on your claim/suggestion then those systems just have to stop making the claims he is testing for. PS: There is also another trend, that is System claiming they fixed the issues Jepsen finds, without submitting themselves again for analysis...but I digress now...

redwood 5y ago
I respectfully disagree. Human language and characteristics around distributed data stores in the real world have inherent ambiguity implicit in them that Jepsen in my view pretends isn't the case. Here's an analogy that may help me communicate how I feel since I realize my message is not landing: Let's say I'm buying a condo in the San Francisco Bay area. And let's say the building that I'm looking to buy in advertisers that it is historic but seismically retrofitted. Then say Jepsen-earthquake-test comes in and shakes the ground beneath the building and shows that the building indeed collapses with enough of an earthquake: would that or would that not be enough information for me to decide whether or not the seismically retrofitted building is good enough for my needs? There's a lot of ambiguity in answer that question.
- belter 5y ago
  Although I agree with your point, on language and its ambiguity, I would argue that is a different claim of the one you made above, and that I replied to. When he demonstrated that Riak was dropping 30-70% of writes, even with the strongest consistency settings, or that Mongo had multiple scenarios of data loss, we are not talking about the subtleties of the English idiom
  - redwood 5y ago
    But the contrived nature of Jepsen tests is so different than the real world. In the real world no system behaves exactly the same as it was designed to behave; the real world has cosmic rays, earthquakes and everything in between. So no statement about a software system can ever be simply accepted as a fundamental fact and so to prove that software systems break is misleading for what most users need to understand about said systems. A Jepsen style test optimized not for bending to show where things break but instead for showing likely real world style situations with an eval of which are most likely to arise would be far more valuable for people
    - eklavya 5y ago
      We do know that some databases fare much better than others and that’s useful to many. In your analogy it would be many builders claiming their buildings to be earthquake proof with only some actually being it. Thanks to Jepsen, customers know, where to buy.
      - redwood 5y ago
        I'm genuinely curious to hear which database is the winner from your perspective based on Jepsen's insight?
        aphyr 5y ago
        If you find out, let me know! Folks are always asking me "What database is best?" and I'm just shuffling my feet and going "Uhhhh, well, systems are really hard to build correctly, there's lots of tradeoffs, you probably want something with a proven replication model under the hood but so much depends on workload and operational characteristics beyond just safety... If you find a database you love please tell me about it because I like hearing people's stories..." FWIW, I think you're raising really good questions in this thread. Qualitative safety is highly contextual, depending on fault model, significance of individual operations, concurrency, throughput, latency demands, operational characteristics, data volume, etc etc., and I try to touch on that in the "Toward a Culture of Safety" section in this report. Hopefully that resonated for you.

cmiles74 5y ago

I disagree that the analysis articles from Jepsen, taken as whole, are arguing for a "perfect distributed system without trade offs". In my experience these articles are fairly even-handed, indeed, many products make some changes based on the analysis and then contract for a follow-up. That the "tone and implication is as if somehow there is some alternative panacea without tradeoffs..." is not something I observed in this analysis or the previous ones that I have read. In my opinion the Rethink DB 2.1.5 report went fairly well for that project. If the claims are aligned with the reality of the product, it's clear the Jepsen report will highlight that. https://jepsen.io/analyses/rethinkdb-2-1-5 https://jepsen.io/analyses/rethinkdb-2-1-5

TheDong 5y ago

If you want to look at a Jepsen test which shows a system mostly working as designed, the etcd 3.4.3 one (https://jepsen.io/analyses/etcd-3.4.3 https://jepsen.io/analyses/etcd-3.4.3) is a much nicer read. Quoting from the discussion section, "etcd 3.4.3 lived up to its claims for key-value operations: we observed nothing but strict-serializable consistency for reads, writes, and even multi-key transactions, during process pauses, crashes, clock skew, network partitions, and membership changes" That's what a successful test looks like. Sure, they found correctness issues with the lock api, but they were able to lend some confidence to etcd's core api. There might be bugs, but jepsen didn't observe any. Sure, that's not proof that systems don't have tradeoffs, but it's proof that jepsen doesn't literally always find consistency issues or other core problems. Said another way, Jepsen is testing for correctness issues in systems. They either find some or they don't. These results are interesting, even if they are not a panacea. There are other tradeoffs, but for most systems, correctness is important enough it merits consideration on its own right. I don't think readers of jepsen misunderstand what's being tested or what it means, nor do they misunderstand that there are other tradeoffs to consider, so I think your comment is off the mark. By analogy, if we were reading a post about "I load-tested this bridge which claims to support 50 tons of weight, and it broke at 5 tons", no one would be saying "yeah, but there are tradeoffs for bridges. This post just makes it clear that there's tradeoffs. If you make the bridge stronger, it would be more expensive, and to imply there's not tradeoffs is misleading. An astute reader might deduce that they should never drive on bridges again". I don't think that would be a reasonable interpretation of such a post, nor do I think the interpretation you espouse here portrays an accurate sentiment.