8 ms·

I respectfully disagree. Human language and characteristics around distributed data stores in the real world have inherent ambiguity implicit in them that Jepse

I respectfully disagree. Human language and characteristics around distributed data stores in the real world have inherent ambiguity implicit in them that Jepsen in my view pretends isn't the case. Here's an analogy that may help me communicate how I feel since I realize my message is not landing: Let's say I'm buying a condo in the San Francisco Bay area. And let's say the building that I'm looking to buy in advertisers that it is historic but seismically retrofitted. Then say Jepsen-earthquake-test comes in and shakes the ground beneath the building and shows that the building indeed collapses with enough of an earthquake: would that or would that not be enough information for me to decide whether or not the seismically retrofitted building is good enough for my needs? There's a lot of ambiguity in answer that question.

belter 5y ago
Although I agree with your point, on language and its ambiguity, I would argue that is a different claim of the one you made above, and that I replied to. When he demonstrated that Riak was dropping 30-70% of writes, even with the strongest consistency settings, or that Mongo had multiple scenarios of data loss, we are not talking about the subtleties of the English idiom
- redwood 5y ago
  But the contrived nature of Jepsen tests is so different than the real world. In the real world no system behaves exactly the same as it was designed to behave; the real world has cosmic rays, earthquakes and everything in between. So no statement about a software system can ever be simply accepted as a fundamental fact and so to prove that software systems break is misleading for what most users need to understand about said systems. A Jepsen style test optimized not for bending to show where things break but instead for showing likely real world style situations with an eval of which are most likely to arise would be far more valuable for people
  - eklavya 5y ago
    We do know that some databases fare much better than others and that’s useful to many. In your analogy it would be many builders claiming their buildings to be earthquake proof with only some actually being it. Thanks to Jepsen, customers know, where to buy.
    - redwood 5y ago
      I'm genuinely curious to hear which database is the winner from your perspective based on Jepsen's insight?
      - aphyr 5y ago
        If you find out, let me know! Folks are always asking me "What database is best?" and I'm just shuffling my feet and going "Uhhhh, well, systems are really hard to build correctly, there's lots of tradeoffs, you probably want something with a proven replication model under the hood but so much depends on workload and operational characteristics beyond just safety... If you find a database you love please tell me about it because I like hearing people's stories..." FWIW, I think you're raising really good questions in this thread. Qualitative safety is highly contextual, depending on fault model, significance of individual operations, concurrency, throughput, latency demands, operational characteristics, data volume, etc etc., and I try to touch on that in the "Toward a Culture of Safety" section in this report. Hopefully that resonated for you.
        redwood 5y ago
        I think it's a refreshingly mature section, And my hats off to you for going there. Unfortunately I don't think the typical reader who I really do believe loves Jepsen outrage porn is going there
      - deleted 5y ago
        [deleted]
    - TheDong 5y ago
      Note, this analogy is slightly off for using the phrase "some actually being it". jepsen cannot prove "earthquake proof" (correctness). Rather, it would be "Builders claim their buildings are earthquake proof, and jepson was able to show a subset collapse in earthquakes. The rest may or may not be earthquake proof". That's still very valuable. It's really valuable to know when something is wrong. It would be more valuable to know that something is definitely right (correct / "earthquake proof"), but jepsen cannot prove that.
  - dmitriid 5y ago
    > But the contrived nature of Jepsen tests is so different than the real world. The real world is significantly more "contrived" than anything Jepsen can come up with. Transactions timing out, Nodes losing transactions even after they've been acknowledged etc.? None of this is contrived.
  - Rapzid 5y ago
    Jepsen is all about verifying claims and communicating limitations. Companies hire them for this as due diligence. They fix real world bugs based on the results. > what most users need to understand about said systems The target audience of these reports are the system builders and the software engineers building services on top of them; not end-users consuming higher level services.
  - camgunz 5y ago
    You make some reasonable points, but I would say a couple things here: - If we're sticking with your example further up-thread, you'd buy a house that was advertised as "can withstand a 4.0 magnitude earthquake", that then failed when subjected to a 4.0 magnitude earthquake. - 4.0 magnitude earthquakes happen all of the time [0]. More or less what I'm saying is, engineers generally don't think DBs lose data, and when they start coming up with ways that might happen (node failure, network failure, clock desync), distributed DBs assure them with algorithms and configuration knobs. Aphyr puts those assurances to the test, which is so, so valuable to us all. It's also worth saying that this space is pretty technically complicated. All the DB engineers I know use some form of Jepsen-style testing (or Jepsen itself) because it's amazingly great. [0]: https://research.google/pubs/pub40801/ https://research.google/pubs/pub40801/
  - stefan_ 5y ago
    This is such a clown take. Read any root cause analysis of your favorite cloud operator and you will find all of the scenarios that Jepsen tests for in those. And you know what's even better? They can be solved!
dmitriid 5y ago
> I respectfully disagree. Human language and characteristics around distributed data stores in the real world have inherent ambiguity implicit in them that Jepsen in my view pretends isn't the case. We've been doing distributed systems for over 50 years now. There's nothing ambiguous in either the language Jepsen uses or the claims he is examining. The crypto shills chose to pretend the language is ambiguous and invent their own definitions on purpose. Including the ridiculous "nah, everyone understands that when we speak in present tense it means we mean some unspecified point in a nebulous future".

belter 5y ago

Although I agree with your point, on language and its ambiguity, I would argue that is a different claim of the one you made above, and that I replied to. When he demonstrated that Riak was dropping 30-70% of writes, even with the strongest consistency settings, or that Mongo had multiple scenarios of data loss, we are not talking about the subtleties of the English idiom

redwood 5y ago
But the contrived nature of Jepsen tests is so different than the real world. In the real world no system behaves exactly the same as it was designed to behave; the real world has cosmic rays, earthquakes and everything in between. So no statement about a software system can ever be simply accepted as a fundamental fact and so to prove that software systems break is misleading for what most users need to understand about said systems. A Jepsen style test optimized not for bending to show where things break but instead for showing likely real world style situations with an eval of which are most likely to arise would be far more valuable for people
- eklavya 5y ago
  We do know that some databases fare much better than others and that’s useful to many. In your analogy it would be many builders claiming their buildings to be earthquake proof with only some actually being it. Thanks to Jepsen, customers know, where to buy.
  - redwood 5y ago
    I'm genuinely curious to hear which database is the winner from your perspective based on Jepsen's insight?
    - aphyr 5y ago
      If you find out, let me know! Folks are always asking me "What database is best?" and I'm just shuffling my feet and going "Uhhhh, well, systems are really hard to build correctly, there's lots of tradeoffs, you probably want something with a proven replication model under the hood but so much depends on workload and operational characteristics beyond just safety... If you find a database you love please tell me about it because I like hearing people's stories..." FWIW, I think you're raising really good questions in this thread. Qualitative safety is highly contextual, depending on fault model, significance of individual operations, concurrency, throughput, latency demands, operational characteristics, data volume, etc etc., and I try to touch on that in the "Toward a Culture of Safety" section in this report. Hopefully that resonated for you.
      - redwood 5y ago
        I think it's a refreshingly mature section, And my hats off to you for going there. Unfortunately I don't think the typical reader who I really do believe loves Jepsen outrage porn is going there
    - deleted 5y ago
      [deleted]
  - TheDong 5y ago
    Note, this analogy is slightly off for using the phrase "some actually being it". jepsen cannot prove "earthquake proof" (correctness). Rather, it would be "Builders claim their buildings are earthquake proof, and jepson was able to show a subset collapse in earthquakes. The rest may or may not be earthquake proof". That's still very valuable. It's really valuable to know when something is wrong. It would be more valuable to know that something is definitely right (correct / "earthquake proof"), but jepsen cannot prove that.
- dmitriid 5y ago
  > But the contrived nature of Jepsen tests is so different than the real world. The real world is significantly more "contrived" than anything Jepsen can come up with. Transactions timing out, Nodes losing transactions even after they've been acknowledged etc.? None of this is contrived.
- Rapzid 5y ago
  Jepsen is all about verifying claims and communicating limitations. Companies hire them for this as due diligence. They fix real world bugs based on the results. > what most users need to understand about said systems The target audience of these reports are the system builders and the software engineers building services on top of them; not end-users consuming higher level services.
- camgunz 5y ago
  You make some reasonable points, but I would say a couple things here: - If we're sticking with your example further up-thread, you'd buy a house that was advertised as "can withstand a 4.0 magnitude earthquake", that then failed when subjected to a 4.0 magnitude earthquake. - 4.0 magnitude earthquakes happen all of the time [0]. More or less what I'm saying is, engineers generally don't think DBs lose data, and when they start coming up with ways that might happen (node failure, network failure, clock desync), distributed DBs assure them with algorithms and configuration knobs. Aphyr puts those assurances to the test, which is so, so valuable to us all. It's also worth saying that this space is pretty technically complicated. All the DB engineers I know use some form of Jepsen-style testing (or Jepsen itself) because it's amazingly great. [0]: https://research.google/pubs/pub40801/ https://research.google/pubs/pub40801/
- stefan_ 5y ago
  This is such a clown take. Read any root cause analysis of your favorite cloud operator and you will find all of the scenarios that Jepsen tests for in those. And you know what's even better? They can be solved!

dmitriid 5y ago

> I respectfully disagree. Human language and characteristics around distributed data stores in the real world have inherent ambiguity implicit in them that Jepsen in my view pretends isn't the case. We've been doing distributed systems for over 50 years now. There's nothing ambiguous in either the language Jepsen uses or the claims he is examining. The crypto shills chose to pretend the language is ambiguous and invent their own definitions on purpose. Including the ridiculous "nah, everyone understands that when we speak in present tense it means we mean some unspecified point in a nebulous future".