5 ms·

I don't find it surprising that so many distributed databases fail at ensuring consistency; it's a very hard problem. What really gets me is that they deliberat

• chralieboy 11y ago
  That's really been the most enlightening part of the series for me. Reinforces that the hardest part in engineering is rarely the technical problem. Distributed databases are really f*ing hard, but infinitely harder if the people can't work together or don't open up to faults.
  • bbrazil 11y ago
    From experience I think some of the problem is that not everyone appreciates the importance of correctness in this sort of system. At the very least you should clearly documenting your expected failure modes, so that it's possible to build correct things on top of it. In part it's hard to convince everyone of the importance of spending time on what looks like an unlikely corner case until after the outage - when it also becomes much harder to fix as you usually need to build this into the core design of your system.
• acconsta 11y ago
  >I don't find it surprising that so many distributed databases fail at ensuring consistency; it's a very hard problem It's not hard — it's slow. Sending reads and writes through Paxos or Raft gives you sequential consistency. But not surprisingly, touching a quorum of nodes for every operation is too slow to be practical for many workloads. And that's usually fine — most data aren't bank account balances.
  • simoncion 11y ago
    Though, it's not even remotely okay if your documentation and marketing material indicates that consistency isn't being sacrificed for performance. :) I suspect that no experienced programmer expects to get data safety for free. We do expect that the documentation doesn't lie to us when describing the level of data safety provided.
    • antirez 11y ago
      I totally agree, just a note: write safety and consistency are not the same thing. An example is a CP store that breaks linearizability in order to serve reads without asking other nodes, but just providing the local value. Writes are as safe as a true CP store, but the consistency model is different and weaker. It is true that many times the replication model employed also has effects on the ability of writes to survive, I just mentioned the above example to show it is not always the case.
    • acconsta 11y ago
      Right, that's the real value here — holding databases accountable for their claims. Zookeeper clearly documented that writes are consistent but reads aren't, so it "passed" Jepsen cleanly. The Etcd guys claimed reads are linearizable, so Aphyr tested for linearizability and it failed. https://aphyr.com/posts/291-call-me-maybe-zookeeper https://aphyr.com/posts/291-call-me-maybe-zookeeper https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul
  • deleted 11y ago
    [deleted]
  • nemo44x 11y ago
    I think you touched on a key point a lot of people don't consider. Not all data is the same nor does it have to be treated the same. Many distributed systems will choke on a piece of data here and there and may not give the appropriate response. But you need to consider what you're working on and if that's OK from time to time given the advantages the system you're considering offers. What might be considered an appropriate data store for one data set may not be for another. And the type of information a lot of distributed systems are handling are far from bank accounts, for example. And if you're handling ultra important, sensitive data there are techniques that have been available for many years (two phase commit, for example) that can help. I love this series but I'm blown away by how many engineers here automatically assume a system is a failure because it doesn't pass a certain type of test from time to time. I do agree with the series that the marketing materials shouldn't claim things, however. Companies should be honest with what their systems weaknesses currently are.
• antirez 11y ago
  Every time the topic of consistency in distributed systems arise I see this idea mentioned in a way or the other that the problem is how hard strong consistency is. To be honest is a problem solved a long time ago by consensus algorithms: there is a lot more work to do in the future in order to explore other useful consistency models, but if you want strong consistency you know you have "just" to use a consensus algorithm like Paxos or Raft or Viewstamped replication or whatever, and you end with a CP system. Of course it is non trivial even to implement those systems correctly, but the point is, most implementations tested by Aphyr don't attempt to do this at all most of the times. There will be bugs but in the end if your goal is to write a consistent store, you can do it. Bugs in wanna-be CP systems and/or failed attempts at implementing CP because of home-made protocols with flaws are part of the issue and part of what Aphyr's research uncovers, but IMHO the central point is a different one: CP systems have performance limits so many real world systems don't even try to be CP, which is fine, it's up to the designer to pick the DB design and tradeoffs, but you need to document it properly. What happens during partitions? What consistency model the system employs if it does not feature strong consistency? Is the system Available during partitions? And so forth. As long as the documentation is honest, it's up to the user to understand, for its use case, if the system is a good fit or not. What we are seeing often is an attempt to informally document what should be well specified. Sometimes there are also incorrect things stated in bold letters in the documentation, like "the system is consistent but sacrifices partition tolerance" which does not mean anything in the context of partitions actually happening in the real world. Aphyr's effort and our collective effort as people working, implementing and using databases, in my opinion, is not to end with databases designed all the same way or all providing strong guarantees, but with clear understandable and honest documentation describing the system behavior.
  • baudehlo 11y ago
    The only thing I'd take out of your comment is "all providing strong guarantees". I'd say that if you document that it does provide those guarantees that it damn well better do that (modulo bugs of course).
  • rdtsc 11y ago
    > but IMHO the central point is a different one: CP systems have performance limits so many real world systems don't even try to be CP They try on paper, so to speak. They might have a CP component, but it is not in the datapath -- it is there for maybe cluster configuration, or say to elect a leader. And then they can claim they have CP because everything goes through the leader, and the leader was elected by a CP component. However without realizing they just pushed all the error cases into corners -- when the membership changes (nodes die, die in unpredictable ways, become partitioned, new nodes added...). They never test that extensively, which is a hard thing to do, because there are so many cases and combinations. The bottom line is even if they have a Paxos or Raft component in their system doesn't mean their data is stored consistently.