When a distributed database is designed, you must navigate and optimize several complex technical tradeoffs to meet the architecture and product objectives. The

When a distributed database is designed, you must navigate and optimize several complex technical tradeoffs to meet the architecture and product objectives. The specific set of tradeoffs made -- and they are different for every platform -- will determine the kinds of data models and workloads that the database will be suitable for, especially if performance and scalability are critical as in this case. The reason distributed databases tend to be buggy, especially in the first iterations, is straightforward if not simple to address. While it is convenient to describe technical design tradeoffs as a set of discrete, independent things, in real implementation they are all interconnected in subtle, complex, nuanced ways. Modifying one design tradeoff in code can have unanticipated consequences for other intended tradeoffs. In other words, there isn't a set of simple tradeoffs, there is a single extremely high-dimensionality tradeoff that is being optimized. Not only are complex high-dimensionality design elements difficult to reason about when writing code the first time, any changes to the code may shift how the tradeoffs interact in non-obvious ways. Humans have finite cognitive budgets, so unless it is obvious that a code change has the potential to have unintended side effects, we generally don't spend the time to fully verify this fact. I can't tell you how many times I've seen tiny innocuous code changes alter the behavior of distributed databases in surprising ways. This is also why once the core code seems to be correct, people are reluctant to modify it if that can be avoided at all.