23 ms·

NewSQL databases fail to guarantee consistency and I blame Spanner

510 points by evanweaver 8y ago

jpalomaki 8y ago
And majority of developers don’t need NoSQL or NewSQL - they are well off with PostgreSQL. I think the old and tried should be the default choice here and other paths taken only after careful consideration and good justifications.
- threeseed 8y ago
  Or how about instead of just blindly picking a system. You spend a week or two to evaluate a few that you think might mesh well for your data structures. PostgreSQL doesn't work well for many, many common use cases.
abadid 8y ago
I'm the author of that post. I'm happy to respond to comments on the post on this thread for the next several hours. You can also leave comments on the post itself, and I will respond there at any time.
- plopz 8y ago
  I've only ever worked on small projects so I'm not at all familiar with these very high-scale distributed databases but from the post it seems to indicate that Spanner is in a league of its own because it integrated hardware into the mix where the others are software only. What are the differences in scale between the two categories mentioned?
  - abadid 8y ago
    Yes, Spanner is quite unusual in the distributed database world in how hardware is a pretty important part of their solution. Other systems may claim important integrations with hardware, but for Spanner, the architecture really relies on particular hardware assumptions. To answer your question about scale: there is no real practical difference in scalability between the two categories discussed in the post. Partitioned consensus has better theoretical scalability, but I am not aware of any real-world workload that can not be handled by global consensus with batching.
    - deleted 8y ago
      [deleted]
    - ryanobjc 8y ago
      I think this "global consensus with batching does everything partitioned does" is a very much theory vs practice type of statement. As in, in theory there is no difference between theory and practice :-) I've seen those batched consensus systems, and honestly, you're kidding yourself if you think they can handle a million qps. Just transmit time of the data on ethernet would become an issue alone! Even with 40 gig - transmit time never becomes free. So now you're stuffing 1/10th of a million qps worth of data via a single set of machines (3, 5, 7, 9? Some relatively small amount) Am I misunderstanding you? Hopefully I am!
- evanweaver 8y ago
  Do these concerns also apply in an HTAP or OLAP context e.g. systems like Cloudera's Kudu, which uses Hybrid Time? Or maybe Volt which you also worked on?
  - xtacy 8y ago
    OLAP systems tend to be read only (for analytics) so the question of transactions and consistency isn't really applicable.
    - kwillets 8y ago
      They don't load themselves.
  - kwillets 8y ago
    Vertica seems to include some of these observations, such as global consistency and group commit. I believe these are easier to achieve (lower overhead) in OLAP due to fewer, larger transactions.
  - nickerten 8y ago
    I've worked on a system that loaded data into Kudu in near real time and simultaneously ran queries on the data. Kudu has no transactions, consistency is eventual which was sufficient due to our near-real time constraint, however you do need a stable NTP source. We have lost data when the cluster could not get a reliable NTP connection, decided to shut down and tablet servers' data files became corrupted.
- jgalentine007 8y ago
  I recently came across CockroachDB and thought it's capabilities interesting, almost too good to be true. I also have been looking at Citus Data which shards and distributes transactions, are you aware of any consistency shortcomings with it?
  - cube2222 8y ago
    CockroachDB's performance is dependent on time synchronization. If a node detects it's too far behind, it will commit suicide. However, before it detects it, there is a possibility of stale reads. https://www.cockroachlabs.com/docs/stable/recommended-production-settings.html#clock-synchronization https://www.cockroachlabs.com/docs/stable/recommended-produc...
    - jgalentine007 8y ago
      Thanks for pointing that out - I have yet to rtfm and dive deep. I wonder how frequently time sync problems occur in virtual environments after ntp syncing - I've seen pretty erratic behavior on virtual active directory domain controllers even after syncing with hyper-v and vmware.
      - brazzledazzle 8y ago
        It’s been a long time since I messed with domain controllers but I believe Microsoft used to have explicit guides for handling time on virtual DCs. At that time we kept around a a couple hardware DCs to be safe but I do remember having the VMware agent correct the time could result in some bad results. I think it was because it immediately fixed the time and didn’t slowly correct the drift but it’s been a long time so my memory could be off.
      - amaccuish 8y ago
        Time shouldn't be a massive issue for AD no? It's a vector clock, not a UTC clock. The UTC clock is only used to solve conflicts no?
        8y ago
- xtacy 8y ago
  Nice writeup, but given the title, I was hoping you would also have a constructive test case that shows that these systems fail to meet their guarantees (like Jepsen did). :) I wonder if any of the aforementioned systems (Calvin/Spanner/YugaByte) that can opportunistically commit transactions and detect issues and roll back + retry all within the scope of the RPC so it can still conform to linearisability requirement?
- mehrdadn 8y ago
  The problem isn't a lack of very accurate clocks per se, right? What you need is an accurate bound on clock error, whatever it might be. It sounds to me that non-Googles are specifying bound parameters that, unlike Google, they cannot guarantee. Doesn't that mean the blame goes to whomever put inaccurate bound parameters into the software? Why blame the algorithm for garbage input?
  - CydeWeys 8y ago
    The problem is that the bound on clock error directly affects your performance. So if you're willing to accept, say, a second in clock error, then all transactions will take a minimum of one second. That level of performance is going to be unacceptable in many situations. The potential clock error on VMs without dedicated time-keeping hardware is so large that performance turns into absolute garbage.
    - mehrdadn 8y ago
      I would understand if the complaint was that Spanner is too slow without expensively accurate clocks and synchronization. But the complaint is that Spanner fails to guarantee consistency, which doesn't make sense to me. The requirements clearly include giving a valid clock bound, so if you give an invalid clock bound, it's clearly your fault for getting incorrect results, not Spanner's!
      - CydeWeys 8y ago
        Spanner does guarantee consistency, thanks to its use of hardware atomic clocks and GPS. It's alternatives like CockroachDB that don't have this dedicated hardware that can fail to guarantee consistency if clocks get of sync (a problem that can't happen in Spanner). Spanner is really fast and massively parallelizable.
        mvijaykarthik 8y ago
        We recently open-sourced https://github.com/rubrikinc/kronos https://github.com/rubrikinc/kronos for the exact same problem. Coincidentally I shared that on Show HN just today: https://news.ycombinator.com/item?id=18037609 https://news.ycombinator.com/item?id=18037609
- joemag 8y ago
  Hi, you used DynamoDB as an example of a weakly consistent system in the opening paragraph, but it actually supports both modes [1]. The point of confusion might come from the fact that the service described in 2007 Dynamo paper was an inspiration for DynamoDB, rather than DynamoDB itself. Disclaimer, I work for AWS, but not on DynamoDB team. [1] https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
  - abadid 8y ago
    Thanks. I added a parenthetical remark to the post to indicate that I was talking about DynamoDB's default settings.
    - Twirrim 8y ago
      It's not really the default settings, per se. You don't have to change any bit of configuration about your database to get consistency. The DynamoDB API gives you the GetItem API call and a boolean property to choose to make it a consistent read. It's left as a very simple task for developers leveraging DynamoDB to make the appropriate trade offs on consistent or inconsistent read. source: Used to work for AWS on a service that heavily leveraged DynamoDB. Not _once_ did we experience any problems with consistency or reliability, despite them and us going through numerous network partitions in that time. The only major issue came towards the end of my time there when DynamoDB had that complete service collapse for several hours. On the sheer scale that DynamoDB operates at, it's more likely to be a question of "How many did we automatically handle this week?" than "How often do we have to deal with network partitions?"
      - dsp1234 8y ago
        From the GetItem docs[0] "GetItem provides an eventually consistent read by default." This seems to meet the definition of "DynamoDB's default settings" [0] - https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html https://docs.aws.amazon.com/amazondynamodb/latest/APIReferen...
- romed 8y ago
  I question your statement that building apps on weakly-consistent systems is so difficult. I’ve worked on very large scale systems that you’ve definitely heard of and probably used that are built atop storage systems with very weak semantics and asynchronous replication. Aren’t such systems existence proofs, or do you think there’s just a huge difference between the abilities of engineers in various organizations?
  - cloakandswagger 8y ago
    He said it was difficult, not impossible. I think it's a fairly noncontroversial statement. Dealing with eventual consistency is always going to be more difficult and require more careful thought and preparation than immediately consistent systems. How many programmers do you think are out there that have only ever worked on systems that use a single RDBMS instance, and what would happen if they tried to apply their techniques to a distributed, eventually consistent environment?
    - wpietri 8y ago
      Exactly. My dad was coding when DBMSes rose to prominence, and it was basically a way to take a bunch of things that were hard to think about and sweep them under the rug. People wrote plenty of good software before they existed, but if you wanted to write a piece of data, you had to think about which disk and where on the disk and exactly the record format. Most programmers just wanted a genie that they could give data to and later ask for it back. It's the same today, but worse. Most programmers still want a simple abstraction that lets them just build things. But now it's not just which sector on which disk, but also which server in which data center on which continent, while withstanding the larger number of failure modes at that scale. When necessary, people can explicitly address that complexity. But it has a big cost, a high cognitive load.
  - groestl 8y ago
    The biggest challenge, in my experience, is explaining the weak consistency guarantees to stakeholders, especially in a QA setting (e.g. product owner demos the product to colleagues, numbers are not immediately up-to-date).
- felixgallo 8y ago
  why didn't you disclose your connections in your post? It appears you've written much the same article for FaunaDB's official blog in the past (https://fauna.com/blog/distributed-consistency-at-scale-spanner-vs-calvin https://fauna.com/blog/distributed-consistency-at-scale-span...).
  - vidarh 8y ago
    The article specifically states that Calvin comes out of his research group and that FaunaDB was inspired by Calvin.
  - abadid 8y ago
    That post was actually quite different than this one. This one focuses on distributed vs. global consensus. But I added a note at the end that clearly documents my connection to FaunaDB. As far as Calvin, the post itself clearly says that it came out of my research group.
- peterwwillis 8y ago
  "Systems that guarantee consistency only experience a necessary reduction in availability in the event of a network partition." Many of the distributed clusters I've maintained had crap infrastructure and no change control, and parts of the clusters were constantly going down from lack of storage, CPU and RAM, or bad changes. The only reason the applications kept working were either (1) the not-broken vnodes continued operating as normal and only broken vnodes were temporarily unavailable, or (2) we shifted traffic to a working region and replication automatically caught up the bad cluster once it was fixed. Clients experienced increased error rates due primarily to these infrastructure problems, and very rarely from network partition. Does your consistent model take this into account, or do you really assume that network partition will be the only problem?
  - mannykannot 8y ago
    It seems you have other problems (crap infrastructure and no change control) to deal with before the issues in this article become your biggest concern, but are not the cases you list themselves partition problems?
    - peterwwillis 8y ago
      They cause partition, but their origin isn't the network. Nobody who runs a large system has perfectly behaving infrastructure. Infrastructure always works better in a lab than in the real world. Even if you imagine your infrastructure is rock-solid, people often make assumptions, like their quota is infinite, or their application will scale past the theoretical limits of individual network segments, i/o bounding, etc. The point is, resources cause problems, and the network is just one of many resources needed by the system. Other resources actually have more constraints on them than the network does. If a resource is constrained, it will impact availability in a highly-consistent model. The author states that simply adding network redundancy would reduce partitions, and infrastructure problems are proof that this is very short-sighted. "You have bigger problems" - no kidding! Hence the weak-consistency model!
  - StreamBright 8y ago
    Even if you maintain your infrastructure properly you run on x86 servers with disks and CPUs that need cooling, using network devices that have fascinating failure scenarios. I guess assuming that your infra is not reliable is a must for any database nowadays.
- reilly3000 8y ago
  Is there a "best-of-both-worlds" approach that could work, or are these two approaches mutually exclusive? I have to imagine that time drift can eventually reconciled with some kind of time delta.
  - abadid 8y ago
    The two approaches seem mutually exclusive to me.
- jakelarkin 8y ago
  won't batching transactions increasing the failure rate by a multiple of (batch size)? how should users reason about that tradeoff?
- Serow225 8y ago
  I think it would be helpful to move the disclaimer to the top of the post, just for clarity's sake. Is there a forecasted date on the release of the independent Jepsen study? Who is performing it? Thanks!!
- CBLT 8y ago
  Great read, thank you for sharing. Do you have any opinion on the design of Eris[0]? Consistency is achieved with extra hardware, but that hardware is a network-level sequencer. [0]: https://syslab.cs.washington.edu/papers/eris-sosp17.pdf https://syslab.cs.washington.edu/papers/eris-sosp17.pdf
  - rwtxn 8y ago
    AFAIK Eris (and previous similar work from the same group) assume network ordering guarantees that can be provided in a single datacenter but probably not in a WAN setting. This discussion is about distributed and geo-replicated databases.
dboreham 8y ago
In case you're wondering : this is a very worthwhile read.
- zzzcpan 8y ago
  Too many false claims for my taste. Especially wrt to eventual consistency and consistency guarantees. Also some misinterpretation of CAP theorem to fit the narrative. Very typical FaunaDB promotional post.
  - haberman 8y ago
    Without some description of what "false claims" and "misinterpretation" you mean, this is a content-free post.
    - jessaustin 8y ago
      It has more "content" than the post to which it replies. "This is good." "No, it is bad, in these particular areas." [EDIT:] Granted, the post could still be wrong, but ISTM accusations of content-freedom tend to be projections.
      - pdpi 8y ago
        "This is a good read" is a statement of opinion. You're free to assign whatever value you like to the op's opinion, but that opinion is "content" in and of itself. "The article has false claim" is a statement of fact. A vague statement of fact about a non-obvious topic that puts no effort into providing concrete examples and reasoning is pretty much worthless.
        jessaustin 8y ago
        Adding qualifiers makes a statement have less content? Please try to imagine using this questionable argument in any other context... It's a controversial topic and TFA was written by someone with a commercial interest. Everything is opinion. The level of comment-policing I see here makes me quite suspicious of everything in TFA. [EDIT:] aargh I'll never learn every discussion that starts with "content-free" cliche goes nowhere... just downvote and move on...
deleted 8y ago
[deleted]
beamatronic 8y ago
Please be sure to distinguish between Couchbase and CouchDB. Couchbase is a CP system.
klodolph 8y ago
Anyone interested in this subject may also find it worthwhile to read the paper describing Spanner: http://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf http://static.googleusercontent.com/media/research.google.co...
haberman 8y ago
> We will trace the failure to guarantee consistency to a controversial design decision made by Spanner that has been tragically and imperfectly emulated in other systems. This post doesn't establish any "controversy" about Spanner's design decision. It only says that it requires special hardware, which other systems attempt to emulate despite not having this specialized hardware. To call this decision "controversial" I think one would need to show that it has some significant problem in the environment it was designed for.
- 1024core 8y ago
  That's because the author is pushing Calvin, which came out of his research group.
- abadid 8y ago
  By controversial, I mean that in the database research community (where I spend most of my time), there is significant disagreement about global consensus vs. Spanner's choice of partitioned consensus.
  - haberman 8y ago
    Disagreement about what though? Does Spanner's solution have an objective problem? Do you or others in your community have specific reasons to believe that it cannot deliver on its promises?
    - abadid 8y ago
      Spanner's approach requires help from hardware and several full time employees maintaining and ensuring the uncertainty guarantees. This increases the cost of the maintaining the system, which for Cloud Spanner is partially passed on to the end users. If you can build a system that doesn't require time synchronization, yet doesn't have any significant drawbacks relative to what Spanner provides, you'd be better off using this alternative system.
      - haberman 8y ago
dekhn 8y ago
This reads like an advertisement for the researcher's system combined with some FUD.
- reinhardt1053 8y ago
  would you mind to elaborate?
  - wsy 8y ago
    Just to mention a few points: - The title implies that Spanner is at fault for something, then the article itself admits that Spanner is totally fine. - Then it goes on to argue that this approach can't possibly work without physical devices for precise time. However, there is no proof whatsoever. Please be a researcher and design an experiment that demonstrates a violated consistency guarantee. - Finally, not disclosing financial relationships with products which are praised in the article is really low, and has only been fixed after this has been pointed out here. And it is not marked as a later addendum, as usual for changed articles.
dgreensp 8y ago
This is the clearest presentation of why CAP is misleading that I’ve ever seen! Wonderful. If the app programmer knows they don’t have global consistency, just consistency per partition, I wonder if in practice there are ways to achieve the necessary application-level guarantees such as in the photo-sharing case (not that it sounds that appealing to need to do so).
- xapata 8y ago
  > application-level guarantees If your database doesn't "guarantee" consistency, but instead provides eventual consistency, your application needs to be aware of how long "eventually" is. In effect, you'd be coding something similar to Spanner's timestamp uncertainty estimation. In the photo sharing example, it'd be reasonable to put a delay on sharing new photos with other users until after permissions changes would have propagated.
  - e12e 8y ago
    > delay on sharing new photos with other users until after permissions changes would have propagated. And then you'd maybe have to map the ux of changing permissions on an existing permission to a three-state: user makes change;ui shows change in progress;change takes effect and is reflected in ui.
tytso 8y ago
The post talks about Spanner using "a specialized hardware solution that uses both GPS and atomic clocks to ensure a minimal clock skew across servers." But what it fails to mention is that is that this time is distributed using a software solution --- NTP, in fact. Google makes its "leap-smeared NTP network" available via Google's Public NTP service. And it's not expensive for someone to buy their own GPS clock and run their own leap-smared NTP service. Yes, it means that someone who installs a NewSQL database will have to set up their own time infrastructure. But that's not hard! There are lots of things about hardware setup which are at a similar level of trickiness, such as UPS with notification so that servers can shutdown gracefully when the batteries are exhausted, locating your servers in on-prem data centers so that if a flood takes out one data center, you have continuity of service elsewhere, etc., etc. Or of course, you can pay a cloud infrastructure provider (which Google happens to provide, but Amazon and Azure also provides similar services) to take care all of these details for you. Heck, if you use Google Compute Platform, you can use the original Spanner service (accept no substitutes :-)
- jrockway 8y ago
  I agree. Attempting to remove the GPS and atomic clocks from the system seems like a premature optimization to me. They are cheap and the ability to order events according to an actual wall clock is a huge benefit. Some basics for less-critical applications: https://github.com/jrockway/beaglebone-gps-clock/blob/master/README.md https://github.com/jrockway/beaglebone-gps-clock/blob/master...
- jakemoshenko 8y ago
  I feel like the author is making a huge omission by not talking about the properties of multi-raft sharded DBs coupled with the TrueTime-like APIs from cloud providers in 2018. He links to a post from the CEO of CockroachDB written in early 2016, and Amazon launched their TimeSync service in 2017. It's likely that Spencer would have something different to say about the matter today.
  - Quekid5 8y ago
    > I feel like the author is making a huge omission by not talking about the properties of multi-raft sharded DBs coupled with the TrueTime-like APIs from cloud providers in 2018. I mean this is all fine and dandy if it works (plus, how do know that it actually works in all the edge cases?), but that's a HUGE amount of complexity. IMO you really need to have a very good reason to even consider anything this complex.
romed 8y ago
I’ve seen comments on HN over the years in which someone Dunning-Kruegers their way into saying that TrueTime is easily replicated. I always wonder if they have sixteen senior SREs in their pocket, because that’s the level of production engineering Google applies to the problem. Time SRE has at various points had take measures up to and including calling the USAF and telling them their satellites are fucked up. If you don’t have the staff for this, the easiest way to get access to TrueTime is probably to just use Cloud Spanner.
- sabujp 8y ago
  the only solution to this is to launch your own satellites :)
  - reilly3000 8y ago
    If we seal the surface of, then boil the ocean we can launch our own time satellites with completely sustainable propulsion.
- Cshelton 8y ago
  Also, I believe all the major cloud providers provide a "TrueTime" API service. I forgot the name that AWS uses, but you can call it on your EC2 instances and make sure your hosts are all in sync. It's pretty cool.
  - otterley 8y ago
    AFAIK AWS is only offering NTP service with a GPS source (i.e. stratum 1). TrueTime appears to be a service offering that is a step above that in terms of the guarantees it provides.
    - Cshelton 8y ago
      Ah, I've never had a use case for anything more accurate than NTP
gregwebs 8y ago
In the post and another comment here it is stated that Calvin can handle any real-world workload. However, according to my reading of the Calvin paper, one must understand the update keys before starting the transaction. I also experienced limitations when trying to use FaunaDB: it doesn't support ad-hoc queries and it only allows for indexed queries. I really like the Calvin protocol, and is does seem perfectly suited to many application workloads, but it is odd to me though, to see Calvin presented as purely superior to all alternatives. It seems like more research and work needs to be done to create systems that can address its shortcomings around transactions that query for keys (3.2.1 Dependent transactions in the paper), including ad-hoc queries, and even interactive queries (with a transaction). Disclaimer: I work on TiDB/TiKV. TiKV (distributed K/V store) uses the percolator model for transactions to ensure linearizability, but TiDB SQL (built on top) allows write skew.
- freels 8y ago
  Support for dependent reads is a subject that the Calvin paper only touches on briefly, describing one strategy of using "reconnaissance reads" to determine the key set. In FaunaDB, this is formalized as an optimistic concurrency control mechanism that combines snapshot reads at the coordinator and read-write conflict detection within the transaction engine. By the time a transaction is ready to commit the entire key set is known. I wrote a blog post that goes into more detail here: https://fauna.com/blog/acid-transactions-in-a-globally-distributed-database https://fauna.com/blog/acid-transactions-in-a-globally-distr... Better analytics support, including ad-hoc queries is on our roadmap. That being said, requiring indexes was a design choice: First and foremost, we want FaunaDB to be the best operational database. It is a lot easier to write fast, predictable queries if they cannot fall back to a table scan! While FaunaDB requires you to create indexes to support the views your workload requires, the reward is a clearer understanding of how queries perform. We've built indexes to be pretty low commitment, however. You can add or drop them on the fly based on how your workload evolves over time.
  - gregwebs 8y ago
    Thanks for the link to the explanation. It seems like FaunaDB is doing the hard work to show how Calvin can actually be used for all types of workloads.
sabujp 8y ago
since spanner is distributed on the backend, any function that uses multiple sql commands that might be called or ingested by multiple sharded servers which are then sent to spanner, should always have these calls sent in a single transaction
CydeWeys 8y ago
It's definitely true that putting the burden of consistency on developers (instead of on the DB) results in a lot more tricky work for developers. On my project, which started six years ago, we use Cloud Datastore, because Cloud Spanner hadn't come out yet. It results in complicated, painful code that would be completely unnecessary with stronger transactional guarantees. Some examples: https://github.com/google/nomulus/blob/master/java/google/registry/backup/CommitLogCheckpointStrategy.java https://github.com/google/nomulus/blob/master/java/google/re... https://github.com/google/nomulus/blob/master/java/google/registry/model/ofy/CommitLoggedWork.java https://github.com/google/nomulus/blob/master/java/google/re... https://github.com/google/nomulus/blob/master/java/google/registry/model/index/ForeignKeyIndex.java https://github.com/google/nomulus/blob/master/java/google/re... It's no surprise that we're currently running screaming to something with stronger transactional guarantees.
- wikibob 8y ago
  I’m guessing that Nomulus is your project? I ran across it before and just wanted to say it’s really cool that this is open sourced.
  - CydeWeys 8y ago
    I'm the tech lead of it. Glad to hear that you've heard of it before, and yeah, I think it's pretty cool it's open sourced too, which is why I made it happen! https://opensource.googleblog.com/2016/10/introducing-nomulus-open-source.html https://opensource.googleblog.com/2016/10/introducing-nomulu...
  - guillon 8y ago
    Any idea if any company is planning to create an offer dedicates To .BRAND new gTLDs?
- erikb 8y ago
  Why does anybody need to provide consistency? Often you don't have complete consistency anyways. There are bugs, there are time delays, there is parallel processing. Why even have the requirement?
peterwwillis 8y ago
Distributed databases are like systems of government. Some are worse than others, but all of them suck. I'm not aware of any study that shows that X type of database reduces bugs, increases availability, and makes customers happier. Pick one that fits your application and deal with the suckiness.
- erikb 8y ago
  Why not take this pain and think about other forms of architecture? Maybe a consistent source of all truth is a bullshit requirement?
- threeseed 8y ago
  Distributed databases underpin every major web app today e.g. Facebook, Instagram, Google, Apple, Spotify etc. So clearly they are helping to increase availability and make customers happy.
nemothekid 8y ago
>Systems that guarantee consistency only experience a necessary reduction in availability in the event of a network partition. As networks become more redundant, partitions become an increasingly rare event. And even if there is a partition, it is still possible for the majority partition to be available In my experience, yes network partitions are incredibly rare. However 99% of my distributed ststem partitions have little do with the network. When running databases on a cloud environment network partitions can occur for a variety of reasons that don’t actually include the network link between databases: 1. The host database is written in a GC’d language and experiences a pathological GC pause. 2. The Virtual machine is migrated and experiences a pathological pause 3. Google migrates your machine with local SSDs, fucks up that process and you lose all your data on that machine (you do have backups right?) 4. AWS retires your instance and you need to reboot your VM. You may never see these issue if you are running a 3 or 5 cluster database. I began seeing issues like this semi regularly once the cluster grew to 30-40 machines (Cassandra). Now I will agree that none of the issues took down majority, but if your R=3, it really only takes an unlucky partition to fuck up an entire shard
- Quekid5 8y ago
  I may be a bit of an old fart, but this is the exact reasoning behind my decision to never go with "distributed X" if there's a "single-machine X" where you can just vertically scale. If you can afford 3-5 machines/VMs for a cluster you can almost certainly afford a single machine/VM with 2-4x the resources/CPU and chances are that it'll perform just as well (or better) because it doesn't have network latency to contend with. Of course if you're around N >= 7 or N >= 9, then you should perhaps start considering "distributed X". As long as your application is mostly built on very few assumptions about consistency it's usually pretty painless to actually go distributed. Of course, there are legitimate cases where you want to go distributed even with N=3 or N=5, but IME they're very few and far between... but if your 3/5 machines are co-located then it's really likely that the "partitioning" problem is one of the scenarios where they actually all go down simultaneously or the availability goes to 0 because you can't actually reach any of the machines (which are likely to be on the 'same' network).
  - 8y ago
mr_pickles 8y ago
No mention of Clustrix? The Spanner paper was in 2012. Clustrix started working on a "NewSQL" database in 2006 and had a product out in 2010.
- antoncohen 8y ago
  One big difference is that Clustrix (YC'06) was a single datacenter consistent cluster, while Spanner can do multi-region consistency. It will be interesting to see what MariaDB does with Clustrix, now that they have acquired them, fingers crossed that they open source the technology.
openloop 8y ago
Bad spanner. Graph databases with consistency.
deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
idubrov 8y ago
Any comments on how FoundationDB might be the same / different?
- Dave_Rosenthal 8y ago
  FoundationDB adheres to the highest level of consistency (linearizability) and, unlike Spanner or its derivatives, does not rely on clocks to achieve this. I'm surprised FoundationDB is not mentioned in the article. Maybe it didn't help the author make his point, but it is both the most mature of the new breed of ACID+noSQL databases and completely free/open source.
  - riku_iki 8y ago
    One reason is may be because it is not 'SQL' DB, which is probably prerequisite for NewSQL category.
  - grogers 8y ago
    According to their docs, FoundationDB only provides serializable isolation, so it isn't the same as the others in the post which offer strict serializable isolation (multi-key version of linearizability). Without strong clocks you can't have strict serializablity and scale beyond a single log (but one log can get you pretty damn far). TBH most people probably only need serializable transactions anyway. https://apple.github.io/foundationdb/developer-guide.html#transactions-in-foundationdb https://apple.github.io/foundationdb/developer-guide.html#tr...
    - ryanworl 8y ago
      While that documentation isn’t wrong, FoundationDB also provides external consistency. https://forums.foundationdb.org/t/keyspace-partitions-performance/168/7 https://forums.foundationdb.org/t/keyspace-partitions-perfor...
- chubot 8y ago
  I'm pretty sure FoundationDB still relies on the more traditional model of hardware: "many fast, identically-spec'd machines on a rack, with fast reliable networking". In other words, I don't think you can run Foundation DB across geographically dispersed data centers, as you can with Spanner, CockroachDB, and others. Running on cloud VMs is a very hostile environment for a database, as others have pointed out in this thread. Though I'm happy to hear from someone who knows more about FoundationDB.
jules 8y ago
The CAP theorem has been truly disastrous for databases. The CAP theorem simply says that if you have a database on 2 servers and the connection between those serves goes down, then queries against one server don't see new updates from the other server, so you either have to give up consistency (serve stale data) or give up availability (one of the servers refuses to process further requests). That's all that CAP is, but somehow half the industry has been convinced that slapping a fancy name on this concept is a justification for giving up consistency (even when the network and all servers are fully functional) to retain availability. The A for availability in CAP means that ALL database servers are fully available, which is unnecessary in practice, because clients can switch to the other servers. Giving up consistency introduces big engineering challenges. You're getting something that most people don't need in return for a large cost.
- Dave_Rosenthal 8y ago
  This is exactly right, and perhaps the most cogent explanation of the CAP theorem on the internet. For a longer explanation of the same idea which includes a concrete example of how you can get "high availability" in a CP system, see: https://apple.github.io/foundationdb/cap-theorem.html https://apple.github.io/foundationdb/cap-theorem.html
- gowld 8y ago
  You can't always "choose another server" while maintaining low latency operations.
- jshen 8y ago
  "The A for availability in CAP means that ALL database servers are fully available" Is this true? I always thought it meant that clients could continue to read and write to "the database" which could include the client switching to another node. There is nothing in CAP theorem about latency, so switching, even if it adds high latency, is fine by CAP theorem. This lack of accounting for latency is what makes CAP theorem less useful than a lot of people realize IMO.
  - zzzcpan 8y ago
sseth 8y ago
The author talks about high throughput achieved via batching. But he has not mentioned latency implications of batching, or perhaps I missed that in the article. Wouldn't batching lead to an increase in transaction latency, even if we achieve higher throughput?
cbsmith 8y ago
Seems to have a powerful straw man there about eventual consistency. The point of AP systems is not 100% availability, but rather higher availability. By the same reasoning, one should never do CP, because it is also not possible to have 100% consistency. Disk/memory/network corruption, even with ECC can overwhelm the ability to maintain consistency.
deleted 8y ago
[deleted]
acje 8y ago
I don’t know for sure, but to me the AP vs CP interpretation seems only to be a true limitation for distributed systems of exactly two nodes. I also like the blogs point that availability is not ever 100%, but I think the added cost of availability levels when going from an eventual consistency system to a linearizable one is underestimated because performance is going to be a significant availability factor, not only failuremodes as discussed.
omneity 8y ago
It's becoming harder to evaluate the guarantees of the most recent so called "cloud-scale" databases, Spanner or CockroachDB for instance. Is there anything out there like a standard benchmark to compare these offerings more accurately?
rbranson 8y ago
Maybe I’m missing something, but the way this reads seems to attribute Spanner’s concurrency control entirely to TrueTime. Spanner still uses partitioned Paxos to establish consensus and 2PC for multi-partition write transactions.
erikb 8y ago
Consistency is a requirement that holds us back deeply. Implementing constency with eventually voted know-all masters are also holding us back. Looking at this makes an engineers heart cringe. In fact the current architecture is only slightly better than what we had in 2005, more than 10 years ago. And if it runs not on google or amazon clouds, and therefore on UNRELIABLE networks, the voting itself can fail and you have a centralized system without center. And while we claim that we need a central point of knowledge that is at sync throughout the whole system, the whole world AROUND the IT works completely without being in sync and completely without knowing it all (sometimes even having no knowledge or assumed but wrong knowledge), at a scale that IT systems only to some degree come close to nowadays.
bnastic 8y ago
> With 10ms batches, Calvin was able to achieve a throughput of over 500,000 transactions per second. For comparison, Amazon.com and NASDAQ likely process no more than 10,000 orders/trades per second even during peak workloads. I haven’t worked with NASDAQ stream directly, but knowing how fast equities tick I find this “10,000 orders/sec” estimate quite low. Not to mention that 10ms delay in confirming an order would be really terrible.
- fastest963 8y ago
  According to NASDAQ's website [1]: > Able to consistently sustain an order rate of over 100,000 orders per second at sub-40 microsecond average latency [1]: https://business.nasdaq.com/market-tech/marketplaces/trading https://business.nasdaq.com/market-tech/marketplaces/trading
deleted 8y ago
[deleted]
apaprocki 8y ago
> a specialized hardware solution that uses both GPS and atomic clocks to ensure a minimal clock skew across servers Maybe I'm wrong, but pretty much anyone with a DC is going to use Microsemi (previously Symmetricom) grandmaster clocks with all the bells-and-whistles including the internal Rubidium atomic oscillator along with PTP to keep everything in sync with multiple layers of redundancy. A specialized hardware solution that uses "both GPS and atomic clocks" to guarantee ~15ns RMS to UTC are just a shopping cart away.
andras_gerlits 8y ago
I have a whitepaper about scalable consistency guarantees using optimistic transactions. It should solve these very problems. https://medium.com/@andrasgerlits/optimistic-acid-transactions-4f193844bb5f https://medium.com/@andrasgerlits/optimistic-acid-transactio...
rkarthik007 8y ago
CTO of YugaByte here. We firmly stand by our claims, and I wanted to explain more. From the post by Daniel: << CockroachDB, to its credit, has acknowledged that by only incorporating Spanner’s software innovations, the system cannot guarantee CAP consistency (which, as mentioned above, is linearizability). YugaByte, however, continues to claim a guarantee of consistency. I would advise people not to trust this claim. YugaByte, by virtue of its Spanner roots, will run into consistency violations when the local clock on a server suddenly jumps beyond the skew uncertainty window. >> The statement about YugaByte DB is incorrect. 1. With respect to CAP, both Cockroach DB (https://www.cockroachlabs.com/blog/limits-of-the-cap-theorem/ https://www.cockroachlabs.com/blog/limits-of-the-cap-theorem...) and YugaByte DB (https://docs.yugabyte.com/latest/faq/architecture/#how-can-yugabyte-db-be-both-cp-and-ha-at-the-same-time https://docs.yugabyte.com/latest/faq/architecture/#how-can-y...) are CP databases with HA and there is really no difference in the claims. 2. With respect to Isolation level in ACID, YugaByte DB does not make the linearizability (called external consistency by Google Spanner) claim. YugaByte DB offers Snapshot Isolation (detects write-write conflicts) today and Serializable isolation (detect read-write and write-write conflicts) is in the roadmap (https://docs.yugabyte.com/latest/architecture/transactions/isolation-levels/ https://docs.yugabyte.com/latest/architecture/transactions/i...). 3. We have publicly claimed that we do rely on NTP and max clock skew bounds to guarantee consistency. For example, slide 43 of our NorCal DB Day talk (https://www.slideshare.net/YugaByte/yugabyte-db-architecture-storage-engine-and-transactions https://www.slideshare.net/YugaByte/yugabyte-db-architecture...) we mention we are “relying on bounded clock sync (NTP, AWS Time Sync, etc).”
- abadid 8y ago
  Looks like you cross-posted this comment here and on my blog, so I'll also cross-post in my response. I am quite confused by your statement that YugaByte does not claim linearizability. The C of CAP is linearizability (see the original CAP theorem https://users.ece.cmu.edu/~adrian/731-sp04/readings/GL-cap.pdf https://users.ece.cmu.edu/~adrian/731-sp04/readings/GL-cap.p...). By claiming to be CP from CAP, you are claiming linearizability. CockroachDB also makes the same CP claim. However, they explicitly walk back from this claim (https://www.cockroachlabs.com/blog/living-without-atomic-clocks/ https://www.cockroachlabs.com/blog/living-without-atomic-clo...). In contrast, YugaByte documentation makes no effort to walk back from this claim. Rather, the documentation seems to indicate that YugaByte is linearizable at: https://blog.yugabyte.com/jepsen-testing-on-yugabyte-db-database/ https://blog.yugabyte.com/jepsen-testing-on-yugabyte-db-data... and https://docs.yugabyte.com/latest/develop/learn/acid-transactions/ https://docs.yugabyte.com/latest/develop/learn/acid-transact.... Also, I would encourage you to read the Herlihy-Wing paper on linearizability published in 1990. You seem to be confusing linearizability with ACID isolation levels, which are actually a different concept. Peter Bailis has a good blog post on the difference: http://www.bailis.org/blog/linearizability-versus-serializability/ http://www.bailis.org/blog/linearizability-versus-serializab.... On your third point, we agree. However, the point of my post is that relying on max clock skew without hardware support is dangerous. I’m going so far as to say that it’s so dangerous that it is incorrect to claim consistency guarantees when you make such assumptions. I would really encourage YugaByte to consider using global consensus instead of partitioned consensus. I believe you will find it much easier to support serializability (which as you said, is on your road map) and linearizability.
voidmain 8y ago
This article lacks any reference to FoundationDB, which offers external consistency and serializable distributed transactions without trusting clocks in any way. We designed it starting in 2009 and so its lineage is independent of either Calvin or Spanner. FDB doesn't have an actively developed SQL layer at the moment, so I guess you could say it isn't a "NewSQL" database, but none of the properties under discussion have much to do with the query language.
- uluyol 8y ago
  As best as I could find, FoundationDB doesn't provide strict serializability (i.e. linearizability + serializability) and doesn't tackle multi-DC deployments. So I wouldn't put it in the same class as FaunaDB or Spanner.
  - jscissr 8y ago
    There is a "Datacenter-aware mode": https://apple.github.io/foundationdb/configuration.html#datacenter-aware-mode https://apple.github.io/foundationdb/configuration.html#data... Here is some discussion about linearizabilty in fdb: https://news.ycombinator.com/item?id=16884882 https://news.ycombinator.com/item?id=16884882
    - uluyol 8y ago
      Without knowing exactly how reads and writes are replicated, I am skeptical. I have gone through the technical documentation and haven't found many details. For reference, I've done work in storage and consensus algorithms and I can tell you for a fact that without using a consensus algorithm for either reconfiguration or request propagation, you will have consistency violations. I would love to be proven wrong, as more systems with strong consistency guarantees is better, but for now, I don't believe that foundation db provides stronger guarantees than serializable reads and writes.
      - voidmain 8y ago
        FoundationDB uses a consensus algorithm for reconfiguration, but not in the (happy path) transaction pipeline. It provides (by default) strict serializability (i.e. serializability and external consistency/linearizability) for arbitrary, ad hoc, interactive transactions, and it's expected to provide excellent performance when every single transaction is cross node (so e.g. indexes can be efficiently updated this way). It provides better fault tolerance than consensus replicated databases typically can because it needs only N+1 replicas instead of 2N+1 to survive N faults (it keeps 2N+1 replicas of tiny configuration for consensus, if course). It has the best testing story in the industry and is used at scale by, among others, the largest company in the world. Because it doesn't have consensus in the datapath it can have lower latencies than consensus replicated databases in multi region deployments, it also supports asynchronous replication, and an upcoming feature will provide a unique option for multi region failover with sub geographic write latencies, maintaining full transaction durability in failover, as if in synchronous replication, except for the exceptionally rare case where the failure of multiple datacenters in a region are exactly simultaneous. Besides its extensive documentation, you can read its source code and run its deterministic simulation tests yourself if you are interested (it's Apache licensed). Skepticism on these points was reasonable when we originally launched it in 2012 but is getting a little silly in 2018.
redwood 8y ago
Odd to see the most popular distributed operational database system in the world left out here.
rifung 8y ago
Probably a dumb question but if you need all servers to talk to each other to achieve consensus, how do you deal with network latency? Do people just keep data in separate zones so they aren't globally replicated?
shaklee3 8y ago
I'm hoping aphyr will comment on this...
bbcbasic 8y ago
MongoDb is webscale. It scales right up. Sharding is the secret webscale sauce.
yoava 8y ago
I like the realistic view of noSql vs CAP and the different tradeoffs. We do need to talk more about the disadvantages of noSql systems and all the new database alternatives. Which brings me to my point - what the author fails to talk about is why spanner has taken the design decision they have made. He does claim scalability, but that is a very general word. I believe the spanner decision is to avoid making a global voting, as in global across the entire world, from North America, to Asia, Europe and even as far as Australia. Such a global voting will take a lot of time - impose seconds of later on any write. I think the author, when he talks about having a single global vote with a small window, thinks in terms of a single database, maybe two regions on the Continental USA, as apposed to spanner trying to be Geo distributed.
AtlasBarfed 8y ago
The context of this article still seems to constrain consistency to guarantees for single atomic units of data (atomic at the node level). But the second you have multiple-node transactions (your data after all in a truly distributed system will exist on mulitple nodes), then the single-node reliability become dependent on multiple data exchanges across networks for confirmations. Then your network partition exposure starts to snowball. Same thing for joins. Great if your joins are perfectly served by a node-local sharding strategy, but when you retrieve from multiple nodes, again your network partitioning risk starts to compound. In AWS you will see noisy neighbor networks and network unreliability. Your VMs get yanked. Your EBS might experience an IO pause. GC pauses. Or simply one of your nodes might get swamped by a query spike.
jbkavungal 8y ago
Interesting article

510 points by evanweaver 8y ago

And majority of developers don’t need NoSQL or NewSQL - they are well off with PostgreSQL. I think the old and tried should be the default choice here and other paths taken only after careful consideration and good justifications.

threeseed 8y ago
Or how about instead of just blindly picking a system. You spend a week or two to evaluate a few that you think might mesh well for your data structures. PostgreSQL doesn't work well for many, many common use cases.

abadid 8y ago

I'm the author of that post. I'm happy to respond to comments on the post on this thread for the next several hours. You can also leave comments on the post itself, and I will respond there at any time.

plopz 8y ago
I've only ever worked on small projects so I'm not at all familiar with these very high-scale distributed databases but from the post it seems to indicate that Spanner is in a league of its own because it integrated hardware into the mix where the others are software only. What are the differences in scale between the two categories mentioned?
- abadid 8y ago
  Yes, Spanner is quite unusual in the distributed database world in how hardware is a pretty important part of their solution. Other systems may claim important integrations with hardware, but for Spanner, the architecture really relies on particular hardware assumptions. To answer your question about scale: there is no real practical difference in scalability between the two categories discussed in the post. Partitioned consensus has better theoretical scalability, but I am not aware of any real-world workload that can not be handled by global consensus with batching.
  - deleted 8y ago
    [deleted]
  - ryanobjc 8y ago
    I think this "global consensus with batching does everything partitioned does" is a very much theory vs practice type of statement. As in, in theory there is no difference between theory and practice :-) I've seen those batched consensus systems, and honestly, you're kidding yourself if you think they can handle a million qps. Just transmit time of the data on ethernet would become an issue alone! Even with 40 gig - transmit time never becomes free. So now you're stuffing 1/10th of a million qps worth of data via a single set of machines (3, 5, 7, 9? Some relatively small amount) Am I misunderstanding you? Hopefully I am!
evanweaver 8y ago
Do these concerns also apply in an HTAP or OLAP context e.g. systems like Cloudera's Kudu, which uses Hybrid Time? Or maybe Volt which you also worked on?
- xtacy 8y ago
  OLAP systems tend to be read only (for analytics) so the question of transactions and consistency isn't really applicable.
  - kwillets 8y ago
    They don't load themselves.
- kwillets 8y ago
  Vertica seems to include some of these observations, such as global consistency and group commit. I believe these are easier to achieve (lower overhead) in OLAP due to fewer, larger transactions.
- nickerten 8y ago
  I've worked on a system that loaded data into Kudu in near real time and simultaneously ran queries on the data. Kudu has no transactions, consistency is eventual which was sufficient due to our near-real time constraint, however you do need a stable NTP source. We have lost data when the cluster could not get a reliable NTP connection, decided to shut down and tablet servers' data files became corrupted.
jgalentine007 8y ago
I recently came across CockroachDB and thought it's capabilities interesting, almost too good to be true. I also have been looking at Citus Data which shards and distributes transactions, are you aware of any consistency shortcomings with it?
- cube2222 8y ago
  CockroachDB's performance is dependent on time synchronization. If a node detects it's too far behind, it will commit suicide. However, before it detects it, there is a possibility of stale reads. https://www.cockroachlabs.com/docs/stable/recommended-production-settings.html#clock-synchronization https://www.cockroachlabs.com/docs/stable/recommended-produc...
  - jgalentine007 8y ago
    Thanks for pointing that out - I have yet to rtfm and dive deep. I wonder how frequently time sync problems occur in virtual environments after ntp syncing - I've seen pretty erratic behavior on virtual active directory domain controllers even after syncing with hyper-v and vmware.
    - brazzledazzle 8y ago
      It’s been a long time since I messed with domain controllers but I believe Microsoft used to have explicit guides for handling time on virtual DCs. At that time we kept around a a couple hardware DCs to be safe but I do remember having the VMware agent correct the time could result in some bad results. I think it was because it immediately fixed the time and didn’t slowly correct the drift but it’s been a long time so my memory could be off.
    - amaccuish 8y ago
      Time shouldn't be a massive issue for AD no? It's a vector clock, not a UTC clock. The UTC clock is only used to solve conflicts no?
      - 8y ago
xtacy 8y ago
Nice writeup, but given the title, I was hoping you would also have a constructive test case that shows that these systems fail to meet their guarantees (like Jepsen did). :) I wonder if any of the aforementioned systems (Calvin/Spanner/YugaByte) that can opportunistically commit transactions and detect issues and roll back + retry all within the scope of the RPC so it can still conform to linearisability requirement?
mehrdadn 8y ago
The problem isn't a lack of very accurate clocks per se, right? What you need is an accurate bound on clock error, whatever it might be. It sounds to me that non-Googles are specifying bound parameters that, unlike Google, they cannot guarantee. Doesn't that mean the blame goes to whomever put inaccurate bound parameters into the software? Why blame the algorithm for garbage input?
- CydeWeys 8y ago
  The problem is that the bound on clock error directly affects your performance. So if you're willing to accept, say, a second in clock error, then all transactions will take a minimum of one second. That level of performance is going to be unacceptable in many situations. The potential clock error on VMs without dedicated time-keeping hardware is so large that performance turns into absolute garbage.
  - mehrdadn 8y ago
    I would understand if the complaint was that Spanner is too slow without expensively accurate clocks and synchronization. But the complaint is that Spanner fails to guarantee consistency, which doesn't make sense to me. The requirements clearly include giving a valid clock bound, so if you give an invalid clock bound, it's clearly your fault for getting incorrect results, not Spanner's!
    - CydeWeys 8y ago
      Spanner does guarantee consistency, thanks to its use of hardware atomic clocks and GPS. It's alternatives like CockroachDB that don't have this dedicated hardware that can fail to guarantee consistency if clocks get of sync (a problem that can't happen in Spanner). Spanner is really fast and massively parallelizable.
      - mvijaykarthik 8y ago
        We recently open-sourced https://github.com/rubrikinc/kronos https://github.com/rubrikinc/kronos for the exact same problem. Coincidentally I shared that on Show HN just today: https://news.ycombinator.com/item?id=18037609 https://news.ycombinator.com/item?id=18037609
joemag 8y ago
Hi, you used DynamoDB as an example of a weakly consistent system in the opening paragraph, but it actually supports both modes [1]. The point of confusion might come from the fact that the service described in 2007 Dynamo paper was an inspiration for DynamoDB, rather than DynamoDB itself. Disclaimer, I work for AWS, but not on DynamoDB team. [1] https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
- abadid 8y ago
  Thanks. I added a parenthetical remark to the post to indicate that I was talking about DynamoDB's default settings.
  - Twirrim 8y ago
    It's not really the default settings, per se. You don't have to change any bit of configuration about your database to get consistency. The DynamoDB API gives you the GetItem API call and a boolean property to choose to make it a consistent read. It's left as a very simple task for developers leveraging DynamoDB to make the appropriate trade offs on consistent or inconsistent read. source: Used to work for AWS on a service that heavily leveraged DynamoDB. Not _once_ did we experience any problems with consistency or reliability, despite them and us going through numerous network partitions in that time. The only major issue came towards the end of my time there when DynamoDB had that complete service collapse for several hours. On the sheer scale that DynamoDB operates at, it's more likely to be a question of "How many did we automatically handle this week?" than "How often do we have to deal with network partitions?"
    - dsp1234 8y ago
      From the GetItem docs[0] "GetItem provides an eventually consistent read by default." This seems to meet the definition of "DynamoDB's default settings" [0] - https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html https://docs.aws.amazon.com/amazondynamodb/latest/APIReferen...
romed 8y ago
I question your statement that building apps on weakly-consistent systems is so difficult. I’ve worked on very large scale systems that you’ve definitely heard of and probably used that are built atop storage systems with very weak semantics and asynchronous replication. Aren’t such systems existence proofs, or do you think there’s just a huge difference between the abilities of engineers in various organizations?
- cloakandswagger 8y ago
  He said it was difficult, not impossible. I think it's a fairly noncontroversial statement. Dealing with eventual consistency is always going to be more difficult and require more careful thought and preparation than immediately consistent systems. How many programmers do you think are out there that have only ever worked on systems that use a single RDBMS instance, and what would happen if they tried to apply their techniques to a distributed, eventually consistent environment?
  - wpietri 8y ago
    Exactly. My dad was coding when DBMSes rose to prominence, and it was basically a way to take a bunch of things that were hard to think about and sweep them under the rug. People wrote plenty of good software before they existed, but if you wanted to write a piece of data, you had to think about which disk and where on the disk and exactly the record format. Most programmers just wanted a genie that they could give data to and later ask for it back. It's the same today, but worse. Most programmers still want a simple abstraction that lets them just build things. But now it's not just which sector on which disk, but also which server in which data center on which continent, while withstanding the larger number of failure modes at that scale. When necessary, people can explicitly address that complexity. But it has a big cost, a high cognitive load.
- groestl 8y ago
  The biggest challenge, in my experience, is explaining the weak consistency guarantees to stakeholders, especially in a QA setting (e.g. product owner demos the product to colleagues, numbers are not immediately up-to-date).
felixgallo 8y ago
why didn't you disclose your connections in your post? It appears you've written much the same article for FaunaDB's official blog in the past (https://fauna.com/blog/distributed-consistency-at-scale-spanner-vs-calvin https://fauna.com/blog/distributed-consistency-at-scale-span...).
- vidarh 8y ago
  The article specifically states that Calvin comes out of his research group and that FaunaDB was inspired by Calvin.
- abadid 8y ago
  That post was actually quite different than this one. This one focuses on distributed vs. global consensus. But I added a note at the end that clearly documents my connection to FaunaDB. As far as Calvin, the post itself clearly says that it came out of my research group.
peterwwillis 8y ago
"Systems that guarantee consistency only experience a necessary reduction in availability in the event of a network partition." Many of the distributed clusters I've maintained had crap infrastructure and no change control, and parts of the clusters were constantly going down from lack of storage, CPU and RAM, or bad changes. The only reason the applications kept working were either (1) the not-broken vnodes continued operating as normal and only broken vnodes were temporarily unavailable, or (2) we shifted traffic to a working region and replication automatically caught up the bad cluster once it was fixed. Clients experienced increased error rates due primarily to these infrastructure problems, and very rarely from network partition. Does your consistent model take this into account, or do you really assume that network partition will be the only problem?
- mannykannot 8y ago
  It seems you have other problems (crap infrastructure and no change control) to deal with before the issues in this article become your biggest concern, but are not the cases you list themselves partition problems?
  - peterwwillis 8y ago
    They cause partition, but their origin isn't the network. Nobody who runs a large system has perfectly behaving infrastructure. Infrastructure always works better in a lab than in the real world. Even if you imagine your infrastructure is rock-solid, people often make assumptions, like their quota is infinite, or their application will scale past the theoretical limits of individual network segments, i/o bounding, etc. The point is, resources cause problems, and the network is just one of many resources needed by the system. Other resources actually have more constraints on them than the network does. If a resource is constrained, it will impact availability in a highly-consistent model. The author states that simply adding network redundancy would reduce partitions, and infrastructure problems are proof that this is very short-sighted. "You have bigger problems" - no kidding! Hence the weak-consistency model!
- StreamBright 8y ago
  Even if you maintain your infrastructure properly you run on x86 servers with disks and CPUs that need cooling, using network devices that have fascinating failure scenarios. I guess assuming that your infra is not reliable is a must for any database nowadays.
reilly3000 8y ago
Is there a "best-of-both-worlds" approach that could work, or are these two approaches mutually exclusive? I have to imagine that time drift can eventually reconciled with some kind of time delta.
- abadid 8y ago
  The two approaches seem mutually exclusive to me.
jakelarkin 8y ago
won't batching transactions increasing the failure rate by a multiple of (batch size)? how should users reason about that tradeoff?
Serow225 8y ago
I think it would be helpful to move the disclaimer to the top of the post, just for clarity's sake. Is there a forecasted date on the release of the independent Jepsen study? Who is performing it? Thanks!!
CBLT 8y ago
Great read, thank you for sharing. Do you have any opinion on the design of Eris[0]? Consistency is achieved with extra hardware, but that hardware is a network-level sequencer. [0]: https://syslab.cs.washington.edu/papers/eris-sosp17.pdf https://syslab.cs.washington.edu/papers/eris-sosp17.pdf
- rwtxn 8y ago
  AFAIK Eris (and previous similar work from the same group) assume network ordering guarantees that can be provided in a single datacenter but probably not in a WAN setting. This discussion is about distributed and geo-replicated databases.

dboreham 8y ago

In case you're wondering : this is a very worthwhile read.

zzzcpan 8y ago
Too many false claims for my taste. Especially wrt to eventual consistency and consistency guarantees. Also some misinterpretation of CAP theorem to fit the narrative. Very typical FaunaDB promotional post.
- haberman 8y ago
  Without some description of what "false claims" and "misinterpretation" you mean, this is a content-free post.
  - jessaustin 8y ago
    It has more "content" than the post to which it replies. "This is good." "No, it is bad, in these particular areas." [EDIT:] Granted, the post could still be wrong, but ISTM accusations of content-freedom tend to be projections.
    - pdpi 8y ago
      "This is a good read" is a statement of opinion. You're free to assign whatever value you like to the op's opinion, but that opinion is "content" in and of itself. "The article has false claim" is a statement of fact. A vague statement of fact about a non-obvious topic that puts no effort into providing concrete examples and reasoning is pretty much worthless.
      - jessaustin 8y ago
        Adding qualifiers makes a statement have less content? Please try to imagine using this questionable argument in any other context... It's a controversial topic and TFA was written by someone with a commercial interest. Everything is opinion. The level of comment-policing I see here makes me quite suspicious of everything in TFA. [EDIT:] aargh I'll never learn every discussion that starts with "content-free" cliche goes nowhere... just downvote and move on...

deleted 8y ago

[deleted]

beamatronic 8y ago

Please be sure to distinguish between Couchbase and CouchDB. Couchbase is a CP system.

klodolph 8y ago

Anyone interested in this subject may also find it worthwhile to read the paper describing Spanner: http://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf http://static.googleusercontent.com/media/research.google.co...

haberman 8y ago

> We will trace the failure to guarantee consistency to a controversial design decision made by Spanner that has been tragically and imperfectly emulated in other systems. This post doesn't establish any "controversy" about Spanner's design decision. It only says that it requires special hardware, which other systems attempt to emulate despite not having this specialized hardware. To call this decision "controversial" I think one would need to show that it has some significant problem in the environment it was designed for.

1024core 8y ago
That's because the author is pushing Calvin, which came out of his research group.
abadid 8y ago
By controversial, I mean that in the database research community (where I spend most of my time), there is significant disagreement about global consensus vs. Spanner's choice of partitioned consensus.
- haberman 8y ago
  Disagreement about what though? Does Spanner's solution have an objective problem? Do you or others in your community have specific reasons to believe that it cannot deliver on its promises?
  - abadid 8y ago
    Spanner's approach requires help from hardware and several full time employees maintaining and ensuring the uncertainty guarantees. This increases the cost of the maintaining the system, which for Cloud Spanner is partially passed on to the end users. If you can build a system that doesn't require time synchronization, yet doesn't have any significant drawbacks relative to what Spanner provides, you'd be better off using this alternative system.
    - haberman 8y ago

dekhn 8y ago

This reads like an advertisement for the researcher's system combined with some FUD.

reinhardt1053 8y ago
would you mind to elaborate?
- wsy 8y ago
  Just to mention a few points: - The title implies that Spanner is at fault for something, then the article itself admits that Spanner is totally fine. - Then it goes on to argue that this approach can't possibly work without physical devices for precise time. However, there is no proof whatsoever. Please be a researcher and design an experiment that demonstrates a violated consistency guarantee. - Finally, not disclosing financial relationships with products which are praised in the article is really low, and has only been fixed after this has been pointed out here. And it is not marked as a later addendum, as usual for changed articles.

dgreensp 8y ago

This is the clearest presentation of why CAP is misleading that I’ve ever seen! Wonderful. If the app programmer knows they don’t have global consistency, just consistency per partition, I wonder if in practice there are ways to achieve the necessary application-level guarantees such as in the photo-sharing case (not that it sounds that appealing to need to do so).

xapata 8y ago
> application-level guarantees If your database doesn't "guarantee" consistency, but instead provides eventual consistency, your application needs to be aware of how long "eventually" is. In effect, you'd be coding something similar to Spanner's timestamp uncertainty estimation. In the photo sharing example, it'd be reasonable to put a delay on sharing new photos with other users until after permissions changes would have propagated.
- e12e 8y ago
  > delay on sharing new photos with other users until after permissions changes would have propagated. And then you'd maybe have to map the ux of changing permissions on an existing permission to a three-state: user makes change;ui shows change in progress;change takes effect and is reflected in ui.

tytso 8y ago

The post talks about Spanner using "a specialized hardware solution that uses both GPS and atomic clocks to ensure a minimal clock skew across servers." But what it fails to mention is that is that this time is distributed using a software solution --- NTP, in fact. Google makes its "leap-smeared NTP network" available via Google's Public NTP service. And it's not expensive for someone to buy their own GPS clock and run their own leap-smared NTP service. Yes, it means that someone who installs a NewSQL database will have to set up their own time infrastructure. But that's not hard! There are lots of things about hardware setup which are at a similar level of trickiness, such as UPS with notification so that servers can shutdown gracefully when the batteries are exhausted, locating your servers in on-prem data centers so that if a flood takes out one data center, you have continuity of service elsewhere, etc., etc. Or of course, you can pay a cloud infrastructure provider (which Google happens to provide, but Amazon and Azure also provides similar services) to take care all of these details for you. Heck, if you use Google Compute Platform, you can use the original Spanner service (accept no substitutes :-)

jrockway 8y ago
I agree. Attempting to remove the GPS and atomic clocks from the system seems like a premature optimization to me. They are cheap and the ability to order events according to an actual wall clock is a huge benefit. Some basics for less-critical applications: https://github.com/jrockway/beaglebone-gps-clock/blob/master/README.md https://github.com/jrockway/beaglebone-gps-clock/blob/master...
jakemoshenko 8y ago
I feel like the author is making a huge omission by not talking about the properties of multi-raft sharded DBs coupled with the TrueTime-like APIs from cloud providers in 2018. He links to a post from the CEO of CockroachDB written in early 2016, and Amazon launched their TimeSync service in 2017. It's likely that Spencer would have something different to say about the matter today.
- Quekid5 8y ago
  > I feel like the author is making a huge omission by not talking about the properties of multi-raft sharded DBs coupled with the TrueTime-like APIs from cloud providers in 2018. I mean this is all fine and dandy if it works (plus, how do know that it actually works in all the edge cases?), but that's a HUGE amount of complexity. IMO you really need to have a very good reason to even consider anything this complex.

romed 8y ago

I’ve seen comments on HN over the years in which someone Dunning-Kruegers their way into saying that TrueTime is easily replicated. I always wonder if they have sixteen senior SREs in their pocket, because that’s the level of production engineering Google applies to the problem. Time SRE has at various points had take measures up to and including calling the USAF and telling them their satellites are fucked up. If you don’t have the staff for this, the easiest way to get access to TrueTime is probably to just use Cloud Spanner.

sabujp 8y ago
the only solution to this is to launch your own satellites :)
- reilly3000 8y ago
  If we seal the surface of, then boil the ocean we can launch our own time satellites with completely sustainable propulsion.
Cshelton 8y ago
Also, I believe all the major cloud providers provide a "TrueTime" API service. I forgot the name that AWS uses, but you can call it on your EC2 instances and make sure your hosts are all in sync. It's pretty cool.
- otterley 8y ago
  AFAIK AWS is only offering NTP service with a GPS source (i.e. stratum 1). TrueTime appears to be a service offering that is a step above that in terms of the guarantees it provides.
  - Cshelton 8y ago
    Ah, I've never had a use case for anything more accurate than NTP

gregwebs 8y ago

In the post and another comment here it is stated that Calvin can handle any real-world workload. However, according to my reading of the Calvin paper, one must understand the update keys before starting the transaction. I also experienced limitations when trying to use FaunaDB: it doesn't support ad-hoc queries and it only allows for indexed queries. I really like the Calvin protocol, and is does seem perfectly suited to many application workloads, but it is odd to me though, to see Calvin presented as purely superior to all alternatives. It seems like more research and work needs to be done to create systems that can address its shortcomings around transactions that query for keys (3.2.1 Dependent transactions in the paper), including ad-hoc queries, and even interactive queries (with a transaction). Disclaimer: I work on TiDB/TiKV. TiKV (distributed K/V store) uses the percolator model for transactions to ensure linearizability, but TiDB SQL (built on top) allows write skew.

freels 8y ago
Support for dependent reads is a subject that the Calvin paper only touches on briefly, describing one strategy of using "reconnaissance reads" to determine the key set. In FaunaDB, this is formalized as an optimistic concurrency control mechanism that combines snapshot reads at the coordinator and read-write conflict detection within the transaction engine. By the time a transaction is ready to commit the entire key set is known. I wrote a blog post that goes into more detail here: https://fauna.com/blog/acid-transactions-in-a-globally-distributed-database https://fauna.com/blog/acid-transactions-in-a-globally-distr... Better analytics support, including ad-hoc queries is on our roadmap. That being said, requiring indexes was a design choice: First and foremost, we want FaunaDB to be the best operational database. It is a lot easier to write fast, predictable queries if they cannot fall back to a table scan! While FaunaDB requires you to create indexes to support the views your workload requires, the reward is a clearer understanding of how queries perform. We've built indexes to be pretty low commitment, however. You can add or drop them on the fly based on how your workload evolves over time.
- gregwebs 8y ago
  Thanks for the link to the explanation. It seems like FaunaDB is doing the hard work to show how Calvin can actually be used for all types of workloads.

sabujp 8y ago

since spanner is distributed on the backend, any function that uses multiple sql commands that might be called or ingested by multiple sharded servers which are then sent to spanner, should always have these calls sent in a single transaction

CydeWeys 8y ago

It's definitely true that putting the burden of consistency on developers (instead of on the DB) results in a lot more tricky work for developers. On my project, which started six years ago, we use Cloud Datastore, because Cloud Spanner hadn't come out yet. It results in complicated, painful code that would be completely unnecessary with stronger transactional guarantees. Some examples: https://github.com/google/nomulus/blob/master/java/google/registry/backup/CommitLogCheckpointStrategy.java https://github.com/google/nomulus/blob/master/java/google/re... https://github.com/google/nomulus/blob/master/java/google/registry/model/ofy/CommitLoggedWork.java https://github.com/google/nomulus/blob/master/java/google/re... https://github.com/google/nomulus/blob/master/java/google/registry/model/index/ForeignKeyIndex.java https://github.com/google/nomulus/blob/master/java/google/re... It's no surprise that we're currently running screaming to something with stronger transactional guarantees.

wikibob 8y ago
I’m guessing that Nomulus is your project? I ran across it before and just wanted to say it’s really cool that this is open sourced.
- CydeWeys 8y ago
  I'm the tech lead of it. Glad to hear that you've heard of it before, and yeah, I think it's pretty cool it's open sourced too, which is why I made it happen! https://opensource.googleblog.com/2016/10/introducing-nomulus-open-source.html https://opensource.googleblog.com/2016/10/introducing-nomulu...
- guillon 8y ago
  Any idea if any company is planning to create an offer dedicates To .BRAND new gTLDs?
erikb 8y ago
Why does anybody need to provide consistency? Often you don't have complete consistency anyways. There are bugs, there are time delays, there is parallel processing. Why even have the requirement?

peterwwillis 8y ago

Distributed databases are like systems of government. Some are worse than others, but all of them suck. I'm not aware of any study that shows that X type of database reduces bugs, increases availability, and makes customers happier. Pick one that fits your application and deal with the suckiness.

erikb 8y ago
Why not take this pain and think about other forms of architecture? Maybe a consistent source of all truth is a bullshit requirement?
threeseed 8y ago
Distributed databases underpin every major web app today e.g. Facebook, Instagram, Google, Apple, Spotify etc. So clearly they are helping to increase availability and make customers happy.

nemothekid 8y ago

>Systems that guarantee consistency only experience a necessary reduction in availability in the event of a network partition. As networks become more redundant, partitions become an increasingly rare event. And even if there is a partition, it is still possible for the majority partition to be available In my experience, yes network partitions are incredibly rare. However 99% of my distributed ststem partitions have little do with the network. When running databases on a cloud environment network partitions can occur for a variety of reasons that don’t actually include the network link between databases: 1. The host database is written in a GC’d language and experiences a pathological GC pause. 2. The Virtual machine is migrated and experiences a pathological pause 3. Google migrates your machine with local SSDs, fucks up that process and you lose all your data on that machine (you do have backups right?) 4. AWS retires your instance and you need to reboot your VM. You may never see these issue if you are running a 3 or 5 cluster database. I began seeing issues like this semi regularly once the cluster grew to 30-40 machines (Cassandra). Now I will agree that none of the issues took down majority, but if your R=3, it really only takes an unlucky partition to fuck up an entire shard

Quekid5 8y ago
I may be a bit of an old fart, but this is the exact reasoning behind my decision to never go with "distributed X" if there's a "single-machine X" where you can just vertically scale. If you can afford 3-5 machines/VMs for a cluster you can almost certainly afford a single machine/VM with 2-4x the resources/CPU and chances are that it'll perform just as well (or better) because it doesn't have network latency to contend with. Of course if you're around N >= 7 or N >= 9, then you should perhaps start considering "distributed X". As long as your application is mostly built on very few assumptions about consistency it's usually pretty painless to actually go distributed. Of course, there are legitimate cases where you want to go distributed even with N=3 or N=5, but IME they're very few and far between... but if your 3/5 machines are co-located then it's really likely that the "partitioning" problem is one of the scenarios where they actually all go down simultaneously or the availability goes to 0 because you can't actually reach any of the machines (which are likely to be on the 'same' network).
- 8y ago

mr_pickles 8y ago

No mention of Clustrix? The Spanner paper was in 2012. Clustrix started working on a "NewSQL" database in 2006 and had a product out in 2010.

antoncohen 8y ago
One big difference is that Clustrix (YC'06) was a single datacenter consistent cluster, while Spanner can do multi-region consistency. It will be interesting to see what MariaDB does with Clustrix, now that they have acquired them, fingers crossed that they open source the technology.

openloop 8y ago

Bad spanner. Graph databases with consistency.

deleted 8y ago

[deleted]

deleted 8y ago

[deleted]

deleted 8y ago

[deleted]

idubrov 8y ago

Any comments on how FoundationDB might be the same / different?

Dave_Rosenthal 8y ago
FoundationDB adheres to the highest level of consistency (linearizability) and, unlike Spanner or its derivatives, does not rely on clocks to achieve this. I'm surprised FoundationDB is not mentioned in the article. Maybe it didn't help the author make his point, but it is both the most mature of the new breed of ACID+noSQL databases and completely free/open source.
- riku_iki 8y ago
  One reason is may be because it is not 'SQL' DB, which is probably prerequisite for NewSQL category.
- grogers 8y ago
  According to their docs, FoundationDB only provides serializable isolation, so it isn't the same as the others in the post which offer strict serializable isolation (multi-key version of linearizability). Without strong clocks you can't have strict serializablity and scale beyond a single log (but one log can get you pretty damn far). TBH most people probably only need serializable transactions anyway. https://apple.github.io/foundationdb/developer-guide.html#transactions-in-foundationdb https://apple.github.io/foundationdb/developer-guide.html#tr...
  - ryanworl 8y ago
    While that documentation isn’t wrong, FoundationDB also provides external consistency. https://forums.foundationdb.org/t/keyspace-partitions-performance/168/7 https://forums.foundationdb.org/t/keyspace-partitions-perfor...
chubot 8y ago
I'm pretty sure FoundationDB still relies on the more traditional model of hardware: "many fast, identically-spec'd machines on a rack, with fast reliable networking". In other words, I don't think you can run Foundation DB across geographically dispersed data centers, as you can with Spanner, CockroachDB, and others. Running on cloud VMs is a very hostile environment for a database, as others have pointed out in this thread. Though I'm happy to hear from someone who knows more about FoundationDB.

jules 8y ago

The CAP theorem has been truly disastrous for databases. The CAP theorem simply says that if you have a database on 2 servers and the connection between those serves goes down, then queries against one server don't see new updates from the other server, so you either have to give up consistency (serve stale data) or give up availability (one of the servers refuses to process further requests). That's all that CAP is, but somehow half the industry has been convinced that slapping a fancy name on this concept is a justification for giving up consistency (even when the network and all servers are fully functional) to retain availability. The A for availability in CAP means that ALL database servers are fully available, which is unnecessary in practice, because clients can switch to the other servers. Giving up consistency introduces big engineering challenges. You're getting something that most people don't need in return for a large cost.

Dave_Rosenthal 8y ago
This is exactly right, and perhaps the most cogent explanation of the CAP theorem on the internet. For a longer explanation of the same idea which includes a concrete example of how you can get "high availability" in a CP system, see: https://apple.github.io/foundationdb/cap-theorem.html https://apple.github.io/foundationdb/cap-theorem.html
gowld 8y ago
You can't always "choose another server" while maintaining low latency operations.
jshen 8y ago
"The A for availability in CAP means that ALL database servers are fully available" Is this true? I always thought it meant that clients could continue to read and write to "the database" which could include the client switching to another node. There is nothing in CAP theorem about latency, so switching, even if it adds high latency, is fine by CAP theorem. This lack of accounting for latency is what makes CAP theorem less useful than a lot of people realize IMO.
- zzzcpan 8y ago

sseth 8y ago

The author talks about high throughput achieved via batching. But he has not mentioned latency implications of batching, or perhaps I missed that in the article. Wouldn't batching lead to an increase in transaction latency, even if we achieve higher throughput?

cbsmith 8y ago

Seems to have a powerful straw man there about eventual consistency. The point of AP systems is not 100% availability, but rather higher availability. By the same reasoning, one should never do CP, because it is also not possible to have 100% consistency. Disk/memory/network corruption, even with ECC can overwhelm the ability to maintain consistency.

deleted 8y ago

[deleted]

acje 8y ago

I don’t know for sure, but to me the AP vs CP interpretation seems only to be a true limitation for distributed systems of exactly two nodes. I also like the blogs point that availability is not ever 100%, but I think the added cost of availability levels when going from an eventual consistency system to a linearizable one is underestimated because performance is going to be a significant availability factor, not only failuremodes as discussed.

omneity 8y ago

It's becoming harder to evaluate the guarantees of the most recent so called "cloud-scale" databases, Spanner or CockroachDB for instance. Is there anything out there like a standard benchmark to compare these offerings more accurately?

rbranson 8y ago

Maybe I’m missing something, but the way this reads seems to attribute Spanner’s concurrency control entirely to TrueTime. Spanner still uses partitioned Paxos to establish consensus and 2PC for multi-partition write transactions.

erikb 8y ago

Consistency is a requirement that holds us back deeply. Implementing constency with eventually voted know-all masters are also holding us back. Looking at this makes an engineers heart cringe. In fact the current architecture is only slightly better than what we had in 2005, more than 10 years ago. And if it runs not on google or amazon clouds, and therefore on UNRELIABLE networks, the voting itself can fail and you have a centralized system without center. And while we claim that we need a central point of knowledge that is at sync throughout the whole system, the whole world AROUND the IT works completely without being in sync and completely without knowing it all (sometimes even having no knowledge or assumed but wrong knowledge), at a scale that IT systems only to some degree come close to nowadays.

bnastic 8y ago

> With 10ms batches, Calvin was able to achieve a throughput of over 500,000 transactions per second. For comparison, Amazon.com and NASDAQ likely process no more than 10,000 orders/trades per second even during peak workloads. I haven’t worked with NASDAQ stream directly, but knowing how fast equities tick I find this “10,000 orders/sec” estimate quite low. Not to mention that 10ms delay in confirming an order would be really terrible.

fastest963 8y ago
According to NASDAQ's website [1]: > Able to consistently sustain an order rate of over 100,000 orders per second at sub-40 microsecond average latency [1]: https://business.nasdaq.com/market-tech/marketplaces/trading https://business.nasdaq.com/market-tech/marketplaces/trading

deleted 8y ago

[deleted]

apaprocki 8y ago

> a specialized hardware solution that uses both GPS and atomic clocks to ensure a minimal clock skew across servers Maybe I'm wrong, but pretty much anyone with a DC is going to use Microsemi (previously Symmetricom) grandmaster clocks with all the bells-and-whistles including the internal Rubidium atomic oscillator along with PTP to keep everything in sync with multiple layers of redundancy. A specialized hardware solution that uses "both GPS and atomic clocks" to guarantee ~15ns RMS to UTC are just a shopping cart away.

andras_gerlits 8y ago

I have a whitepaper about scalable consistency guarantees using optimistic transactions. It should solve these very problems. https://medium.com/@andrasgerlits/optimistic-acid-transactions-4f193844bb5f https://medium.com/@andrasgerlits/optimistic-acid-transactio...

rkarthik007 8y ago

CTO of YugaByte here. We firmly stand by our claims, and I wanted to explain more. From the post by Daniel: << CockroachDB, to its credit, has acknowledged that by only incorporating Spanner’s software innovations, the system cannot guarantee CAP consistency (which, as mentioned above, is linearizability). YugaByte, however, continues to claim a guarantee of consistency. I would advise people not to trust this claim. YugaByte, by virtue of its Spanner roots, will run into consistency violations when the local clock on a server suddenly jumps beyond the skew uncertainty window. >> The statement about YugaByte DB is incorrect. 1. With respect to CAP, both Cockroach DB (https://www.cockroachlabs.com/blog/limits-of-the-cap-theorem/ https://www.cockroachlabs.com/blog/limits-of-the-cap-theorem...) and YugaByte DB (https://docs.yugabyte.com/latest/faq/architecture/#how-can-yugabyte-db-be-both-cp-and-ha-at-the-same-time https://docs.yugabyte.com/latest/faq/architecture/#how-can-y...) are CP databases with HA and there is really no difference in the claims. 2. With respect to Isolation level in ACID, YugaByte DB does not make the linearizability (called external consistency by Google Spanner) claim. YugaByte DB offers Snapshot Isolation (detects write-write conflicts) today and Serializable isolation (detect read-write and write-write conflicts) is in the roadmap (https://docs.yugabyte.com/latest/architecture/transactions/isolation-levels/ https://docs.yugabyte.com/latest/architecture/transactions/i...). 3. We have publicly claimed that we do rely on NTP and max clock skew bounds to guarantee consistency. For example, slide 43 of our NorCal DB Day talk (https://www.slideshare.net/YugaByte/yugabyte-db-architecture-storage-engine-and-transactions https://www.slideshare.net/YugaByte/yugabyte-db-architecture...) we mention we are “relying on bounded clock sync (NTP, AWS Time Sync, etc).”

abadid 8y ago
Looks like you cross-posted this comment here and on my blog, so I'll also cross-post in my response. I am quite confused by your statement that YugaByte does not claim linearizability. The C of CAP is linearizability (see the original CAP theorem https://users.ece.cmu.edu/~adrian/731-sp04/readings/GL-cap.pdf https://users.ece.cmu.edu/~adrian/731-sp04/readings/GL-cap.p...). By claiming to be CP from CAP, you are claiming linearizability. CockroachDB also makes the same CP claim. However, they explicitly walk back from this claim (https://www.cockroachlabs.com/blog/living-without-atomic-clocks/ https://www.cockroachlabs.com/blog/living-without-atomic-clo...). In contrast, YugaByte documentation makes no effort to walk back from this claim. Rather, the documentation seems to indicate that YugaByte is linearizable at: https://blog.yugabyte.com/jepsen-testing-on-yugabyte-db-database/ https://blog.yugabyte.com/jepsen-testing-on-yugabyte-db-data... and https://docs.yugabyte.com/latest/develop/learn/acid-transactions/ https://docs.yugabyte.com/latest/develop/learn/acid-transact.... Also, I would encourage you to read the Herlihy-Wing paper on linearizability published in 1990. You seem to be confusing linearizability with ACID isolation levels, which are actually a different concept. Peter Bailis has a good blog post on the difference: http://www.bailis.org/blog/linearizability-versus-serializability/ http://www.bailis.org/blog/linearizability-versus-serializab.... On your third point, we agree. However, the point of my post is that relying on max clock skew without hardware support is dangerous. I’m going so far as to say that it’s so dangerous that it is incorrect to claim consistency guarantees when you make such assumptions. I would really encourage YugaByte to consider using global consensus instead of partitioned consensus. I believe you will find it much easier to support serializability (which as you said, is on your road map) and linearizability.

voidmain 8y ago

This article lacks any reference to FoundationDB, which offers external consistency and serializable distributed transactions without trusting clocks in any way. We designed it starting in 2009 and so its lineage is independent of either Calvin or Spanner. FDB doesn't have an actively developed SQL layer at the moment, so I guess you could say it isn't a "NewSQL" database, but none of the properties under discussion have much to do with the query language.

uluyol 8y ago
As best as I could find, FoundationDB doesn't provide strict serializability (i.e. linearizability + serializability) and doesn't tackle multi-DC deployments. So I wouldn't put it in the same class as FaunaDB or Spanner.
- jscissr 8y ago
  There is a "Datacenter-aware mode": https://apple.github.io/foundationdb/configuration.html#datacenter-aware-mode https://apple.github.io/foundationdb/configuration.html#data... Here is some discussion about linearizabilty in fdb: https://news.ycombinator.com/item?id=16884882 https://news.ycombinator.com/item?id=16884882
  - uluyol 8y ago
    Without knowing exactly how reads and writes are replicated, I am skeptical. I have gone through the technical documentation and haven't found many details. For reference, I've done work in storage and consensus algorithms and I can tell you for a fact that without using a consensus algorithm for either reconfiguration or request propagation, you will have consistency violations. I would love to be proven wrong, as more systems with strong consistency guarantees is better, but for now, I don't believe that foundation db provides stronger guarantees than serializable reads and writes.
    - voidmain 8y ago
      FoundationDB uses a consensus algorithm for reconfiguration, but not in the (happy path) transaction pipeline. It provides (by default) strict serializability (i.e. serializability and external consistency/linearizability) for arbitrary, ad hoc, interactive transactions, and it's expected to provide excellent performance when every single transaction is cross node (so e.g. indexes can be efficiently updated this way). It provides better fault tolerance than consensus replicated databases typically can because it needs only N+1 replicas instead of 2N+1 to survive N faults (it keeps 2N+1 replicas of tiny configuration for consensus, if course). It has the best testing story in the industry and is used at scale by, among others, the largest company in the world. Because it doesn't have consensus in the datapath it can have lower latencies than consensus replicated databases in multi region deployments, it also supports asynchronous replication, and an upcoming feature will provide a unique option for multi region failover with sub geographic write latencies, maintaining full transaction durability in failover, as if in synchronous replication, except for the exceptionally rare case where the failure of multiple datacenters in a region are exactly simultaneous. Besides its extensive documentation, you can read its source code and run its deterministic simulation tests yourself if you are interested (it's Apache licensed). Skepticism on these points was reasonable when we originally launched it in 2012 but is getting a little silly in 2018.

redwood 8y ago

Odd to see the most popular distributed operational database system in the world left out here.

rifung 8y ago

Probably a dumb question but if you need all servers to talk to each other to achieve consensus, how do you deal with network latency? Do people just keep data in separate zones so they aren't globally replicated?

shaklee3 8y ago

I'm hoping aphyr will comment on this...

bbcbasic 8y ago

MongoDb is webscale. It scales right up. Sharding is the secret webscale sauce.

yoava 8y ago

I like the realistic view of noSql vs CAP and the different tradeoffs. We do need to talk more about the disadvantages of noSql systems and all the new database alternatives. Which brings me to my point - what the author fails to talk about is why spanner has taken the design decision they have made. He does claim scalability, but that is a very general word. I believe the spanner decision is to avoid making a global voting, as in global across the entire world, from North America, to Asia, Europe and even as far as Australia. Such a global voting will take a lot of time - impose seconds of later on any write. I think the author, when he talks about having a single global vote with a small window, thinks in terms of a single database, maybe two regions on the Continental USA, as apposed to spanner trying to be Geo distributed.

AtlasBarfed 8y ago

The context of this article still seems to constrain consistency to guarantees for single atomic units of data (atomic at the node level). But the second you have multiple-node transactions (your data after all in a truly distributed system will exist on mulitple nodes), then the single-node reliability become dependent on multiple data exchanges across networks for confirmations. Then your network partition exposure starts to snowball. Same thing for joins. Great if your joins are perfectly served by a node-local sharding strategy, but when you retrieve from multiple nodes, again your network partitioning risk starts to compound. In AWS you will see noisy neighbor networks and network unreliability. Your VMs get yanked. Your EBS might experience an IO pause. GC pauses. Or simply one of your nodes might get swamped by a query spike.

jbkavungal 8y ago

Interesting article