10 ms·
My opinion is exactly opposite: Consistently is overvalued. Requiring consistency in distributed system generally leads to designs that reduces availability.
by vidarh 10y ago
My opinion is exactly opposite: Consistently is overvalued.
Requiring consistency in distributed system generally leads to designs that reduces availability.
Which is one of the reasons that bank transactions generally do not rely on transactional updates against your bank. "Low level" operations as part of settlement may us transactions, but the bank system is "designed" (more like it has grown by accretion) to function almost entirely by settlement and reconciliation rather than holding onto any notion of consistency.
The real world rarely involves having a consistent view of anything. We often design software with consistency guarantees that are pointless because the guarantees can only hold until the data has been output, and are often obsolete before the user has even seen it.
That's not to say that there are no places where consistency matters, but often it matters because of thoughtless designs elsewhere that ends up demanding unnecessary locks and killing throughput, failing if connectivity to some canonical data store happens to be unavailable etc.
The places where we can't design systems to function without consistently guarantees are few and far between.
- kpmah 10y agoBanks are forced to be eventually consistent because of their distributed nature. If it was practical to have a big central ACID database I'm sure that's what they'd use. There is an extra layer of complexity with distributed systems. That said, if the problem has been thought about, I'm happy. My frustration is with people not understanding the trade-offs.
- deleted 10y ago[deleted]
- vidarh 10y agoThe problem is that a big central ACID database enforces strict limits on throughput limited at best by the the speed of light (in reality it's worse). Enforcing a globally consistent views basically requires coordination. Coordination requires that a signal traverses a distance and is acted on no faster than the instruction throughput of a single CPU core. When the sources of operations are geographically spread out, you will be limited based on the fastest communication possible between those locations in addition to the processing latency. Ultimately this means that there are scaling limits to throughput of a fully serialised system that causes real problems all the time. Sometimes a single global truth is still fast enough. Very often it isn't, when e.g. your users are spread out globally. That's why it's important to consider avoiding consistency when it is not necessary.
- sebastianconcpt 10y agoI think you are saying that consistency matters but a system is more robust with settlement and reconciliation where consistency is like a luxury that could optimize the really robust method
- vidarh 10y agoIn most cases what matters is very localised consistency - e.g. I want to see the comments I post on a comment page right away - while global consistency matters much less, as long as the system keeps converging (it may or may not be consistent at any given point in time, but the older a change is, the more likely that it appears the same everywhere, basically).
- nxzero 10y agoThere's a big difference between being availability being predictable & unpredictable - it's called consistency. Picking a pattern, and sticking to it is fundamental to systems flow. Yes, have "randomness" is important, but trying to cultivate chaos from utter disorder is a something at the very least historically speaking people are not good at collectively. Chaos functions best when it's on the edge of a system, not embedded in it.
- kahirsch 10y agoAs you say, the low-level operations in banking do provide atomicity, which are often lacking in distributed databases. Also, the banking system does have the system of compensating transactions (e.g. returned checks) when constraints are violated, as well as an extensive system of auditing to maintain consistency. Do most distributed systems have the ability to detect inconsistency? How do they manage to correct problems when found? Is it mostly manual? I'm genuinely interested in hearing what people have done. I've certainly experienced inconsistencies which were in very low-stakes online situations that were still annoying. For example, a reddit comment that was posted on a thread, but never shows up my user page. Or when I stopped watching a movie on Netflix on my PC to go watch something else with my wife on our TV, but Netflix won't let us because it thinks I'm still watching on the PC. Come to think of it, there was also something that used to happen frequently with Amazon video, though I haven't seen it happen in a while. When I would resume a video, it gave me two choices of times to resume, one based on the device and one based on the server. The odd thing was that I had only ever used one device to watch the video but it was always the server-stored time that was right. Just a bug, I guess, but that does indicate the kind of inconsistencies that have to be resolved somehow.
- vidarh 10y ago> Do most distributed systems have the ability to detect inconsistency? They do if you build them to have them. Your bank account works without requiring consistency because every operation is commutative: You never do anything that says "set account to value x". You have operations like "deposit x" and "withdraw x". For operations that require you move money from one account to another, you need to guarantee that both or none steps gets durably recorded, and you may (or may not, depending on your requirements) insist on local ordering, but even there global consistency is unnecessary. > How do they manage to correct problems when found? Is it mostly manual? I'm genuinely interested in hearing what people have done. You design systems to make them self-correcting: Log events durably first, and then act on the events. If possible, make operations idempotent. If not, either make operations fully commutative, or group them into larger commutative operations and use local transactions (so require local consistency) to ensure they are all applied at once. The point is not to require no atomicity, but to localise any guarantees as much as possible, because the more local they are, the less they affect throughput. On failure, your system needs to be able to determine which operations have been applied, and which have not, and resume processing of operations that have not. > I've certainly experienced inconsistencies which were in very low-stakes online situations that were still annoying. For example, a reddit comment that was posted on a thread, but never shows up my user page. Or when I stopped watching a movie on Netflix on my PC to go watch something else with my wife on our TV, but Netflix won't let us because it thinks I'm still watching on the PC. This type of thing is often down to copping out on guarantees when a system includes multiple data stores, and is often fixed by applying a local transactional event log used to trigger jobs that are guaranteed to get processed. E.g. consider the Reddit example - if you want to be able to effortlessly shard, you might denormalize and store data about the comment both tied to a thread, and tid to a user. If these are in two different databases, you need some form of distributed transactional system. A simple way of achieving that is to write a transactional log locally on each system (as long as operations are design to not need coordination) that ensures that both "attach post to thread" and "attach post to user page" will get logged successfully before giving a success response to the end user. You can do that by using a commit marker in your log. Then you "just" need a job processor that marks jobs done when all operations in a transaction has completed (and that can handle re-running the same ones). But this is something people often fail to do, because it's easier to just write out the code for each step sequentially and deal with the rare instances where a server dies in the middle of processing a request. So the basic pattern for ditching global consistency is to write transaction logs locally, representing each set of operations you need to apply, and put in place a mechanism to ensure each one is applied no matter what. Then use that "everywhere" including in the many places where you should've had guarantees like that, but don't because developers have been lazy.
- bunderbunder 10y agoMy (admittedly limited, so far) experience in the financial industry has indicated that, while it is the norm is to use settlement and reconciliation in situations where making ACID-style guarantees is impossible, ACID-style guarantees are most emphatically _not_ foregone when it isn't necessary to do so. One big reason for that is that the settlement and reconciliation process is easier to manage when the parties involved can at least be confident that their own view of the situation is internally consistent. Gremlin-hunting is a lot less expensive if you can minimize the number of places for gremlins to hide.
- socmag 10y agoThis! There is no single "truth". Under the unfortunate consequences of the CAP theorem and the speed of light, consistency grinds to a halt as the number of observers increases. Time itself would have to stand still in the limit if at each time step every particle in the universe had to agree on the state of the system. What we need to preserve is mostly strong "causal consistency", which means that from a single observers perspective, the system "appears consistent". This article is really good and isn't obscured by NoSQL vs SQL arguments. http://www.grahamlea.com/2016/08/distributed-transactions-microservices-icebergs/ http://www.grahamlea.com/2016/08/distributed-transactions-mi...
- knucklesandwich 10y agoThe bank example really needs to die, its not reflective of what people often use databases for. 1. Bank transactions are easy to express as a join-semilattice[1], which means its intrinsically easy to build a convergent system of bank accounts in a distributed environment. Many problems are not easily expressible as these and usually trade off an unacceptable amount of space to formulate as a join-semilattice and still work with complicated types of transactions. 2. A bank transaction has a small upper bound on the types of inconsistency that can occur: at worst you can make a couple overdrafts. 3. A bank transaction has a trivial method for detecting and handling an inconsistency. Detection is easy because its just a check that the transaction you're applying doesn't leave the account with a negative balance. Handling is easy, because the bank can just levy an overdraft charge on an account holder (and since they have the ability to collect debts on overdrawn accounts, this easily works for them). Complicated transactions often don't have a trivial method of detecting inconsistency and handling inconsistency is rarely as simple as being able to punish the user. In fact corrective action (and likewise, inconsistency) is often non-intuitive to users because we expect causality in our interaction with the world. > The real world rarely involves having a consistent view of anything I strongly disagree with this. Most of our interaction with the world is perceived consistently. When I touch an object, the sensation of the object and the force I exert upon the object is immediate and clearly corresponds to my actions. There is no perception of reordering my intentions in the real world. We expect most applications to behave the same way, because the distributed nature of them is often hidden from us. They present themselves as a singular facade, and thus we typically expect our actions to be reflected immediately and in the order we issue them. Furthermore, this availability vs consistency dichotomy is an overstatement of Brewer's theorem. For instance, linearizable algorithms like Raft and Multi-Paxos DO provide high availability (they'll remain available so long as a majority of nodes in the system operate correctly). In fact, Google's most recent database was developed to be highly available and strongly consistent [2]. [1]: https://en.wikipedia.org/wiki/Semilattice https://en.wikipedia.org/wiki/Semilattice [2]: http://research.google.com/archive/spanner.html http://research.google.com/archive/spanner.html
- kpmah 10y agoDo you (or anyone) know of a better example than the bank account? I wouldn't be against changing it. I wanted to communicate through a fairly succinct and understandable example that there are potentially high stakes, and I wanted to show a 'NoSQL' example rather than a micro-services one. It was the first one that came to mind.
- 4ad 10y agoThe bank example is terrible. The inconsistency is an absolute PITA for the consumer. The delays are are a terrible, terrible user experience, and the consumer has to spend more money in the end. It's extremely expensive, both in money the bank has to spend on auditing and in the legal framework that has to be put in place. The bank example is the perfect argument for consistency, not against.
- vidarh 10y agoIt's a perfect argument for shorter transaction processing times. The bank inconsistency is there because historically it was impossible to guarantee consistency without making the processing times far worse. E.g. want to make a payment via check? Sorry, need to send a courier to the branch holding your account to verify the funds are there, and then afterwards send a courier to inform of the updated account state before another payment would be allowed. But the shorter you make the processing time, the less benefit you get from introducing consistency requirements, to the point where the thing that is important to address is the remaining cases where processing transactions take time, not the consistency (at least as long as banks also don't take the piss by taking advantage of the lax consistency to charge unwarranted overdraft fees).
- novaleaf 10y agoabsolute need for consistency is rare, unless deal with anything remotely similar to "financial transactions". Thankfully the author's premise of "NoSql == inconsistency" is wrong in at least one case. Google Cloud Datastore offers ACID compliance. I use that to great effect (I run a subscription SaaS with usage quotas)
- dasil003 10y agoI guess whether it's overvalued or undervalued depends on whom we're talking about. Your point is well taken, but I think the real problem is that inexperienced programmers want the best of both worlds. In other words, they design systems that require consistency to function correctly because it's easier, but then they do not actually guarantee said consistency, because hey, schemaless is so much faster than boring old SQL. The way I like to think about it (and mind you, I'm an early-stage kind of guy, so I rarely work on huge systems) is that consistency guarantees give you a lot of leverage, and while it's almost unattainable in the real world, in the computing world we have extremely powerful CPUs and networks that allow communication between any two points on the globe in less than a second. There is frankly a huge number of applications that can be built on an ACID database and never have to worry about the offline or distributed use cases that would require building a proper distributed system.
- karmakaze 10y agoFor the specific case of distributed datastores, by threshold for considering any NoSQL/NewSQL is scaled writes and consistent reads. Designing a system based on consistent reads is much simpler without quirks that need to be explained to end-users.
- superuser2 10y ago>designs that reduces availability. It's a slimy abuse of language to call a service "available" when it's giving incorrect output and corrupting its persistent state. I can write a program with lots of nines that is blazingly faster than its competitors if its behavior is allowed to be incorrect.
- user5994461 10y ago> I can write a program with lots of nines that is blazingly faster than its competitors if its behavior is allowed to be incorrect. Congratulations! You just understood distributed systems, or put more simply the real world of app development. Most of the programs in the real world ARE allowed to be incorrect at times. Corollary: Most of the programs in the real world do NOT need to be correct every single time. These statements should be on top of system design and requirement gathering classes.
- vidarh 10y agoA lot of the time there is no "correct" output. E.g. a search result from a search engine is outdated the moment it has been generated - new pages come into existence all the time, and besides the index is unlikely to be up todate. What matters is whether "close enough" is sufficient, and a large proportion of time it is: E.g. for the search engine example don't care if I get every single page that matched my query right at the moment I pressed submit. I care that I find something that meets my expectations. Getting "good enough" results and getting them at a reasonable time beats not getting results or waiting ages because the system is insisting on maintaining global consistency. Yes, there are cases where "good enough" means you need global consistency, but they are far fewer than people tend to think.