9 ms·
I used to agree until I was forced to use an eventually consistent system. Both have their place, but I think you get better architected, more reliable softwar
by jedberg 9y ago
I used to agree until I was forced to use an eventually consistent system.
Both have their place, but I think you get better architected, more reliable software with an eventually consistent system.
In and eventually consistent system, the programmer must consider what to do in the case of data failure. I think this is a good thing. When building a user authentication system, who better to decide what to do when you can't access the user database than the person writing the user auth system? Should it fail open or closed? Well if you're a bank, you should probably fail closed. If you're Netflix? Fail open. The programmer should know the business case better than the database, which will alway fail closed.
Sure, in a consistent system the programmer could think about that, but they aren't forced to, which I think is good.
Also, it means that joins must be done in your application. Sometimes its good to have the programmer deal with that too, because it means they will have to really consider what data they need and be much more familiar with how much data is needed to create the join. If you're just relying on the database to do the joins, it might be hiding a lot of load from you, especially if you inadvertently create multiple table scans. In an eventually consistent system, you know if you created a table scan, because you have to pull back a full table's worth of data (or not, because you saw that issue and fixed it).
I just feels like the programmer gets better insight into what is happening to the data and can make better decisions on what to do when they are closer to the business cases the software is being built for.
- Retra 9y agoI don't think you can just set a line like that. Eventually the programmer might want to consider what to do in the case of data failure. Otherwise, why even write shared code? The programmer might be wise to write all their software from scratch, affording every consideration where they know best, but they certainly won't want to start there. And this says nothing for teams where people leave and enter over time and persistent experience is ephemeral.
- jedberg 9y agoThis is a fair point. Maybe the engineer doesn't think about it, but at least some engineer wrote a library that takes care of it, instead of relying on the database to do it. And even with a consistent system, maybe they are still using a library. But my point was eventually consistent systems, in general, force you to think about these things earlier on. Of course there are exceptions in both directions.
- taeric 9y agoI'm not sure I follow your example. Is anyone actually "failing open" for user authentication? Why? I think I can understand some of your point. Trust should be engineered into the system, not assumed. That is a lot of work and not easy, though. (I think this ultimately lands on the block chain idea of the world. I'm fairly ignorant there.) Strong consistency just means that you can expect any accesses you performed to be consistent in time with each other, right? To that end, it is nice not having to effectively do a "loopUntilChangeReflected" after every write and checking to see if it is safe to do the next thing. Which you have to do in an eventually consistent world.
- IgorPartola 9y agoNot that I necessarily agree, but what GP is saying is that say you have a distributed DB backing your auth system, and there is a partition. The user tries to log in and you can’t be sure that their account is still valid because of the partition, but because all they are trying to do is watch a movie, just let them in based on the data that the partition you do have access to. This is in contrast to a bank which should always operate only when there is no partition.
- taeric 9y agoThat makes sense, but how do you encode it? The auth system now does not return authorized or not authorized, but authorized or maybe authorized? Seems much easier to just rely on auth being a yes or no. Possibly with an expiration for how long a yes can be trusted. That is, how is it simpler to assume that the data you get back could be wrong? Now, anyone involved with the system seems to have to know how everyone involved in the system interprets results.
- jedberg 9y agoIt allows the person writing the business software layer to make a decision that fits that business use case independent of how others interpret it. But no one else needs to know how it works. If the "play movie" service calls the auth service and says "should I let this play", the auth service returns "authorized" or didn't give a response or said "maybe authorized" or whatever. The "play movie" service makes the business decision that makes sense for playing a movie, in this case, "go ahead". But the billing service maybe says "I'm only gonna let you in with a confirmed yes". So it's possible the billing service will give you an error in the same conditions that the movie service does not, but those are business decisions instead of technical ones, which is good.
- cookiecaper 9y agoI agree with the principle that good development practices will try to get the coder to think through the potential failure modes of the code they've written. Unfortunately, I think you've come to the opposite conclusion in this case. When jedberg writes the code for a system backed by an eventually-consistent data store, he asks "What happens if this value is stale and how can I structure the code to cope with that in a sane and reasonable way?" But when $Random_Dev writes the code for such a system, he asks some combination of "What does eventually consistent mean?", "What do you mean I won't get back the right data?", and "Isn't that the DBA's problem?" The core issue being that the average dev will punctuate this line of questioning with things that shift the blame elsewhere; if not the DBA, then "Well, it's good enough for [INSERT_TRENDY_COMPANY_HERE] [ reddit? ;) ], so I think we'll be OK!" or "I don't think that will ever happen to us, our reads get consistent quickly enough". Just whatever will punt the issue down the road. To be honest, I believe that this impulse is the impetus behind the rise in NoSQL in the first place. Devs resent DBAs and SQL. They, very simply, want a place to shove their data, a place to read it out, and to not know anything else about the in between. I have seen many pitiful architectures that clearly turned on this consideration (and we're seeing it all over again with cloud orchestration). Good developers will dig in and search out the meaning of the data whether the failure case is handled at the DB level or not. Highly-structured systems like a traditional RDBMS are very good for resiliency because they require someone to put some definition and formality around the dataset to get basic functionality, and they are generally extremely careful about ensuring the things that come in and out of them are safe and accurate. That's the kind of backing that we really need, especially when there are many developers who can't be trusted to even try to understand more than the bare minimum necessary to get a positive quarterly review.
- jedberg 9y agoYour point is valid, but it makes me sad. We basically come from opposite viewpoints -- I think most devs will do the right thing, you think most will do the wrong thing. Honestly, you're probably right and I'm probably wrong, but I like to hope that I will always work in a place where I'm right. :) (It was Netflix BTW, not reddit. For the most part reddit used strongly consistent databases (Postgres), but we did use some eventual consistency in the caching layer).
- daxfohl 9y agoWow, I so strongly disagree with this. I'm currently working on a system backed by DocumentDB, and had a solution for the core of our application written out in a 13-line javascript stored procedure. The edict came down from above: no stored procedures because scalability. No analysis, no real-world data (we're currently doing a few transactions per day; my understanding is that stackoverflow uses SQL server and scales just fine), just idealism. So we set off to push our stored proc logic into a lazily-consistent background job. I hated the idea initially, but the result turned me off of eventual consistency permanently. This is because, not only did we develop an edict against stored procedures, but we then developed an edict that everything must be done using the lowest consistency model possible. We ended up settling on session consistency. TBH we discovered an algorithm that didn't even require session consistency, but the latency would have been to great. Again, no particular analysis or rationale for the edict, just idealism. The end result was a system that, for each update, required six background jobs to be launched, which each launched their own cascading jobs that patched up the data. And our records all had to be peppered with version fields and updated-by-job fields and version-of-foreign-key fields that these fixup jobs had to test against to make sure they're not operating against stale data or running over each other, such that this crap actually outweighs the actual data in the records. And these background jobs manage state in Table Storage and transfer messages via Service Bus, so two more SPOFs not to mention all the extra compute resource cost. It's a huge mess of a jenga puzzle that's taken months to implement and code review and test, and now even six months later we're still finding edge cases that aren't handled correctly (or are they? you have to constantly go through from the start to really understand it). In addition, since things are never guaranteed to be in a consistent state, that means we could have foreign keys that don't reference anything, or circular graphs where cycles should be disallowed, or whatever else at any moment in time, and so we have to fudge this stuff when presenting data to the user, and it reduces our ability to do any sort of meaningful statistics on our data. To remove 13 lines of javascript. It's just a mess. And then what happened is a couple new requirements came in that couldn't be reasonably handled in a lazy fashion, so we ended up using stored procs for that logic anyway. Now, to be clear, the resulting system is strictly more scalable than the stored proc based one. One could in theory just throw more machines at any load and it'll continue to work. So there's that. But a) the load beyond which the stored proc system could scale is about 3 orders of magnitude higher than even our optimistic usage estimates in the foreseeable future, and b) the lazy algorithm requires so much additional compute overhead that, while infinitely scalable, would we even be able to afford it anymore? So for my money, give me a strongly consistent system by default. I don't even particularly understand the argument given above. With a strongly consistent system, you can still choose to do things like join in the client if there's a reason for it. But going the opposite way and forcing consistency to be unassumed everywhere produces the mess we're dwelling in now. Now, certainly there are cases where eventual consistency is better: high-write-volume on not-very-relational data where point-in-time accuracy isn't super business-critical is where it typically shines. At the "edges" of your system, essentially. But I do want to point out some of the pain that can be encountered if it's chosen for the wrong use case, and make sure that anyone who does use it in their core application logic knows what they're in for, and puts some more thought into whether it's what they really want.
- mayank 9y agoI think you may have some misconceptions about what consistency means in this context. > In and eventually consistent system, the programmer must consider what to do in the case of data failure. Eventually consistency has nothing to do with data failure; the real-world effect of eventually consistency is stale data, which you can't generally tell is stale. > When building a user authentication system, who better to decide what to do when you can't access the user database than the person writing the user auth system? I think you're talking about availability, not (eventual) consistency. > Also, it means that joins must be done in your application. There is nothing preventing you from doing joins in a strongly consistent database. Google Spanner supports JOINs for example [1]. > In an eventually consistent system, you know if you created a table scan, because you have to pull back a full table's worth of data This also has nothing to do with eventual consistency. [1] https://cloud.google.com/spanner/docs/query-syntax#join-hints https://cloud.google.com/spanner/docs/query-syntax#join-hint...
- jetcata 9y agoAgree with above, I'm not sure why that comment is most upvoted, it's poorly written and the author doesn't seem to have a good grasp of distributed systems concepts.
- hueving 9y agoVague "I agree" criticisms make for poor reading. Provide specific counter examples like the parent or don't bother commenting.
- jedberg 9y agoI was using "eventually consistent system" as a shorthand for "multi-master database that is eventually consistent" which may have been confusing.
- jedberg 9y agoI was using "eventually consistent" as a shorthand for "multi-master highly-available key-value distributed database". Most people who don't work in the space regularly consider them the same thing. Basically Cassandra/DynamoDB vs Cloud Spanner. > Eventually consistency has nothing to do with data failure; the real-world effect of eventually consistency is stale data, which you can't generally tell is stale. You're right, that is the highly available distributed part, but it goes along with eventual consistency because if you plan for dealing with stale data then those same concepts protect you from unavailable data. > I think you're talking about availability, not (eventual) consistency. Again, the two go hand in hand. Deal with stale data and you've dealt with unavailable data. > There is nothing preventing you from doing joins in a strongly consistent database. You misread -- I was saying that with a multi-master key-value system you have to do joins in the app. > This also has nothing to do with eventual consistency. Same, I was talking about Dynamo-like systems.