3 ms·

So this might not be the right place for this, but i'm curious. How do people deal with "eventual consistency"? In my head once a transaction is done, it's ev

So this might not be the right place for this, but i'm curious. How do people deal with "eventual consistency"? In my head once a transaction is done, it's everywhere and everyone has access to it. What happens if 2 nodes try to modify the same data at the same time? Or what happens if you insert on one node, then query on another before it propagates? And if the answer to those questions are what I think they are (that bad stuff happens), how do you setup your application to avoid doing it?

hinkley 10y ago
Always wondered this myself. There seems to be a general fondness for writing to the database and then immediately reading back from it to make a decision. Some variant of that show up in the requirements quite often, usually in the form of confirmation pages or overview screens following right after an update, so the user knows what they did took effect. And almost as often you'll find that some member or members of your team have made the assumption that the whole interaction will occur on one machine talking to one database instance. With session affinity you might be able to create a convincing illusion that this is working, but now with microservices you're intentionally hitting many servers. If checkout and order history are on two different servers, how does eventual consistency work?
- LoSboccacc 10y ago
  >There seems to be a general fondness for writing to the database and then immediately reading back from it to make a decision one way I saw banking software do it was keeping deltas of the changes thorough a wizard like process and then commit it all at once in a single transaction, at which point if there was a conflict the wizard restarted with up to date data. conflict detection was done by a sequential id keyed on the accounts touched on the operation, so that any operation modifying the account before your delta was applied would knock your commit off
  - hinkley 10y ago
    Banking and the credit industry are a good example of eventually consistent systems. But you can see the inconsistencies and you're expected to just ignore them. I guess what I'm asking is how do you push back on requirements and inattentive coworkers that make these compromises unworkable, because it seems so pervasive to me.
sanderjd 10y ago
This is a really big question that quickly gets into distributed systems theory in general, but in part, the idea is to recognize that a lot of times the "bad stuff" is not so bad. For instance, if someone updates their profile picture on Facebook, and I make a request that should include their profile picture, but their update hasn't propagated to the node I'm reading from, I just get the old profile picture, and that's a-ok. There are definitely nasty things (like simultaneous updates) that you have to be aware of, but for a lot of applications, there are a lot of cases where out of date data makes little difference.
- Klathmon 10y ago
  I guess the part I have trouble with is how do you separate what can't be delayed from what can? Is it just a matter of keeping the information on 2 separate systems, or are there tools that let you for instance mark one query as "take your time and make sure everything is 100% up to date before committing this" and another as "return whatever you got, it's not important"?
  - dberg 10y ago
    One solution is to cookie persist the user to the primary site and have a TTL to allow for some replication lag. Facebook does something similar here.
  - pas 10y ago
    Cassandra, MongoDB and Redis can do that to some degree. http://cassandra.apache.org/doc/latest/architecture/dynamo.html#tunable-consistency http://cassandra.apache.org/doc/latest/architecture/dynamo.h... http://hector-client.github.io/hector/build/html/content/consistency_level.html http://hector-client.github.io/hector/build/html/content/con... Tunable durability, and some minimal isolation (atomicity) guarantees for Mongo: https://docs.mongodb.com/manual/reference/write-concern/ https://docs.mongodb.com/manual/reference/write-concern/ and https://docs.mongodb.com/manual/core/read-isolation-consistency-recency/ https://docs.mongodb.com/manual/core/read-isolation-consiste... http://redis.io/commands/WAIT http://redis.io/commands/WAIT I guess something like this can and will eventually find its way into any practical and effective distributed data store.
cookiecaper 10y ago
"Eventual consistency" usually means a few seconds rather than a few hours. If your users can tolerate hitting refresh when they see the stale data, or if your application is such that users don't have the option to use the newly-inputted or updated information for a short while, it may not be a problem. It's probably not the type of thing you can just drop in. It will fit some use cases and not others. You'll want to design your application around an eventually-consistent system to prevent and/or handle conflicts in a sane manner. Some systems like Riak have a built-in complex resolution scheme based on "vector clocks", which record information about the version of the data that was received and can tell which entry is newer. [0] [0] http://docs.basho.com/riak/kv/2.1.4/developing/usage/conflict-resolution/ http://docs.basho.com/riak/kv/2.1.4/developing/usage/conflic...
hvo 10y ago
It may interest you to study this a bit to understand MVCC (Multiversion Concurrency Control). https://www.postgresql.org/docs/9.6/static/mvcc.html https://www.postgresql.org/docs/9.6/static/mvcc.html
Florin_Andrei 10y ago
> How do people deal with "eventual consistency"? In my head once a transaction is done, it's everywhere and everyone has access to it. Different strokes for different folks. Synchronous replication: the whole cluster is a single unit. COMMIT only returns when it's been truly committed to all nodes. OTOH, it's slow, and may or may not tolerate partitions. Asynchronous replication: COMMIT returns when it's gone into the current node. It is then the cluster's job to propagate it to the rest. This is a lot faster, and can deal with network partitions. OTOH there is usually a state lag between distant nodes. Different applications may need either sync or async, depending on requirements.

hinkley 10y ago

Always wondered this myself. There seems to be a general fondness for writing to the database and then immediately reading back from it to make a decision. Some variant of that show up in the requirements quite often, usually in the form of confirmation pages or overview screens following right after an update, so the user knows what they did took effect. And almost as often you'll find that some member or members of your team have made the assumption that the whole interaction will occur on one machine talking to one database instance. With session affinity you might be able to create a convincing illusion that this is working, but now with microservices you're intentionally hitting many servers. If checkout and order history are on two different servers, how does eventual consistency work?

LoSboccacc 10y ago
>There seems to be a general fondness for writing to the database and then immediately reading back from it to make a decision one way I saw banking software do it was keeping deltas of the changes thorough a wizard like process and then commit it all at once in a single transaction, at which point if there was a conflict the wizard restarted with up to date data. conflict detection was done by a sequential id keyed on the accounts touched on the operation, so that any operation modifying the account before your delta was applied would knock your commit off
- hinkley 10y ago
  Banking and the credit industry are a good example of eventually consistent systems. But you can see the inconsistencies and you're expected to just ignore them. I guess what I'm asking is how do you push back on requirements and inattentive coworkers that make these compromises unworkable, because it seems so pervasive to me.

sanderjd 10y ago

This is a really big question that quickly gets into distributed systems theory in general, but in part, the idea is to recognize that a lot of times the "bad stuff" is not so bad. For instance, if someone updates their profile picture on Facebook, and I make a request that should include their profile picture, but their update hasn't propagated to the node I'm reading from, I just get the old profile picture, and that's a-ok. There are definitely nasty things (like simultaneous updates) that you have to be aware of, but for a lot of applications, there are a lot of cases where out of date data makes little difference.

Klathmon 10y ago
I guess the part I have trouble with is how do you separate what can't be delayed from what can? Is it just a matter of keeping the information on 2 separate systems, or are there tools that let you for instance mark one query as "take your time and make sure everything is 100% up to date before committing this" and another as "return whatever you got, it's not important"?
- dberg 10y ago
  One solution is to cookie persist the user to the primary site and have a TTL to allow for some replication lag. Facebook does something similar here.
- pas 10y ago
  Cassandra, MongoDB and Redis can do that to some degree. http://cassandra.apache.org/doc/latest/architecture/dynamo.html#tunable-consistency http://cassandra.apache.org/doc/latest/architecture/dynamo.h... http://hector-client.github.io/hector/build/html/content/consistency_level.html http://hector-client.github.io/hector/build/html/content/con... Tunable durability, and some minimal isolation (atomicity) guarantees for Mongo: https://docs.mongodb.com/manual/reference/write-concern/ https://docs.mongodb.com/manual/reference/write-concern/ and https://docs.mongodb.com/manual/core/read-isolation-consistency-recency/ https://docs.mongodb.com/manual/core/read-isolation-consiste... http://redis.io/commands/WAIT http://redis.io/commands/WAIT I guess something like this can and will eventually find its way into any practical and effective distributed data store.

cookiecaper 10y ago

"Eventual consistency" usually means a few seconds rather than a few hours. If your users can tolerate hitting refresh when they see the stale data, or if your application is such that users don't have the option to use the newly-inputted or updated information for a short while, it may not be a problem. It's probably not the type of thing you can just drop in. It will fit some use cases and not others. You'll want to design your application around an eventually-consistent system to prevent and/or handle conflicts in a sane manner. Some systems like Riak have a built-in complex resolution scheme based on "vector clocks", which record information about the version of the data that was received and can tell which entry is newer. [0] [0] http://docs.basho.com/riak/kv/2.1.4/developing/usage/conflict-resolution/ http://docs.basho.com/riak/kv/2.1.4/developing/usage/conflic...

hvo 10y ago

It may interest you to study this a bit to understand MVCC (Multiversion Concurrency Control). https://www.postgresql.org/docs/9.6/static/mvcc.html https://www.postgresql.org/docs/9.6/static/mvcc.html

Florin_Andrei 10y ago

> How do people deal with "eventual consistency"? In my head once a transaction is done, it's everywhere and everyone has access to it. Different strokes for different folks. Synchronous replication: the whole cluster is a single unit. COMMIT only returns when it's been truly committed to all nodes. OTOH, it's slow, and may or may not tolerate partitions. Asynchronous replication: COMMIT returns when it's gone into the current node. It is then the cluster's job to propagate it to the rest. This is a lot faster, and can deal with network partitions. OTOH there is usually a state lag between distant nodes. Different applications may need either sync or async, depending on requirements.