13 ms·

Consistency is Consistently Undervalued

220 points by kpmah 10y ago

yyyuuu 10y ago
I wandered lonely as a cloud That floats on high o'er vales and hills, When all at once I saw a crowd, A host, of golden daffodils; Beside the lake, beneath the trees, Fluttering and dancing in the breeze.
yyyuuu 10y ago
Continuous as the stars that shine And twinkle on the milky way, They stretched in never-ending line Along the margin of a bay: Ten thousand saw I at a glance, Tossing their heads in sprightly dance.
- dang 10y ago
  We're always up for the Romantics (though I prefer Coleridge), but maybe not like this. Your comments have been inappropriate for Hacker News lately, especially the one where you impersonated another user. This sort of pattern will get an account banned, so please stick to civil, substantive comments only.
olalonde 10y ago
And in case you think there's a general solution to that problem, there isn't: https://en.wikipedia.org/wiki/CAP_theorem https://en.wikipedia.org/wiki/CAP_theorem Still, it's funny how banking seems to be the canonical example for why we need transactions given that most banking transactions are inconsistent (http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on-why-banks-are-base-not-acid-availability.html http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on...).
- kpmah 10y ago
  Well, they're eventually consistent. It's not like it hasn't been thought through. :) It's a useful example because it shows how there can be serious consequences for getting it wrong.
  - yyyuuu 10y ago
    yes .. you are right
- kpmah 10y ago
  Oh, in addition to my other reply I should point out that the 'C' in CAP is not the same thing as the 'C' in ACID. Be careful with that one, it can cause confusion! http://blog.thislongrun.com/2015/03/the-confusing-cap-and-acid-wording.html http://blog.thislongrun.com/2015/03/the-confusing-cap-and-ac...
bullen 10y ago
But consider this: You are using mysql, you make a transaction with say deposit and withdraw. What happens on the mysql machine if you pull the plug exactly when mysql has done the deposit but not the withdraw? The ONLY difference between SQL transactions and NoSQL microservice transactions is the time between the parts of a transaction. Personally I use a JSON file with state to execute my NoSQL microservice transactions and it's alot more scalable than having a pre-internet era SQL legacy design hogging all my time and resources.
- lomnakkus 10y ago
  > What happens on the mysql machine if you pull the plug exactly when mysql has done the deposit but not the withdraw? Nothing bad -- the deposit will never be seen. It sounds like you need to read up on what ACID means. (EDIT: Clarification)
- XorNot 10y ago
  This is not how database transactions work. A transaction means that the amended state is not visible to any readers until it has been fully committed. Pulling the plug means the transaction is lost, and the writer knows that it was not committed (and so can try again).
- hmottestad 10y ago
  >> What happens on the mysql machine if you pull the plug exactly when mysql has done the deposit but not the withdraw? Nothing happens. And the client discovers that the connection to the db was closed before the 2 phase commit was done and gives an error to the user.
- matthewmacleod 10y ago
  I don't mean to be unkind, but is this meant to be a parody? There are no "parts of a transaction" because a transaction is definitionally atomic. Transactional file modification is a fairly tricky problem, and I'd be surprised if you'd actually implemented a safe system in that manner. What's certain though is that spinning up MySQL or Postgres and using it to store simple records is essentially a zero-cost setup task - so I doubt it's ever going to "hog all your time and resources".
fagnerbrack 10y ago
Why isn't the OP using Event Sourcing "commands" for the "Bank Accounts" example?
- victorNicollet 10y ago
  I believe Event Sourcing opens a whole other can of worms which would detract from the point of the article, such as whether the event streams have a well-defined order (especially if each account is its own aggregate), or whether the resulting event should be "Transaction Completed" (assumes pre-conditions were checked) or "Transaction Attempted" (checks pre-conditions before altering state).
  - partisan 10y ago
    My teams' event sourcing implementation stores and publishes events in a well defined order within a specific aggregate. There are scenarios where a two-phase commit is used to ensure that invariants across aggregates are maintained. We use an ACID compliant database to store our domain events and we project our events into a relational schema. When projecting events, we use transactions to make sure the database updates from a specific event are either all applied or not at all.
    - victorNicollet 10y ago
      Yes, this is a typical way of doing things in the CQRS/ES world, but there are many things left unsaid: - would you have a single aggregate for "all bank accounts" so that one financial transaction equals one event, or is there a good reason to have multiple aggregates ? - is command execution transactional, or is it possible for another thread to generate (possibly conflicting) events between the time you check a precondition and the time you write the intended events ? My team uses an ES implementation with very strong guarantees: one aggregate per macro-service, and command execution is transactional, but I'm fairly certain that this is not the way CQRS/ES is described or recommended.
      - partisan 10y ago
fagnerbrack 10y ago
In case anyone is wondering what an "atomic change" means in database terminology: https://www.gnu.org/software/emacs/manual/html_node/elisp/Atomic-Changes.html https://www.gnu.org/software/emacs/manual/html_node/elisp/At...
calind 10y ago
I think it's a bad example because this should not be the way to develop in this kind (microservices) of systems. In these environments you atomically create objects in your application's "local" storage and have a reconciliation loop for creating objects in other services or deleting these orphan "local" objects.
- kpmah 10y ago
  I think it's a good example because this should not be the way to develop this kind of system - and yet people do it this way! :)
morgo 10y ago
Good article. I've stopped using bank transfers as an example for Acid transactions, and instead talk about social features: - if I change a privacy setting in Facebook or remove access to a user, these changes should be atomic and durable - transactions offer a good semantic of which to make these changes. They can be staged in queries, but nothing is successful until after a commit. - without transactions durability is hard to offer. You would essentially need to make each query flush to disk, rather tha each transaction. Much more expensive.
xanton 10y ago
Transactions can have different isolation levels. And sometimes the problem at hand can be implemented using transactions with weak isolation levels which are not that hard to implement using your favorite NoSQL database that support CAS operation. I recommend this article: http://rystsov.info/2012/09/01/cas.html http://rystsov.info/2012/09/01/cas.html
- kpmah 10y ago
  Yes, be aware of the transactional capabilities of your database. Some ACID databases in their default configuration don't provide all transactional guarantees!
  - brandur 10y ago
    I just wanted to add that the Postgres documentation [1] and Wikipedia [2] are both really well written when it comes to explaining each isolation level and the guarantees that it provides. Well worth the reading time. [1] https://www.postgresql.org/docs/current/static/transaction-iso.html https://www.postgresql.org/docs/current/static/transaction-i... [2] https://en.wikipedia.org/wiki/Isolation_(database_systems)#Isolation_levels https://en.wikipedia.org/wiki/Isolation_(database_systems)#I...
brandur 10y ago
Amen. Whether or not the article's example is a good one, in a world without consistency you need to worry about state between _any_ two database operations in the system, so there's nearly unlimited opportunity for this class of error in almost any application found in the real world. The truly nefarious aspect of NoSQL stores is that the problems that arise from giving up ACID often aren't obvious until your new product is actually in production and failures that you didn't plan for start to appear. Once you're running a NoSQL system of considerable size, you're going to have a sizable number of engineers who are spending significant amounts of their time thinking about and repairing data integrity problems that arise from even minor failures that are happening every single day. There is really no general fix for this; it's going to be a persistent operational tax that stays with your company as long as the NoSQL store does. The same isn't true for an ACID database. You may eventually run into scaling bottle necks (although not nearly as soon as most people think), transactions are darn close to magic in how much default resilience they give to your system. If an unexpected failure occurs, you can roll back the transaction that you're running in, and in almost 100% of cases this turns out to be a "good enough" solution, leaving your application state sane and data integrity sound. In the long run, ACID databases pay dividends in allowing an engineering team to stay focused on building new features instead of getting lost in the weeds of never ending daily operational work. NoSQL stores on the other hand are more akin to an unpaid credit card bill, with unpaid interest continuing to compound month after month.
- socmag 10y ago
  A database is an abstract concept, and ACID transactions on databases have nothing to do with the interface semantics used to interact with the shared state being managed. You can have non-ACIDic SQL databases and ACID NoSQL databases. While it is true that many KV stores do not present a fully consistent view over distributed data, there are some that do. It is very important that people shed this idea that only SQL databases can have strong consistency, that's for sure.
  - brandur 10y ago
    > It is very important that people shed this idea that only SQL databases can have strong consistency, that's for sure. Why? In practice, people are going to be building things using the same common paths: Postgres, Mongo, MySQL, Redis, etc., and from a purely pragmatic standpoint, the SQL databases that you're likely to use come with strong ACID guarantees while the NoSQL databases don't. If you can point me to a NoSQL data store that provides high availability, similar atomicity, consistency, and isolation properties provided by a strong isolation level in Postgres/MySQL/Oracle/SQL Server, and has a good track record of production use, I'm all ears.
muteor 10y ago
If anyone is interested in this sort of thing, I found this a great article: http://www.grahamlea.com/2016/08/distributed-transactions-microservices-icebergs/ http://www.grahamlea.com/2016/08/distributed-transactions-mi...
vidarh 10y ago
My opinion is exactly opposite: Consistently is overvalued. Requiring consistency in distributed system generally leads to designs that reduces availability. Which is one of the reasons that bank transactions generally do not rely on transactional updates against your bank. "Low level" operations as part of settlement may us transactions, but the bank system is "designed" (more like it has grown by accretion) to function almost entirely by settlement and reconciliation rather than holding onto any notion of consistency. The real world rarely involves having a consistent view of anything. We often design software with consistency guarantees that are pointless because the guarantees can only hold until the data has been output, and are often obsolete before the user has even seen it. That's not to say that there are no places where consistency matters, but often it matters because of thoughtless designs elsewhere that ends up demanding unnecessary locks and killing throughput, failing if connectivity to some canonical data store happens to be unavailable etc. The places where we can't design systems to function without consistently guarantees are few and far between.
- kpmah 10y ago
  Banks are forced to be eventually consistent because of their distributed nature. If it was practical to have a big central ACID database I'm sure that's what they'd use. There is an extra layer of complexity with distributed systems. That said, if the problem has been thought about, I'm happy. My frustration is with people not understanding the trade-offs.
  - deleted 10y ago
    [deleted]
  - vidarh 10y ago
    The problem is that a big central ACID database enforces strict limits on throughput limited at best by the the speed of light (in reality it's worse). Enforcing a globally consistent views basically requires coordination. Coordination requires that a signal traverses a distance and is acted on no faster than the instruction throughput of a single CPU core. When the sources of operations are geographically spread out, you will be limited based on the fastest communication possible between those locations in addition to the processing latency. Ultimately this means that there are scaling limits to throughput of a fully serialised system that causes real problems all the time. Sometimes a single global truth is still fast enough. Very often it isn't, when e.g. your users are spread out globally. That's why it's important to consider avoiding consistency when it is not necessary.
mettamage 10y ago
Before I read this article, the following question popped into my mind and it miiiiiight be tangentially related -- yea probably not, blame the title ;) When taking the concept of consistency, does consistency have an effect that is akin to compound interest? For example, imagine someone doing the same thing year after year diligently. (S)he'd increase his or her skill say 10% a year (have no clue what realistic numbers are). Would that mean that the compound interest effect would occur? I phrased it really naievely, because while the answer is "yes" in those circumstances (1.1 ^ n). I'm overlooking a lot and have no clue what I overlook. I know it's off-topic, it's what I thought when I read the title and I never thought about it before, so I'm a bit too curious at the moment ;)
sz4kerto 10y ago
The universally hated JEE can do distributed transactions by default. Yes, with pitfalls, but it can. (It is usually hated by devs who have never used it properly.)
- jamesblonde 10y ago
  The problem with JEE is when you leave Java Transactions - i.e., interact with some external system as part of a transaction. Then there is no way around writing the transaction compensation/recovery/reconciliation logic to handle partial failures. On the other point, I agree that JEE is unfairly maligned. We build a modern AngularJS/Material frontend on a JavaEE 7 backend and get all the benefits of a scalable, secure enterprise platform from Java EE.
ah- 10y ago
You can use event logs and eventual consistency to solve this problem. Basically you make the transfer of money an event that is then atomically committed to an event log. The two bank accounts then eventually incorporate that state. See http://www.grahamlea.com/2016/08/distributed-transactions-microservices-icebergs/ http://www.grahamlea.com/2016/08/distributed-transactions-mi... But I agree that often life is easier if you just keep things simpler. If you require strong consistency like with the user/profile don't make that state distributed. If you do make it distributed you need to live with less consistency.
- erikpukinskis 10y ago
  If we're talking about user created event, I think you should have a permanent log anyway. Otherwise if there is any kind of bug or any kind of weirdness in your data, how are you supposed to figure out what the heck happened? You can try to outwit chaos by prevent all inconsistency and bad data, but that'd a game you will eventually lose. From a more religious perspective, every action a user takes is sacred data that you should not lose, unless you deliberately want to lose it for privacy reasons. You can rely on your software to have no bugs and never confuse your user, but again that's a game you eventually use. I would rather keep that action log so I have a place to rebuild from when things are lost. Otherwise your only choice is to reverse engineer your data. Data which, if not technically corrupt, is corrupted from a human standpoint. And if you want undo you need a log and playback anyway. To me data consistency at the database level is not a real solution to the problem. It is a good tool, but it only solves a very narrow slice of predictable bugs. It doesn't help at all with the inevitable bugs you can't predict. A log based approach gives you a powerful tool in all kinds of tough situations.
acjohnson55 10y ago
I sort of agree. The examples in the article are ways in which people play fast and loose with consistency, often using a NoSQL store that has poor support for atomicity and isolation. This is a helpful message, because I've definitely seen naively designed systems start to develop all sorts of corruption when used at scale. The answer for many low-throughput applications is to just use Postgres. Both Django and Rails, by default, work with relational databases and leverage transactions for consistency. Then, there is the rise of microservices to consider. In this case, I also agree with the author that it becomes crucial to understand that the number of states your data model can be in can potentially multiply, since transactional updates are very difficult to do. But I feel like on the opposite side of the spectrum of sophistication are people working on well-engineered eventually consistent data systems, with techniques like event sourcing, and a strong understanding of the hazards. There's a compelling argument that this more closely models the real world and unlocks scalability potential that is difficult or impossible to match with a fully consistent, ACID-compliant database. Interestingly, in a recent project, I decided to layer in strict consistency on top of event sourcing underpinnings (Akka Persistence). My project has low write volume, but also no tolerance for the latency of a write conflict resolution strategy. That resulted in a library called Atomic Store [1]. [1] https://github.com/artsy/atomic-store https://github.com/artsy/atomic-store
toolslive 10y ago
The problem is that when the system does not guarantee consistency, you force the application developer using the system to solve that problem. Each application developed, will have to solve the same problem. Besides the fact the same effort is done over and over again, you also are forcing application developers to solve a problem for which they probably do not have the right skill set. In short, that strategy is wasteful (replicating work) and risky (they'll make mistakes)
xarien 10y ago
Depends on your POV. Startups undervalue it and corporations overvalue it. At the end of the day, it's just risk management.
phamilton 10y ago
This profile example is missing the better approach: avoid the dependency of creating the user before creating the profile. Create the profile with a generated uuid. Once that succeeds, then create the user with the same uuid. If you build a system that allows orphaned profiles (by just ignoring them) then you avoid the need to deal with potentially missing profiles. This is essentially implementing MVCC. Write all your data with a new version and then as a final step write to a ledger declaring the new version to be valid. In this case, creating the user is writing to that ledger.
- kpmah 10y ago
  Well, we're playing make believe with the requirements. In fairyland the user and the profile need to exist together. Only the fairies know why! If you can relax the requirements, you can relax the constraints.
  - phamilton 10y ago
    That's the point. Requirements are rarely as strict as we presume they are.
matt_wulfeck 10y ago
Maybe I'm crazy, but I never see atomic libraries that are called like this: bank_account2.deposit(amount) bank_account1.deposit(amount) Isn't this kind of thing always called in some atomic batch operation? transact.batch([ account[a] = -8, account[b] = 8 ]).submit()
- JoachimSchipper 10y ago
  Yes; "with atomic...:" seems to be pseudo-syntax.
marknadal 10y ago
Disclaimer: I work on distributed systems and have spoken around the world on them, so I am very biased. I think something a lot of people miss is that the universe itself is not strongly consistent. This is Einstein's theory of relativity. It fundamentally takes time for information to travel. So if strong consistency is not viable, even at a physics level, what can we do instead? That is why our team here at http://gun.js.org/ http://gun.js.org/ believe in CRDTs. What are CRDTs? They are data structures that are mathematically proven to produce the same results on retries - so even if the power goes out, or the network fails, you can safely re-attempt the update. This means you fundamentally don't need "transactions" or any of these jargon words that people often throw out. Sadly CRDTs are becoming another one of those jargons, despite how simple they are in reality.
- nickpsecurity 10y ago
  "This is Einstein's theory of relativity. It fundamentally takes time for information to travel." Which is why Google's hack in Spanner and F1 RDBMS of simply using or waiting out the known time delay was so clever. Also relies on a tool that was a test of Einstein's theory. ;) "That is why our team here at http://gun.js.org/ http://gun.js.org/ believe in CRDTs." Thanks for the link. I'll look into it. Plus the liberal license that gives it a chance of making a dent in commercial sector or in combo with BSD/Apache projects.
chajath 10y ago
To actually do a distributed transaction, I would look into algorithms such as 2PC or 3PC. Although before going there, I would seriously consider consolidating different backends into one scalable option (banking transaction example is a bit contrived, although I gather the author is just trying to make a point). At the service level integration, we can leverage a reliable message queue middleware to make sure a task is eventually delivered and handled (or be put in a dead letter queue so we can do a batch clean up) Also as a general principle, I would make each of those sub-transactions to be idempotent, so that retrying multiple times won't hurt, and there would be a natural way of picking the winner if there are conflicting ongoing commit / retry attempts.

220 points by kpmah 10y ago

I wandered lonely as a cloud That floats on high o'er vales and hills, When all at once I saw a crowd, A host, of golden daffodils; Beside the lake, beneath the trees, Fluttering and dancing in the breeze.

yyyuuu 10y ago

Continuous as the stars that shine And twinkle on the milky way, They stretched in never-ending line Along the margin of a bay: Ten thousand saw I at a glance, Tossing their heads in sprightly dance.

dang 10y ago
We're always up for the Romantics (though I prefer Coleridge), but maybe not like this. Your comments have been inappropriate for Hacker News lately, especially the one where you impersonated another user. This sort of pattern will get an account banned, so please stick to civil, substantive comments only.

olalonde 10y ago

And in case you think there's a general solution to that problem, there isn't: https://en.wikipedia.org/wiki/CAP_theorem https://en.wikipedia.org/wiki/CAP_theorem Still, it's funny how banking seems to be the canonical example for why we need transactions given that most banking transactions are inconsistent (http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on-why-banks-are-base-not-acid-availability.html http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on...).

kpmah 10y ago
Well, they're eventually consistent. It's not like it hasn't been thought through. :) It's a useful example because it shows how there can be serious consequences for getting it wrong.
- yyyuuu 10y ago
  yes .. you are right
kpmah 10y ago
Oh, in addition to my other reply I should point out that the 'C' in CAP is not the same thing as the 'C' in ACID. Be careful with that one, it can cause confusion! http://blog.thislongrun.com/2015/03/the-confusing-cap-and-acid-wording.html http://blog.thislongrun.com/2015/03/the-confusing-cap-and-ac...

bullen 10y ago

But consider this: You are using mysql, you make a transaction with say deposit and withdraw. What happens on the mysql machine if you pull the plug exactly when mysql has done the deposit but not the withdraw? The ONLY difference between SQL transactions and NoSQL microservice transactions is the time between the parts of a transaction. Personally I use a JSON file with state to execute my NoSQL microservice transactions and it's alot more scalable than having a pre-internet era SQL legacy design hogging all my time and resources.

lomnakkus 10y ago
> What happens on the mysql machine if you pull the plug exactly when mysql has done the deposit but not the withdraw? Nothing bad -- the deposit will never be seen. It sounds like you need to read up on what ACID means. (EDIT: Clarification)
XorNot 10y ago
This is not how database transactions work. A transaction means that the amended state is not visible to any readers until it has been fully committed. Pulling the plug means the transaction is lost, and the writer knows that it was not committed (and so can try again).
hmottestad 10y ago
>> What happens on the mysql machine if you pull the plug exactly when mysql has done the deposit but not the withdraw? Nothing happens. And the client discovers that the connection to the db was closed before the 2 phase commit was done and gives an error to the user.
matthewmacleod 10y ago
I don't mean to be unkind, but is this meant to be a parody? There are no "parts of a transaction" because a transaction is definitionally atomic. Transactional file modification is a fairly tricky problem, and I'd be surprised if you'd actually implemented a safe system in that manner. What's certain though is that spinning up MySQL or Postgres and using it to store simple records is essentially a zero-cost setup task - so I doubt it's ever going to "hog all your time and resources".

fagnerbrack 10y ago

Why isn't the OP using Event Sourcing "commands" for the "Bank Accounts" example?

victorNicollet 10y ago
I believe Event Sourcing opens a whole other can of worms which would detract from the point of the article, such as whether the event streams have a well-defined order (especially if each account is its own aggregate), or whether the resulting event should be "Transaction Completed" (assumes pre-conditions were checked) or "Transaction Attempted" (checks pre-conditions before altering state).
- partisan 10y ago
  My teams' event sourcing implementation stores and publishes events in a well defined order within a specific aggregate. There are scenarios where a two-phase commit is used to ensure that invariants across aggregates are maintained. We use an ACID compliant database to store our domain events and we project our events into a relational schema. When projecting events, we use transactions to make sure the database updates from a specific event are either all applied or not at all.
  - victorNicollet 10y ago
    Yes, this is a typical way of doing things in the CQRS/ES world, but there are many things left unsaid: - would you have a single aggregate for "all bank accounts" so that one financial transaction equals one event, or is there a good reason to have multiple aggregates ? - is command execution transactional, or is it possible for another thread to generate (possibly conflicting) events between the time you check a precondition and the time you write the intended events ? My team uses an ES implementation with very strong guarantees: one aggregate per macro-service, and command execution is transactional, but I'm fairly certain that this is not the way CQRS/ES is described or recommended.
    - partisan 10y ago

fagnerbrack 10y ago

In case anyone is wondering what an "atomic change" means in database terminology: https://www.gnu.org/software/emacs/manual/html_node/elisp/Atomic-Changes.html https://www.gnu.org/software/emacs/manual/html_node/elisp/At...

calind 10y ago

I think it's a bad example because this should not be the way to develop in this kind (microservices) of systems. In these environments you atomically create objects in your application's "local" storage and have a reconciliation loop for creating objects in other services or deleting these orphan "local" objects.

kpmah 10y ago
I think it's a good example because this should not be the way to develop this kind of system - and yet people do it this way! :)

morgo 10y ago

Good article. I've stopped using bank transfers as an example for Acid transactions, and instead talk about social features: - if I change a privacy setting in Facebook or remove access to a user, these changes should be atomic and durable - transactions offer a good semantic of which to make these changes. They can be staged in queries, but nothing is successful until after a commit. - without transactions durability is hard to offer. You would essentially need to make each query flush to disk, rather tha each transaction. Much more expensive.

xanton 10y ago

Transactions can have different isolation levels. And sometimes the problem at hand can be implemented using transactions with weak isolation levels which are not that hard to implement using your favorite NoSQL database that support CAS operation. I recommend this article: http://rystsov.info/2012/09/01/cas.html http://rystsov.info/2012/09/01/cas.html

kpmah 10y ago
Yes, be aware of the transactional capabilities of your database. Some ACID databases in their default configuration don't provide all transactional guarantees!
- brandur 10y ago
  I just wanted to add that the Postgres documentation [1] and Wikipedia [2] are both really well written when it comes to explaining each isolation level and the guarantees that it provides. Well worth the reading time. [1] https://www.postgresql.org/docs/current/static/transaction-iso.html https://www.postgresql.org/docs/current/static/transaction-i... [2] https://en.wikipedia.org/wiki/Isolation_(database_systems)#Isolation_levels https://en.wikipedia.org/wiki/Isolation_(database_systems)#I...

brandur 10y ago

Amen. Whether or not the article's example is a good one, in a world without consistency you need to worry about state between _any_ two database operations in the system, so there's nearly unlimited opportunity for this class of error in almost any application found in the real world. The truly nefarious aspect of NoSQL stores is that the problems that arise from giving up ACID often aren't obvious until your new product is actually in production and failures that you didn't plan for start to appear. Once you're running a NoSQL system of considerable size, you're going to have a sizable number of engineers who are spending significant amounts of their time thinking about and repairing data integrity problems that arise from even minor failures that are happening every single day. There is really no general fix for this; it's going to be a persistent operational tax that stays with your company as long as the NoSQL store does. The same isn't true for an ACID database. You may eventually run into scaling bottle necks (although not nearly as soon as most people think), transactions are darn close to magic in how much default resilience they give to your system. If an unexpected failure occurs, you can roll back the transaction that you're running in, and in almost 100% of cases this turns out to be a "good enough" solution, leaving your application state sane and data integrity sound. In the long run, ACID databases pay dividends in allowing an engineering team to stay focused on building new features instead of getting lost in the weeds of never ending daily operational work. NoSQL stores on the other hand are more akin to an unpaid credit card bill, with unpaid interest continuing to compound month after month.

socmag 10y ago
A database is an abstract concept, and ACID transactions on databases have nothing to do with the interface semantics used to interact with the shared state being managed. You can have non-ACIDic SQL databases and ACID NoSQL databases. While it is true that many KV stores do not present a fully consistent view over distributed data, there are some that do. It is very important that people shed this idea that only SQL databases can have strong consistency, that's for sure.
- brandur 10y ago
  > It is very important that people shed this idea that only SQL databases can have strong consistency, that's for sure. Why? In practice, people are going to be building things using the same common paths: Postgres, Mongo, MySQL, Redis, etc., and from a purely pragmatic standpoint, the SQL databases that you're likely to use come with strong ACID guarantees while the NoSQL databases don't. If you can point me to a NoSQL data store that provides high availability, similar atomicity, consistency, and isolation properties provided by a strong isolation level in Postgres/MySQL/Oracle/SQL Server, and has a good track record of production use, I'm all ears.

muteor 10y ago

If anyone is interested in this sort of thing, I found this a great article: http://www.grahamlea.com/2016/08/distributed-transactions-microservices-icebergs/ http://www.grahamlea.com/2016/08/distributed-transactions-mi...

vidarh 10y ago

My opinion is exactly opposite: Consistently is overvalued. Requiring consistency in distributed system generally leads to designs that reduces availability. Which is one of the reasons that bank transactions generally do not rely on transactional updates against your bank. "Low level" operations as part of settlement may us transactions, but the bank system is "designed" (more like it has grown by accretion) to function almost entirely by settlement and reconciliation rather than holding onto any notion of consistency. The real world rarely involves having a consistent view of anything. We often design software with consistency guarantees that are pointless because the guarantees can only hold until the data has been output, and are often obsolete before the user has even seen it. That's not to say that there are no places where consistency matters, but often it matters because of thoughtless designs elsewhere that ends up demanding unnecessary locks and killing throughput, failing if connectivity to some canonical data store happens to be unavailable etc. The places where we can't design systems to function without consistently guarantees are few and far between.

kpmah 10y ago
Banks are forced to be eventually consistent because of their distributed nature. If it was practical to have a big central ACID database I'm sure that's what they'd use. There is an extra layer of complexity with distributed systems. That said, if the problem has been thought about, I'm happy. My frustration is with people not understanding the trade-offs.
- deleted 10y ago
  [deleted]
- vidarh 10y ago
  The problem is that a big central ACID database enforces strict limits on throughput limited at best by the the speed of light (in reality it's worse). Enforcing a globally consistent views basically requires coordination. Coordination requires that a signal traverses a distance and is acted on no faster than the instruction throughput of a single CPU core. When the sources of operations are geographically spread out, you will be limited based on the fastest communication possible between those locations in addition to the processing latency. Ultimately this means that there are scaling limits to throughput of a fully serialised system that causes real problems all the time. Sometimes a single global truth is still fast enough. Very often it isn't, when e.g. your users are spread out globally. That's why it's important to consider avoiding consistency when it is not necessary.

mettamage 10y ago

Before I read this article, the following question popped into my mind and it miiiiiight be tangentially related -- yea probably not, blame the title ;) When taking the concept of consistency, does consistency have an effect that is akin to compound interest? For example, imagine someone doing the same thing year after year diligently. (S)he'd increase his or her skill say 10% a year (have no clue what realistic numbers are). Would that mean that the compound interest effect would occur? I phrased it really naievely, because while the answer is "yes" in those circumstances (1.1 ^ n). I'm overlooking a lot and have no clue what I overlook. I know it's off-topic, it's what I thought when I read the title and I never thought about it before, so I'm a bit too curious at the moment ;)

sz4kerto 10y ago

The universally hated JEE can do distributed transactions by default. Yes, with pitfalls, but it can. (It is usually hated by devs who have never used it properly.)

jamesblonde 10y ago
The problem with JEE is when you leave Java Transactions - i.e., interact with some external system as part of a transaction. Then there is no way around writing the transaction compensation/recovery/reconciliation logic to handle partial failures. On the other point, I agree that JEE is unfairly maligned. We build a modern AngularJS/Material frontend on a JavaEE 7 backend and get all the benefits of a scalable, secure enterprise platform from Java EE.

ah- 10y ago

You can use event logs and eventual consistency to solve this problem. Basically you make the transfer of money an event that is then atomically committed to an event log. The two bank accounts then eventually incorporate that state. See http://www.grahamlea.com/2016/08/distributed-transactions-microservices-icebergs/ http://www.grahamlea.com/2016/08/distributed-transactions-mi... But I agree that often life is easier if you just keep things simpler. If you require strong consistency like with the user/profile don't make that state distributed. If you do make it distributed you need to live with less consistency.

erikpukinskis 10y ago
If we're talking about user created event, I think you should have a permanent log anyway. Otherwise if there is any kind of bug or any kind of weirdness in your data, how are you supposed to figure out what the heck happened? You can try to outwit chaos by prevent all inconsistency and bad data, but that'd a game you will eventually lose. From a more religious perspective, every action a user takes is sacred data that you should not lose, unless you deliberately want to lose it for privacy reasons. You can rely on your software to have no bugs and never confuse your user, but again that's a game you eventually use. I would rather keep that action log so I have a place to rebuild from when things are lost. Otherwise your only choice is to reverse engineer your data. Data which, if not technically corrupt, is corrupted from a human standpoint. And if you want undo you need a log and playback anyway. To me data consistency at the database level is not a real solution to the problem. It is a good tool, but it only solves a very narrow slice of predictable bugs. It doesn't help at all with the inevitable bugs you can't predict. A log based approach gives you a powerful tool in all kinds of tough situations.

acjohnson55 10y ago

I sort of agree. The examples in the article are ways in which people play fast and loose with consistency, often using a NoSQL store that has poor support for atomicity and isolation. This is a helpful message, because I've definitely seen naively designed systems start to develop all sorts of corruption when used at scale. The answer for many low-throughput applications is to just use Postgres. Both Django and Rails, by default, work with relational databases and leverage transactions for consistency. Then, there is the rise of microservices to consider. In this case, I also agree with the author that it becomes crucial to understand that the number of states your data model can be in can potentially multiply, since transactional updates are very difficult to do. But I feel like on the opposite side of the spectrum of sophistication are people working on well-engineered eventually consistent data systems, with techniques like event sourcing, and a strong understanding of the hazards. There's a compelling argument that this more closely models the real world and unlocks scalability potential that is difficult or impossible to match with a fully consistent, ACID-compliant database. Interestingly, in a recent project, I decided to layer in strict consistency on top of event sourcing underpinnings (Akka Persistence). My project has low write volume, but also no tolerance for the latency of a write conflict resolution strategy. That resulted in a library called Atomic Store [1]. [1] https://github.com/artsy/atomic-store https://github.com/artsy/atomic-store

toolslive 10y ago

The problem is that when the system does not guarantee consistency, you force the application developer using the system to solve that problem. Each application developed, will have to solve the same problem. Besides the fact the same effort is done over and over again, you also are forcing application developers to solve a problem for which they probably do not have the right skill set. In short, that strategy is wasteful (replicating work) and risky (they'll make mistakes)

xarien 10y ago

Depends on your POV. Startups undervalue it and corporations overvalue it. At the end of the day, it's just risk management.

phamilton 10y ago

This profile example is missing the better approach: avoid the dependency of creating the user before creating the profile. Create the profile with a generated uuid. Once that succeeds, then create the user with the same uuid. If you build a system that allows orphaned profiles (by just ignoring them) then you avoid the need to deal with potentially missing profiles. This is essentially implementing MVCC. Write all your data with a new version and then as a final step write to a ledger declaring the new version to be valid. In this case, creating the user is writing to that ledger.

kpmah 10y ago
Well, we're playing make believe with the requirements. In fairyland the user and the profile need to exist together. Only the fairies know why! If you can relax the requirements, you can relax the constraints.
- phamilton 10y ago
  That's the point. Requirements are rarely as strict as we presume they are.

matt_wulfeck 10y ago

Maybe I'm crazy, but I never see atomic libraries that are called like this: bank_account2.deposit(amount) bank_account1.deposit(amount) Isn't this kind of thing always called in some atomic batch operation? transact.batch([ account[a] = -8, account[b] = 8 ]).submit()

JoachimSchipper 10y ago
Yes; "with atomic...:" seems to be pseudo-syntax.

marknadal 10y ago

Disclaimer: I work on distributed systems and have spoken around the world on them, so I am very biased. I think something a lot of people miss is that the universe itself is not strongly consistent. This is Einstein's theory of relativity. It fundamentally takes time for information to travel. So if strong consistency is not viable, even at a physics level, what can we do instead? That is why our team here at http://gun.js.org/ http://gun.js.org/ believe in CRDTs. What are CRDTs? They are data structures that are mathematically proven to produce the same results on retries - so even if the power goes out, or the network fails, you can safely re-attempt the update. This means you fundamentally don't need "transactions" or any of these jargon words that people often throw out. Sadly CRDTs are becoming another one of those jargons, despite how simple they are in reality.

nickpsecurity 10y ago
"This is Einstein's theory of relativity. It fundamentally takes time for information to travel." Which is why Google's hack in Spanner and F1 RDBMS of simply using or waiting out the known time delay was so clever. Also relies on a tool that was a test of Einstein's theory. ;) "That is why our team here at http://gun.js.org/ http://gun.js.org/ believe in CRDTs." Thanks for the link. I'll look into it. Plus the liberal license that gives it a chance of making a dent in commercial sector or in combo with BSD/Apache projects.

chajath 10y ago

To actually do a distributed transaction, I would look into algorithms such as 2PC or 3PC. Although before going there, I would seriously consider consolidating different backends into one scalable option (banking transaction example is a bit contrived, although I gather the author is just trying to make a point). At the service level integration, we can leverage a reliable message queue middleware to make sure a task is eventually delivered and handled (or be put in a dead letter queue so we can do a batch clean up) Also as a general principle, I would make each of those sub-transactions to be idempotent, so that retrying multiple times won't hurt, and there would be a natural way of picking the winner if there are conflicting ongoing commit / retry attempts.