13 ms·
Eric Brewer on Why Banks Are BASE Not ACID – Availability Is Revenue (2013)
- brandur 10y agoInteresting article, but it's really talking about ATM networks rather than the broader "banks". A bank's core systems could still hugely benefit from having an ACID compliant database even if any individual ATM node could still be allowed to operate independent for some amount of time to maximize availability. Consistency is a feature whose importance is regularly downplayed by hawkers of NoSQL databases. I've seen firsthand what happens when a NoSQL database scales out and it's not pretty — a significant number of people end up spending a significant amount of their time trying to figure out how to maintain data integrity across failures in the system, even the kind of minor failures that are occurring every day. A lot of the time that involves going in and cleaning up data by hand. There is no general solution for this problem, and it'll be a persistent operational tax that dogs you as long as your NoSQL database stays in use.
- justincormack 10y agoHistorically banks were distributed - your account was at a particular branch, and transfers needed a remote protocol.
- brockers 10y agoNo the article uses ATMs as an example but the general accounting principals apply regardless. Consider paper checks or IOUs. Humanity literally has thousands of years doing eventual financial consistency while ACID compliance has only existed for the last 50 or so. I'm not saying ACID isn't good (or that NoSQL is a panacea) but we have long had exceedingly successful financial systems without ACID.
- brokencode 10y agoWhile interesting, this is not satisfying as a general argument for non-ACID databases. Incrementing and decrementing are about the simplest possible operations, and even the article admitted that these are commutative, so the order doesn't matter. It is indeed much easier to work with operations that can be ordered arbitrarily, but I think that most real-world problems are sequential.
- droro 10y agoThe conclusions of this article have far more implications for the future of blockchain technologies than for the future of databases. Bitcoin and Ethereum and their ilk have, out of necessity, valued ACID qualities above all else- including availability and responsiveness. Does it make sense that completing a $1 transaction would require the same amount of resources as a $100,000 transaction? The traditional banking system has the flexibility to expend fewer resources on the $1 transaction, but Bitcoin does not- by design. Interestingly, I think it is the inconvenience of the traditional banking system that gives it this flexibility. If financial transactions were something that could be done instantaneously, by a computer, without having to provide personal information to set up an account, then stealing money in 100,000 $1 transactions might be easier than stealing $100,000 in one big heist. Add a little inconvenience to each transaction and this problem goes away. Unless blockchain transactions are extremely cheap, it will limit their competitiveness with the traditional banking system for this reason.
- cloudjacker 10y agoYou can get a benevolent miner friend to process your blockchain transaction for free Gotta send them the signature and broadcast it directly to them
- amscanne 10y ago> Bitcoin and Ethereum and their ilk have, out of necessity, valued ACID qualities above all else They definitely don't have ACID properties. The blockchain gives you something like eventual consistency, but it's probabilistic. You have atomicity within a single block, but given the acceptance of the block is only eventually consistent, I'm not sure it's meaningful.
- fauigerzigerk 10y agoThis is a very selective view of banking. In financial reporting you have a lot of situations where you need a consistent point-in-time snapshot of your data or you might be in very big trouble. It's true that clearing of payments and trades is usually a multi-step process that isn't done in one huge distributed transaction. But implementing each stage of that process without ACID transactions seems like a daunting task to me. So I think Eric Brewer's remarks lack a bit of nuance and detail, but he's not entirely wrong of course.
- reqres 10y agoCould you be specific. In what particular areas do you need a consistent point-in-time snapshot? Having worked in finance myself, as part of a team that procured and replaced an international core banking system with a state-of-the-art, multi-hundred million dollar successor; we've never had a strong consistency requirement. Much weaker forms consistency sufficed for their needs and the requirements of regulators. Also what big trouble do you mean? I do not know of any financial regulators that expect banks to have a totally consistent point-in-time view of their balance sheet. Furthermore the Basel Accords and similar banking supervision rules do not mandate such a requirement. I be very much grateful if you could provide me with this information, as I can pass it to my former colleagues who will be quite surprised to discover this requirement.
- BoorishBears 10y agoThat last paragraph implies you're less interested in finding out and more interested in insisting those requirements don't exist because you worked on a project that overlaps with the incredibly broad term "financial reporting". The term is so broad I'd almost say your anecdotal experience is more likely to be irrelevant than not.
- reqres 10y agoA core banking system captures and stores all the transactions and business a bank performs. Without this a bank would not be able to function let alone produce a meaningful financial report.
- danjoc 10y ago(2013) It would be interesting to know if BASE at ATMs had anything to do with http://www.theatlantic.com/international/archive/2016/05/japan-atm-theft/483902/ http://www.theatlantic.com/international/archive/2016/05/jap... Coordinated attack over two hours? A little DDOS + BASE perhaps?
- sfifs 10y agoI find quite often in India that whem networks are down, ATMs go down displaying a network failure error message on screen. I've never heard of anyone being able to overdraw an account. Credit cards also need authorization both through PIN on terminal and bank authorization over the networks (which do fail from time to time). Over the Internet, two factor authentication is pretty much mandatory especially for credit cards. Therefore this whole Banks don't value consistency argument feels very fishy to my experience. Is this a uniquely American thing like lack of chip and pin?
- carlob 10y agoI don't know about India, but I know that in many western European countries overdraft fees such as those that are common in the US would be considered above the maximum legal interest rate, because of that the bank has no big incentive in letting you withdraw money you don't have. Consider also how much more widespread chip and pin cards are on this side of the pond, and you're starting to make me suspect that the wider adoption of chip and pin might have been helped by usury laws.
- PaulHoule 10y agoIt is easy to laugh at Bitcoin but with Bitcoin you can send $15 from New Jersey to France immediately but the fee to do a bank transfer is many times that. I wrote a check for $130,000 dollars that bounced once because the teller deposited the check I got for selling a house in somebody else's account. Mistakes happen, but the bank is able to figure it out and (hopefully) make it right - that is why the system is built to tolerate slop.
- karmajunkie 10y agoIf you'd sent it to the wrong wallet, OTOH, you'd just be at the mercy of the wallet owner to return the BTC. These are real problems, and by the time bitcoin proponents find adequate solutions for them, what they will have created is going to look an awful lot like the banking system.
- wtbob 10y ago> If you'd sent it to the wrong wallet, OTOH, you'd just be at the mercy of the wallet owner to return the BTC. IIRC, a wallet is just a public key, so nearly any incorrect account number will be a public key for which no-one has created the private key — thus there would be no-one to return the money. This really isn't a problem, I think, because I'd rather have to rely on me and my computing equipment not committing errors rather than have to rely on some random teller not making errors.
- threeseed 10y agoYour point makes no sense. If some random teller makes a mistake the bank can just organise for the transaction to be reversed.
- aianus 10y ago> If some random teller makes a mistake the bank can just organise for the transaction to be reversed. It is not that easy to reverse an international wire transfer, especially if the funds have already been withdrawn.
- deleted 10y ago[deleted]
- marknadal 10y agoThis is such an important article for people to read, especially people so bent on strong consistency. I've done talks on this for our database (eventually consistent, http://gun.js.org/ http://gun.js.org/ ), and people always ask about banking. Yes banks should use strongly consistent approaches, but they don't - user experience trumps perfect academics. And having Eric Brewer say this is fantastic evidence.
- lotyrin 10y agoWhat do you do when your reservation app gives two people the same table? What do you do when your auction app thinks there are two winning bidders? It's not about "perfect academics" it's about risk management. How do you prevent or manage inconsistencies.
- marknadal 10y agoThese are great questions and I hope I can convince you with some simple answers. We cannot get this problem if we treat the table as its own thing. So for instance, rather than having a "reservation" that has an "assignedTable" column with a string of "2A", if instead we have a table that has a property of "7pm" with a value of "Mr & Mrs Smith" then this value cannot be a string munged with the other couple. This easily applies to many other areas, like finance. Rather than having a "purchase" record that has a "debit" amount on it, make the software model the real world of currency. Instead you have a "dollar" record with a unique serial number, it has a "owner" property of "lotyrin". No matter how many times we concurrently write to this property, the end result must necessarily be 1 value. This is the way humans have been thinking for thousands of years. And guess what, I sat down with a large airline and had to warn them "we're not Strongly Consistent" and they laughed at me saying "you realize we've been booking seat reservations before there was internet, before you were born, and before there was cheap telephony. Seat reservation has never been strongly consistent - we used to have hundreds of travel agents booking seats and it would take 2 weeks before we would hear about it." Again, I want to emphasize this is exactly the reason that perfect academics doesn't always help - it is a solution to a problem in isolation, devoid of its real world context. I am in no way saying the academics aren't useful or interesting, but I am saying that even Eric Brewer himself recognizes where academics aren't helpful. But to stress your point, let's take your example further. Rather than assuming the restaurant actually has a credit card authorization server that is the source of authority, let us pretend it is more of a bitcoin crowd hosted reservation service. Now despite our "Mr & Mrs Smith" value being straightforward, we wind up having a replication problem. One server has it as that value, and another peer-to-peer server has it as "Mr & Mrs Lionheart". What happens? Well, under our database (like an Open Source Firebase, see the link in my original comment), this is why realtime updates and push notifications are so handy. Even if 2 couples reserve the same table on different peer servers at roughly the same time (such that when the peer servers checked availability there was no conflict), when the new update saves it gets pushed out to all the peers and runs through a conflict resolution algorithm locally. So all peers sync based off of push, not an outdated pull. In fact, we made a great explainer video for this type of situation, see http://gun.js.org/explainers/conflict_resolution/lexical_sorting.html http://gun.js.org/explainers/conflict_resolution/lexical_sor... (might change the URL naming scheme in future, so careful of a broken link). But I'm going to again assume you aren't talking about "what practically happens" but are intentionally trying to push for the extreme edge case bugs. If so, let us add that the peers are offline while these updates happen (because yes, this is something we support, being offline-first) so therefore cannot sync with each other fast enough for the 2 couples to realize one of them got ousted. We are now only left with two scenarios: A) Because when the app tries to save the data it gets an error that the push notification failed due to the network being down, we display a warning to the user that the "Reservation is pending, proceed at your own risk or try again later". Or B) Because we are being academic, we yes go ahead and make the reservation (unaware of the conflict). How do we address it? Well... Finally, both the Lionhearts and Smiths show up at the same time. Initially they argue both asserting "I have the reservation" until a hostess steps in. The hostess apologizes and says "We are terribly sorry, this is our mistake and let us fix it for you." They then offer: A) Would one of you like to sit at the open table over here, that somebody canceled last minute on? B) If (A) is not available, then here are free drinks* for all of you, but who would like a 20% discount* off their next dinner? Next open table should be in about 45 minutes? *Total loss is 75cents on sugar water and $10 of discount, but profit GAIN is $50 (because had the reservation app said "booked out" the 2nd couple would have never turned into a customer), leaving the restaurant at a net of +$39.25 at the end of the night. Again, let us make it more extreme, both couples are uncompromising, saying "We cannot do that, we have a show at 9pm", then there are only 2 remaining situations: A) You flip a coin and 1 couple leaves angry and never comes back to use the 20% discount because they are so angry the coin did not choose them. (Note, the restaurant still made a profit filling out the seat though) OR B) You basically flip a coin and 1 couple leaves bummed, but enjoying their free drink and looking forward to coming back another time with a 20% discount. (Note, restaurant still makes a positive profit) The moral of the story is that you want to create Win-Win options for everybody. Why? Because as you say, it is all about risk management and the probabilities that Murphy's Law is going to strike so many times (multi-continuously-offline-peers-that-do-not-error-with-above-average-unsatisfiable-angry-humans) such that you get an angry customer with a free drink and a coupon is extraordinarily unlikely. And if you have not already calculated for ever losing 75cents on sugar water plus a discount, then something is wrong with your risk management department - not with your algorithm. With that, I'll leave the bidding app to your imagination. The trick question for finding simple solution is: What would humans do before the telephone? I hope this was a pleasure to read, and I want to thank you for calling me out by asking the glaring question. If you have any concerns or counter arguments, please let me know. I would be happy to hear them. If you like what you see here though, check out the rest of our work on the Open Source database. Cheers!
- erikb 10y agoI could only read a third of the article before starting to crinch too much. If this was on purpose and I missed the same conclusion I am presenting here, sorry. The article sounds like "this of course is the smart way, make money first, worry about details later". But if you think about it this is very, very wrong. First, if any system will fail drastically at some point. You wait long enough and something unexpected happens. Second, if this consistency tolerance becomes widely known people will try, and at some point succeed, to cheat the system big time. If any of these two scenarios happens, the whole system breaks. And then, what do you do? You can only get so many bail outs from the tax payers. The smart way, of course, is to not be too greedy and stick close to a relatively consistent scenario.
- hackuser 10y ago1234 days ago: https://news.ycombinator.com/item?id=5642010 https://news.ycombinator.com/item?id=5642010