5 ms·

This is such an important article for people to read, especially people so bent on strong consistency. I've done talks on this for our database (eventually cons

This is such an important article for people to read, especially people so bent on strong consistency. I've done talks on this for our database (eventually consistent, http://gun.js.org/ http://gun.js.org/ ), and people always ask about banking. Yes banks should use strongly consistent approaches, but they don't - user experience trumps perfect academics. And having Eric Brewer say this is fantastic evidence.

lotyrin 10y ago
What do you do when your reservation app gives two people the same table? What do you do when your auction app thinks there are two winning bidders? It's not about "perfect academics" it's about risk management. How do you prevent or manage inconsistencies.
- marknadal 10y ago
  These are great questions and I hope I can convince you with some simple answers. We cannot get this problem if we treat the table as its own thing. So for instance, rather than having a "reservation" that has an "assignedTable" column with a string of "2A", if instead we have a table that has a property of "7pm" with a value of "Mr & Mrs Smith" then this value cannot be a string munged with the other couple. This easily applies to many other areas, like finance. Rather than having a "purchase" record that has a "debit" amount on it, make the software model the real world of currency. Instead you have a "dollar" record with a unique serial number, it has a "owner" property of "lotyrin". No matter how many times we concurrently write to this property, the end result must necessarily be 1 value. This is the way humans have been thinking for thousands of years. And guess what, I sat down with a large airline and had to warn them "we're not Strongly Consistent" and they laughed at me saying "you realize we've been booking seat reservations before there was internet, before you were born, and before there was cheap telephony. Seat reservation has never been strongly consistent - we used to have hundreds of travel agents booking seats and it would take 2 weeks before we would hear about it." Again, I want to emphasize this is exactly the reason that perfect academics doesn't always help - it is a solution to a problem in isolation, devoid of its real world context. I am in no way saying the academics aren't useful or interesting, but I am saying that even Eric Brewer himself recognizes where academics aren't helpful. But to stress your point, let's take your example further. Rather than assuming the restaurant actually has a credit card authorization server that is the source of authority, let us pretend it is more of a bitcoin crowd hosted reservation service. Now despite our "Mr & Mrs Smith" value being straightforward, we wind up having a replication problem. One server has it as that value, and another peer-to-peer server has it as "Mr & Mrs Lionheart". What happens? Well, under our database (like an Open Source Firebase, see the link in my original comment), this is why realtime updates and push notifications are so handy. Even if 2 couples reserve the same table on different peer servers at roughly the same time (such that when the peer servers checked availability there was no conflict), when the new update saves it gets pushed out to all the peers and runs through a conflict resolution algorithm locally. So all peers sync based off of push, not an outdated pull. In fact, we made a great explainer video for this type of situation, see http://gun.js.org/explainers/conflict_resolution/lexical_sorting.html http://gun.js.org/explainers/conflict_resolution/lexical_sor... (might change the URL naming scheme in future, so careful of a broken link). But I'm going to again assume you aren't talking about "what practically happens" but are intentionally trying to push for the extreme edge case bugs. If so, let us add that the peers are offline while these updates happen (because yes, this is something we support, being offline-first) so therefore cannot sync with each other fast enough for the 2 couples to realize one of them got ousted. We are now only left with two scenarios: A) Because when the app tries to save the data it gets an error that the push notification failed due to the network being down, we display a warning to the user that the "Reservation is pending, proceed at your own risk or try again later". Or B) Because we are being academic, we yes go ahead and make the reservation (unaware of the conflict). How do we address it? Well... Finally, both the Lionhearts and Smiths show up at the same time. Initially they argue both asserting "I have the reservation" until a hostess steps in. The hostess apologizes and says "We are terribly sorry, this is our mistake and let us fix it for you." They then offer: A) Would one of you like to sit at the open table over here, that somebody canceled last minute on? B) If (A) is not available, then here are free drinks* for all of you, but who would like a 20% discount* off their next dinner? Next open table should be in about 45 minutes? *Total loss is 75cents on sugar water and $10 of discount, but profit GAIN is $50 (because had the reservation app said "booked out" the 2nd couple would have never turned into a customer), leaving the restaurant at a net of +$39.25 at the end of the night. Again, let us make it more extreme, both couples are uncompromising, saying "We cannot do that, we have a show at 9pm", then there are only 2 remaining situations: A) You flip a coin and 1 couple leaves angry and never comes back to use the 20% discount because they are so angry the coin did not choose them. (Note, the restaurant still made a profit filling out the seat though) OR B) You basically flip a coin and 1 couple leaves bummed, but enjoying their free drink and looking forward to coming back another time with a 20% discount. (Note, restaurant still makes a positive profit) The moral of the story is that you want to create Win-Win options for everybody. Why? Because as you say, it is all about risk management and the probabilities that Murphy's Law is going to strike so many times (multi-continuously-offline-peers-that-do-not-error-with-above-average-unsatisfiable-angry-humans) such that you get an angry customer with a free drink and a coupon is extraordinarily unlikely. And if you have not already calculated for ever losing 75cents on sugar water plus a discount, then something is wrong with your risk management department - not with your algorithm. With that, I'll leave the bidding app to your imagination. The trick question for finding simple solution is: What would humans do before the telephone? I hope this was a pleasure to read, and I want to thank you for calling me out by asking the glaring question. If you have any concerns or counter arguments, please let me know. I would be happy to hear them. If you like what you see here though, check out the rest of our work on the Open Source database. Cheers!

lotyrin 10y ago

What do you do when your reservation app gives two people the same table? What do you do when your auction app thinks there are two winning bidders? It's not about "perfect academics" it's about risk management. How do you prevent or manage inconsistencies.

marknadal 10y ago
These are great questions and I hope I can convince you with some simple answers. We cannot get this problem if we treat the table as its own thing. So for instance, rather than having a "reservation" that has an "assignedTable" column with a string of "2A", if instead we have a table that has a property of "7pm" with a value of "Mr & Mrs Smith" then this value cannot be a string munged with the other couple. This easily applies to many other areas, like finance. Rather than having a "purchase" record that has a "debit" amount on it, make the software model the real world of currency. Instead you have a "dollar" record with a unique serial number, it has a "owner" property of "lotyrin". No matter how many times we concurrently write to this property, the end result must necessarily be 1 value. This is the way humans have been thinking for thousands of years. And guess what, I sat down with a large airline and had to warn them "we're not Strongly Consistent" and they laughed at me saying "you realize we've been booking seat reservations before there was internet, before you were born, and before there was cheap telephony. Seat reservation has never been strongly consistent - we used to have hundreds of travel agents booking seats and it would take 2 weeks before we would hear about it." Again, I want to emphasize this is exactly the reason that perfect academics doesn't always help - it is a solution to a problem in isolation, devoid of its real world context. I am in no way saying the academics aren't useful or interesting, but I am saying that even Eric Brewer himself recognizes where academics aren't helpful. But to stress your point, let's take your example further. Rather than assuming the restaurant actually has a credit card authorization server that is the source of authority, let us pretend it is more of a bitcoin crowd hosted reservation service. Now despite our "Mr & Mrs Smith" value being straightforward, we wind up having a replication problem. One server has it as that value, and another peer-to-peer server has it as "Mr & Mrs Lionheart". What happens? Well, under our database (like an Open Source Firebase, see the link in my original comment), this is why realtime updates and push notifications are so handy. Even if 2 couples reserve the same table on different peer servers at roughly the same time (such that when the peer servers checked availability there was no conflict), when the new update saves it gets pushed out to all the peers and runs through a conflict resolution algorithm locally. So all peers sync based off of push, not an outdated pull. In fact, we made a great explainer video for this type of situation, see http://gun.js.org/explainers/conflict_resolution/lexical_sorting.html http://gun.js.org/explainers/conflict_resolution/lexical_sor... (might change the URL naming scheme in future, so careful of a broken link). But I'm going to again assume you aren't talking about "what practically happens" but are intentionally trying to push for the extreme edge case bugs. If so, let us add that the peers are offline while these updates happen (because yes, this is something we support, being offline-first) so therefore cannot sync with each other fast enough for the 2 couples to realize one of them got ousted. We are now only left with two scenarios: A) Because when the app tries to save the data it gets an error that the push notification failed due to the network being down, we display a warning to the user that the "Reservation is pending, proceed at your own risk or try again later". Or B) Because we are being academic, we yes go ahead and make the reservation (unaware of the conflict). How do we address it? Well... Finally, both the Lionhearts and Smiths show up at the same time. Initially they argue both asserting "I have the reservation" until a hostess steps in. The hostess apologizes and says "We are terribly sorry, this is our mistake and let us fix it for you." They then offer: A) Would one of you like to sit at the open table over here, that somebody canceled last minute on? B) If (A) is not available, then here are free drinks* for all of you, but who would like a 20% discount* off their next dinner? Next open table should be in about 45 minutes? *Total loss is 75cents on sugar water and $10 of discount, but profit GAIN is $50 (because had the reservation app said "booked out" the 2nd couple would have never turned into a customer), leaving the restaurant at a net of +$39.25 at the end of the night. Again, let us make it more extreme, both couples are uncompromising, saying "We cannot do that, we have a show at 9pm", then there are only 2 remaining situations: A) You flip a coin and 1 couple leaves angry and never comes back to use the 20% discount because they are so angry the coin did not choose them. (Note, the restaurant still made a profit filling out the seat though) OR B) You basically flip a coin and 1 couple leaves bummed, but enjoying their free drink and looking forward to coming back another time with a 20% discount. (Note, restaurant still makes a positive profit) The moral of the story is that you want to create Win-Win options for everybody. Why? Because as you say, it is all about risk management and the probabilities that Murphy's Law is going to strike so many times (multi-continuously-offline-peers-that-do-not-error-with-above-average-unsatisfiable-angry-humans) such that you get an angry customer with a free drink and a coupon is extraordinarily unlikely. And if you have not already calculated for ever losing 75cents on sugar water plus a discount, then something is wrong with your risk management department - not with your algorithm. With that, I'll leave the bidding app to your imagination. The trick question for finding simple solution is: What would humans do before the telephone? I hope this was a pleasure to read, and I want to thank you for calling me out by asking the glaring question. If you have any concerns or counter arguments, please let me know. I would be happy to hear them. If you like what you see here though, check out the rest of our work on the Open Source database. Cheers!