4 ms·

The 2010 is really important here. And Stonebraker is thinking about local databases systems and was a bit upset but the NoSQL movement push at the time. And h

by belter 8mo ago

The 2010 is really important here. And Stonebraker is thinking about local databases systems and was a bit upset but the NoSQL movement push at the time. And he is making a mistake in claiming the partitions are "exceedingly rare". Again he is not thinking about a global distributed cloud across continents. The real world works with Eventual Consistency. Embrace it, for most 90% of the Business Scenarios its the best option: https://i.ibb.co/DtxrRH3/eventual-consistency.png https://i.ibb.co/DtxrRH3/eventual-consistency.png

hobs 8mo ago
I would say quite the opposite - most business have little need for eventual consistency and at a small scale its not even a requirement for any database you would reasonably used, way more than 90% of companies don't need eventual consistency.
- belter 8mo ago
  No. The real world is full of eventual consistency, and we simply operate around it. :-) Think about a supermarket: If the store is open 24/7, prices change constantly, and some items still have the old price tag until shelves get refreshed. The system converges over time. Or airlines: They must overbook, because if they wait for perfect certainty, planes fly half empty. They accept inconsistency and correct later with compensation. Even banking works this way. All database books have the usual “you can’t debit twice, so you need transactions”…bullshit. But think of a money transfer across banks and possibly across countries? Not globally atomic... What if you transfer money to an account that was closed an hour ago in another system? The transfer doesn’t instantly fail everywhere. It’s posted as credit/debit, then reconciliation runs later, and you eventually get a reversal. Same with stock markets: Trades happen continuously, but final clearing and settlement occur after the fact. And technically DNS is eventual consistency by design. You update a record, but the world sees it gradually as caches expire. Yet the internet works. Distributed systems aren’t broken when they’re eventually consistent. They’re mirroring how real systems work: commit locally, reconcile globally, compensate when needed.
  - awesome_dude 8mo ago
    > Even banking works this way. All database books have the usual “you can’t debit twice, so you need transactions”…bullshit. But think of a money transfer across banks and possibly across countries? Not globally atomic.. Banking is my "go to" anology when it comes to eventual consistency because 1: We use banking almost universally the same ways, and 2: we understand fully the eventual consistency employed (even though we don't think about it) Allow me to elaborate. When I was younger we had "cheque books" which meant that I could write a cheque (or check if you're American) and give it to someone in lieu of cash, they would take the cheque to the bank, and, after a period of time their bank would deposit funds into their account, and my bank would debit funds from mine - that delay is eventual consistency. That /style/ of banking might have gone for some people, but the principle remains the same, this very second my bank account is showing me two "balances", the "current" balance and the "available" balance. Those two numbers are not equal, but they will /eventually/ be consistent. The reason that they are not consistent is because I have used my debit card, which is really a credit arrangement that my bank has negotiated with Visa or Mastercard, etc. Whereby I have paid for some goods/services with my debit card, Visa has guaranteed the merchant that they will be paid (with some exceptions) and Visa have placed a hold on the balance of my account for the amount. At some point - it might be overnight, it might be in a few days, there will be a reconciliation where actual money will be paid by my bank to Visa to settle the account, and Visa will pay the merchant's bank some money to settle the debt. Once that reconciliation takes place to everyone's satisfaction, my account balances will be consistent.
    - da_chicken 8mo ago
      No, this is confusing how the financial institutions operate as a business with how the data store that backs those institutions operates as a technology. You can certainly operate your financial system with a double entry register and delayed reconciliation due to the use of credit and the nature of various forms of promissory notes, but you're going to want the data store behind the scenes to be fully consistent with recording those transactions regardless of how long they might take to reconcile. If you don't know that your register is consistent, what are you even reconciling against? What you're arguing is akin to arguing that because computers store data in volatile RAM and that data will often differ from what is on disk, that you shouldn't have to worry about file system consistency or the integrity of private address spaces. After all, they aren't going to match anyways.
anonymars 8mo ago
Remember also that "partition" is not "yes or no" but rather a latency threshold. If the network is connected but a call now takes 30 seconds instead of milliseconds, that is probably a partition
- convolvatron 8mo ago
  this is likely wrong. the issue with partitions is that we can no longer communicate at all, thus we can't end up in the same state. If we have poor performance, thats certainly something that worth putting machinery in to adapt to, but its not at all in the same class as 'I can't talk to you and I dont know what you're doing at all' fro a correctness standpoint edit: yeah ok, since failure detection is being driven by timers by necessity, then sure. the tradeoff we're making between the interval under which we're unable to make progress vs the upheaval caused by announcing a failure.
  - anonymars 8mo ago
    Yeah, I glossed over a few steps. There's likely a latency threshold beyond which you should abort, and then it is a partition (after all, that's what TCP is doing under the hood if it sends a packet and doesn't get a response). One should be so lucky to have an operation fail immediately, rather than lumber on until it times out (holding resources hostage all the while)!
majormajor 8mo ago
> And he is making a mistake in claiming the partitions are "exceedingly rare". Again he is not thinking about a global distributed cloud across continents. Any time an AWS region or AZ goes down we see a lot of popular services go nearly-completely-down. And it's generally fine. One thing I appreciate about AWS is that (operating "live" in just a single AZ or even single region) I've seen far fewer partition-causing networking hiccups than when my coworkers and I were responsible for wiring and tuning our own networks for our own hardware in datacenters.

by belter 8mo ago

hobs 8mo ago

I would say quite the opposite - most business have little need for eventual consistency and at a small scale its not even a requirement for any database you would reasonably used, way more than 90% of companies don't need eventual consistency.

belter 8mo ago
No. The real world is full of eventual consistency, and we simply operate around it. :-) Think about a supermarket: If the store is open 24/7, prices change constantly, and some items still have the old price tag until shelves get refreshed. The system converges over time. Or airlines: They must overbook, because if they wait for perfect certainty, planes fly half empty. They accept inconsistency and correct later with compensation. Even banking works this way. All database books have the usual “you can’t debit twice, so you need transactions”…bullshit. But think of a money transfer across banks and possibly across countries? Not globally atomic... What if you transfer money to an account that was closed an hour ago in another system? The transfer doesn’t instantly fail everywhere. It’s posted as credit/debit, then reconciliation runs later, and you eventually get a reversal. Same with stock markets: Trades happen continuously, but final clearing and settlement occur after the fact. And technically DNS is eventual consistency by design. You update a record, but the world sees it gradually as caches expire. Yet the internet works. Distributed systems aren’t broken when they’re eventually consistent. They’re mirroring how real systems work: commit locally, reconcile globally, compensate when needed.
- awesome_dude 8mo ago
  > Even banking works this way. All database books have the usual “you can’t debit twice, so you need transactions”…bullshit. But think of a money transfer across banks and possibly across countries? Not globally atomic.. Banking is my "go to" anology when it comes to eventual consistency because 1: We use banking almost universally the same ways, and 2: we understand fully the eventual consistency employed (even though we don't think about it) Allow me to elaborate. When I was younger we had "cheque books" which meant that I could write a cheque (or check if you're American) and give it to someone in lieu of cash, they would take the cheque to the bank, and, after a period of time their bank would deposit funds into their account, and my bank would debit funds from mine - that delay is eventual consistency. That /style/ of banking might have gone for some people, but the principle remains the same, this very second my bank account is showing me two "balances", the "current" balance and the "available" balance. Those two numbers are not equal, but they will /eventually/ be consistent. The reason that they are not consistent is because I have used my debit card, which is really a credit arrangement that my bank has negotiated with Visa or Mastercard, etc. Whereby I have paid for some goods/services with my debit card, Visa has guaranteed the merchant that they will be paid (with some exceptions) and Visa have placed a hold on the balance of my account for the amount. At some point - it might be overnight, it might be in a few days, there will be a reconciliation where actual money will be paid by my bank to Visa to settle the account, and Visa will pay the merchant's bank some money to settle the debt. Once that reconciliation takes place to everyone's satisfaction, my account balances will be consistent.
  - da_chicken 8mo ago
    No, this is confusing how the financial institutions operate as a business with how the data store that backs those institutions operates as a technology. You can certainly operate your financial system with a double entry register and delayed reconciliation due to the use of credit and the nature of various forms of promissory notes, but you're going to want the data store behind the scenes to be fully consistent with recording those transactions regardless of how long they might take to reconcile. If you don't know that your register is consistent, what are you even reconciling against? What you're arguing is akin to arguing that because computers store data in volatile RAM and that data will often differ from what is on disk, that you shouldn't have to worry about file system consistency or the integrity of private address spaces. After all, they aren't going to match anyways.

anonymars 8mo ago

Remember also that "partition" is not "yes or no" but rather a latency threshold. If the network is connected but a call now takes 30 seconds instead of milliseconds, that is probably a partition

convolvatron 8mo ago
this is likely wrong. the issue with partitions is that we can no longer communicate at all, thus we can't end up in the same state. If we have poor performance, thats certainly something that worth putting machinery in to adapt to, but its not at all in the same class as 'I can't talk to you and I dont know what you're doing at all' fro a correctness standpoint edit: yeah ok, since failure detection is being driven by timers by necessity, then sure. the tradeoff we're making between the interval under which we're unable to make progress vs the upheaval caused by announcing a failure.
- anonymars 8mo ago
  Yeah, I glossed over a few steps. There's likely a latency threshold beyond which you should abort, and then it is a partition (after all, that's what TCP is doing under the hood if it sends a packet and doesn't get a response). One should be so lucky to have an operation fail immediately, rather than lumber on until it times out (holding resources hostage all the while)!

majormajor 8mo ago

> And he is making a mistake in claiming the partitions are "exceedingly rare". Again he is not thinking about a global distributed cloud across continents. Any time an AWS region or AZ goes down we see a lot of popular services go nearly-completely-down. And it's generally fine. One thing I appreciate about AWS is that (operating "live" in just a single AZ or even single region) I've seen far fewer partition-causing networking hiccups than when my coworkers and I were responsible for wiring and tuning our own networks for our own hardware in datacenters.