6 ms·

I’ve had this debate with people numerous times and I think the “you don’t need strong consistency” crowd has really gotten it backwards. These same people will

I’ve had this debate with people numerous times and I think the “you don’t need strong consistency” crowd has really gotten it backwards. These same people will also agree with you that premature optimization is bad, but then argue that application developers should be forced to grapple with looser isolation models for vague reasons like “latency” and “costs”. Most applications don’t push their databases that hard, and the ones that do usually have their most expensive queries as large “read only” queries where strict serializability of other transactions adds little additional overhead. It’s already hard enough to write correct software, we should be pushing hard problems like transaction isolation deep into the database so they can be solved correctly once, instead of 1000s of times incorrectly. Most applications that depend on strong consistency in their database could be rewritten to require less strict guarantees, but why? Strict serializability should be the default. Engineers working in performance sensitive settings and who really know what they’re doing can always loosen this constraint in the narrow part of the application where it matters. It’s crazy to me that anyone would ask a developer to justify why they need strict serializability. It should be the opposite, you should be forced to justify why you’re using a looser isolation level for performance reasons just like we force developers to justify why they’re introducing more performant, but less readable code!

liuliu 4y ago
The post has more leg than you give it credit for. The author thinks serializable is table stake, and linearizability can be implemented cheaply (through sharding). Strict-serializable however, cannot really break "scale-out" barrier with some serious compromises (either time-bounds it (Spanner's 7ms bounds), or central ordering service that cannot scale out). It additionally provides a proof that strict-serializable != serializable + linearizable and then go on to question what you responded: does strict-serializable worth all these troubles?
- richieartoul 4y ago
  In my opinion the answer is unequivocally, yes, it is worth it, and developers should accept nothing less from modern databases. Even with strict serializability, developers will still be surprised by transactional behavior in distributed environments, but at least strict serializability provides a relatively straight forward framework for developers to reason about with almost no “ifs, ands, or buts” except for some confusing behavior around idempotency and transaction retries. As soon as you drop below strict serializability, the cognitive load of reasoning about your applications correctness increases dramatically. Most developers just accept this, but it’s not the right trade off for most applications IMO. If this was some theoretical question that we hadn’t figured out if it was practical or possible yet, I would have more sympathy for this line of thinking, but it’s not anymore. Spanner, CockroachDB, and FoundationDB have all proven that these systems can be scaled, and economically so. FoundationDB scales incredibly well with great performance characteristics. That’s evidence enough to me that the cost of strict serializability is worth it for 99% of applications.
  - _benedict 4y ago
    CockroachDB is not strict serializable. It is linearizable+serializable+(some other guarantee I forget). Only Spanner currently offers properly scalable strict serializability, as FoundationDB clusters have a size limit (that is very forgiving, and enough for most use cases). Apache Cassandra is working on providing scalable (without restriction) strict serializable transactions[1], and they should arrive fairly soon. So far as I am aware, at this time it will be the only distributed database besides Spanner offering fully scalable and fast global transactions with this level of isolation. [1] https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-15%3A+General+Purpose+Transactions?preview=/188744725/188744736/Accord.pdf https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-15... (Disclaimer) I'm one of the authors
    - richieartoul 4y ago
      You’re right about CockroachDB, my mistake. I think your characterization of Cassandra as “fully scalable” in a way that other systems are not is misleading, but I won’t argue it with you :p
      - _benedict 4y ago
        I'm not talking about Cassandra being more generally scalable than other systems, only that no system as scalable as Cassandra offers strict serializable transactions besides Spanner (including Cassandra, today)
        richieartoul 4y ago
        What does scalable mean in this context? In general, Cassandra is much more efficient at writes and can handle higher volumes of write throughput assuming identical hardware, but from what I’ve seen FDB is as efficient (if not more so) for read heavy workloads than Cassandra. A small FDB cluster with no tuning running on small VMs will easily saturate each nodes 10gb NICs for read queries without breaking a sweat. The cluster size limitations are also greatly overstated and mostly a hold over from stale documentation from 2013. You can (and companies do) run FDB clusters with 100s or 1000s of nodes without issue. Cassandra supports this too but in my experience it gets quite difficult to operate at that point, although I’m sure recent versions have gotten much better about this. If you only care about raw write throughout I think your statement is mostly accurate, but for general purpose workloads I would argue that FDB is a system “as scalable as Cassandra” that also offers strictly serializable (and interactive!) transactions which makes it much more broadly applicable to a wide variety of use cases.
- mamcx 4y ago
  Is important to take in context the SIZE. A lot of Ink is wrongly interpreted because people not think about the SIZE of the (data/problem). MOST, if not ALL(99%) of the thinking about this stuff is for the very end of the scale: Terabytes, dozen/thousand of machines, insane latency, etc. MOST, if not ALL(99%) of the ACTUAL data people deal with, the one that matters, is in the low of gigabytes, at best. One, maybe 2 servers, answer in a second(is) is more than ok. You can fit most databases in an iPhone with spare to work. Instead, the real trouble that people face is wrong schema/query designs, and introducing bugs and other stuff that the RDBMs thankfully correct/protect against. Making the primary store MORE strict is a big win to alleviate the real issues.
- aidenn0 4y ago
  Is scale-out something most services need though? TFA points out that it's likely only a subset of the data that needs strict-serializable; the flip-side of that is that it's likely only a subset of the data needs scale-out.
jrockway 4y ago
Yeah, I always use strict serializability by default. The reason is because I have no model for reasoning about less strict levels of isolation. For example, with "default" isolation in Postgres, can you transfer money between two bank accounts atomically? The answer, I'm pretty sure, is no! At that point, everything you "know" about databases is out the window. You're basically crossing your fingers and hoping that nothing weird is happening, and if it does, hey it was only one user a day, 99.9% of users are perfectly happy and didn't lose their home and family! That's within our SLA! I think the article is talking about distributed systems where it's costly to guarantee ordering among multiple entity groups (shards for example), and there are "never" transactions across those entity groups. For example, the database hosting your website and the database backing my website can't talk to each other, so it doesn't really matter that transactions that we execute against our database don't appear in a defined order; we don't even know that the other exists. I do agree with the sentiment that if those two databases are at the same company, eventually you'll be burned by not being able to do transactions across them. Ever try to make an atomic change to two git repositories? Not possible, but often quite annoying. This is doubly annoying when you start having internal sync processes because two teams used two different database to ultimately build, what looks like to the user, one app. My philosophy is that if strict serializability ever makes it impossible to run my service, I'm already so rich that I can just sell the service and let someone else deal with the problem. So far, no dice on that.
- sitkack 4y ago
  > Ever try to make an atomic change to two git repositories? I know your intent wasn't to discuss this. But you could 1) combine them 2) use tags or git hashes as your URI 3) create a third repo that uses git submodules to combine them by said git hash Trying to maintain global state by having both gitrepos be compatible at the same wall time isn't as important as it seems. There are lots of ways to meaningfully deal with this.
closeparen 4y ago
The hard part about working at scale is not that you need to use more efficient algorithms or combine computers to handle the load. It is that all the weird and spooky non-reproducible edge case stuff is happening constantly & will actually demand your attention. The lower your scale, the more you can get away with using a "high scalability" database. Who cares how concurrent transactions are reconciled when you've never seen one? I would wager that most users aren't actually handling the caveats at application layer. They're just not hitting the weird stuff often enough to ever really consider it.
jiggawatts 4y ago
I've wasted a lot of brain cycles dreaming up my "ideal" database engine, and as you've said, a key feature would be strict serializability. In my opinion the trick to implementing this without too many performance issues is to add the HTTP cookies feature to database engine protocols. Every transaction with the DB engine should require a client "transaction sequence logical clock" cookie. The returned result set should return the updated cookie, which should then be "threaded" through all the way to a web browser cookie or GUI client local store. That way, individual users will always be telling the database engine (or cluster!) that they need to see their previous transactions in a linear order, but concurrent users could potentially have a slightly difference perspective on things. Cache entries could also be tagged with the clock cookies and then used to verify if they're still valid or need to be refreshed. An important feature would be to have a union operator on the clock cookies that provides a new cookie that is "in the future" of all provided clock cookies. This would allow cross-user queries to correctly reflect the required input transactions. Similarly, it should be possible to use a special "now" cookie, which is identical to typical single-server DB behaviour with serializable transactions. Last but not least, "now minus t seconds" would produce the equivalent to what you get if you do read-only queries against an asynchronous replica. This would allow combinations not possible with a naive DB engine that always enforces a single serialized transaction sequence, but the default behaviour would be very safe and consistent. The computer science for this exists. There are lamport timestamp implementations using compressed tree representations, for example. The timestamps could be tracked per table, which would allow transactions across unrelated tables to be independently serialized. Essentially, transactions should form a directed acyclic graph much like Git commits instead of a strictly linear linked list. There are many analogies: Git commit hashes are a type of cookie that can be passed around and used for various things, you need to provide a predecessor commit hash for new commits, and the hashes are used to keep caches (local repos) 100% consistent with the central transaction history, etc...
- naasking 4y ago
  If you want a git-like DAG you might be better off with using a versioning API directly. Take a look at concurrent revisions: https://www.microsoft.com/en-us/research/project/concurrent-revisions/ https://www.microsoft.com/en-us/research/project/concurrent-...

liuliu 4y ago

The post has more leg than you give it credit for. The author thinks serializable is table stake, and linearizability can be implemented cheaply (through sharding). Strict-serializable however, cannot really break "scale-out" barrier with some serious compromises (either time-bounds it (Spanner's 7ms bounds), or central ordering service that cannot scale out). It additionally provides a proof that strict-serializable != serializable + linearizable and then go on to question what you responded: does strict-serializable worth all these troubles?

richieartoul 4y ago
In my opinion the answer is unequivocally, yes, it is worth it, and developers should accept nothing less from modern databases. Even with strict serializability, developers will still be surprised by transactional behavior in distributed environments, but at least strict serializability provides a relatively straight forward framework for developers to reason about with almost no “ifs, ands, or buts” except for some confusing behavior around idempotency and transaction retries. As soon as you drop below strict serializability, the cognitive load of reasoning about your applications correctness increases dramatically. Most developers just accept this, but it’s not the right trade off for most applications IMO. If this was some theoretical question that we hadn’t figured out if it was practical or possible yet, I would have more sympathy for this line of thinking, but it’s not anymore. Spanner, CockroachDB, and FoundationDB have all proven that these systems can be scaled, and economically so. FoundationDB scales incredibly well with great performance characteristics. That’s evidence enough to me that the cost of strict serializability is worth it for 99% of applications.
- _benedict 4y ago
  CockroachDB is not strict serializable. It is linearizable+serializable+(some other guarantee I forget). Only Spanner currently offers properly scalable strict serializability, as FoundationDB clusters have a size limit (that is very forgiving, and enough for most use cases). Apache Cassandra is working on providing scalable (without restriction) strict serializable transactions[1], and they should arrive fairly soon. So far as I am aware, at this time it will be the only distributed database besides Spanner offering fully scalable and fast global transactions with this level of isolation. [1] https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-15%3A+General+Purpose+Transactions?preview=/188744725/188744736/Accord.pdf https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-15... (Disclaimer) I'm one of the authors
  - richieartoul 4y ago
    You’re right about CockroachDB, my mistake. I think your characterization of Cassandra as “fully scalable” in a way that other systems are not is misleading, but I won’t argue it with you :p
    - _benedict 4y ago
      I'm not talking about Cassandra being more generally scalable than other systems, only that no system as scalable as Cassandra offers strict serializable transactions besides Spanner (including Cassandra, today)
      - richieartoul 4y ago
        What does scalable mean in this context? In general, Cassandra is much more efficient at writes and can handle higher volumes of write throughput assuming identical hardware, but from what I’ve seen FDB is as efficient (if not more so) for read heavy workloads than Cassandra. A small FDB cluster with no tuning running on small VMs will easily saturate each nodes 10gb NICs for read queries without breaking a sweat. The cluster size limitations are also greatly overstated and mostly a hold over from stale documentation from 2013. You can (and companies do) run FDB clusters with 100s or 1000s of nodes without issue. Cassandra supports this too but in my experience it gets quite difficult to operate at that point, although I’m sure recent versions have gotten much better about this. If you only care about raw write throughout I think your statement is mostly accurate, but for general purpose workloads I would argue that FDB is a system “as scalable as Cassandra” that also offers strictly serializable (and interactive!) transactions which makes it much more broadly applicable to a wide variety of use cases.
mamcx 4y ago
Is important to take in context the SIZE. A lot of Ink is wrongly interpreted because people not think about the SIZE of the (data/problem). MOST, if not ALL(99%) of the thinking about this stuff is for the very end of the scale: Terabytes, dozen/thousand of machines, insane latency, etc. MOST, if not ALL(99%) of the ACTUAL data people deal with, the one that matters, is in the low of gigabytes, at best. One, maybe 2 servers, answer in a second(is) is more than ok. You can fit most databases in an iPhone with spare to work. Instead, the real trouble that people face is wrong schema/query designs, and introducing bugs and other stuff that the RDBMs thankfully correct/protect against. Making the primary store MORE strict is a big win to alleviate the real issues.
aidenn0 4y ago
Is scale-out something most services need though? TFA points out that it's likely only a subset of the data that needs strict-serializable; the flip-side of that is that it's likely only a subset of the data needs scale-out.

jrockway 4y ago

Yeah, I always use strict serializability by default. The reason is because I have no model for reasoning about less strict levels of isolation. For example, with "default" isolation in Postgres, can you transfer money between two bank accounts atomically? The answer, I'm pretty sure, is no! At that point, everything you "know" about databases is out the window. You're basically crossing your fingers and hoping that nothing weird is happening, and if it does, hey it was only one user a day, 99.9% of users are perfectly happy and didn't lose their home and family! That's within our SLA! I think the article is talking about distributed systems where it's costly to guarantee ordering among multiple entity groups (shards for example), and there are "never" transactions across those entity groups. For example, the database hosting your website and the database backing my website can't talk to each other, so it doesn't really matter that transactions that we execute against our database don't appear in a defined order; we don't even know that the other exists. I do agree with the sentiment that if those two databases are at the same company, eventually you'll be burned by not being able to do transactions across them. Ever try to make an atomic change to two git repositories? Not possible, but often quite annoying. This is doubly annoying when you start having internal sync processes because two teams used two different database to ultimately build, what looks like to the user, one app. My philosophy is that if strict serializability ever makes it impossible to run my service, I'm already so rich that I can just sell the service and let someone else deal with the problem. So far, no dice on that.

sitkack 4y ago
> Ever try to make an atomic change to two git repositories? I know your intent wasn't to discuss this. But you could 1) combine them 2) use tags or git hashes as your URI 3) create a third repo that uses git submodules to combine them by said git hash Trying to maintain global state by having both gitrepos be compatible at the same wall time isn't as important as it seems. There are lots of ways to meaningfully deal with this.

closeparen 4y ago

The hard part about working at scale is not that you need to use more efficient algorithms or combine computers to handle the load. It is that all the weird and spooky non-reproducible edge case stuff is happening constantly & will actually demand your attention. The lower your scale, the more you can get away with using a "high scalability" database. Who cares how concurrent transactions are reconciled when you've never seen one? I would wager that most users aren't actually handling the caveats at application layer. They're just not hitting the weird stuff often enough to ever really consider it.

jiggawatts 4y ago

I've wasted a lot of brain cycles dreaming up my "ideal" database engine, and as you've said, a key feature would be strict serializability. In my opinion the trick to implementing this without too many performance issues is to add the HTTP cookies feature to database engine protocols. Every transaction with the DB engine should require a client "transaction sequence logical clock" cookie. The returned result set should return the updated cookie, which should then be "threaded" through all the way to a web browser cookie or GUI client local store. That way, individual users will always be telling the database engine (or cluster!) that they need to see their previous transactions in a linear order, but concurrent users could potentially have a slightly difference perspective on things. Cache entries could also be tagged with the clock cookies and then used to verify if they're still valid or need to be refreshed. An important feature would be to have a union operator on the clock cookies that provides a new cookie that is "in the future" of all provided clock cookies. This would allow cross-user queries to correctly reflect the required input transactions. Similarly, it should be possible to use a special "now" cookie, which is identical to typical single-server DB behaviour with serializable transactions. Last but not least, "now minus t seconds" would produce the equivalent to what you get if you do read-only queries against an asynchronous replica. This would allow combinations not possible with a naive DB engine that always enforces a single serialized transaction sequence, but the default behaviour would be very safe and consistent. The computer science for this exists. There are lamport timestamp implementations using compressed tree representations, for example. The timestamps could be tracked per table, which would allow transactions across unrelated tables to be independently serialized. Essentially, transactions should form a directed acyclic graph much like Git commits instead of a strictly linear linked list. There are many analogies: Git commit hashes are a type of cookie that can be passed around and used for various things, you need to provide a predecessor commit hash for new commits, and the hashes are used to keep caches (local repos) 100% consistent with the central transaction history, etc...

naasking 4y ago
If you want a git-like DAG you might be better off with using a versioning API directly. Take a look at concurrent revisions: https://www.microsoft.com/en-us/research/project/concurrent-revisions/ https://www.microsoft.com/en-us/research/project/concurrent-...