6 ms·

The post has more leg than you give it credit for. The author thinks serializable is table stake, and linearizability can be implemented cheaply (through shardi

by liuliu 4y ago

The post has more leg than you give it credit for. The author thinks serializable is table stake, and linearizability can be implemented cheaply (through sharding). Strict-serializable however, cannot really break "scale-out" barrier with some serious compromises (either time-bounds it (Spanner's 7ms bounds), or central ordering service that cannot scale out). It additionally provides a proof that strict-serializable != serializable + linearizable and then go on to question what you responded: does strict-serializable worth all these troubles?

richieartoul 4y ago
In my opinion the answer is unequivocally, yes, it is worth it, and developers should accept nothing less from modern databases. Even with strict serializability, developers will still be surprised by transactional behavior in distributed environments, but at least strict serializability provides a relatively straight forward framework for developers to reason about with almost no “ifs, ands, or buts” except for some confusing behavior around idempotency and transaction retries. As soon as you drop below strict serializability, the cognitive load of reasoning about your applications correctness increases dramatically. Most developers just accept this, but it’s not the right trade off for most applications IMO. If this was some theoretical question that we hadn’t figured out if it was practical or possible yet, I would have more sympathy for this line of thinking, but it’s not anymore. Spanner, CockroachDB, and FoundationDB have all proven that these systems can be scaled, and economically so. FoundationDB scales incredibly well with great performance characteristics. That’s evidence enough to me that the cost of strict serializability is worth it for 99% of applications.
- _benedict 4y ago
  CockroachDB is not strict serializable. It is linearizable+serializable+(some other guarantee I forget). Only Spanner currently offers properly scalable strict serializability, as FoundationDB clusters have a size limit (that is very forgiving, and enough for most use cases). Apache Cassandra is working on providing scalable (without restriction) strict serializable transactions[1], and they should arrive fairly soon. So far as I am aware, at this time it will be the only distributed database besides Spanner offering fully scalable and fast global transactions with this level of isolation. [1] https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-15%3A+General+Purpose+Transactions?preview=/188744725/188744736/Accord.pdf https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-15... (Disclaimer) I'm one of the authors
  - richieartoul 4y ago
    You’re right about CockroachDB, my mistake. I think your characterization of Cassandra as “fully scalable” in a way that other systems are not is misleading, but I won’t argue it with you :p
    - _benedict 4y ago
      I'm not talking about Cassandra being more generally scalable than other systems, only that no system as scalable as Cassandra offers strict serializable transactions besides Spanner (including Cassandra, today)
      - richieartoul 4y ago
        What does scalable mean in this context? In general, Cassandra is much more efficient at writes and can handle higher volumes of write throughput assuming identical hardware, but from what I’ve seen FDB is as efficient (if not more so) for read heavy workloads than Cassandra. A small FDB cluster with no tuning running on small VMs will easily saturate each nodes 10gb NICs for read queries without breaking a sweat. The cluster size limitations are also greatly overstated and mostly a hold over from stale documentation from 2013. You can (and companies do) run FDB clusters with 100s or 1000s of nodes without issue. Cassandra supports this too but in my experience it gets quite difficult to operate at that point, although I’m sure recent versions have gotten much better about this. If you only care about raw write throughout I think your statement is mostly accurate, but for general purpose workloads I would argue that FDB is a system “as scalable as Cassandra” that also offers strictly serializable (and interactive!) transactions which makes it much more broadly applicable to a wide variety of use cases.
  - bitbckt 4y ago
    FaunaDB also offers scalable strict serializability. Disclosure: I work on FaunaDB.
    - _benedict 4y ago
      I believe Fauna inherits some scalability issues from Calvin, in that given NxN communication is necessary for transactions to be processed, a catastrophic failure in one shard can bring the entire database offline. Is that correct? It may not happen in practice today, but as your clusters grow your exposure to such a failure is increased, and the NxN communication overhead grows does it not? It's certainly more scalable than other systems offering this isolation today (besides Spanner), but algorithmically at least I believe the proposal we are developing for Cassandra is strictly more scalable, as the cost and failure exposure grow only with the transaction scope, not the cluster. Not to ding FaunaDB, though. It probably is the most scalable database offering this level of isolation that is deployable on your own hardware - assuming it is? I know it is primarily a managed service, like Spanner. I'm also not aware how large any real world clusters have gotten with FaunaDB in practice as yet. Do you have any data on that?
mamcx 4y ago
Is important to take in context the SIZE. A lot of Ink is wrongly interpreted because people not think about the SIZE of the (data/problem). MOST, if not ALL(99%) of the thinking about this stuff is for the very end of the scale: Terabytes, dozen/thousand of machines, insane latency, etc. MOST, if not ALL(99%) of the ACTUAL data people deal with, the one that matters, is in the low of gigabytes, at best. One, maybe 2 servers, answer in a second(is) is more than ok. You can fit most databases in an iPhone with spare to work. Instead, the real trouble that people face is wrong schema/query designs, and introducing bugs and other stuff that the RDBMs thankfully correct/protect against. Making the primary store MORE strict is a big win to alleviate the real issues.
aidenn0 4y ago
Is scale-out something most services need though? TFA points out that it's likely only a subset of the data that needs strict-serializable; the flip-side of that is that it's likely only a subset of the data needs scale-out.

by liuliu 4y ago

richieartoul 4y ago

In my opinion the answer is unequivocally, yes, it is worth it, and developers should accept nothing less from modern databases. Even with strict serializability, developers will still be surprised by transactional behavior in distributed environments, but at least strict serializability provides a relatively straight forward framework for developers to reason about with almost no “ifs, ands, or buts” except for some confusing behavior around idempotency and transaction retries. As soon as you drop below strict serializability, the cognitive load of reasoning about your applications correctness increases dramatically. Most developers just accept this, but it’s not the right trade off for most applications IMO. If this was some theoretical question that we hadn’t figured out if it was practical or possible yet, I would have more sympathy for this line of thinking, but it’s not anymore. Spanner, CockroachDB, and FoundationDB have all proven that these systems can be scaled, and economically so. FoundationDB scales incredibly well with great performance characteristics. That’s evidence enough to me that the cost of strict serializability is worth it for 99% of applications.

_benedict 4y ago
CockroachDB is not strict serializable. It is linearizable+serializable+(some other guarantee I forget). Only Spanner currently offers properly scalable strict serializability, as FoundationDB clusters have a size limit (that is very forgiving, and enough for most use cases). Apache Cassandra is working on providing scalable (without restriction) strict serializable transactions[1], and they should arrive fairly soon. So far as I am aware, at this time it will be the only distributed database besides Spanner offering fully scalable and fast global transactions with this level of isolation. [1] https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-15%3A+General+Purpose+Transactions?preview=/188744725/188744736/Accord.pdf https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-15... (Disclaimer) I'm one of the authors
- richieartoul 4y ago
  You’re right about CockroachDB, my mistake. I think your characterization of Cassandra as “fully scalable” in a way that other systems are not is misleading, but I won’t argue it with you :p
  - _benedict 4y ago
    I'm not talking about Cassandra being more generally scalable than other systems, only that no system as scalable as Cassandra offers strict serializable transactions besides Spanner (including Cassandra, today)
    - richieartoul 4y ago
      What does scalable mean in this context? In general, Cassandra is much more efficient at writes and can handle higher volumes of write throughput assuming identical hardware, but from what I’ve seen FDB is as efficient (if not more so) for read heavy workloads than Cassandra. A small FDB cluster with no tuning running on small VMs will easily saturate each nodes 10gb NICs for read queries without breaking a sweat. The cluster size limitations are also greatly overstated and mostly a hold over from stale documentation from 2013. You can (and companies do) run FDB clusters with 100s or 1000s of nodes without issue. Cassandra supports this too but in my experience it gets quite difficult to operate at that point, although I’m sure recent versions have gotten much better about this. If you only care about raw write throughout I think your statement is mostly accurate, but for general purpose workloads I would argue that FDB is a system “as scalable as Cassandra” that also offers strictly serializable (and interactive!) transactions which makes it much more broadly applicable to a wide variety of use cases.
- bitbckt 4y ago
  FaunaDB also offers scalable strict serializability. Disclosure: I work on FaunaDB.
  - _benedict 4y ago
    I believe Fauna inherits some scalability issues from Calvin, in that given NxN communication is necessary for transactions to be processed, a catastrophic failure in one shard can bring the entire database offline. Is that correct? It may not happen in practice today, but as your clusters grow your exposure to such a failure is increased, and the NxN communication overhead grows does it not? It's certainly more scalable than other systems offering this isolation today (besides Spanner), but algorithmically at least I believe the proposal we are developing for Cassandra is strictly more scalable, as the cost and failure exposure grow only with the transaction scope, not the cluster. Not to ding FaunaDB, though. It probably is the most scalable database offering this level of isolation that is deployable on your own hardware - assuming it is? I know it is primarily a managed service, like Spanner. I'm also not aware how large any real world clusters have gotten with FaunaDB in practice as yet. Do you have any data on that?

mamcx 4y ago

Is important to take in context the SIZE. A lot of Ink is wrongly interpreted because people not think about the SIZE of the (data/problem). MOST, if not ALL(99%) of the thinking about this stuff is for the very end of the scale: Terabytes, dozen/thousand of machines, insane latency, etc. MOST, if not ALL(99%) of the ACTUAL data people deal with, the one that matters, is in the low of gigabytes, at best. One, maybe 2 servers, answer in a second(is) is more than ok. You can fit most databases in an iPhone with spare to work. Instead, the real trouble that people face is wrong schema/query designs, and introducing bugs and other stuff that the RDBMs thankfully correct/protect against. Making the primary store MORE strict is a big win to alleviate the real issues.

aidenn0 4y ago

Is scale-out something most services need though? TFA points out that it's likely only a subset of the data that needs strict-serializable; the flip-side of that is that it's likely only a subset of the data needs scale-out.