3 ms·
Sharding is a great way to scale, but comes with costs. One cost is that you need a separate transaction layer "on top of" the shards if you want to offer trans
by mjb 6y ago
Sharding is a great way to scale, but comes with costs. One cost is that you need a separate transaction layer "on top of" the shards if you want to offer transactions. If you want to offer strong isolation (such as snapshot or serializability), good performance, and fault tolerance, that transaction layer can be quite complex. Scaling a single RSM is much cleaner conceptually, and (depending on the model) can offer very strong semantics (strict serializability would be typical). Having a single system image (rather than trying to simulate one through distributed transaction protocols) also removes complexity in other areas, such as making it easier to do large read queries MVCC-style.
I don't think that being able to scale an RSM up makes anything possible that wasn't possible before, but it is convenient in all kinds of ways.
- electricshampo1 6y agoI agree with basically everything you said. However, this would only apply in relatively narrow cases where your workload can be supported by a single RSM. There is an absolute limit for the sequencing alone from a single leader, even after you apply the tricks mentioned in the paper. Geo optimizations (US users data stored in US, EU users data stored in EU, etc) for latency while maintaining a single database view also become challenging (impossible?) without sharding. Fault tolerance is also slightly improved with many shards because you expose a smaller fraction of your database to transient downtime (re-election, catchup) or slowness when the node containing the leader fails or becomes slow.