3 ms·

Isn't sharding orthogonal to this? It seems completely possible to have a system where the individual data partitions are asynchronously replicated, but the met

Isn't sharding orthogonal to this? It seems completely possible to have a system where the individual data partitions are asynchronously replicated, but the metadata of leader leases and locations for all the partitions are listed on an separate strongly-consistent fault-tolerant source. I think Vitess does something like that, and certainly Bigtable [1] was an early implementation using Chubby to point to the "root" tablet. [1] https://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf https://static.googleusercontent.com/media/research.google.c...

gregwebs 8y ago
Sharding is generally used to refer to a particular type of partitioning: physically distributing data on different machines with the purpose of scaling horizontally (to increase write capacity). What many NewSQL databases use is a logical partitioning scheme where data is partitioned into <= 1 GB chunks. This makes it easy to both scale horizontally and achieve fault tolerance (and change these characteristics without interrupting service). In some systems that component may be coupled and hidden away. In TiDB it is a separately deployed component called the Placement Driver (which embeds etcd, a Chubby equivalent): https://pingcap.com/blog/2016-11-09-Deep-Dive-into-TiKV/#placement-driver https://pingcap.com/blog/2016-11-09-Deep-Dive-into-TiKV/#pla... With sharding you do still need a system to locate your data. A big difference is that most sharding systems are not designed to distribute transactions (or possibly even joins) across shards. So Vitess has a special 2PC for cross-shard transactions with poorer performance than a single shard transaction.