3 ms·

horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas There's one feature that I'd like to see

by tremon 5y ago

horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas There's one feature that I'd like to see in that area: partitioned writable replicas. In the same vein that you can partition the table storage across an index, I'd like it to be possible to assign different writers to different parts of a table/database. Of course, you'd still need a single primary replica to handle the transactions that transcend the configured partitions, but we already have routing engines that can transparently redirect an incoming query to any available replica, so it's partly there. There's probably corner cases lurking that I can't even think of, but in my mind it's the only thing missing from building a truly web-scale (multiple zones, multiple datacenters) ACID-preserving relational database.

taffer 5y ago
This is essentially the idea of VoltDB [1]: All tables are partitioned and each partition is assigned to a single thread. As long as each query only touches a single partition, you get linear horizontal scalability and serializable isolation. The drawback is that if a query touches multiple partitions, only that one query can be executed on the entire cluster, so this architecture only works for very specific OLTP workloads. [1] https://www.voltdb.com/ https://www.voltdb.com/
klodolph 5y ago
> There's one feature that I'd like to see in that area: partitioned writable replicas. This is getting more common these days. Google’s Spanner database does this. The database is divided into “tablets” which are each given their own Paxos state. Spanner moves tablets between server instances to do load balancing. There’s no need for you to do the partitioning manually, although if you run into performance problems, you may need to understand how tablets work. Spanner’s definitely not unique in doing this, but Spanner does have a nice writeup of how it works: https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf https://static.googleusercontent.com/media/research.google.c... (PDF)
Andys 5y ago
Checkout TiDB. It sounds too good to be true, and it is, because transactions must be co-ordinated across multiple nodes on the network. Thus the write throughput can be very high but the write latency is also very high.
derekperkins 5y ago
Vitess does just that, letting you seamlessly scale writes in the same way you would scale reads today, scaling linearly with additional nodes.
Negitivefrags 5y ago
I’ve manually implemented this approach before and I would be a little scared of doing it automatically. When you do it manually you are very aware of queries that need to go to multiple shards and really do whatever it takes to avoid them. We sharded by account id and all an individual users data would be on that shard. Basically the only queries that need to transcend the shard are when you need to enumerate accounts by some non-ID value like an email address. You have to ask all the shards in that case. Things like creating an account are tricky if you want to make sure that each has a unique email address. You need to use two-phase commit across all the databases to do it reliably.
erichocean 5y ago
> I'd like it to be possible to assign different writers to different parts of a table/database If you're happy letting the database do that for you, that's how current CockroachDB works.

by tremon 5y ago

taffer 5y ago

This is essentially the idea of VoltDB [1]: All tables are partitioned and each partition is assigned to a single thread. As long as each query only touches a single partition, you get linear horizontal scalability and serializable isolation. The drawback is that if a query touches multiple partitions, only that one query can be executed on the entire cluster, so this architecture only works for very specific OLTP workloads. [1] https://www.voltdb.com/ https://www.voltdb.com/

klodolph 5y ago

> There's one feature that I'd like to see in that area: partitioned writable replicas. This is getting more common these days. Google’s Spanner database does this. The database is divided into “tablets” which are each given their own Paxos state. Spanner moves tablets between server instances to do load balancing. There’s no need for you to do the partitioning manually, although if you run into performance problems, you may need to understand how tablets work. Spanner’s definitely not unique in doing this, but Spanner does have a nice writeup of how it works: https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf https://static.googleusercontent.com/media/research.google.c... (PDF)

Andys 5y ago

Checkout TiDB. It sounds too good to be true, and it is, because transactions must be co-ordinated across multiple nodes on the network. Thus the write throughput can be very high but the write latency is also very high.

derekperkins 5y ago

Vitess does just that, letting you seamlessly scale writes in the same way you would scale reads today, scaling linearly with additional nodes.

Negitivefrags 5y ago

I’ve manually implemented this approach before and I would be a little scared of doing it automatically. When you do it manually you are very aware of queries that need to go to multiple shards and really do whatever it takes to avoid them. We sharded by account id and all an individual users data would be on that shard. Basically the only queries that need to transcend the shard are when you need to enumerate accounts by some non-ID value like an email address. You have to ask all the shards in that case. Things like creating an account are tricky if you want to make sure that each has a unique email address. You need to use two-phase commit across all the databases to do it reliably.

erichocean 5y ago

> I'd like it to be possible to assign different writers to different parts of a table/database If you're happy letting the database do that for you, that's how current CockroachDB works.