4 ms·

What about sequences? The example shows an auto-incrementing user ID. How’s that possible without contention between all shards? Is the proxy responsible for se

What about sequences? The example shows an auto-incrementing user ID. How’s that possible without contention between all shards? Is the proxy responsible for sequences? What about foreign keys? Do they all have to live on the same shard? How do you do distributed transactions? On cross-shard reads: how do you do sorting? And cross-shard joins? I’d love to be proven wrong, but I suspect the 768 servers look like 1 only on the very surface, and you’ll get wildly different characteristics from cross-shard and single-shard queries. I personally would prefer if they _didn’t_ look like 1 if they can’t behave like 1.

random3 3mo ago
A 767 servers KV store should be enough for everyone
vkazanov 3mo ago
Of course 768 servers NEVER behave as 1. This is physically impossible. Global services using relational dbs typically severely restrict queries that run against the cluster. So no joins, no intervals, no grouping, etc. Transactional queries are usually limited to something like "get a single record, preferably from cache". For many typical web services this can go VERY FAR. Only a handful of global services needs more than a few dozen database servers and a caching cluster. In fact, i have seen major businesses running off a pair of very big postgres instances. Analytical stuff is extracted into dedicated storages optimized for throughput, like Snowflake or Redshift or BigQuery.
- ahk-dev 3mo ago
  This seems like the important distinction: making the infrastructure look like one database to the application is different from making it behave like one unrestricted relational database. At what point does hiding the sharding become counterproductive? I imagine teams still need a fairly deep understanding of shard keys, query routing, and failure modes to avoid accidentally expensive cross-shard operations.
  - vkazanov 3mo ago
    Yes! Distributed systems introduce severe restrictions on what can be reasonably done at scale. Having a single connection string is one thing, being able to do a massive JOIN is another (and should only be ever done in analytical databases). The question is not "when sharding becomes counter-productive" but "when it starts making sense". With sharding something somewhere has to know how to route queries to subsets of data. So it is a complexity price paid for being able to scale. If one can avoid paying this cost then he should. And USUALLY cross-shard queries are not just expensive but simply impossible in operational clusters. Like, if you do COUNT on a table, you only count within a single db shard table.
    - drdexebtjl 3mo ago
      Parent commenter was not asking about when sharding becomes counterproductive, it was asking about when hiding sharding from the application becomes counterproductive. The point is that it’s a leaky abstraction. It should not be at the database proxy level. Instead, the application should be responsible to route queries to the correct shard. That way it can’t make cross-shard queries or cross-shard transactions accidentally, it must do so explicitly.
- samlambert 3mo ago
  > Transactional queries are usually limited to something like "get a single record, preferably from cache". simply wrong
  - chaps 3mo ago
    Care to explain? Not getting a good vibe from you, Mr. CEO.
    - marysol5 3mo ago
      Getting developers to work with the infrastructure we have, and not make everyone's lives substantially harder is near impossible.
  - vkazanov 3mo ago
    Well, operational dbs in practical circumstances are used for queries that take predictable time to complete. "Get a row" "Get ten rows" Etc, etc. All constant, all predictable, all not requiring unpredictable time to run. Also caches nicely. What is "simply wrong" here?
jackb4040 3mo ago
I had this question too, even their "Database Scaling" course doesn't get to it. https://vitess.io/docs/faq/sharding/advanced/ https://vitess.io/docs/faq/sharding/advanced/ Looks like it's not drop-in, you need to heavily modify your data and indexes structurally to make sharding performant. Cross-shard joins are possible but need to be designed for explicitly (to be fair, indexes also need to be designed explicitly in regular SQL). IDK if a drop-in solution is possible or if this is an information-theoretical limitation.
citrin_ru 3mo ago
> The example shows an auto-incrementing user ID. How’s that possible without contention between all shards? Why not divide ID range (63bit?) by the maximum planned number of shards and then set on each shard it's min/max value so ranges will not overlap?
- chiph 3mo ago
  Because if the business grows, then you will need to add servers, and reallocate each existing shard's min/max values. If you left room to grow (divide amongst 2^31 instead of 2^63) you can go a long way before you have to solve that problem. If the business turns out to be wildly successful, you'd probably set up a whole new pool anyway and migrate over (and get a new set of pre-allocated ID ranges). This might be the perfect case to use an UUIDv7 as a key instead of a number.
  - prirun 3mo ago
    Consistent hashing helps solve this. You don't divide into shards based on the id, but on the consistent hash of the id. Then when you have to add shards, it's easier to distribute data to the new shards. If you have 3 shards and expand to 4, only 1/4th of the data from each shard gets moved to the new shard.
javier2 3mo ago
low-traffic auto increment like user creation can be relatively performant by reserving batches up front, then using those in each shard. But the use-case for this is mostly if you do not need, or have very limited use for anything cross-shard.
bddicken 3mo ago
A lot of what you're referring to is dictated by the sharding strategy. Vitess and Neki both let you configure this via the VSchema / data topology (That's what I'm getting at here) https://planetscale.com/blog/making-768-servers-look-like-1#how-does-it-know https://planetscale.com/blog/making-768-servers-look-like-1#... This decision matters a ton. I drilled into this specific point on sharding in one of my articles from awhile back: https://planetscale.com/blog/database-sharding https://planetscale.com/blog/database-sharding This is something you face any time you shard data, no matter the system. Single-shard queries are preferred to cross-shard ones. We want 99% of what we do to be single shard, but occasionally cross-shard work is unavoidable.
- drdexebtjl 3mo ago
  > We want 99% of what we do to be single shard, but occasionally cross-shard work is unavoidable. Why optimize for the 1% and put the logic in a database proxy, then? It’s a leaky abstraction. Require the application to explicitly and loudly make a query for each shard and collate the results.
  - kbrajesh176 3mo ago
    Exactly. sharding is for scaling usecase where queries can be handled within a shard(co-partitioning). If sharding strategy can assign keys from two tables where join is needed you can get achieve same performance. In another usecase where joins are not needed realtime, you can build another system where results are precomputed and cached. I consider sharding as methodology for scaling instead of onefit all solution. Overall it depends on usecase and sound sharding strategy for performance.

random3 3mo ago

A 767 servers KV store should be enough for everyone

vkazanov 3mo ago

Of course 768 servers NEVER behave as 1. This is physically impossible. Global services using relational dbs typically severely restrict queries that run against the cluster. So no joins, no intervals, no grouping, etc. Transactional queries are usually limited to something like "get a single record, preferably from cache". For many typical web services this can go VERY FAR. Only a handful of global services needs more than a few dozen database servers and a caching cluster. In fact, i have seen major businesses running off a pair of very big postgres instances. Analytical stuff is extracted into dedicated storages optimized for throughput, like Snowflake or Redshift or BigQuery.

ahk-dev 3mo ago
This seems like the important distinction: making the infrastructure look like one database to the application is different from making it behave like one unrestricted relational database. At what point does hiding the sharding become counterproductive? I imagine teams still need a fairly deep understanding of shard keys, query routing, and failure modes to avoid accidentally expensive cross-shard operations.
- vkazanov 3mo ago
  Yes! Distributed systems introduce severe restrictions on what can be reasonably done at scale. Having a single connection string is one thing, being able to do a massive JOIN is another (and should only be ever done in analytical databases). The question is not "when sharding becomes counter-productive" but "when it starts making sense". With sharding something somewhere has to know how to route queries to subsets of data. So it is a complexity price paid for being able to scale. If one can avoid paying this cost then he should. And USUALLY cross-shard queries are not just expensive but simply impossible in operational clusters. Like, if you do COUNT on a table, you only count within a single db shard table.
  - drdexebtjl 3mo ago
    Parent commenter was not asking about when sharding becomes counterproductive, it was asking about when hiding sharding from the application becomes counterproductive. The point is that it’s a leaky abstraction. It should not be at the database proxy level. Instead, the application should be responsible to route queries to the correct shard. That way it can’t make cross-shard queries or cross-shard transactions accidentally, it must do so explicitly.
samlambert 3mo ago
> Transactional queries are usually limited to something like "get a single record, preferably from cache". simply wrong
- chaps 3mo ago
  Care to explain? Not getting a good vibe from you, Mr. CEO.
  - marysol5 3mo ago
    Getting developers to work with the infrastructure we have, and not make everyone's lives substantially harder is near impossible.
- vkazanov 3mo ago
  Well, operational dbs in practical circumstances are used for queries that take predictable time to complete. "Get a row" "Get ten rows" Etc, etc. All constant, all predictable, all not requiring unpredictable time to run. Also caches nicely. What is "simply wrong" here?

jackb4040 3mo ago

I had this question too, even their "Database Scaling" course doesn't get to it. https://vitess.io/docs/faq/sharding/advanced/ https://vitess.io/docs/faq/sharding/advanced/ Looks like it's not drop-in, you need to heavily modify your data and indexes structurally to make sharding performant. Cross-shard joins are possible but need to be designed for explicitly (to be fair, indexes also need to be designed explicitly in regular SQL). IDK if a drop-in solution is possible or if this is an information-theoretical limitation.

citrin_ru 3mo ago

> The example shows an auto-incrementing user ID. How’s that possible without contention between all shards? Why not divide ID range (63bit?) by the maximum planned number of shards and then set on each shard it's min/max value so ranges will not overlap?

chiph 3mo ago
Because if the business grows, then you will need to add servers, and reallocate each existing shard's min/max values. If you left room to grow (divide amongst 2^31 instead of 2^63) you can go a long way before you have to solve that problem. If the business turns out to be wildly successful, you'd probably set up a whole new pool anyway and migrate over (and get a new set of pre-allocated ID ranges). This might be the perfect case to use an UUIDv7 as a key instead of a number.
- prirun 3mo ago
  Consistent hashing helps solve this. You don't divide into shards based on the id, but on the consistent hash of the id. Then when you have to add shards, it's easier to distribute data to the new shards. If you have 3 shards and expand to 4, only 1/4th of the data from each shard gets moved to the new shard.

javier2 3mo ago

low-traffic auto increment like user creation can be relatively performant by reserving batches up front, then using those in each shard. But the use-case for this is mostly if you do not need, or have very limited use for anything cross-shard.

bddicken 3mo ago

A lot of what you're referring to is dictated by the sharding strategy. Vitess and Neki both let you configure this via the VSchema / data topology (That's what I'm getting at here) https://planetscale.com/blog/making-768-servers-look-like-1#how-does-it-know https://planetscale.com/blog/making-768-servers-look-like-1#... This decision matters a ton. I drilled into this specific point on sharding in one of my articles from awhile back: https://planetscale.com/blog/database-sharding https://planetscale.com/blog/database-sharding This is something you face any time you shard data, no matter the system. Single-shard queries are preferred to cross-shard ones. We want 99% of what we do to be single shard, but occasionally cross-shard work is unavoidable.

drdexebtjl 3mo ago
> We want 99% of what we do to be single shard, but occasionally cross-shard work is unavoidable. Why optimize for the 1% and put the logic in a database proxy, then? It’s a leaky abstraction. Require the application to explicitly and loudly make a query for each shard and collate the results.
- kbrajesh176 3mo ago
  Exactly. sharding is for scaling usecase where queries can be handled within a shard(co-partitioning). If sharding strategy can assign keys from two tables where join is needed you can get achieve same performance. In another usecase where joins are not needed realtime, you can build another system where results are precomputed and cached. I consider sharding as methodology for scaling instead of onefit all solution. Overall it depends on usecase and sound sharding strategy for performance.