4 ms·

Curious if there is a sharding scheme that can scale indefinitely for both the total size of data and total number of data objects? S3 apparently can do it, but

Curious if there is a sharding scheme that can scale indefinitely for both the total size of data and total number of data objects? S3 apparently can do it, but I’ve never seen any publication talking about it. I’m aware of popular techniques, but they all have practical limits. For instance, consistency hashing seems good, but in reality too big a cluster will lead to increasingly severe membership flapping. Placement lookup seems flexible and flexible, but will eventually lead to the challenge of scaling up the placement driver for its too large index or metadata.

deleted 2y ago
[deleted]
oceanplexian 2y ago
I worked at LiveJournal and we were sharding there since the late 90s. We ended up with 50 +/- Database pairs and 1 Global Master DB pair serving millions of users in the US (They all started on MySQL and my team moved them over to Percona) We would cut the shards manually when a database would fill up with too many users. We only had to do it once or twice a year, since the only data on the global master was a reference to the user id and the database where your data was stored, it scaled well enough to run the #1 or #2 largest social media site in the US. Even with early 00s tech you could store more rows in a DB than the population of the USA. When indexed the lookup can be done in milliseconds. If a user database got too hot we would move half the users off it and cut a new user DB, and update their references. Held together with a lot of Perl scripts and duct tape, a far simpler time.
jerf 2y ago
Total separation between individual smaller DBs scales indefinitely. If you can define domains that need no real-time cross-referencing at all, then this works. This isn't a snarky answer because that defines a fairly specific situation, and has its own specific pitfalls, but it is a viable answer. For instance, there's a lot of situations out in the world where you don't really need all your customer data piled into one database. You can easily shard out one database per customer. But a characteristic pitfall of that approach is the generally pathological distribution of customer size. You'll often end up with That One Customer that is your biggest customer and therefore blows out your database size in their own shard. While it's nice that you can at least isolate the rest of your customers from the blast radius, a system that you wrote from top to bottom otherwise to be in a single database per customer can be very, very difficult to fix for That One Customer... and guess what, they're also your biggest customer so management is going to come down on you like a sack of bricks to keep them running. This is a lot more viable in 2024 than it was in 2004. You can scale that single database up to a lot more power nowadays, between all the CPU improvements and those glorious, glorious SSDs. But as is the way of things in engineering, where every benefit comes with corresponding risks and costs, that means that if you do hit the DB barrier with That One Customer, you'll hit it all the harder, because now you're entrenched with a 64-core DB on terabytes of RAM, and all the corresponding code complexity behind it, that you're going to try to redesign on the fly to work with within-customer sharding, which involves a lot more code than when someone blew out DBs in 2004. This comes with its own issues when you have to update the DB schema too, although, so does the One Big Database.
- hintymad 2y ago
  Would this help designing an object store, though? Case in point, S3 has unlimited bucket size, and they achieved that adding new servers will always increase the performance and reliability of the S3 cluster (they call it taking the advantage of the scale). I'd imagine that the key to achieve such is the sharding strategy, or the strategy of data placement in general. In contrast, MinIO makes it O(1) operation to find the node to write, but it has to ask all the participating nodes, at least a subset of them, to figure out which node to read from for a given object. To that end, MinIO's salability will hit a wall at some point.

deleted 2y ago

[deleted]

oceanplexian 2y ago

I worked at LiveJournal and we were sharding there since the late 90s. We ended up with 50 +/- Database pairs and 1 Global Master DB pair serving millions of users in the US (They all started on MySQL and my team moved them over to Percona) We would cut the shards manually when a database would fill up with too many users. We only had to do it once or twice a year, since the only data on the global master was a reference to the user id and the database where your data was stored, it scaled well enough to run the #1 or #2 largest social media site in the US. Even with early 00s tech you could store more rows in a DB than the population of the USA. When indexed the lookup can be done in milliseconds. If a user database got too hot we would move half the users off it and cut a new user DB, and update their references. Held together with a lot of Perl scripts and duct tape, a far simpler time.

jerf 2y ago

Total separation between individual smaller DBs scales indefinitely. If you can define domains that need no real-time cross-referencing at all, then this works. This isn't a snarky answer because that defines a fairly specific situation, and has its own specific pitfalls, but it is a viable answer. For instance, there's a lot of situations out in the world where you don't really need all your customer data piled into one database. You can easily shard out one database per customer. But a characteristic pitfall of that approach is the generally pathological distribution of customer size. You'll often end up with That One Customer that is your biggest customer and therefore blows out your database size in their own shard. While it's nice that you can at least isolate the rest of your customers from the blast radius, a system that you wrote from top to bottom otherwise to be in a single database per customer can be very, very difficult to fix for That One Customer... and guess what, they're also your biggest customer so management is going to come down on you like a sack of bricks to keep them running. This is a lot more viable in 2024 than it was in 2004. You can scale that single database up to a lot more power nowadays, between all the CPU improvements and those glorious, glorious SSDs. But as is the way of things in engineering, where every benefit comes with corresponding risks and costs, that means that if you do hit the DB barrier with That One Customer, you'll hit it all the harder, because now you're entrenched with a 64-core DB on terabytes of RAM, and all the corresponding code complexity behind it, that you're going to try to redesign on the fly to work with within-customer sharding, which involves a lot more code than when someone blew out DBs in 2004. This comes with its own issues when you have to update the DB schema too, although, so does the One Big Database.

hintymad 2y ago
Would this help designing an object store, though? Case in point, S3 has unlimited bucket size, and they achieved that adding new servers will always increase the performance and reliability of the S3 cluster (they call it taking the advantage of the scale). I'd imagine that the key to achieve such is the sharding strategy, or the strategy of data placement in general. In contrast, MinIO makes it O(1) operation to find the node to write, but it has to ask all the participating nodes, at least a subset of them, to figure out which node to read from for a given object. To that end, MinIO's salability will hit a wall at some point.