5 ms·
Sharding based on customer is an attractive way to get to shards, potentially with minimal interference in ops or data model, but truly sucks when your biggest
by wiredfool 6y ago
Sharding based on customer is an attractive way to get to shards, potentially with minimal interference in ops or data model, but truly sucks when your biggest customers don’t fit on one shard anymore.
- kamikaz1k 6y agoHas that ever happened to you? Would love to read about that story and hopefully the solution...
- juancampa 6y agoThe article addresses this particular point > Each keyspace is a logical collection of data that roughly scales by the same factor — number of users, teams, and channels. Say goodbye to only sharding by team, and to team hot-spots!
- sougou 6y agoThe way to reason about this is: If a customer had a huge business, what key would they shard by? Then choose that as the sharding key for all customers.
- philsnow 6y agosougou is one of the original vitess devs, he would know this very well. Hi Sugu!
- atombender 6y agoThe solution is to have different shard sizes. Your largest customer needs to fit on the largest shard size (extra large VMs or whatever).
- wiredfool 6y agoFins, but there comes a time when your largest customer doesn’t fit on your largest machine. I wouldn’t say that killed my first startup, but it certainly didn’t help.
- atombender 6y agoI guess I was assuming a shard was more than one machine. If a customer can't be horizontally scaled across nodes, then of course there's an upper limit on how large a shard can be. Database like CockroachDB and TiDB help here.
- continuations 6y agoHow big is your largest customer though? You can easily buy a machine with several TB of RAM, several hundreds of TB of SSDs in RAID giving you millions of IOPS, quad-socket 256 cores. How likely is it that a single machine cannot handle a single customer?
- jjeaff 6y agoYa, this is always what I think about when I see people spending lots of effort on "scalability". You either need an enormous client base or are doing something very specialized that requires tons of memory or cpu per customer. I would think 99% of all internet companies could likely handle everything on a single massive machine, or two, for redundancy. Assuming they were somewhat optimized.
- derekperkins 6y agoThat introduces different problems. Applying MySQL replication logs is single threaded by default and generally doesn't scale as well as the master, so it becomes increasingly difficult to keep replicas from lagging. As your database gets into tens or hundreds of TBs, now backups and restores take hours or days to complete. Any DR scenario becomes an existential threat. For large tables, schema changes can take weeks. This often results in developers doing their own custom table sharding strategies, even though they might still be on a single machine. I'm not saying that you can't architect around these problems, but operating a db at massive scale on a single machine isn't the obvious win that it seems to be.
- chevman 6y agoYeah, seems like the key point here is be careful architecting your infrastructure around a somewhat arbitrary construct like 'customer'. The relevant entity to scale around for Slack is likely 'active users' or even 'messages'. Which may map loosely to 'customers', but that's almost incidental.