3 ms·
Everyone who has worked with sharding at scale knows the pain when there's this one terrible key (your biggest customer, hottest key, largest blob) that just ru
by pornel 3y ago
Everyone who has worked with sharding at scale knows the pain when there's this one terrible key (your biggest customer, hottest key, largest blob) that just ruins load-balancing of the shards.
- scottlamb 3y ago100% true, but in fairness work stealing doesn't completely solve that either. You often end up locking the shard anyway because lockless data structures are limited and hard. And if you're doing something like a key/value store, you're probably sharding to decide which server to direct traffic to, and so you have the same problem at that layer. (There are things that can help, of course, e.g. <https://aws.amazon.com/blogs/architecture/shuffle-sharding-massive-and-magical-fault-isolation/ https://aws.amazon.com/blogs/architecture/shuffle-sharding-m...>.)
- IshKebab 3y agoYeah maybe, but how many people are running these high performance async servers on 2 core machines?