3 ms·
Consistent hashing is a common baseline, but it's not the hardest problem in this. You still have to move some data when you shuffle nodes and the devil is in
by moe 16y ago
Consistent hashing is a common baseline, but it's not the hardest problem in this.
You still have to move some data when you shuffle nodes and the devil is in the detail there; when/how to move chunks without interrupting access or violating integrity, how to ensure n-copies (if redundancy is required), how to prevent hot-spots. And of course all the aggregation/routing to allow complex queries that span nodes.
That's the reason why there are very few out-of-the-box sharding solutions. It's just very tough to get right, and one size won't fit all.