3 ms·
How about just using consistent hashing? http://en.wikipedia.org/wiki/Consistent_hashing http://en.wikipedia.org/wiki/Consistent_hashing In this scheme, you c
by chubot 16y ago
How about just using consistent hashing? http://en.wikipedia.org/wiki/Consistent_hashing http://en.wikipedia.org/wiki/Consistent_hashing
In this scheme, you can add and remove servers without mving all your data.
There is apparently a memcached plugin to do this: http://www.mikeperham.com/2009/01/14/consistent-hashing-in-memcache-client/ http://www.mikeperham.com/2009/01/14/consistent-hashing-in-m...
- antirez 16y agoThe idea of presharding is exactly to avoid using consistent hashing. With memcached consistent hashing works as you are only interested in relocating the smallest amount of keys when adding nodes, you don't need to rehash, as memcached is a cache: eventually the new values will be created at the right position. With Redis the idea does not work well, you need to move things among instances. Instead with presharding this is not required.
- moe 16y agoConsistent hashing is a common baseline, but it's not the hardest problem in this. You still have to move some data when you shuffle nodes and the devil is in the detail there; when/how to move chunks without interrupting access or violating integrity, how to ensure n-copies (if redundancy is required), how to prevent hot-spots. And of course all the aggregation/routing to allow complex queries that span nodes. That's the reason why there are very few out-of-the-box sharding solutions. It's just very tough to get right, and one size won't fit all.