5 ms·
Hash-based partitioning has a big problem that when you change the hash size all of the data moves around. Eventually you'll need to do a lookup-based partitio
by tdicola 11y ago
Hash-based partitioning has a big problem that when you change the hash size all of the data moves around. Eventually you'll need to do a lookup-based partitioning scheme. You also probably want control over where some users live since you don't want two super hot users on the same server.
- TheLoneWolfling 11y agoThere are hash schemes that prevent that, however. Look up consistent hashing. And you could probably have a (small) hashmap that allows overrides for the most used pages.
- twic 11y agoMy thought too. If they have half a million sites, and only 1% are hot enough to need deliberate placement, then they only need to store five thousand special cases, which is a few megabytes of memory, easily stored on each load balancer and loaded at boot. Use hashing for the rest. Ideally consistent hashing, or rendezvous hashing, which i just read about on Wikipedia so must be good: http://en.wikipedia.org/wiki/Rendezvous_hashing http://en.wikipedia.org/wiki/Rendezvous_hashing
- kid0m4n 11y agoWon't CDN come into force for the two super hot users? I would gone with consistent hashing as well....
- BillinghamJ 11y agoYou could make the hash, say, 2 bytes long. That would allow for up to 65536 servers. Say you have 8 servers at present, you'd allocate 8192 sequential keys to each server. If you need more capacity, use 16 servers instead, and move half the files from each server to the corresponding new server. Thus exactly half the data is moved each time the number of servers is doubled or halved.