3 ms·
Thank you for your kind words and that critical insight - "closed solution". Sorry for my poor explanation. When you shard data, you could shard the record id
by samsquire 4y ago
Thank you for your kind words and that critical insight - "closed solution".
Sorry for my poor explanation.
When you shard data, you could shard the record identifier and keep the data on a particular server -OR- you can shard the data itself and mapreduce it.
For example, a file system locally shards the file data into inodes. The sum of all the inodes = the data of the file.
I am trying to split an integer into shards, the sum of the parts is the value.
If account balance is defined as the SUM of all server's amounts, then each server stores a proportion of the data for that user.
If there are 8 servers and a user has £8000 in their bank balance, then you try store £1000 in each server, so any server can serve requests without coordination. If you user wants to spend £25, then that's fine, just deduct the local data. If you the user wants to spend £1001, then you need a globally consistent read since that would drop the balance below 0 for the local replica, and you need to check the other replicas to see if the user has enough.
I think it is related to the ID generation problem. If you have 1000 servers and you need to generate monotonically incrasing ID numbers without clashes between servers, how do you do it without synchronization? Do you split an integer / 1000 and give each server that batch to assign ID numbers out of?
Writing this out has lead me to think that you could run two clusters. One cluster serves sharded data. The other serves a stream of updates from the other servers and keeps a globally consistent value.
With probability, most transactions shall be below the server's proportion and shall be served locally without coordination. Coordination is only necessary rarely - for large purchases.