3 ms·
I'd try to build a microsharding system. Depending on the amount of data you want to store, go for 256 or 65k virtual shards and allocate a few shards on each D
by andr 18y ago
I'd try to build a microsharding system. Depending on the amount of data you want to store, go for 256 or 65k virtual shards and allocate a few shards on each DB server. A hash of the key will give you its shard and it's easy to keep a shard map in memory.
I'd advise going with BDB or whatever key-value store is fastest those days instead of MySQL. And an application-server layer to take care of the microsharding.
Also, try to put most of the work on application servers (things like joins, sorting, calculations etc), because those are much easier to scale than database nodes.
If possible, use in-memory logs instead of writing to the disk during the 1 hour and write the log to disk after the rush hour is over. Consider using a RAM-only database + UPSes, if your data is not that much.