3 ms·
The described problem - imbalance in shard allocation - is not one that's specific to MongoDB. In fact one of the huge choices one has to make when deploying
by rit 16y ago
The described problem - imbalance in shard allocation - is not one that's specific to MongoDB.
In fact one of the huge choices one has to make when deploying say, Cassandra, is the partitioning algorithm. Sequential partitioning (saying say, users a-l go on Partition 1 and m-z go on #2) gives you certain capabilities for range queries but also risks you overloading if one particular user is significantly larger than others.
Cassandra recommends random partitioning as a way of better balancing your data across shards.
You're going to run into the same problem on just about any sharded setup (With any software) - how do you make sure that you are distributing new chunks/documents/rows in a way that doesn't overload any given server.
- tav 16y agoOr your datastore can take care of doing that for you, c.f. BigTable. This is one of my main gripes with most of the current set of NoSQL offerings — they leave too many decisions in the hands of developers. Whilst it would definitely be advantageous for all developers to understand the intricacies of various CPU and OS scheduling algorithms, it's not an issue that most developers have to deal with directly. The App Engine datastore, in particular, proves that it is possible to create NoSQL datastores which don't force developers to think about issues like load balancing.
- mike_esspe 16y agoApp engine had big downtime too, due to problem with database re-balancing (though downtime was not as long, as foursquare's).
- rit 16y agoWell - in many cases they leave the decisions in the hands of developers because they have made a conscious choice to let the developers decide. BigTable is more than anything a filesystem - it is NOT a database. It lacks true user configurable indexes, custom sorting, querying etc. These features and the data storage requirements to support them are a very different prospect from what a filesystem (even a distributed one) needs.