Interesting writeup. The thing it seems to not address is that a different sharding model might have not worked for foursquare's data access needs. In this http

Interesting writeup. The thing it seems to not address is that a different sharding model might have not worked for foursquare's data access needs. In this http://news.ycombinator.com/item?id=1769982 http://news.ycombinator.com/item?id=1769982 HN post harryh alludes to the fact that using user ID as the shard key was a design decision. Their current design requires that an entire user's checkin history be able to be loaded for calculating badges so sharding by user ID might have been necessary to guarantee not having the performance overhead of multi-server retrievals. From reading the various write ups it seems like the real problems (using the current load all data for badge calculations method) came from getting too close to the max memory and then the issue that foursquare's checkins weren't filling full pages so migrating data didn't actually give nodes 1 & 2 any place to write new data to (in memory) until after compaction could be performed.