What antirez is getting at there is actually a much harder (and more interesting) problem that could be generalized as something like "efficient data locality f

What antirez is getting at there is actually a much harder (and more interesting) problem that could be generalized as something like "efficient data locality for mixed latency access". However, assuming that all data must actually be in memory (as stated in the posted email), you don't actually solve the Mongo problem with more efficient organization of the data set, though compacting could be considered a sub-problem of the one that antirez describes. But as I noted in my earlier comment, compacting wouldn't have actually solved their problems, it just would have delayed them. It's reasonable to ask if all of their data truly needs to be hot, but even there, you'd eventually hit diminishing returns as you approached the threshold where your active set couldn't fit in memory, and there smarter data organization wouldn't actually fix things once you started pulling chunks out for sharding; you'd still need to recompact.