I'm not sure how similar this will be but Twitter have a bit of information on how they indexed "roughly half a trillion documents " when they moved to indexing

I'm not sure how similar this will be but Twitter have a bit of information on how they indexed "roughly half a trillion documents " when they moved to indexing all tweets here: https://blog.twitter.com/2014/building-a-complete-tweet-index https://blog.twitter.com/2014/building-a-complete-tweet-inde... Edit: I don't think Twitter do nearly the same filtering with this and their social graph so maybe not as close as I first thought.