It usually boils down to the following simple principles and techniques: A: Probabilistic data structures and algorithms For example you can quickly estimate i

It usually boils down to the following simple principles and techniques: A: Probabilistic data structures and algorithms For example you can quickly estimate if a very large set contains a certain key with bloom filters or estimate cardinality with hyperloglog. B: Divide and conquer Just divide your data to workable pieces and combine partial results at the end. Hadoop is a popular infrastructure for doing this. Other example is doing pieces of work in memory and saving it to disk by chunks and reducing it later to get the final result.