When dealing with very very large datasets, the ability to filter fast is important. So it becomes a 2-stage (actually n-stage...) problem: filter the raw data,

When dealing with very very large datasets, the ability to filter fast is important. So it becomes a 2-stage (actually n-stage...) problem: filter the raw data, and remove the parts that are known irrelevant (alternately include only the 'maybes'), then do the more intensive processing on the smaller resultant data-set. When done in parallel it looks a lot like map-reduce :)