This seems like a good use case for some sort of spam filtering. Maybe the web is that big, but what percentage of it is data someone will ever care about? I wo

This seems like a good use case for some sort of spam filtering. Maybe the web is that big, but what percentage of it is data someone will ever care about? I wonder if you could make a good-enough search with agressive filtering of pages before they enter the index.