On the other hand, most of that is stuff that human curation can rightfully reject as not worth linking and indexing. We don't need to index the entire web, we

On the other hand, most of that is stuff that human curation can rightfully reject as not worth linking and indexing. We don't need to index the entire web, we need to index the good stuff (that is likely to stay there anyway and be worth archiving).