Isn't the point of a crawler to crawl nearly everything? Of course they might weigh some things much heigher (recrawl wikipedia much more frequently, for examp

Isn't the point of a crawler to crawl nearly everything? Of course they might weigh some things much heigher (recrawl wikipedia much more frequently, for example), but you don't know what might be interesting as a search result until you've crawled it!