3 ms·
I agree that crawlers can't solve the problem, the trick is trying to avoid contributing to the problem. Take another well known example, DNS reflection attacks
by ChuckMcM 13y ago
I agree that crawlers can't solve the problem, the trick is trying to avoid contributing to the problem. Take another well known example, DNS reflection attacks. Clearly DNS can't solve the DDOS "problem" but it can avoid helping out by not recursively resolving queries for untrusted clients.
So any time the crawler can be a bit more aware about what it is heading into, the saves it time not looking at pages which we know apriori we would not be indexing anyway. And that (which pages should be indexed) can only be answered from the set of pages that are crawled.