4 ms·
The author should use something like "crawl data" instead of "index". An index is the end result of analyzing crawled web pages. It's a cool idea though becaus
by mmaunder 15y ago
The author should use something like "crawl data" instead of "index". An index is the end result of analyzing crawled web pages.
It's a cool idea though because Yahoo sucks up a ton of my bandwidth and delivers very little in SEO traffic. On most of my sites now I have a Yahoo bot specific Crawl-Delay in robots.txt of 60 seconds, which pretty much bans them.