3 ms·
If that page doesn't exist, then according to specification, they don't and you can crawl any page.
by jdrock 17y ago
If that page doesn't exist, then according to specification, they don't and you can crawl any page.
- twoz 17y agoUpdate: Some Yahoo! subdomains have 'em. Yes: http://search.yahoo.com/robots.txt http://groups.yahoo.com/robots.txt http://realestate.yahoo.com/robots.txt No: http://maps.yahoo.com/robots.txt http://omg.yahoo.com/robots.txt