3 ms·
If the URL is a publicly accessible webserver with no robots.txt telling them to stay off it, I wouldn't be surprised if it gets fed to the crawler.
by angusgr 15y ago
If the URL is a publicly accessible webserver with no robots.txt telling them to stay off it, I wouldn't be surprised if it gets fed to the crawler.
- eli 15y agoGoogle can even index files listed in robots.txt (it just doesn't crawl them)