5 ms·
Presumably you'd want Google to index pages, just not try to discover pages by brute forcing the URL? Is there any way to do that in robots.txt? Someone did me
by ancarda 8y ago
Presumably you'd want Google to index pages, just not try to discover pages by brute forcing the URL?
Is there any way to do that in robots.txt? Someone did mention adjusting the crawl rate in Webmaster Tools -- would the author need to do this for every search engine?
- dymk 8y agoHe can submit a sitemap to Google if he’d like a particular structure crawled. Sounds like his issue is mainly with Google, not other search engines, so it’s not an O(n) problem he’s got to solve.
- greglindahl 8y agoSubmitting a sitemap might cause Google to crawl those pages, but it certainly won't stop google from crawling additional pages like the ones mentioned in the blog post. Lots of sitemaps are incomplete or outdated or completely invalid or whatever.
- dymk 8y agoI feel like you're still misunderstanding just how simple the fix is. If he wants to, he can just disallow the portions of his site that he claims Google is scraping via a robots.txt, and he can also submit a sitemap to the portions that he does want crawled.
- greglindahl 8y agoOK, so let's look at the top example of unwanted crawling from the article: /blog/index.php?/categories/14-Forensics (wanted) /blog/index.php?/categories/14-Forensic (unwanted) /blog/index.php?/categories/14-Forensi (unwanted) /blog/index.php?/categories/14-Forens (unwanted) How do you propose blocking the last 3, without having an enormous robots.txt?
- pas 8y agoI think if you do a Disallow: * and also submit an explicit sitemap, then the sitemap would 'win' for those explicit URLs, no?
- _asummers 8y agoYou can return a `X-Robots-Tag: none` header for the inner URLs. It's a thing I throw into APIs I write, just in case Google manages to find it.