6 ms·
Submitting a sitemap might cause Google to crawl those pages, but it certainly won't stop google from crawling additional pages like the ones mentioned in the b
by greglindahl 8y ago
Submitting a sitemap might cause Google to crawl those pages, but it certainly won't stop google from crawling additional pages like the ones mentioned in the blog post. Lots of sitemaps are incomplete or outdated or completely invalid or whatever.
- dymk 8y agoI feel like you're still misunderstanding just how simple the fix is. If he wants to, he can just disallow the portions of his site that he claims Google is scraping via a robots.txt, and he can also submit a sitemap to the portions that he does want crawled.
- greglindahl 8y agoOK, so let's look at the top example of unwanted crawling from the article: /blog/index.php?/categories/14-Forensics (wanted) /blog/index.php?/categories/14-Forensic (unwanted) /blog/index.php?/categories/14-Forensi (unwanted) /blog/index.php?/categories/14-Forens (unwanted) How do you propose blocking the last 3, without having an enormous robots.txt?
- pas 8y agoI think if you do a Disallow: * and also submit an explicit sitemap, then the sitemap would 'win' for those explicit URLs, no?