2 ms·
I feel like you're still misunderstanding just how simple the fix is. If he wants to, he can just disallow the portions of his site that he claims Google is scr
by dymk 8y ago
I feel like you're still misunderstanding just how simple the fix is. If he wants to, he can just disallow the portions of his site that he claims Google is scraping via a robots.txt, and he can also submit a sitemap to the portions that he does want crawled.
- greglindahl 8y agoOK, so let's look at the top example of unwanted crawling from the article: /blog/index.php?/categories/14-Forensics (wanted) /blog/index.php?/categories/14-Forensic (unwanted) /blog/index.php?/categories/14-Forensi (unwanted) /blog/index.php?/categories/14-Forens (unwanted) How do you propose blocking the last 3, without having an enormous robots.txt?
- pas 8y agoI think if you do a Disallow: * and also submit an explicit sitemap, then the sitemap would 'win' for those explicit URLs, no?