4 ms·
OK, so let's look at the top example of unwanted crawling from the article: /blog/index.php?/categories/14-Forensics (wanted) /blog/index.php?/categories/1
by greglindahl 8y ago
OK, so let's look at the top example of unwanted crawling from the article:
/blog/index.php?/categories/14-Forensics (wanted)
/blog/index.php?/categories/14-Forensic (unwanted)
/blog/index.php?/categories/14-Forensi (unwanted)
/blog/index.php?/categories/14-Forens (unwanted)
How do you propose blocking the last 3, without having an enormous robots.txt?
- pas 8y agoI think if you do a Disallow: * and also submit an explicit sitemap, then the sitemap would 'win' for those explicit URLs, no?