4 ms·
Strange advice. Not sure if I understood what they mean. One important use case to exclude sections of your website is to not pollute the sitemap which Google
by greenspot 10y ago
Strange advice. Not sure if I understood what they mean.
One important use case to exclude sections of your website is to not pollute the sitemap which Google crawls or to be more precise--the daily crawl volume Google allocates to your site. If you let every page be crawled more important pages get crawled less. Example: In the past, you created a content category which didn't turn out successful. Before you remove this category with plenty of links which would result in crawl errors it would be smarter to exclude them in the ROBOTS file and focus on your core categories.