3 ms·
Google will still index the url even if you block them from crawling the page via robots.txt. They will index the lane, and it can still rank well. Google just
by bhartzer 5y ago
Google will still index the url even if you block them from crawling the page via robots.txt. They will index the lane, and it can still rank well. Google just puts up a message in the results saying they’re not allowed to crawl the page.