3 ms·
Google will put forbidden pages in its index. It doesn't scrape them. (The URL to the page exists even without visiting the page.)
by tedunangst 10y ago
Google will put forbidden pages in its index. It doesn't scrape them. (The URL to the page exists even without visiting the page.)