3 ms·
Yes, site operators actually block non-Googlebot crawlers. See the example [0] from https://news.ycombinator.com/item?id=25538842 https://news.ycombinator.com/
by loeg 6y ago
Yes, site operators actually block non-Googlebot crawlers. See the example [0] from https://news.ycombinator.com/item?id=25538842 https://news.ycombinator.com/item?id=25538842 .
Spoofing crawler identity completely defeats the point of the honor-system robots.txt.