4 ms·
I would assume that any site that was implementing JS-level blocks also has the appropriate robots.txt file in place.
by klenwell 7y ago
I would assume that any site that was implementing JS-level blocks also has the appropriate robots.txt file in place.
- greglindahl 7y agoThat's not true in the actual web, however. The best example is a large number of unimportant sites that send 429 errors for /robots.txt if they think it's a scraper. A 4xx result for robots.txt is considered to mean no robots.txt for most crawlers. So the website is getting the reverse of what it thought it was getting.