2 ms·
What can search engines do about user-agent based content differentiation? Say my robots.txt allows Googlebot and nothing else. If Google attempts to double-che
by oefrha 4y ago
What can search engines do about user-agent based content differentiation? Say my robots.txt allows Googlebot and nothing else. If Google attempts to double-check with a covert user agent, robots.txt is violated. Assign humans to review reported pages? It’s pretty easy to swamp a manual system like that. Just forget about robots.txt?
- foobarbecue 4y agoI remember reading on HN years ago that Google bots have never honored robots.txt, but I don't actually know
- hombre_fatal 4y agorobots.txt is just a guideline between well-meaning actors for the majority of their traffic, like helping a bot not waste its time nor your bandwidth by crawling dynamically-generated, endless-scrolling /calendar.php pages. Google does use it to that extent. It's not a firewall. Seems like you're describing cloaking (https://developers.google.com/search/docs/advanced/guidelines/cloaking https://developers.google.com/search/docs/advanced/guideline...), one of the oldest SEO tricks, and you can imagine that search engines started defeating it on Day 2 of crawling the web.