4 ms·
They are doing a mistake .... Never ever ever use robots.txt to prevent indexing. Robots.txt is only for preventing crawling. Rather detect bots from the user
by JimWestergren 14y ago
They are doing a mistake ....
Never ever ever use robots.txt to prevent indexing.
Robots.txt is only for preventing crawling. Rather detect bots from the user agent and throw a 404 header with a die and include a meta noindex as well just in case. Facebook recently got millions of secret URLS indexed in Google with emails in them - and yes crawling was blocked with robots.txt.
- nulluk 14y agoI wouldn't recommend detecting bots, Google will see it as clocking and badly penalise you for it as it goes against there guidelines: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=66355 http://support.google.com/webmasters/bin/answer.py?hl=en&... Returning a noindex meta or header should be enough for the honest crawlers, if your worried about dishonest crawlers then your fighting a loosing battle and have a different problem all together.
- JimWestergren 14y agoYes meta noindex is the standard way. But what if you don't want to waste server resources in bots crawling thousands of meta noindex pages? Perhaps you are using some heavy SQL queries on those pages. You can block crawling with robots.txt but then Google won't see the noindex and URLs will be indexed. If you block and send a 404 to bots I think that's fine. They will see a blank page - nothing to gain from that in ranking. So cloaking, perhaps yes, but I don't think it would be risky.
- neurotech1 14y agoYou could return an error 403 to the bot, if detected. Google may penalize that page in the rankings, but not the whole site. If you display spammy keyword loaded page to the bot, then Google may ban/penalize the whole site.