3 ms·
It's not against the robots.txt rules. In Facebook's robots.txt file, there is a section for the Googlebot (as well as several other bots) User-agent: Google
by eel 13y ago
It's not against the robots.txt rules. In Facebook's robots.txt file, there is a section for the Googlebot (as well as several other bots)
User-agent: Googlebot
Disallow: /ac.php
...
A bot is supposed to follow the its own User-agent section. If that is not found, it should follow the User-agent: * section. [1]
[1] http://www.robotstxt.org/orig.html http://www.robotstxt.org/orig.html
- techaddict009 13y agoAny supporting link reference for this ?
- eel 13y agoI edited the original comment, but yes, it is in this document: http://www.robotstxt.org/orig.html http://www.robotstxt.org/orig.html The key paragraph is: If the value is '*', the record describes the default access policy for any robot that has not matched any of the other records. It is not allowed to have multiple such records in the "/robots.txt" file.
- techaddict009 13y agoThanks for sharing this !
- hiby007 13y agoGreat Reply, thanks.