4 ms·
Any supporting link reference for this ?
by techaddict009 13y ago
Any supporting link reference for this ?
- eel 13y agoI edited the original comment, but yes, it is in this document: http://www.robotstxt.org/orig.html http://www.robotstxt.org/orig.html The key paragraph is: If the value is '*', the record describes the default access policy for any robot that has not matched any of the other records. It is not allowed to have multiple such records in the "/robots.txt" file.
- techaddict009 13y agoThanks for sharing this !