I edited the original comment, but yes, it is in this document: http://www.robotstxt.org/orig.html http://www.robotstxt.org/orig.html The key paragraph is:

I edited the original comment, but yes, it is in this document: http://www.robotstxt.org/orig.html http://www.robotstxt.org/orig.html The key paragraph is: If the value is '*', the record describes the default access policy for any robot that has not matched any of the other records. It is not allowed to have multiple such records in the "/robots.txt" file.