4 ms·Formalizing the Robots Exclusion Protocol Specification4 points by manjana 6y agomanjana 6y agoAlso: https://tools.ietf.org/html/draft-koster-rep-02 https://tools.ietf.org/html/draft-koster-rep-02