4 ms·
The strictness is in what may be listed in the robots txt, not the User-Agent header as sent by bots. the example given in the linked draft standard[0] makes th
by steventhedev 7y ago
The strictness is in what may be listed in the robots txt, not the User-Agent header as sent by bots. the example given in the linked draft standard[0] makes this abundantly clear that it's on the bot to understand how to interpret the corresponding lines of robots.txt.
Of course, in practice robots.txt tend to look less like [1] and more like [2].
[0]: https://tools.ietf.org/html/draft-rep-wg-topic#section-2.2.1 https://tools.ietf.org/html/draft-rep-wg-topic#section-2.2.1
[1]: https://github.com/robots.txt https://github.com/robots.txt
[2]: https://wpengine.com/robots.txt https://wpengine.com/robots.txt
- jchw 7y agoSorry, I mean for matching, and I did try to imply it was a limitation of the standard and not the library. Though to avoid confusion, I do personally think keeping the user agent minimal is wise, since users might have difficulty guessing what value to use if it differs sufficiently from the real user agent that's sent.