2 ms·
It’s not logical to assume anything about a standard merely from a 30-year-old filename when you can just read the documentation. > WWW Robots (also called wan
by JimDabell 2y ago
It’s not logical to assume anything about a standard merely from a 30-year-old filename when you can just read the documentation.
> WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages.
— http://www.robotstxt.org/orig.html http://www.robotstxt.org/orig.html