3 ms·

Its fairly logical to assume that robots.txt governs robots (empahsis in "bots") not just crawlers, if they are only intended to block crawlers why aren't they

Its fairly logical to assume that robots.txt governs robots (empahsis in "bots") not just crawlers, if they are only intended to block crawlers why aren't they called crawlers.txt instead and remove all ambiguity?

bluish29 2y ago
That's a historical question. At this time, most if not all the bots were either search engines or archival. The name was even "RobotsNotWanted.txt" at the beginning but made "robots.txt" for simplicity. To give another example, Internet Archive stopped respecting it a couple of years ago, and they discuss this point (crawlers vs other bots) here [1]. [1] https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- mattigames 2y ago
  If it was uniquely an historical question then another text file to handle AI requests would exist by now, e.g. ai-bots.txt, but it hasn't and likely never will, they don't want to even have to pretend to comply with creator requests about forbidding (or not) the usage of their sites.
- pseudalopex 2y ago
  You meant search bots and other bots? Internet Archive's bot is a crawler. They showed no difference between search bots and archive bots. robots.txt was never for SEO alone. Sites exclude print versions so people see more ads and links to other pages. Sites exclude search pages to conserve resources. They said sites exclude large files for costs. And they can't think sites want sensitive areas like administrative pages archived. Really Internet Archive stopped respecting robots.txt because they wanted to archive what sites didn't want them to archive. Many sites disallowed Internet Archive specifically. Many sites allowed specific bots. Many sites disallowed all bots and meant all bots. And hiding old snapshots when a new domain owner changed robots.txt was a self inflicted problem. robots.txt says what to crawl or not now. They knew all of this.
alexey-salmin 2y ago
There's more than one way to define what a bot is. You can make a request by typing the url in chrome, or by asking an AI tool to do so. Both start from user intent, both heavily rely on complicated software to work. It's fairly logical to assume that bots don't have an intent and users do. It's not the only available interpretation though.
JimDabell 2y ago
It’s not logical to assume anything about a standard merely from a 30-year-old filename when you can just read the documentation. > WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages. — http://www.robotstxt.org/orig.html http://www.robotstxt.org/orig.html
Dylan16807 2y ago
> Its fairly logical to assume that robots.txt governs robots (empahsis in "bots") not just crawlers It's plenty logical. That doesn't make it correct. > if they are only intended to block crawlers why aren't they called crawlers.txt instead and remove all ambiguity? Ha. Ask HTTP Referer. A million standards have quirks in them that we're stuck with.

bluish29 2y ago

That's a historical question. At this time, most if not all the bots were either search engines or archival. The name was even "RobotsNotWanted.txt" at the beginning but made "robots.txt" for simplicity. To give another example, Internet Archive stopped respecting it a couple of years ago, and they discuss this point (crawlers vs other bots) here [1]. [1] https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

mattigames 2y ago
If it was uniquely an historical question then another text file to handle AI requests would exist by now, e.g. ai-bots.txt, but it hasn't and likely never will, they don't want to even have to pretend to comply with creator requests about forbidding (or not) the usage of their sites.
pseudalopex 2y ago
You meant search bots and other bots? Internet Archive's bot is a crawler. They showed no difference between search bots and archive bots. robots.txt was never for SEO alone. Sites exclude print versions so people see more ads and links to other pages. Sites exclude search pages to conserve resources. They said sites exclude large files for costs. And they can't think sites want sensitive areas like administrative pages archived. Really Internet Archive stopped respecting robots.txt because they wanted to archive what sites didn't want them to archive. Many sites disallowed Internet Archive specifically. Many sites allowed specific bots. Many sites disallowed all bots and meant all bots. And hiding old snapshots when a new domain owner changed robots.txt was a self inflicted problem. robots.txt says what to crawl or not now. They knew all of this.

alexey-salmin 2y ago

There's more than one way to define what a bot is. You can make a request by typing the url in chrome, or by asking an AI tool to do so. Both start from user intent, both heavily rely on complicated software to work. It's fairly logical to assume that bots don't have an intent and users do. It's not the only available interpretation though.

JimDabell 2y ago

It’s not logical to assume anything about a standard merely from a 30-year-old filename when you can just read the documentation. > WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages. — http://www.robotstxt.org/orig.html http://www.robotstxt.org/orig.html

Dylan16807 2y ago

> Its fairly logical to assume that robots.txt governs robots (empahsis in "bots") not just crawlers It's plenty logical. That doesn't make it correct. > if they are only intended to block crawlers why aren't they called crawlers.txt instead and remove all ambiguity? Ha. Ask HTTP Referer. A million standards have quirks in them that we're stuck with.