3 ms·

Just to go a little bit more into detail on this, because the article and most of the conversation here is based on a big misunderstanding: robots.txt governs

Just to go a little bit more into detail on this, because the article and most of the conversation here is based on a big misunderstanding: robots.txt governs crawlers. Fetching a single user-specified URL is not crawling. Crawling is when you automatically follow links to continue fetching subsequent pages. Perplexity’s documentation that the article links to describes how their crawler works. That is not the piece of software that fetches individual web pages when a user asks for them. That’s just a regular user-agent, because it’s acting as an agent for the user. The distinction between crawling and not crawling has been very firmly established for decades. You can see it in action with wget. If you fetch a specific URL with `wget https://www.example.com https://www.example.com` then wget will just fetch that URL. It will not fetch robots.txt at all. If you tell wget to act recursively with `wget --recursive https://www.example.com https://www.example.com` to crawl that website, then wget will fetch `https://www.example.com https://www.example.com`, look for links on the page, then if it finds any links to other pages, it will fetch `https://www.example.com/robots.txt https://www.example.com/robots.txt` to check if it is permitted to fetch any subsequent links. This is the difference between fetching a web page and crawling a website. Perplexity is following the very well established norms here.

mattigames 2y ago
Its fairly logical to assume that robots.txt governs robots (empahsis in "bots") not just crawlers, if they are only intended to block crawlers why aren't they called crawlers.txt instead and remove all ambiguity?
- bluish29 2y ago
  That's a historical question. At this time, most if not all the bots were either search engines or archival. The name was even "RobotsNotWanted.txt" at the beginning but made "robots.txt" for simplicity. To give another example, Internet Archive stopped respecting it a couple of years ago, and they discuss this point (crawlers vs other bots) here [1]. [1] https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
  - mattigames 2y ago
    If it was uniquely an historical question then another text file to handle AI requests would exist by now, e.g. ai-bots.txt, but it hasn't and likely never will, they don't want to even have to pretend to comply with creator requests about forbidding (or not) the usage of their sites.
  - pseudalopex 2y ago
    You meant search bots and other bots? Internet Archive's bot is a crawler. They showed no difference between search bots and archive bots. robots.txt was never for SEO alone. Sites exclude print versions so people see more ads and links to other pages. Sites exclude search pages to conserve resources. They said sites exclude large files for costs. And they can't think sites want sensitive areas like administrative pages archived. Really Internet Archive stopped respecting robots.txt because they wanted to archive what sites didn't want them to archive. Many sites disallowed Internet Archive specifically. Many sites allowed specific bots. Many sites disallowed all bots and meant all bots. And hiding old snapshots when a new domain owner changed robots.txt was a self inflicted problem. robots.txt says what to crawl or not now. They knew all of this.
- alexey-salmin 2y ago
  There's more than one way to define what a bot is. You can make a request by typing the url in chrome, or by asking an AI tool to do so. Both start from user intent, both heavily rely on complicated software to work. It's fairly logical to assume that bots don't have an intent and users do. It's not the only available interpretation though.
- JimDabell 2y ago
  It’s not logical to assume anything about a standard merely from a 30-year-old filename when you can just read the documentation. > WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages. — http://www.robotstxt.org/orig.html http://www.robotstxt.org/orig.html
- Dylan16807 2y ago
  > Its fairly logical to assume that robots.txt governs robots (empahsis in "bots") not just crawlers It's plenty logical. That doesn't make it correct. > if they are only intended to block crawlers why aren't they called crawlers.txt instead and remove all ambiguity? Ha. Ask HTTP Referer. A million standards have quirks in them that we're stuck with.

mattigames 2y ago

Its fairly logical to assume that robots.txt governs robots (empahsis in "bots") not just crawlers, if they are only intended to block crawlers why aren't they called crawlers.txt instead and remove all ambiguity?

bluish29 2y ago
That's a historical question. At this time, most if not all the bots were either search engines or archival. The name was even "RobotsNotWanted.txt" at the beginning but made "robots.txt" for simplicity. To give another example, Internet Archive stopped respecting it a couple of years ago, and they discuss this point (crawlers vs other bots) here [1]. [1] https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- mattigames 2y ago
  If it was uniquely an historical question then another text file to handle AI requests would exist by now, e.g. ai-bots.txt, but it hasn't and likely never will, they don't want to even have to pretend to comply with creator requests about forbidding (or not) the usage of their sites.
- pseudalopex 2y ago
  You meant search bots and other bots? Internet Archive's bot is a crawler. They showed no difference between search bots and archive bots. robots.txt was never for SEO alone. Sites exclude print versions so people see more ads and links to other pages. Sites exclude search pages to conserve resources. They said sites exclude large files for costs. And they can't think sites want sensitive areas like administrative pages archived. Really Internet Archive stopped respecting robots.txt because they wanted to archive what sites didn't want them to archive. Many sites disallowed Internet Archive specifically. Many sites allowed specific bots. Many sites disallowed all bots and meant all bots. And hiding old snapshots when a new domain owner changed robots.txt was a self inflicted problem. robots.txt says what to crawl or not now. They knew all of this.
alexey-salmin 2y ago
There's more than one way to define what a bot is. You can make a request by typing the url in chrome, or by asking an AI tool to do so. Both start from user intent, both heavily rely on complicated software to work. It's fairly logical to assume that bots don't have an intent and users do. It's not the only available interpretation though.
JimDabell 2y ago
It’s not logical to assume anything about a standard merely from a 30-year-old filename when you can just read the documentation. > WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages. — http://www.robotstxt.org/orig.html http://www.robotstxt.org/orig.html
Dylan16807 2y ago
> Its fairly logical to assume that robots.txt governs robots (empahsis in "bots") not just crawlers It's plenty logical. That doesn't make it correct. > if they are only intended to block crawlers why aren't they called crawlers.txt instead and remove all ambiguity? Ha. Ask HTTP Referer. A million standards have quirks in them that we're stuck with.