If you aren’t recursively fetching pages, you aren’t a robot and shouldn’t follow robots.txt: > WWW Robots (also called wanderers or spiders) are programs that

If you aren’t recursively fetching pages, you aren’t a robot and shouldn’t follow robots.txt: > WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages. — http://www.robotstxt.org/orig.html http://www.robotstxt.org/orig.html When these companies crawl the web looking for training data, they should obey.robots.txt. When a user asks them about a particular page (e.g. “summarise this page”), they should ignore robots.txt All of these articles purport to show that these companies are not doing the former by testing the behaviour of the latter. That’s a misunderstanding of how robots.txt is supposed to work.