robots.txt is nothing more than a way of politely asking the world not to recursively crawl through links on a site. It has no connection to whether someone is

robots.txt is nothing more than a way of politely asking the world not to recursively crawl through links on a site. It has no connection to whether someone is allowed to access the data or what they're allowed to do with it.