3 ms·
robots.txt files are not mechanisms for authorizing access; they are guidelines designed to instruct user-agents on relevant resources. They essentially say to
by jalfresi 11y ago
robots.txt files are not mechanisms for authorizing access; they are guidelines designed to instruct user-agents on relevant resources. They essentially say to the user-agent "the content marked as 'disallow' should not be crawled for indexing purposes for unspecified reasons"
Adhering to a robots.txt file is not mandatory, though it does make you a good netizen/curries favour with server admins.
- smitherfield 11y agoPeople have actually gotten into trouble for accessing subdirectories which aren't accessible via links from the site's index page. But I'm not sure if that extends to robots.txt, at least as it pertains to stuff which is intended for public consumption. Otherwise people who host sites containing illegal content would just put the entire site in robots.txt.