robots.txt only tells bots not to crawl the website. It doesn't say anything about indexing or archival of pages. It would be perfectly possible to have the Wa

robots.txt only tells bots not to crawl the website. It doesn't say anything about indexing or archival of pages. It would be perfectly possible to have the Wayback Machine respect the robots.txt and not have it crawl or archive any new pages, whilst making pages that have already been archived accessible unless a specific user agent has been denied.