I doubt a bot could legally agree to a license put into robots.txt, even if it were able to make sense of it, and a human is never expected to read it. The pur

I doubt a bot could legally agree to a license put into robots.txt, even if it were able to make sense of it, and a human is never expected to read it. The purpose of robots.txt is to guide bots away from circular links and such that would result in bogging down the site and causing undue amounts of nonsense traffic. The purpose of robots.txt not access control. EDIT: typo fix