That document should not be considered legally binding. It is not a contract. Robots.txt is not even really a standard, best it was is a draft, set to expire o

That document should not be considered legally binding. It is not a contract. Robots.txt is not even really a standard, best it was is a draft, set to expire on January 2nd next year. The purpose of the file was to preserve performance, not to hide anything. It targets crawlers not API calls. Consent of a non-single person business entity takes months at the very least, if it ever happens. It's easier for them to deny (incl. lawyers) and intimidate than compete and handle support tickets. Welcome to capitalism.