I was about to agree that robots.txt prohibitions should be considered a form of authorization. But I think what is being argued is that "if it's publicly avai

I was about to agree that robots.txt prohibitions should be considered a form of authorization. But I think what is being argued is that "if it's publicly available on a URL, it's available for any client to download and use." I think the latter argument holds more water, as since they are making it publicly available it is implicit authorization.