This interpretation makes caching illegal and puts routing in a gray area. If not instructing against caching is implied consent to redistribute the content, th

This interpretation makes caching illegal and puts routing in a gray area. If not instructing against caching is implied consent to redistribute the content, then you're essentially agreeing with me. robots.txt is indeed intended for crawling, but if it's there and you redistribute someone's content anyway, I'd consider it less defensible.