Implied agreement. The law acknowledges standard practice and expectations. That's what it is built on. If you put something up on the web, then although you

Implied agreement. The law acknowledges standard practice and expectations. That's what it is built on. If you put something up on the web, then although you have copyright, you're also implying permission for people to be able to do what they normally do with it (at a minimum, view it in a web browser). Similarly, if you've put up a robots.txt, then you're also quite clearly giving permission to crawlers (within certain bounds). There's also reasonableness and taking into account what robots.txt is capable of, of course. You clearly aren't implying permission for me to DoS you. Explicitly agreed terms trumps this. If it could be shown that the person operating the crawler had agreed to an AUP, then that would change things. I imagine that this would make this case quite different from Google crawling your site.