> Again: robots.txt is intended to control crawler behavior, not search index visibility. I'm obviously not asking the question clearly, I'm wanting to stop bo

> Again: robots.txt is intended to control crawler behavior, not search index visibility. I'm obviously not asking the question clearly, I'm wanting to stop bots from crawling (it's scraping that annoys me), not search engines from listing URIs. If I want to completely stop a bot from crawling my site (in the sense of "retrieving my content"), won't robots.txt prevent that? Even in the case of the bot having obtained a valid list of my URIs but not the pages contents from a 3rd party source? Lets say I email you a list of URIs on my site. My robots.txt forbids all crawlers. Are you allowed to give the list of URIs to your bot and retrieve the content?