I think the biggest argument for honoring robots.txt is that sites, especially old sites, can have a lot of really highly resource intensive pages. I don't wan

I think the biggest argument for honoring robots.txt is that sites, especially old sites, can have a lot of really highly resource intensive pages. I don't want someone crawling a page that has 800+ DB calls... for example. Yes, I should optimize the page, or whatever... but really it may be useful for that admin, or 1 out of 10,000 users, who uses the page. It's not ideal to have someone crawl all those pages at once. I think they should honor robots.txt, and the meta tag version on specific pages or links -- given the site publisher went out of their way to give instructions to crawlers it seems reasonable to honor those requests.