Google's stated goal is to provide people access to information. But in reality, they provide access to what's easily accessible to them. Any robots.txt tool is

Google's stated goal is to provide people access to information. But in reality, they provide access to what's easily accessible to them. Any robots.txt tool is far too blunt of an instrument. It does not respect other highly valued goals of privacy, competitive fairness, and comprehensiveness, among others. To put it more starkly, to assist with privacy, there should be a robots.txt tag/line that allows scraping, but prohibits indexing anything that is tagged as "personally identifiable information". It should be easy to Google what the newest bankruptcy law is, but should be a bit harder to figure out if a specific person filed for bankruptcy. Similarly, with comprehensiveness, robots.txt should be able to say "don't index these pages unless you also index those pages". For example, a medical site should be able to say, you can index these several name-brand drugs but only if you also index their generic equivalents. And as far as fairness, a better robots.txt should be able to tell Google to index the portions of its content to attract clicks, but prevent them from scraping their most valuable content. For example, Yelp may allow them to index a restaraunt's overall rating, but not summarize their reviews.