For the most part I agree, but I feel there are grey areas. Things like web browsers (which are not robots) can access the content as though they are from a hum

For the most part I agree, but I feel there are grey areas. Things like web browsers (which are not robots) can access the content as though they are from a human. But what about extensions or apps that do things in the background, such as caching the contents of several pages for offline viewing. Is that now considered a bot. The robotstxt.org site states that a robot "should" obey the rules. "should" is not a legal term that implies compliance. "must" would have been more appropriate to indicate enforcement.