Indeed. Human activity is cacheable as it respects Zipf's or a power law. Spider activity is not cacheable. Most pages are GETed within a few multiples of ea

Indeed. Human activity is cacheable as it respects Zipf's or a power law. Spider activity is not cacheable. Most pages are GETed within a few multiples of each other. Probably a good way to detect bot traffic is put a URL on a sitemap, and have no links to it anywhere else.