We ( http://samuru.com http://samuru.com ) limit our crawling, and we honor robots.txt We also use Google Infrastructure, so we come from the same IP's as Googl

We ( http://samuru.com http://samuru.com ) limit our crawling, and we honor robots.txt We also use Google Infrastructure, so we come from the same IP's as Google Bot. We probably crawl 50 pages for every visitor we deliver. There are a lot more pages on the web than people on the planet. So the ratios will always favor the bots.