I have done a bit of scrapping with ruby mechanize, when we hit limits have circumvented by proxy and tor google as a search engine crawls most all sites, but

I have done a bit of scrapping with ruby mechanize, when we hit limits have circumvented by proxy and tor google as a search engine crawls most all sites, but offers very few usable stuff to other bots http://www.google.com/robots.txt http://www.google.com/robots.txt Disallow 247 Allow 41