2 ms·
Yeah, just like Google bot crawls the web putting enormous load on people's servers (G-Bot can pull hundreds of thousands of pages per day from a single server)
by visarga 7y ago
Yeah, just like Google bot crawls the web putting enormous load on people's servers (G-Bot can pull hundreds of thousands of pages per day from a single server), while being very sensitive to automated searches and banning your ass (IP) in a heart beat. They sure don't want to be crawled by anyone.
- orf 7y agoIf your site cannot handle the moderate 3-5 requests per second it would take for google to make hundreds of thousands (500k) of requests a day, then I hate to break it to you but you have bigger problems at hand.