4 ms·
When our startup was just getting going, search spiders were an unexpected problem for us. Our project was a bit odd, in that we had millions of documents avai
by joblessjunkie 16y ago
When our startup was just getting going, search spiders were an unexpected problem for us.
Our project was a bit odd, in that we had millions of documents available to browse. We wanted them all to be available to search engines, but it was an operational headache to keep that much data moving.
Our normal daily traffic might have been 10,000 users with perhaps 50K page requests. We built our first small web server to handle this load.
We were caught blindsided by the Googlebot, which from day one demanded hundreds of thousands of pages per day. Googlebot offers no option to slow it down, and if you try to slow it down on purpose your search ranking will suffer.
So 90% of our early operational burden was devoted to keeping Googlebot happy. This was an unexpected burden, but it was worth it to be in the Google index, because they sent a lot of traffic our way and kept our fledging startup alive.
But it was definitely not worth it to spend equivalent operational expense to be in marginal search engines that don't refer much traffic. To a site operator, they don't offer any value in exchange for what they demand.
It was very interesting to see the relative volume of requests that we got from Googlebot versus the other crawlers. While Googlebot found us on day one, it took over 3 months before we appeared in Microsoft's index, and even then, they had perhaps only 10% of the coverage of our site when compared to Google.
- ig1 16y agoYou can slow down Googlebot via the Google Webmaster Tools
- joblessjunkie 16y agoAt the time, I recall that our choices were between "too fast" or "way, way too fast".
- rorrr 16y agoYou can request to change the crawl rate. http://www.google.com/support/webmasters/bin/answer.py?answer=48620 http://www.google.com/support/webmasters/bin/answer.py?answe...
- palish 16y ago"You can't change the crawl rate for sites that are not at the root level—for example, www.example.com/folder." What if Googlebot starts at a non-root location (via a link from another site)? Will it always start at root?
- terryjsmith 16y agoI think they just mean that it's a whole site/domain action; that you can't change crawl rate for a sub-directory based site ala Wordpress MU/Multisite with directories.
- ocharles 16y agoThen deny everything until you are ready, but don't deny access to everything but the big boys (who are the ones with the real resources to hammer your site anyway). I'm not saying you were in the wrong, but that is the stance I would (and do, with my current project) take. And of course, when we are ready, we will open the door to all indexing.
- ohashi 16y agoJust because you do this doesn't mean it makes sense for everyone. Allowing Google more access because it gives more reward makes perfect business sense. Why should anyone and everyone be allowed to crawl and scrape my website or nobody allowed? The world isn't a series of black and white decisions.
- z2amiller 16y agoFWIW, most of the "Major Players" support the Crawl-Delay directive in robots.txt, see: http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl... I know at minimum GoogleBot and Yahoo! Slurp respect this value. I believe Google AdsBot (Landing page URL verification for AdWords) also supports this value, but GoogleBot and AdsBot each will crawl at the rate you specify (so if you specify a Crawl-Delay of 1, each bot will crawl at ~1qps). I don't know if it is in the spec, but fractional crawl delays appeared to be respected (Crawl-Delay: 0.25 would result in ~4qps, for example). I too have fought with this problem - at times more than 90% of the capacity of the site I was running was devoted to serving up content for bots. I sympathize with small (and large) companies who don't want to add capacity so that new/random bot can add another <x> QPS to the daily baseline load.