4 ms·
Just out of curiosity, how is this possible? If the web server can't handle being crawled...how can it handle serving web pages? "If your web server is unable
by snake_plissken 12y ago
Just out of curiosity, how is this possible? If the web server can't handle being crawled...how can it handle serving web pages?
"If your web server is unable to handle the volume of crawl requests for resources, it may have a negative impact on our capability to render your pages. If you’d like to ensure that your pages can be rendered by Google, make sure your servers are able to handle crawl requests for resources."
- devNoise 12y agoTry to equate being crawled with the /. effect. The site can server web pages, but not at scale. So if the googlebot comes along to index your site, the webserver may fail under load. Perhaps you're on co-hosting plan and your provider suspends your site because the crawling has put you over one of the limits on your plan.