5 ms·
Try to equate being crawled with the /. effect. The site can server web pages, but not at scale. So if the googlebot comes along to index your site, the webserv
by devNoise 12y ago
Try to equate being crawled with the /. effect. The site can server web pages, but not at scale. So if the googlebot comes along to index your site, the webserver may fail under load. Perhaps you're on co-hosting plan and your provider suspends your site because the crawling has put you over one of the limits on your plan.