I never see this one mentioned: Google crawls the web at a high rate and is able to index new content rapidly. This is possible because every site lets googlebo

I never see this one mentioned: Google crawls the web at a high rate and is able to index new content rapidly. This is possible because every site lets googlebot through. But if you are not already an established search engine everyone will throttle your crawler, and you will have no chance to compete with Google on keeping up with the latest content.