3 ms·

Then deny everything until you are ready, but don't deny access to everything but the big boys (who are the ones with the real resources to hammer your site any

Then deny everything until you are ready, but don't deny access to everything but the big boys (who are the ones with the real resources to hammer your site anyway). I'm not saying you were in the wrong, but that is the stance I would (and do, with my current project) take. And of course, when we are ready, we will open the door to all indexing.

ohashi 16y ago
Just because you do this doesn't mean it makes sense for everyone. Allowing Google more access because it gives more reward makes perfect business sense. Why should anyone and everyone be allowed to crawl and scrape my website or nobody allowed? The world isn't a series of black and white decisions.