3 ms·
What if i don't want Google to crawl my website but i'm ok if others do?
by Tepix 6y ago
What if i don't want Google to crawl my website but i'm ok if others do?
- octoberfranklin 6y agoAs it currently is you can't stop them from using an undeclared IP range to crawl you. This doesn't change that. "Everybody except X is allowed" rules have always been "on your honor" type restrictions. "X" can always claim to be Joe Rando.