3 ms·
Google probably assumes that if you are not disallowing their spider via robots.txt you are consenting to allow Google to index your content.
by ZitchDog 16y ago
Google probably assumes that if you are not disallowing their spider via robots.txt you are consenting to allow Google to index your content.