3 ms·
Couldn’t you just use the user agent if you wanted to do this? Or does google crawl without revealing itself?
by jtsiskin 6y ago
Couldn’t you just use the user agent if you wanted to do this?
Or does google crawl without revealing itself?
- GauntletWizard 6y agoGooglebot is a well known user-agent, but they do verify with impersonation. It's a heavy penalty (usually delisting) for serving them noticeably different content (there's of course leeway, because sites are dynamic)