5 ms·
I have seen sites behave differently if you use a Googlebot UA, but am I missing something or does this merely mean that anyone doing something like this curl
by hannob 6y ago
I have seen sites behave differently if you use a Googlebot UA, but am I missing something or does this merely mean that anyone doing something like this
curl -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html http://www.google.com/bot.html)'
will get Google-level crawler access?
- kirubakaran 6y agoThat would work on website that have a naive check for just user agent. Google also publishes the IP address ranges their crawlers run on. Lot of websites check for that, and there's no way around that. https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...