3 ms·
> Only a select few crawlers are allowed access to the entire web, and Google is given extra special privileges on top of that. Hmm, so set up a VPN on the Goo
by dheera 6y ago
> Only a select few crawlers are allowed access to the entire web, and Google is given extra special privileges on top of that.
Hmm, so set up a VPN on the Google Cloud so you have a Google IP address, use a Google User-Agent, and go!
- jesboat 6y agohttps://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/... describes the procedure for checking "is this source Google it". You couldn't fake it just by running on gcp