3 ms·
I wonder if fake bot detectors can distinguish between any Google IP like GCP instances (i.e. do they simply check the ASN) or crawler specific IPs Or maybe g
by trinovantes 4y ago
I wonder if fake bot detectors can distinguish between any Google IP like GCP instances (i.e. do they simply check the ASN) or crawler specific IPs
Or maybe google crawler also runs on GCP and it's indistinguishable from regular $5 compute users
- windowsworkstoo 4y agoYes, Google and most major search engines enable a RDNS lookup to validate they are really a googlebot
- trinovantes 4y agoTIL. Found this for googlebot https://developers.google.com/static/search/apis/ipranges/googlebot.json https://developers.google.com/static/search/apis/ipranges/go...