3 ms·
Doesn’t that mean you have to know every IP Address used by Google bot now and in the future?
by TedDoesntTalk 5y ago
Doesn’t that mean you have to know every IP Address used by Google bot now and in the future?
- judge2020 5y agoThe way to check googlebot (in a way that will be resistant to expansion of Googlebot's IP ranges in the future) is to perform hostname lookup, with dns lookup as well to verify that the rDNS isn't a lie: https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...
- lucb1e 5y agoIndeed, this was one of the things I considered (note I'm not OP), but then I didn't really want to rely on DNS. https://duckduckgo.com/?q=it's+always+DNS https://duckduckgo.com/?q=it's+always+DNS
- AshamedCaptain 5y agoNot a very hard problem; after all, many websites allow full access to Googlebot IP ranges yet show a paywall to everyone else (including competing search engines). I also happen to ban Google ranges on multiple less-public sites specially since they completely ignore robots.txt and crawl-delay.
- chihuahua 5y agoIs that how archive.vn works? I've always wondered how they are able to get the full text of paywalled sites like Wall Street Journal who give 0 free articles per month.