2 ms·
You think they'll stick to those IPs any more than robots.txt when it comes to that?
by account42 6mo ago
You think they'll stick to those IPs any more than robots.txt when it comes to that?
- alt227 6mo agoYes I do, I also think they will continue identifying themselves in the user-agent header. https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers https://developers.google.com/crawling/docs/crawlers-fetcher...