3 ms·
Have you checked their robots.txt and HTML headers, and spoofed Googlebot’s UA and requested a URL to check the response? Faking that the requesting IP is one
by WalterGR 5y ago
Have you checked their robots.txt and HTML headers, and spoofed Googlebot’s UA and requested a URL to check the response?
Faking that the requesting IP is one that Googlebot uses and still receiving the response is the more difficult part.