3 ms·
Can't you just set your useragent to googlebot? Or something which isn't googlebot but which matches the most common regexes like "Not-Googlebot/2.1"
by oh_sigh 6y ago
Can't you just set your useragent to googlebot? Or something which isn't googlebot but which matches the most common regexes like "Not-Googlebot/2.1"
- Grimm1 6y agoA lot of sites do some variation of these when you set googlebot as your UA, certainly the larger more sophisticated sites do. https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en So unless you have a google domain your sol, it's also just generally frowned upon. We have our own UA WhizeBot with an email contact so you can let us know if our crawler is doing anything you'd rather it not. There have been a few legal cases that protect scraping publicly available information on the web but we'd rather follow robots.txt to avoid the potential for shenanigans in any case.
- ricardo81 6y agoThat's "poor man's cloaking". Most people that genuinely care if it's Googlebot or not will verify it appropriately by doing a DNS lookup and a reverse DNS of that IP to ensure it's one of Google's IPs. 'Faking' the UA is much more likely to annoy site owners and end up with a permanent block.