3 ms·They do know that you can spoof the user agent I regularly do this to crawl sites.by C1sc0cat 8y agoThey do know that you can spoof the user agent I regularly do this to crawl sites.