2 ms·
I would assume Google respects robots.txt, so you should be fine non-abusively scraping their site insofar as you respect their robots.txt
by anonymousab 6y ago
I would assume Google respects robots.txt, so you should be fine non-abusively scraping their site insofar as you respect their robots.txt
- tempestn 6y agoYeah, the tricky thing for those scraped by google is that given google's search monopoly, the sites can't block their scraping entirely, since they need to be shown in search results.
- 1vuio0pswjnm7 6y agoGoogle's robots.txt does not tell the full story. For example, if you include a User-Agent header and put certain strings in it, e.g., "curl/7.47", you will be blocked. echo -e 'GET /search?q=robots.txt HTTP/1.1\r\nHost: www.google.com\r\nUser-Agent: curl/7.47\r\nConnection: close\r\n\r\n' |socat -,ignoreeof ssl:www.google.com,verify=0 The problem with the robots.txt "standard", e.g., ones like Google's with no "crawl-delay" directives, is that it does not define what is a "robot". The query above is obviously not a "robot", but Google, with all it resources, still treats as such. Google probably does more (abusive) scraping than any other entity. Web scraping is in their DNA. It is in their web pages, too. curl https://www.google.com/search/static/gs/animal/m05py0.html|grep scrape