3 ms·
I think that is very different from cloning repos from Github in accordance with the licences. I agree crawling sites to the extent it causes problems is a pro
by graemep 2y ago
I think that is very different from cloning repos from Github in accordance with the licences.
I agree crawling sites to the extent it causes problems is a problem.
Googlebot and Bingbot do follow robots.txt and respect HTTP 429 responses and usually have reasonable default crawl rates.
Is it possible that these are scrapers using fake UA strings?
- 7373737373 2y ago> Is it possible that these are scrapers using fake UA strings? I now believe this is the case
- nextaccountic 2y agoYou need to check if the requests come from Googlebot's IP ranges https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/... https://searchengineland.com/google-publishes-the-list-of-googlebot-ip-addresses-375935 https://searchengineland.com/google-publishes-the-list-of-go... https://developers.google.com/search/apis/ipranges/googlebot.json https://developers.google.com/search/apis/ipranges/googlebot...