3 ms·
Robust crawling with all the gratuitous anti-bot infra out there (requesting the rss feed that is specifically meant to be consumed by machine for a public gov
by PeterStuer 2y ago
Robust crawling with all the gratuitous anti-bot infra out there (requesting the rss feed that is specifically meant to be consumed by machine for a public gov site is 'protected' by Cloudflare's anti-botting. Seriously?) takes way more work than you think.
- marginalia_nu 2y agoThe way to get around anti-bot stuff as a search engine crawler is to be upfront about what you are doing, as well as to respect robots.txt, and rate limits, and use a gratuitous crawl delay (1 sec is the bare minimum). You can register with cloudflare as a search engine crawler and they largely will let your traffic through. If you get blocked by individual sites, you can usually just email them explaining the situation and get unblocked. I'm having very little issues with this.