4 ms·
As someone who does targeted scraping of forums, I can say having a good open API and caching is probably the best way to decrease load. If you use Cloudflare,
by mike_d 3y ago
As someone who does targeted scraping of forums, I can say having a good open API and caching is probably the best way to decrease load.
If you use Cloudflare, turn off their anti-bot stuff. It is far more efficient to let them just serve bots from the cache than having scrapers use tricks to bypass them and go directly to your origin server.
- Spivak 3y agoYeah there's basically no way I would ever read what you just wrote and choose to do anything but block harder even at higher cost. Bots end running around CF would guarantee turning on authenticated origin pull.