3 ms·
Scrapy needs to have sane defaults that do no harm
- sph 1y agoIs there a pluggable fail2ban-like server for HTTP, to rate-limit and block with a 429 any rude bot? Also, how come we haven't yet designed a system of pluggable HTTP pipelines? If you have a custom web app, either you implement rate limiting by hand, or you set that up in your frontend nginx/caddy/apache/etc. It would be much nicer if we had standardized composable HTTP server processes that either handle a request, or pass through to the next one. That way rate limiting would just be a matter of plugging a rate-limiter process just before your web app. I guess I just miss having Elixir's Plug system at the OS level.
- sc68cal 1y agoI think the issue is that services like CloudFlare are relied upon for this type of protection. It is a very large problem domain and it's easier to delegate it to a dedicated service than to build something to solve it
- 2000UltraDeluxe 1y agoModsec should be able to handle it.