2 ms·
How Safe Are Our RSS Feeds from AI Data Scrapers?
- ashutoshstark 8mo agoOne thing I’ve noticed with scraper traffic is that a lot of the basic defenses rely on external infrastructure like Redis or a WAF layer. In a recent project, I experimented with keeping the rate limiter entirely in-memory using ConcurrentHashMap.compute() to implement an atomic sliding window. It worked surprisingly well for smaller services where adding Redis felt like overkill. I wrote up the approach and some of the tradeoffs here if anyone’s curious: https://github.com/ashutosh-stark/velocity-gate https://github.com/ashutosh-stark/velocity-gate