2 ms·
The main challenge is distributed rate-limiting, something I'd hope the framework handles for me. Also having k8s settings that work well in your experience w.r
by verdverm 9mo ago
The main challenge is distributed rate-limiting, something I'd hope the framework handles for me. Also having k8s settings that work well in your experience w.r.t. scaling
- nihalwashere 9mo agoDistributed rate-limiting is intentionally not in the core library, Reader focuses on the scraping primitives and stays unopinionated about orchestration. For multi-node rate limiting, you'd layer that on top: Redis + a simple limiter that gates calls to reader.scrape(). For k8s resource settings, the Docker guide is a good starting point: https://docs.reader.dev/documentation/guides/deployment https://docs.reader.dev/documentation/guides/deployment But I will add some reference examples on how to build a rate-limiting and K8s orchestration layer on top of Reader... Thanks for sharing this :)
- verdverm 9mo agoThere's plenty of what you've built here to go around. It's trivial now to reproduce the basics. Distributed rate-limiting is a hard problem, one people may pay for
- nihalwashere 9mo agoI will add some reference examples on how to build a rate-limiting and K8s orchestration layer on top of Reader soon :)
- verdverm 9mo agoYou're missing the point, I don't want to build it myself. The framework I will actually use will do it for me. If yours does not, it will not be in consideration.