2 ms·

> It is capabale to handle distributed workload with its redis database support sounds like this is limited by redis. For many organizations, this is fine. At

> It is capabale to handle distributed workload with its redis database support sounds like this is limited by redis. For many organizations, this is fine. At my last gig, we used redis for deduplication and it required over 150 redis nodes with deterministic routing and consensus. Redis reportedly could support 200k rps per node, but in our case, we wouldn't see it get passed around 50k rps no matter how we tuned it. An interesting addition to this library would be to use an interface and allow your backing datastore of choice allowing teams to use redis, zookeeper, an in-mem Go instance of the same library, sql, etc. A fun exercise would be to figure out how to make the rate limiting itself distributed so you don't need a single store keeping everything in sync. Maybe a combo of deterministic request routing in a ring topology

dist1ll 3y ago
RPS meaning reads or writes? What's the distribution of message sizes, and how large is your total dataset? What specs (core count, NIC) did each node have? I'm asking because without this info, RPS is not a particularly useful metric. As an extreme example, if your dataset is <1MB, you could likely serve read-heavy requests from your SmartNIC's dcache at close to line rate.
- sethammons 3y ago
  It's been a number of years since I worked on it. I can try to answer your questions. The calls were nearly entirely INCR calls against keys that were typically around 150 bytes long and app logic was based on the return values. I believe each node took up around 20GB of memory when saturated. Redis uses a single CPU, so core count shouldn't matter, no? I'm not entirely sure what our NIC situation was, but it was tuned to handle gigabit traffic.
  - brazzledazzle 3y ago
    Did you tune the file descriptor limits?
    - sethammons 3y ago
      yeah, good call out. We did. One of the things a lot of folks stub their toe on when starting to scale up
dasubhajit 3y ago
> An interesting addition to this library would be to use an interface and allow your backing datastore of choice allowing teams to use redis, zookeeper, an in-mem Go instance of the same library, sql, etc. Thanks for the feedback. I'm gonna implement an in-mem Go instance for local dev, but not sure if that will be enough to use in prod. also, in the next release, I will make redis optional.

dist1ll 3y ago

RPS meaning reads or writes? What's the distribution of message sizes, and how large is your total dataset? What specs (core count, NIC) did each node have? I'm asking because without this info, RPS is not a particularly useful metric. As an extreme example, if your dataset is <1MB, you could likely serve read-heavy requests from your SmartNIC's dcache at close to line rate.

sethammons 3y ago
It's been a number of years since I worked on it. I can try to answer your questions. The calls were nearly entirely INCR calls against keys that were typically around 150 bytes long and app logic was based on the return values. I believe each node took up around 20GB of memory when saturated. Redis uses a single CPU, so core count shouldn't matter, no? I'm not entirely sure what our NIC situation was, but it was tuned to handle gigabit traffic.
- brazzledazzle 3y ago
  Did you tune the file descriptor limits?
  - sethammons 3y ago
    yeah, good call out. We did. One of the things a lot of folks stub their toe on when starting to scale up

dasubhajit 3y ago

> An interesting addition to this library would be to use an interface and allow your backing datastore of choice allowing teams to use redis, zookeeper, an in-mem Go instance of the same library, sql, etc. Thanks for the feedback. I'm gonna implement an in-mem Go instance for local dev, but not sure if that will be enough to use in prod. also, in the next release, I will make redis optional.