3 ms·
Ask HN: Worker queue with multi-dimensional rate limiting
I want to distribute tasks to worker nodes using a queue. The work depends on external resources.
The access to those needs to be rate limited across multiple dimensions (i.e. customer id, city, phone provider, there may be only 1 per customer id / second, 1000/city second and 20 / phone provider/second). The rate limit needs to apply to all worker nodes. My worker nodes share the same network.
How can I achieve that while maintaining a reasonable throughput? Any experience?
My naive attempt would be to use one leaky bucket per dimension implemented in Redis and check those prior accessing the resource. If there's no token in all the necessary buckets available, I would put the request back at the end of the queue.
Are there better ways to implement this?
- hguhghuff 8y agoI’d do it in Postgres and use sql columns for all your qualifiers. Refer SKIP LOCKED
- brudgers 8y agoWhat business problem are you trying to solve?
- jyothi_9 8y agoit is really explainable very well and i got more information from your blog. https://www.kitsonlinetrainings.com/sap-bw-on-hana-online-training.html https://www.kitsonlinetrainings.com/sap-bw-on-hana-online-tr... https://www.kitsonlinetrainings.com/sap-cs-online-training.html https://www.kitsonlinetrainings.com/sap-cs-online-training.h... https://www.kitsonlinetrainings.com/sap-fico-online-training.html https://www.kitsonlinetrainings.com/sap-fico-online-training... https://www.kitsonlinetrainings.com/sap-grc-online-training.html https://www.kitsonlinetrainings.com/sap-grc-online-training.... https://www.kitsonlinetrainings.com/sap-gts-online-training.html https://www.kitsonlinetrainings.com/sap-gts-online-training....
- jyothi_9 8y agoThanks for this great share. https://www.kitsonlinetrainings.com/sap-hana-online-training.html https://www.kitsonlinetrainings.com/sap-hana-online-training... https://www.kitsonlinetrainings.com/sap-hybris-online-training.html https://www.kitsonlinetrainings.com/sap-hybris-online-traini... https://www.kitsonlinetrainings.com/sap-mm-online-training.html https://www.kitsonlinetrainings.com/sap-mm-online-training.h... https://www.kitsonlinetrainings.com/sap-pm-online-training.html https://www.kitsonlinetrainings.com/sap-pm-online-training.h... https://www.kitsonlinetrainings.com/sap-pp-online-training.html https://www.kitsonlinetrainings.com/sap-pp-online-training.h...
- jyothi_9 8y agoThis post is much helpful for us. https://www.kitsonlinetrainings.com/sap-qm-online-training.html https://www.kitsonlinetrainings.com/sap-qm-online-training.h... https://www.kitsonlinetrainings.com/sap-sd-online-training.html https://www.kitsonlinetrainings.com/sap-sd-online-training.h... https://www.kitsonlinetrainings.com/sap-security-online-training.html https://www.kitsonlinetrainings.com/sap-security-online-trai... https://www.kitsonlinetrainings.com/sap-wm-online-training.html https://www.kitsonlinetrainings.com/sap-wm-online-training.h... https://www.kitsonlinetrainings.com/scala-online-training.html https://www.kitsonlinetrainings.com/scala-online-training.ht...