4 ms·
It is absolutely a good idea to write your own task queue. I agree 100%. Most of the time you have a very specific need, and you'll have to do weird stuff (or u
by roflyear 4y ago
It is absolutely a good idea to write your own task queue. I agree 100%. Most of the time you have a very specific need, and you'll have to do weird stuff (or use a library and/or framework that isn't super popular) to support those features.
My example is I needed a task queue that did some basic rate limiting, but across workers. I wanted to be able to have 100 workers and still not hit an endpoint more than (roughly) n-times a second (or minute, or hour, or day, etc..).
There are systems that do this, but they are either complex and require complex backends and/or software installs (other platforms, etc..). Maybe they don't run in your desired environment, or maybe they do - but adding dependencies should not always be done lightly.
Creating my own task queue took a few days and then a few more days of addressing what were for us very minor bugs (duplicate execution of jobs was one).
- arynda 4y agoHow did you end up solving the rate limiting? Were the workers themselves responsible for coordination via a semaphore (e.g. there's already 25 of us, don't consume from this queue) or did you solve it on the scheduling side and only push jobs into their queues once you knew there was capacity?
- roflyear 4y agoScheduling side, we only schedule jobs if there have not been N of that type scheduled during that period. Works pretty good. The jobs are still technically "queued" but they don't get consumed until something opens up (but workers can work on other tasks).
- arynda 4y agoThat's cool. I've been thinking through a similar design. Would this be an accurate description of your approach? queues hold jobs, the scheduler/dispatcher handles load-balancing and rate-limiting, pushing jobs down to workers only when flow-control criteria is met? ┌────────┐ ┌─────────┐ │ queueA ├──┐ ┌──┤worker1 │ └────────┘ │ │ └─────────┘ │ │ ┌────────┐ ├───────────┴┐ ┌─────────┐ │ queueB ├──┤dispatcher ├─┤worker2 │ └────────┘ ├───────────┬┘ └─────────┘ │ │ ┌────────┐ │ │ ┌─────────┐ │ queueC ├──┘ └──┤workerN │ └────────┘ └─────────┘
- roflyear 4y agoYeah, pretty much! Technically I have one real queue but different "tags" to route things (so we can rate limit etc..). But basically the same thing. My system supports as many dispatchers as you want, which is a good addition but makes the logic more complex as you have to be careful with locks so you don't schedule jobs many times, for example. My system also implements retry logic, and a bunch of other stuff, but that isn't absolutely required either.
- calvinmorrison 4y agosorry, how did you draw this/
- arynda 4y agohttps://asciiflow.com/ https://asciiflow.com/
- calvinmorrison 4y agothank you