4 ms·
Work queues are a mostly solved problem, unless you've got unusual needs, like preposterous amounts of throughput. These guys don't have unusual needs, and they
by pjscott 14y ago
Work queues are a mostly solved problem, unless you've got unusual needs, like preposterous amounts of throughput. These guys don't have unusual needs, and they've got a paltry 16k tasks to keep track of. For that kind of job, use a library. You don't want to be the one who writes code to deal with restarting hung workers, or shunting pathological jobs to a retry-later queue, or changing pipeline topology live in production without a hiccup. I've written this stuff -- we had unusual needs -- so take it from me: you don't want to write this stuff.
If you do most of your work in Python, Celery is pretty slick: http://celeryproject.org/ http://celeryproject.org/
If you like Redis -- and who doesn't? -- then Resque or one of the alternate implementations will work well in Ruby or a plethora of other languages:
https://github.com/defunkt/resque https://github.com/defunkt/resque
https://github.com/defunkt/resque/wiki/Alternate-Implementations https://github.com/defunkt/resque/wiki/Alternate-Implementat...
Beanstalkd is pretty generic, and simple to set up. If for some reason you can't use Celery or Resque, maybe have a look at it:
http://kr.github.com/beanstalkd/ http://kr.github.com/beanstalkd/
Distributed task queueing is important and useful stuff, and if you half-ass it, your infrastructure will literally explode, usually at 3:00 AM. Get a good library for it.
- compumike 14y agoIndeed, Celery was a tool we considered using for this task, especially since we're Python/Django based already. And yes, there are a lot of seemingly minor but quite important sticky points as you mentioned (and we listed in our post). I would certainly give these libraries a close look if I were to implement another work queue today. I'm in total agreement with the basic message: distributed task queueing is important and complex, so don't "half-ass it". :)
- Ixiaus 14y agoAMQP is overkill in most cases (I found out from personal experience). The OP has a good point in using Celery. See my comment above if your rendering clients are written in Erlang.
- memset 14y agoCan you clarify what you mean by "use a library?" It's easy to follow the rabbitmq tutorial and build a nice little pipeline for tasks - what is the difference between that and using a library?
- heretohelp 14y agoQuite a lot if you don't want stuff to break.
- pjscott 14y agoThe key question is, "What happens if $INSERT_CALAMITY_HERE?" What happens if a worker crashes? What happens if a few jobs cause the workers to work forever? What happens if a job hits a bug in your worker code and causes it to reliably crash every time? What happens if your queueing server goes down? What happens if you need to migrate your queueing server to another EC2 instance because Amazon has decided to get rid of the physical machine it's running on? Undesirable answers to such questions include "We accidentally drop a job without noticing," or "We have to shut down the whole system in a user-visible way for twenty minutes while we do maintenance," or "All the workers keep crashing and being restarted, flapping eternally until someone notices and manually intervenes." EDIT: Oops, I just noticed that I didn't really answer your question about what I meant by using a library. What I mean is, use one of the more full-featured job queueing libraries like Celery or Resque, rather than trying to roll your own on top of something like RabbitMQ.
- memset 14y agoAh, okay, so something like Celery would handle all of those kinds of cases for you. (Though... doesn't rabbitmq have many of those features built-in? Queue server replication and redundancy and stuff? Though I suppose a dedicated project would handle more of those calamities than I could think of on my own.) Thanks for that!
- Ixiaus 14y agoRabbitMQ doesn't have all that built in at all. RabbitMQ is only an AMQP message broker/exchange/queue agent. It's hard to articulate (and probably should be done in a blog post) all the things that can go wrong, but rest assured, there are a lot of things that can go wrong with RabbitMQ AND with your worker nodes that are built on-top of RabbitMQ. Distributed workloads is a hard problem, but it is a solved problem (for most applications, there are some exceptions thought too).