4 ms·

Can you clarify what you mean by "use a library?" It's easy to follow the rabbitmq tutorial and build a nice little pipeline for tasks - what is the difference

by memset 14y ago

Can you clarify what you mean by "use a library?" It's easy to follow the rabbitmq tutorial and build a nice little pipeline for tasks - what is the difference between that and using a library?

heretohelp 14y ago
Quite a lot if you don't want stuff to break.
pjscott 14y ago
The key question is, "What happens if $INSERT_CALAMITY_HERE?" What happens if a worker crashes? What happens if a few jobs cause the workers to work forever? What happens if a job hits a bug in your worker code and causes it to reliably crash every time? What happens if your queueing server goes down? What happens if you need to migrate your queueing server to another EC2 instance because Amazon has decided to get rid of the physical machine it's running on? Undesirable answers to such questions include "We accidentally drop a job without noticing," or "We have to shut down the whole system in a user-visible way for twenty minutes while we do maintenance," or "All the workers keep crashing and being restarted, flapping eternally until someone notices and manually intervenes." EDIT: Oops, I just noticed that I didn't really answer your question about what I meant by using a library. What I mean is, use one of the more full-featured job queueing libraries like Celery or Resque, rather than trying to roll your own on top of something like RabbitMQ.
- memset 14y ago
  Ah, okay, so something like Celery would handle all of those kinds of cases for you. (Though... doesn't rabbitmq have many of those features built-in? Queue server replication and redundancy and stuff? Though I suppose a dedicated project would handle more of those calamities than I could think of on my own.) Thanks for that!
  - Ixiaus 14y ago
    RabbitMQ doesn't have all that built in at all. RabbitMQ is only an AMQP message broker/exchange/queue agent. It's hard to articulate (and probably should be done in a blog post) all the things that can go wrong, but rest assured, there are a lot of things that can go wrong with RabbitMQ AND with your worker nodes that are built on-top of RabbitMQ. Distributed workloads is a hard problem, but it is a solved problem (for most applications, there are some exceptions thought too).

by memset 14y ago

Can you clarify what you mean by "use a library?" It's easy to follow the rabbitmq tutorial and build a nice little pipeline for tasks - what is the difference between that and using a library?

heretohelp 14y ago

Quite a lot if you don't want stuff to break.

pjscott 14y ago

The key question is, "What happens if $INSERT_CALAMITY_HERE?" What happens if a worker crashes? What happens if a few jobs cause the workers to work forever? What happens if a job hits a bug in your worker code and causes it to reliably crash every time? What happens if your queueing server goes down? What happens if you need to migrate your queueing server to another EC2 instance because Amazon has decided to get rid of the physical machine it's running on? Undesirable answers to such questions include "We accidentally drop a job without noticing," or "We have to shut down the whole system in a user-visible way for twenty minutes while we do maintenance," or "All the workers keep crashing and being restarted, flapping eternally until someone notices and manually intervenes." EDIT: Oops, I just noticed that I didn't really answer your question about what I meant by using a library. What I mean is, use one of the more full-featured job queueing libraries like Celery or Resque, rather than trying to roll your own on top of something like RabbitMQ.

memset 14y ago
Ah, okay, so something like Celery would handle all of those kinds of cases for you. (Though... doesn't rabbitmq have many of those features built-in? Queue server replication and redundancy and stuff? Though I suppose a dedicated project would handle more of those calamities than I could think of on my own.) Thanks for that!
- Ixiaus 14y ago
  RabbitMQ doesn't have all that built in at all. RabbitMQ is only an AMQP message broker/exchange/queue agent. It's hard to articulate (and probably should be done in a blog post) all the things that can go wrong, but rest assured, there are a lot of things that can go wrong with RabbitMQ AND with your worker nodes that are built on-top of RabbitMQ. Distributed workloads is a hard problem, but it is a solved problem (for most applications, there are some exceptions thought too).