5 ms·

> I've never seen this in years of dealing with RabbitMQ. Did you do long running jobs like they did? It's a stereotype, but I don't think they used the techno

> I've never seen this in years of dealing with RabbitMQ. Did you do long running jobs like they did? It's a stereotype, but I don't think they used the technology correctly here -- you're not supposed to hold onto messages for hours before acknowledging. They should have used RabbitMQ just to kick off the job, immediately ACKing the request, and job tracking/completion handled inside... a database.

xahrepap 4y ago
I’ve used RabbitMq to do long running jobs. Jobs that take hours and hours to complete. Occasionally even 1-2 days. It did take some configuring to get it working. Between acking appropriately and the prefetch (qos perhaps? Can’t remember, don’t have it in front of me). We were able to make it work. It was pretty straightforward it never even crossed my mind that this isn’t a correct use case for RMQ. (Used the Java client.)
mark242 4y ago
The short answer is "yes" but the questions that you should be asking are: A) How long am I willing to block the queue for additional consumers, B) How committed am I to getting close to exactly-once processing, and C) how tolerant of consumer failure should I be? Depending on the answer to those three questions is what drives your queue architecture. Note that this has nothing to do with time spent processing messages or "long running jobs". Assume that your producers will be able to spike and generate messages faster than your consumers can process them. This is normal! This is why you have a queue in the first place! If your jobs take 5 seconds or 5 hours, your strategy is influenced by the answers to those three questions. For example -- if you're willing to drop a message if a consumer gets power-cycled, then yeah, you'd immediately ack the request and put it back onto a dead letter queue if your consumer runs into an exception. Alternatively, if you're unwilling to block and you want to be very tolerant of consumer failure, you'd fan out your queues and have your consumers checking multiple queues in parallel. Etc etc etc, you get the drift. Keep in mind also that this isn't specific to RabbitMQ! You'd want to answer the same questions if you were using SQS, or if you were using Kafka, or if you were using 0mq, or if you were using Redis queues, or if you were using pg queues.
- zibarn 4y ago
  Except sqs has a limit for unacked messages of 12 hours
  - js2 4y ago
    If I were designing a system with tasks that took that long, I'd probably want to find a way to checkpoint along the way and break the task down into smaller steps. I'd hate to lose 11 hours and 59 seconds of work if the task fails near the end. You can then use multiple queues to pass the task along between steps.
carrja99 4y ago
Bingo. This is exactly a setup I have seen before as well. Message recieved and acked, process kicked off, job state tracked in redis or similar.
hospadar 4y ago
at which point, if you've got to use a DB to track status, really why bother with the queuing system?
- 4dayworkweek4u 4y ago
  Queues are good for connecting separated systems (like 2 or more separate companies).
- TedDoesntTalk 4y ago
  When you’re dealing with billions of messages, i think queuing systems may be tuned more for it? I’d like to hear why people chose Kafka over some RDBMS tables.
  - RhodesianHunter 4y ago
    Performance and distributed nature.
  - jjeaff 4y ago
    Billions of jobs with hours long time to complete seems like something no one would have the resources for.
    - OOPMan 4y ago
      Just wait till a start convinces someone to give them money for a system that allows anyone, anywhere to queue up infinite loops XD
  - raverbashing 4y ago
    I'm going to be honest, I think some 80% of Kafka users are overengineering it. Kafka has specific use cases but it seems a lot of people just go "ok use Kafka here" and wait for the load that rarely comes
    - eurasiantiger 4y ago
petesergeant 4y ago
To me, the difference here is a job queue vs a message queue
fake-name 4y ago
I do something like this in some projects of mine. RabbitMQ is completely happy to do this. On the other hand, literally every python client library would fall over and/or wedge forever periodically. Celery in particular was a flaming pile of garbage (at least circa 2018). I wound up vendoring the best of the bunch (amqpstorm, fyi), and adding a bunch of additional logic to handle all the various corner cases. That, plus a bunch of careful thought about how the queues are structured, and I've been pushing millions of messages a day without issue. I wasn't able to get a lot of my changes upstream due to the extent I had to alter the assumptions the library made about control flow.
- OOPMan 4y ago
  Don't worry, Celery has always been and will always be a flaming pile of garbage
  - xmcqdpt2 4y ago
    For what it's worth, I found rq to be much nicer than celery but only used it in a relatively constrained fashion.
    - justinclift 4y ago
      rq doesn't interoperate with other languages though, so if you use it then your workers can only be written in Python. :(
valenterry 4y ago
> They should have used RabbitMQ just to kick off the job, immediately ACKing the request, and job tracking/completion handled inside... a database. That's what I think too. Use the message queue for communication and potentially work distribution and the database for the rest like internal state.

xahrepap 4y ago

I’ve used RabbitMq to do long running jobs. Jobs that take hours and hours to complete. Occasionally even 1-2 days. It did take some configuring to get it working. Between acking appropriately and the prefetch (qos perhaps? Can’t remember, don’t have it in front of me). We were able to make it work. It was pretty straightforward it never even crossed my mind that this isn’t a correct use case for RMQ. (Used the Java client.)

mark242 4y ago

The short answer is "yes" but the questions that you should be asking are: A) How long am I willing to block the queue for additional consumers, B) How committed am I to getting close to exactly-once processing, and C) how tolerant of consumer failure should I be? Depending on the answer to those three questions is what drives your queue architecture. Note that this has nothing to do with time spent processing messages or "long running jobs". Assume that your producers will be able to spike and generate messages faster than your consumers can process them. This is normal! This is why you have a queue in the first place! If your jobs take 5 seconds or 5 hours, your strategy is influenced by the answers to those three questions. For example -- if you're willing to drop a message if a consumer gets power-cycled, then yeah, you'd immediately ack the request and put it back onto a dead letter queue if your consumer runs into an exception. Alternatively, if you're unwilling to block and you want to be very tolerant of consumer failure, you'd fan out your queues and have your consumers checking multiple queues in parallel. Etc etc etc, you get the drift. Keep in mind also that this isn't specific to RabbitMQ! You'd want to answer the same questions if you were using SQS, or if you were using Kafka, or if you were using 0mq, or if you were using Redis queues, or if you were using pg queues.

zibarn 4y ago
Except sqs has a limit for unacked messages of 12 hours
- js2 4y ago
  If I were designing a system with tasks that took that long, I'd probably want to find a way to checkpoint along the way and break the task down into smaller steps. I'd hate to lose 11 hours and 59 seconds of work if the task fails near the end. You can then use multiple queues to pass the task along between steps.

carrja99 4y ago

Bingo. This is exactly a setup I have seen before as well. Message recieved and acked, process kicked off, job state tracked in redis or similar.

hospadar 4y ago

at which point, if you've got to use a DB to track status, really why bother with the queuing system?

4dayworkweek4u 4y ago
Queues are good for connecting separated systems (like 2 or more separate companies).
TedDoesntTalk 4y ago
When you’re dealing with billions of messages, i think queuing systems may be tuned more for it? I’d like to hear why people chose Kafka over some RDBMS tables.
- RhodesianHunter 4y ago
  Performance and distributed nature.
- jjeaff 4y ago
  Billions of jobs with hours long time to complete seems like something no one would have the resources for.
  - OOPMan 4y ago
    Just wait till a start convinces someone to give them money for a system that allows anyone, anywhere to queue up infinite loops XD
- raverbashing 4y ago
  I'm going to be honest, I think some 80% of Kafka users are overengineering it. Kafka has specific use cases but it seems a lot of people just go "ok use Kafka here" and wait for the load that rarely comes
  - eurasiantiger 4y ago

petesergeant 4y ago

To me, the difference here is a job queue vs a message queue

fake-name 4y ago

I do something like this in some projects of mine. RabbitMQ is completely happy to do this. On the other hand, literally every python client library would fall over and/or wedge forever periodically. Celery in particular was a flaming pile of garbage (at least circa 2018). I wound up vendoring the best of the bunch (amqpstorm, fyi), and adding a bunch of additional logic to handle all the various corner cases. That, plus a bunch of careful thought about how the queues are structured, and I've been pushing millions of messages a day without issue. I wasn't able to get a lot of my changes upstream due to the extent I had to alter the assumptions the library made about control flow.

OOPMan 4y ago
Don't worry, Celery has always been and will always be a flaming pile of garbage
- xmcqdpt2 4y ago
  For what it's worth, I found rq to be much nicer than celery but only used it in a relatively constrained fashion.
  - justinclift 4y ago
    rq doesn't interoperate with other languages though, so if you use it then your workers can only be written in Python. :(

valenterry 4y ago

> They should have used RabbitMQ just to kick off the job, immediately ACKing the request, and job tracking/completion handled inside... a database. That's what I think too. Use the message queue for communication and potentially work distribution and the database for the rest like internal state.