4 ms·

I disagree with the characterization in #1 (although I can't speak to the Celery particulars). I feel like if you have a job that is critical to your business

I disagree with the characterization in #1 (although I can't speak to the Celery particulars). I feel like if you have a job that is critical to your business process, the job should be persisted to your database and created within the same database transaction as whatever is kicking off the job. Consider how background jobs are typically managed with RabbitMQ, Redis, etc. They are usually created in an "after commit" hook from whatever gets persisted to your relational database. In this scenario, there is a gap between the database transaction being committed and the job being sent to and persisted by RabbitMQ or Redis; during this gap the only record of that task is being held in a process's memory. If this process gets killed suddenly during this gap, that background job will be lost forever. It sounds unlikely, but if RabbitMQ or Redis is down and the process has to sit and retry, waiting for them to come back online, the gap can be sizable.

denibertovic 12y ago
I think you're missing the point. The Celery (or any task queue really) particulars are very important here, cause you don't want background workers hammering your database if they don't need to. Cause the workers wan't work with a AMQP implementation, which the database is not. It's like using a fork instead of a hammer, sure you might get a few nails but it's not the right tool for the job. The systems that use these kinds of tools are usually not structured in a way that they need to wait for something in the database to be stored. By nature they are async tasks and they should be able to run whenever and return sometime in the future, and they will most likely produce some kind of result in the database, so there is no reason to store the job information itself in the database. Jobs are usually not created as hooks after a database commit, so jobs being persisted with database transactions is not quite relevant and Celery has failure mechanism and ways to recover if it was not able to send a task to the broker (ie. RabbitMQ was down). Redis and RabbitMQ do have a mechanism of persisting jobs onto disk as well so they don't get lost when the process is restarted. So there is no way that a job get's lost forever as you say, if you handle all these cases correctly. One more thing, Python's database drivers don't work quite as you've described. Namely they don't (by design) make use of the autocommit feature of the database engine, rather they wrap every sql statement in a transaction, so either way each statement get's executed separately in it's own transaction. This would not guarantee, let's say a db record being added and the job being saved as well. You would have to use explicit atomic blocks (something a kin to what Django >= 1.6 has) to get both things or none to be persisted.
- jaegerpicker 12y ago
  My reply should be closer to this, I'm on painkillers a little foggy but this reply is correct. The db record being saved does not guarantee the job will be saved. Particularly with Django.
- waffle_ss 12y ago
  I'm coming at it from the Ruby angle, in which jobs are often triggered using ActiveRecord after_commit hooks. I admit to being ignorant of the Python/Celery way of doing things so perhaps I am missing the point. I'm talking about jobs being produced atomically with the data that necessitated the background job (I realize not all background jobs are spawned in this fashion). I agree with your point about polling being bad, however as someone pointed out below it's not an issue with Postgres's LISTEN/NOTIFY (and I added a note to the queue_classic gem which makes this easy to take advantage of in MRI Ruby). Obviously I'm aware that Redis and RabbitMQ persist jobs. That's not what I was talking about at all. I think we're on different wavelengths here so I'll let it be. :-)
  - denibertovic 12y ago
    Could be, wavelengths thing I mean. :) I'm not aware of how ruby does it other then hearing about 2 most popular solutions that are used with rails. As for Postgres's PUB/SUB I've replied to the reply that mentioned that and you can't really use that. Would be great if we cold update the broker driver and see if we could get support though.
jaegerpicker 12y ago
I disagree with this, in my experience it's almost always a really bad idea to use the DB as a queue. If rabbitmq is down the process should retry a finite amount of times (usually 3 in our use case) then set a status on the db record. Then you have audits running to pick up records in that state and retry the process once the system is back up and running. That way nothing is lost and you gain all of the benefits of Rabbitmq.
- denibertovic 12y ago
  Precisely. Tnx jaegerpicker.
- waffle_ss 12y ago
  So you are storing a job in the database then - a job whose job it is to send a job to RabbitMQ/Redis :-) Honestly I think that's the ideal way of doing things, however that's not often how you see it done.
  - denibertovic 12y ago
    No, you are not storing the job itself, just a flag that indicates that the job was not done/or even stored. What's more, there really are various ways to handle failure of storing a job on a queue, but in most cases it's just the absence of the desired effect that can be a giveaway. Ie. you know that the job either didn't finish or wasn't even sent to the queue if it's effect (ie. updating the user's friend list from facebook) was not done. Also, take a look at #5 for a great monitoring stats solution.
    - waffle_ss 12y ago
      > No, you are not storing the job itself, just a flag that indicates that the job was not done/or even stored You must be storing enough representation of the job to re-send it though, no? You couldn't do this with a simple flag for all job types I imagine. > in most cases it's just the absence of the desired effect that can be a giveaway Sure, but not all jobs create side effects you can easily look for (e.g. sending an email). And secondly, you'd have to create a "sweeper" process for every type of job you create then.

denibertovic 12y ago

I think you're missing the point. The Celery (or any task queue really) particulars are very important here, cause you don't want background workers hammering your database if they don't need to. Cause the workers wan't work with a AMQP implementation, which the database is not. It's like using a fork instead of a hammer, sure you might get a few nails but it's not the right tool for the job. The systems that use these kinds of tools are usually not structured in a way that they need to wait for something in the database to be stored. By nature they are async tasks and they should be able to run whenever and return sometime in the future, and they will most likely produce some kind of result in the database, so there is no reason to store the job information itself in the database. Jobs are usually not created as hooks after a database commit, so jobs being persisted with database transactions is not quite relevant and Celery has failure mechanism and ways to recover if it was not able to send a task to the broker (ie. RabbitMQ was down). Redis and RabbitMQ do have a mechanism of persisting jobs onto disk as well so they don't get lost when the process is restarted. So there is no way that a job get's lost forever as you say, if you handle all these cases correctly. One more thing, Python's database drivers don't work quite as you've described. Namely they don't (by design) make use of the autocommit feature of the database engine, rather they wrap every sql statement in a transaction, so either way each statement get's executed separately in it's own transaction. This would not guarantee, let's say a db record being added and the job being saved as well. You would have to use explicit atomic blocks (something a kin to what Django >= 1.6 has) to get both things or none to be persisted.

jaegerpicker 12y ago
My reply should be closer to this, I'm on painkillers a little foggy but this reply is correct. The db record being saved does not guarantee the job will be saved. Particularly with Django.
waffle_ss 12y ago
I'm coming at it from the Ruby angle, in which jobs are often triggered using ActiveRecord after_commit hooks. I admit to being ignorant of the Python/Celery way of doing things so perhaps I am missing the point. I'm talking about jobs being produced atomically with the data that necessitated the background job (I realize not all background jobs are spawned in this fashion). I agree with your point about polling being bad, however as someone pointed out below it's not an issue with Postgres's LISTEN/NOTIFY (and I added a note to the queue_classic gem which makes this easy to take advantage of in MRI Ruby). Obviously I'm aware that Redis and RabbitMQ persist jobs. That's not what I was talking about at all. I think we're on different wavelengths here so I'll let it be. :-)
- denibertovic 12y ago
  Could be, wavelengths thing I mean. :) I'm not aware of how ruby does it other then hearing about 2 most popular solutions that are used with rails. As for Postgres's PUB/SUB I've replied to the reply that mentioned that and you can't really use that. Would be great if we cold update the broker driver and see if we could get support though.

jaegerpicker 12y ago

I disagree with this, in my experience it's almost always a really bad idea to use the DB as a queue. If rabbitmq is down the process should retry a finite amount of times (usually 3 in our use case) then set a status on the db record. Then you have audits running to pick up records in that state and retry the process once the system is back up and running. That way nothing is lost and you gain all of the benefits of Rabbitmq.

denibertovic 12y ago
Precisely. Tnx jaegerpicker.
waffle_ss 12y ago
So you are storing a job in the database then - a job whose job it is to send a job to RabbitMQ/Redis :-) Honestly I think that's the ideal way of doing things, however that's not often how you see it done.
- denibertovic 12y ago
  No, you are not storing the job itself, just a flag that indicates that the job was not done/or even stored. What's more, there really are various ways to handle failure of storing a job on a queue, but in most cases it's just the absence of the desired effect that can be a giveaway. Ie. you know that the job either didn't finish or wasn't even sent to the queue if it's effect (ie. updating the user's friend list from facebook) was not done. Also, take a look at #5 for a great monitoring stats solution.
  - waffle_ss 12y ago
    > No, you are not storing the job itself, just a flag that indicates that the job was not done/or even stored You must be storing enough representation of the job to re-send it though, no? You couldn't do this with a simple flag for all job types I imagine. > in most cases it's just the absence of the desired effect that can be a giveaway Sure, but not all jobs create side effects you can easily look for (e.g. sending an email). And secondly, you'd have to create a "sweeper" process for every type of job you create then.