4 ms·

Thanks for the article, very interesting. Just one point on the solution. To my understanding, you prefer a queue implemented on a DB because transactions guara

by LukaAl 11y ago

Thanks for the article, very interesting. Just one point on the solution. To my understanding, you prefer a queue implemented on a DB because transactions guarantee that a task could not start unless the task before as succeeded and committed the required data on the DB. In our environment we run tasks on RabbitMQ/Celery. One of the nice feature that I believe exists also in Sidekiq is that it allows you to create chain (or more complex structure) of task and the worker itself will take care of synchronizing them removing your problem (when a task finishes successfully it commits on the db and triggers the next step). The only problem we had where on entry points were we create the task chains and we fire them before committing the transaction. One solution was manually committing before firing the task. But that was somewhat difficult. What we have implemented was a Python decorator to have the tasks actually fired after the function completion (thus after the transaction commit). In Go we achieve the same result in a more simple way using the defer statement. In my experience, all these solutions are local to the process that fires the task, so there is less risk of interaction with other process, easy to implement and more robust compared to other solution.

brandur 11y ago
> Thanks for the article, very interesting. Thanks! > What we have implemented was a Python decorator to have the tasks actually fired after the function completion (thus after the transaction commit). In Go we achieve the same result in a more simple way using the defer statement. In my experience, all these solutions are local to the process that fires the task, so there is less risk of interaction with other process, easy to implement and more robust compared to other solution. Oh yes, totally. I've seen this same pattern in Ruby before whereby a job enqueue is put on something like an ActiveRecord `after_commit` hook. One (overly) pedantic observation is that this still leaves you with the possibility of having your transaction and data safely committed in your database, but with your job not enqueued if something happens to your process between the time of commit and time of enqueue. Admittedly though, this probably doesn't happen all that often in real life. Probably the best answer I have is that we take this approach for the sheer convenience. We can do things like this: def create_user(email, password) User.transaction do user = User.new email: email user.set_password_hash(password) check_for_abuse! # make an account ID queue job to create record in billing system user.billing_account_id = uuid() async_create_account_in_billing_system(user) # queue job to send an e-mail invite async_send_invite_email(user) create_auditing_record ... end end You can build out very explicit chains of actions that may enqueue jobs or call into other chains that may enqueue their own jobs and all the while never have to worry about any kind of ordering problem while working anything. There are no hidden callbacks anywhere, but you still get to keep perfect correctness: if anything fails at any point, the whole system rolls back like it never happened.
- LukaAl 11y ago
  The observation on failure modes is correct. When you process thousands of tasks every day even problems with low probability happens. What bugs me with the transaction approach is that you loose all the information (except probably for logs). In your example, if I signup on your service and for any reason the process fails (maybe after I received the confirmation that everything is ok) I will end up having no record stored for my account. This is problematic for post mortem diagnosis, for customer support (although its easy to ask the customer redo the signup) and so for and so on. Imagine you are handling payment with an external provider (e.g: Paypal). I could end up being billed without a trace on your system that I've paid. I'm not saying that my approach is correct. I'm saying that just assuming that transactions will save all your problems discount the fact that your system, to be useful, has side effects (not using the term in a strict way) on your customer and possibly on systems outside your organization. I prefer to plan for actively manage the possible failure modes and to detect and correct quickly anomaly. For this reason inconsistent data is sometime better than no data. Another point on the database approach that I haven't thought before: In the past we designed a system that stored and updated user timelines on MySQL. It has always been a nightmare. SQL databases are not designed with an high ratio of write operation in mind. The indexes get quickly fragmented, even deleting entries doesn't immediately deflate the disk consumption etc. I don't see this being a problem immediately applicable to a queue use case. But with your service growing in size there's a risk of hitting scalability problem. Obviously you could react like we have done, use a bigger database, use better disks, create cleanup jobs. But it is bad software engineering and you are just buying more time.
lobster_johnson 11y ago
I actually wrote about this problem recently [1]. (Note that in that comment I'm not describing a job system, but rather a system for keeping one or more secondary stores in sync with a master database.) If you're using an external queue such as RabbitMQ, there's basically no way to guarantee a queue message being fired without a "two-phase" transactional approach. Let's say your use case is creating a user and sending an email verification email. You need to: 1. Start transaction. 2. Create the user. 3. Insert a tuple into an "events to fire" table. 4. Commit. 5. Fire the event. 6. Delete the tuple. This ensures that if the transaction fails, nothing is fired; if the transaction succeeds, we guarantee that the user now exists, and that the command to fire the event also exists. Thus, if step 5 fails, we still have the "events to fire" tuple, and we can have a cron job running every minute that picks up stragglers. (In practice, you'll need to run steps 5-6 in a transaction that exclusively locks the tuple so that a concurrent cron job doesn't do it.) This way, you're risking duplicate events, but you'll never lose the event, unless RabbitMQ falls over. Duplicate events are of course unfortunate and undesirable, but missing events are worse. Of course, with the above solution, it could be argued that your "events to fire" table is a queue itself. The good news is that RabbitMQ is much better at scaling a queue than Postgres, so the duplication of work isn't actually that much of a problem. The "events to fire" table can be optimized in various ways; Postgres supports "unlogged" tables, for example. Another approach which will be available soon is Bottled Water [2], which gives you pretty much the above, except without needing to maintain the "events to fire" table. But it's definitely more complicated. [1] https://news.ycombinator.com/item?id=9490583 https://news.ycombinator.com/item?id=9490583 [2] http://blog.confluent.io/2015/04/23/bottled-water-real-time-integration-of-postgresql-and-kafka/ http://blog.confluent.io/2015/04/23/bottled-water-real-time-...
- adamtj 11y ago
  Unlogged tables aren't durable and might lose events even after a successful commit. If you're willing to make that trade off, then you don't need Postgres in the first place.
  - lobster_johnson 11y ago
    Sure, which is why I said it would be an optimization. But if your PostgreSQL server dies in the middle of that transaction, I would be more worried about RabbitMQ.

by LukaAl 11y ago

brandur 11y ago

> Thanks for the article, very interesting. Thanks! > What we have implemented was a Python decorator to have the tasks actually fired after the function completion (thus after the transaction commit). In Go we achieve the same result in a more simple way using the defer statement. In my experience, all these solutions are local to the process that fires the task, so there is less risk of interaction with other process, easy to implement and more robust compared to other solution. Oh yes, totally. I've seen this same pattern in Ruby before whereby a job enqueue is put on something like an ActiveRecord `after_commit` hook. One (overly) pedantic observation is that this still leaves you with the possibility of having your transaction and data safely committed in your database, but with your job not enqueued if something happens to your process between the time of commit and time of enqueue. Admittedly though, this probably doesn't happen all that often in real life. Probably the best answer I have is that we take this approach for the sheer convenience. We can do things like this: def create_user(email, password) User.transaction do user = User.new email: email user.set_password_hash(password) check_for_abuse! # make an account ID queue job to create record in billing system user.billing_account_id = uuid() async_create_account_in_billing_system(user) # queue job to send an e-mail invite async_send_invite_email(user) create_auditing_record ... end end You can build out very explicit chains of actions that may enqueue jobs or call into other chains that may enqueue their own jobs and all the while never have to worry about any kind of ordering problem while working anything. There are no hidden callbacks anywhere, but you still get to keep perfect correctness: if anything fails at any point, the whole system rolls back like it never happened.

LukaAl 11y ago
The observation on failure modes is correct. When you process thousands of tasks every day even problems with low probability happens. What bugs me with the transaction approach is that you loose all the information (except probably for logs). In your example, if I signup on your service and for any reason the process fails (maybe after I received the confirmation that everything is ok) I will end up having no record stored for my account. This is problematic for post mortem diagnosis, for customer support (although its easy to ask the customer redo the signup) and so for and so on. Imagine you are handling payment with an external provider (e.g: Paypal). I could end up being billed without a trace on your system that I've paid. I'm not saying that my approach is correct. I'm saying that just assuming that transactions will save all your problems discount the fact that your system, to be useful, has side effects (not using the term in a strict way) on your customer and possibly on systems outside your organization. I prefer to plan for actively manage the possible failure modes and to detect and correct quickly anomaly. For this reason inconsistent data is sometime better than no data. Another point on the database approach that I haven't thought before: In the past we designed a system that stored and updated user timelines on MySQL. It has always been a nightmare. SQL databases are not designed with an high ratio of write operation in mind. The indexes get quickly fragmented, even deleting entries doesn't immediately deflate the disk consumption etc. I don't see this being a problem immediately applicable to a queue use case. But with your service growing in size there's a risk of hitting scalability problem. Obviously you could react like we have done, use a bigger database, use better disks, create cleanup jobs. But it is bad software engineering and you are just buying more time.

lobster_johnson 11y ago

I actually wrote about this problem recently [1]. (Note that in that comment I'm not describing a job system, but rather a system for keeping one or more secondary stores in sync with a master database.) If you're using an external queue such as RabbitMQ, there's basically no way to guarantee a queue message being fired without a "two-phase" transactional approach. Let's say your use case is creating a user and sending an email verification email. You need to: 1. Start transaction. 2. Create the user. 3. Insert a tuple into an "events to fire" table. 4. Commit. 5. Fire the event. 6. Delete the tuple. This ensures that if the transaction fails, nothing is fired; if the transaction succeeds, we guarantee that the user now exists, and that the command to fire the event also exists. Thus, if step 5 fails, we still have the "events to fire" tuple, and we can have a cron job running every minute that picks up stragglers. (In practice, you'll need to run steps 5-6 in a transaction that exclusively locks the tuple so that a concurrent cron job doesn't do it.) This way, you're risking duplicate events, but you'll never lose the event, unless RabbitMQ falls over. Duplicate events are of course unfortunate and undesirable, but missing events are worse. Of course, with the above solution, it could be argued that your "events to fire" table is a queue itself. The good news is that RabbitMQ is much better at scaling a queue than Postgres, so the duplication of work isn't actually that much of a problem. The "events to fire" table can be optimized in various ways; Postgres supports "unlogged" tables, for example. Another approach which will be available soon is Bottled Water [2], which gives you pretty much the above, except without needing to maintain the "events to fire" table. But it's definitely more complicated. [1] https://news.ycombinator.com/item?id=9490583 https://news.ycombinator.com/item?id=9490583 [2] http://blog.confluent.io/2015/04/23/bottled-water-real-time-integration-of-postgresql-and-kafka/ http://blog.confluent.io/2015/04/23/bottled-water-real-time-...

adamtj 11y ago
Unlogged tables aren't durable and might lose events even after a successful commit. If you're willing to make that trade off, then you don't need Postgres in the first place.
- lobster_johnson 11y ago
  Sure, which is why I said it would be an optimization. But if your PostgreSQL server dies in the middle of that transaction, I would be more worried about RabbitMQ.