3 ms·
> Thanks for the article, very interesting. Thanks! > What we have implemented was a Python decorator to have the tasks actually fired after the function comp
by brandur 11y ago
> Thanks for the article, very interesting.
Thanks!
> What we have implemented was a Python decorator to have the tasks actually fired after the function completion (thus after the transaction commit). In Go we achieve the same result in a more simple way using the defer statement. In my experience, all these solutions are local to the process that fires the task, so there is less risk of interaction with other process, easy to implement and more robust compared to other solution.
Oh yes, totally. I've seen this same pattern in Ruby before whereby a job enqueue is put on something like an ActiveRecord `after_commit` hook.
One (overly) pedantic observation is that this still leaves you with the possibility of having your transaction and data safely committed in your database, but with your job not enqueued if something happens to your process between the time of commit and time of enqueue. Admittedly though, this probably doesn't happen all that often in real life.
Probably the best answer I have is that we take this approach for the sheer convenience. We can do things like this:
def create_user(email, password)
User.transaction do
user = User.new email: email
user.set_password_hash(password)
check_for_abuse!
# make an account ID queue job to create record in billing system
user.billing_account_id = uuid()
async_create_account_in_billing_system(user)
# queue job to send an e-mail invite
async_send_invite_email(user)
create_auditing_record
...
end
end
You can build out very explicit chains of actions that may enqueue jobs or call into other chains that may enqueue their own jobs and all the while never have to worry about any kind of ordering problem while working anything. There are no hidden callbacks anywhere, but you still get to keep perfect correctness: if anything fails at any point, the whole system rolls back like it never happened.
- LukaAl 11y agoThe observation on failure modes is correct. When you process thousands of tasks every day even problems with low probability happens. What bugs me with the transaction approach is that you loose all the information (except probably for logs). In your example, if I signup on your service and for any reason the process fails (maybe after I received the confirmation that everything is ok) I will end up having no record stored for my account. This is problematic for post mortem diagnosis, for customer support (although its easy to ask the customer redo the signup) and so for and so on. Imagine you are handling payment with an external provider (e.g: Paypal). I could end up being billed without a trace on your system that I've paid. I'm not saying that my approach is correct. I'm saying that just assuming that transactions will save all your problems discount the fact that your system, to be useful, has side effects (not using the term in a strict way) on your customer and possibly on systems outside your organization. I prefer to plan for actively manage the possible failure modes and to detect and correct quickly anomaly. For this reason inconsistent data is sometime better than no data. Another point on the database approach that I haven't thought before: In the past we designed a system that stored and updated user timelines on MySQL. It has always been a nightmare. SQL databases are not designed with an high ratio of write operation in mind. The indexes get quickly fragmented, even deleting entries doesn't immediately deflate the disk consumption etc. I don't see this being a problem immediately applicable to a queue use case. But with your service growing in size there's a risk of hitting scalability problem. Obviously you could react like we have done, use a bigger database, use better disks, create cleanup jobs. But it is bad software engineering and you are just buying more time.