3 ms·
The observation on failure modes is correct. When you process thousands of tasks every day even problems with low probability happens. What bugs me with the tra
by LukaAl 11y ago
The observation on failure modes is correct. When you process thousands of tasks every day even problems with low probability happens.
What bugs me with the transaction approach is that you loose all the information (except probably for logs). In your example, if I signup on your service and for any reason the process fails (maybe after I received the confirmation that everything is ok) I will end up having no record stored for my account. This is problematic for post mortem diagnosis, for customer support (although its easy to ask the customer redo the signup) and so for and so on. Imagine you are handling payment with an external provider (e.g: Paypal). I could end up being billed without a trace on your system that I've paid. I'm not saying that my approach is correct. I'm saying that just assuming that transactions will save all your problems discount the fact that your system, to be useful, has side effects (not using the term in a strict way) on your customer and possibly on systems outside your organization. I prefer to plan for actively manage the possible failure modes and to detect and correct quickly anomaly. For this reason inconsistent data is sometime better than no data.
Another point on the database approach that I haven't thought before: In the past we designed a system that stored and updated user timelines on MySQL. It has always been a nightmare. SQL databases are not designed with an high ratio of write operation in mind. The indexes get quickly fragmented, even deleting entries doesn't immediately deflate the disk consumption etc. I don't see this being a problem immediately applicable to a queue use case. But with your service growing in size there's a risk of hitting scalability problem. Obviously you could react like we have done, use a bigger database, use better disks, create cleanup jobs. But it is bad software engineering and you are just buying more time.