4 ms·
Author here. Some comments to the comments: Yes, I do effectively argue for (re-)implementing a queue system (for certain situations), even though you already
by stolsvik 4y ago
Author here. Some comments to the comments:
Yes, I do effectively argue for (re-)implementing a queue system (for certain situations), even though you already employ a message-oriented architecture! But this is not necessarily meant as an explicit, separate DB table: If you have an orders table, you can use that table as your "Work queue". So, if some event makes 10k orders fulfillable, you will now have to fulfill these, and if you use a queue system, that might entail putting one message on the queue for each order. You could now choose to stick all those 10k jobs on a queue in one go, or you could pace them out. I argue that I would go for the explicit pacing if the fulfill process is complex, consisting of multiple hops between systems, and hence queues - and I try to put forward some arguments in the article.
And yes, the "fabric" of messaging endpoints should really be able to handle any number of messages on the queue. The downstream processors should already have their concurrency (number of parallel consumers/executors) tuned so that they do not trash the external resources like databases. However, several of those sub processes, or the external resources they touch (e.g .databases), are probably used by other processes during the day. Some of these might be human interactive like looking up a customer. By throwing out all the orders on the fabric in one go, you might still make the system annoyingly sluggish due to interaction between all the concurrent jobs: There will probably be a difference between "flat out" processing of hundreds of thousands of jobs, and what happens otherwise throughout the day. Having an ability to control this from the top is nice. And I do point out ways to achieve such pacing using just the MQ (e.g. use a choke-point to pace), but I would still prefer explicit pacing, i.e. explicitly put smaller subsets of the total 10k orders onto the queue, since this also gives other nice effects:
The most important points are the ones about introspection, monitor and control. These are elements which I feel are essential to feeling confident about the system health, and in particular for being confident when you for example make larger refactorings to the process. Yes, you can have monitoring and introspection by having the order table be updated by the processing of each order (i.e. update some field, describing which stage of the processing it is in, and if you get error conditions). But by just throwing all the orders on the messaging fabric in one go, your reduce your ability to explicitly halt or abort if some day things start failing, or to easily tune down the pacing after a large refactoring just to be extra careful the first time it runs in prod. You can even throw in an automatic halting: If e.g. more than 100, or 2% of current orders, fail processing, then stop the issuer. Yes, you can use your logs to see what went wrong, but it is nicer to have just a few such processes to understand, than 10k processes that failed, with redelivery-retries kicking in, and overall general massive mess.
Wrt. the argument that the messaging system could give you this control: None of the classic MQ systems, like ActiveMQ, Artemis or RabbitMQ, gives you good domain-specific introspection abilities, that is, see the orders, customer info etc. And that is how I like my MQ: I want it to be a extremely efficient and robust, but rather dumb, message broker. I want it to pass messages. I do not want the message broker to orchestrate and control my processes - that I want to do myself. Mats3 (https://mats3.io/ https://mats3.io/) is an attempt to make such orchestrating of multiple steps simpler when using messaging - as simple as coding it in a sequential, top-to-bottom coding style, while still having each step be fully independent and asynchronous, reaping the benefits of a message-oriented architecture.