2 ms·

With a real-time database, each worker could subscribe to be notified when the set of tasks changed, but it would require a separate locking mechanism in the ap

by paulgb 4y ago

With a real-time database, each worker could subscribe to be notified when the set of tasks changed, but it would require a separate locking mechanism in the application to ensure that each task is only attempted by one worker. (Imagine a scenario in which a task arrives when there are multiple workers idle.) With RabbitMQ, you ensure* that each task is only attempted by one worker at a time, and you don't have to do anything special to ensure that at the application level. *I'm simplifying a bit, there are edge cases where e.g. you lose a worker that has already started a task but not completed it.

jgalentine007 4y ago
Interestingly enough I picked RabbitMQ for the same reasons for a distributed execution system I designed at work. RabbitMQ provides a lot of nice to have features and flexibility when it comes to delivering messages between endpoints. The fact that you get access control, easy encryption, multiple protocols (mqtt/amqp/stomp etc), excellent reliability and scalability rolled into a mature system is pretty awesome. The concepts are easy to understand and get real code working. The interfaces with RabbitMQ are simple enough that you could replace it with something else without having to untangle a whole lot. The other killer feature for me was async event listening which cuts out a lot of overhead from polling / long polling.