3 ms·
Two suggestions: 1. RabbitMQ saves on some of the operational complexity, especially at low volume. I've used it for similarly durability-critical applications
by zbentley 9y ago
Two suggestions:
1. RabbitMQ saves on some of the operational complexity, especially at low volume. I've used it for similarly durability-critical applications with a combination of:
- Publisher Confirms.
- Durable queue and message metadata.
- Multiple nodes with HA (synchronous replication) in pause-minority mode (sacrifice availability when there's a partition).
- Fanout-duplicating messages into two queues, hooking up main consumers to one, and a periodic backup that drains the other to a backup location (a separate, non-replicated RabbitMQ via federation, or a text/db file or whatnot). This deals with RMQ's achilles heel, which is its synchronous replication system that can fail in interesting ways. Restoring a backup taken like this isn't automatic, but I've found that even adding a second decoupled RMQ instance into the mix is sufficient to significantly mitigate data loss due to disasters or RMQ failures/bugs.
All of those things will slow RMQ down a bit from its "crazy high throughput" use case, but your volume requirements are low, and the slowness will not be significant (at worst network time + 10s of ms rather than 1s of ms).
The configuration of each of those can be done mostly via idempotent messaging operations from your clients, with a bit of interaction (programmatic or manual) via the very user-friendly management HTTP API/site.
For even more delivery/execution confirmation, you can use more advanced messaging flows with AMQP transactions to get better assurance of whether or not a particular operation was completed or needs to be restored from backup.
2. Use a relational database. The reliability/transaction guarantees of something like Postgres are incredibly nice to have, simple to use, and incredibly widely supported even compared to things as popular as Kafka/JMS/NSQ/RMQ. At low volume, using a properly-tuned, replicated, dedicated database as a message bus or queue (via polling, triggers, or other Postgres features) tends to be almost as easy to set up as a real queue that prioritizes user-friendliness (like RabbitMQ), much easier to use and reason about in client code, and much more reliable.
Edit: syntax, missed words.