6 ms·
Initially when developing Alchemy we had a look at NSQ http://nsq.io http://nsq.io and found it a little difficult (that was a year or so ago, so might be bette
by grahar64 11y ago
Initially when developing Alchemy we had a look at NSQ http://nsq.io http://nsq.io and found it a little difficult (that was a year or so ago, so might be better now). Then we started looking at RabbitMQ and thought it fit our requirements better. I have not heard of NATS but will definitely have a look see. At the moment Alchemy is tied to RabbitMQ pretty tightly, but abstracting and supporting many queue solutions would be good.
I don't like pushing the retry into the client side. Since microservice by necessity have lots of communication between them that can be quite a bit of code across all the services. I would rather the architecture deal with it and just ensure that endpoints are idempotent so calls can be retried without adding client complexity. This is a personal preference, and in some cases clients do need to deal with retires, but I just like it not to be the default.
In the services folder there is this https://github.com/LoyaltyNZ/alchemy-framework/blob/master/services/zglobal_rabbitmq.service https://github.com/LoyaltyNZ/alchemy-framework/blob/master/s... pretty good RabbitMQ HA setup, with:
1. cluster_partition_handling set to autoheal so that it will do its best to recover (a few lost messages is infinitely better than a broken system)
2. queue_master_locator is min-master, so queues are mastered on nodes where the least amount of other queues are mastered. This will balance the queues across the clusters meaning if a node goes down then there will be minimal amount of queues to recreate
3. A mirror policy to mirror every queue (this will only mirror service queues because response queues are exclusive) , this will make the system a bit slower, but makes it much more robust.
This is enough to handle split brain (although this is difficult to test) and nodes going down and coming back (much easier to test).
Cheers for the comment :)
- spotman 11y agoJust curious, what did you find difficult about NSQ? I moved one large project from RabbitMQ to NSQ over a year ago and haven't looked back. It has just been wonderful to work with and build on top of. Anything you didn't like about NSQ or pitfalls to watch out for from your experience?
- deepanchor 11y agoCurious about this too. I've found NSQ to be rock solid and easy to setup and work with.
- lobster_johnson 11y agoDestructive cluster recovery, patched with mirroring raises the question of how suitable RabbitMQ is for the job in the first place. Mirroring for RPC requests, think about it for a second! Mirroring. For RPC! RabbitMQ's autohealing just solves your problem in the wrong way. Yes, it will usually fix itself (if it doesn't die with a Mnesia inconsistent_database error), but it will discard messages, and you won't know which ones. Meanwhile, NATS will forward messages to subscribers as long as there's a clear path. There are no network partition issues because the queues don't have RabbitMQ's strict, total ordering. Note that RabbitMQ is notoriously sensitive to partitions; one small blip and it gets its knickers in a twist. This is why I recommend increasing the net_ticktime option to something like 180 so you're less exposed. Having done this for a long time, my advice is that making the client more intelligent is always the better option. If you're relunctant, consider a sidecar proxy like Linkerd [1] which can handle the gritty details for you. [1] https://linkerd.io/ https://linkerd.io/