8 ms·

Alchemy Micro-Service Framework: Using RabbitMQ Instead of HTTP

149 points by grahar64 11y ago

kitwalker12 11y ago
this is helpful. we already use rabbitmq for our golang services. This would ease the task of setting up publishers and subscribers in the node APIs we're building.
atombender 11y ago
As someone who has used RabbitMQ in production for many years, you should rather consider using NATS [1] for RPC. RabbitMQ's high availability support is, frankly, terrible [2]. It's a single point of failure no matter how you turn it, because it cannot merge conflicting queues that result from a split-brain situation. Partitions can happen not just on network outage, but also in high-load situations. NATS is also a lot faster [3], and its client network protocol is so simple that you can implement a client in a couple hundred lines in any language. Compare to AMQP, which is complex, often implemented wrong, and requires a lot of setup (at the very least: declare exchanges, declare queues, then bind them) on the client side. NATS does topic-based pub/sub out of the box, no schema required. (Re performance, relying on ACK/NACK with RPC is a bad idea. The better solution is to move retrying into the client side and rely on timeouts, and of course error replies.) RabbitMQ is one of the better message queue implementations for scenarios where you need the bigger features it provides: durability (on-disk persistence), transactions, cross-data center replication (shovel/federation plugins), hierarchical topologies and so on. [1] http://nats.io http://nats.io [2] https://aphyr.com/posts/315-jepsen-rabbitmq https://aphyr.com/posts/315-jepsen-rabbitmq [3] http://bravenewgeek.com/dissecting-message-queues/ http://bravenewgeek.com/dissecting-message-queues/
- sixftmonster 11y ago
  Would this library be better off utilizing NATS instead of RabbitMQ baring any requirements a person might have for the persistence and other features you mentioned?
  - lobster_johnson 11y ago
    Yes, I think so.
- grahar64 11y ago
  Initially when developing Alchemy we had a look at NSQ http://nsq.io http://nsq.io and found it a little difficult (that was a year or so ago, so might be better now). Then we started looking at RabbitMQ and thought it fit our requirements better. I have not heard of NATS but will definitely have a look see. At the moment Alchemy is tied to RabbitMQ pretty tightly, but abstracting and supporting many queue solutions would be good. I don't like pushing the retry into the client side. Since microservice by necessity have lots of communication between them that can be quite a bit of code across all the services. I would rather the architecture deal with it and just ensure that endpoints are idempotent so calls can be retried without adding client complexity. This is a personal preference, and in some cases clients do need to deal with retires, but I just like it not to be the default. In the services folder there is this https://github.com/LoyaltyNZ/alchemy-framework/blob/master/services/zglobal_rabbitmq.service https://github.com/LoyaltyNZ/alchemy-framework/blob/master/s... pretty good RabbitMQ HA setup, with: 1. cluster_partition_handling set to autoheal so that it will do its best to recover (a few lost messages is infinitely better than a broken system) 2. queue_master_locator is min-master, so queues are mastered on nodes where the least amount of other queues are mastered. This will balance the queues across the clusters meaning if a node goes down then there will be minimal amount of queues to recreate 3. A mirror policy to mirror every queue (this will only mirror service queues because response queues are exclusive) , this will make the system a bit slower, but makes it much more robust. This is enough to handle split brain (although this is difficult to test) and nodes going down and coming back (much easier to test). Cheers for the comment :)
  - spotman 11y ago
    Just curious, what did you find difficult about NSQ? I moved one large project from RabbitMQ to NSQ over a year ago and haven't looked back. It has just been wonderful to work with and build on top of. Anything you didn't like about NSQ or pitfalls to watch out for from your experience?
    - deepanchor 11y ago
      Curious about this too. I've found NSQ to be rock solid and easy to setup and work with.
- chuhnk 11y ago
  +1 to this. Used rabbit extensively in production but now #10 on the list. Nats is #1 as a transport message bus in my opinion.
- bjflanne 11y ago
  Also felt this post by Diogo in the NATS community was interesting here: http://www.diogogmt.com/2016/02/08/benchmarking-nats-and-rest/ http://www.diogogmt.com/2016/02/08/benchmarking-nats-and-res...
  - lobster_johnson 11y ago
    Impressive numbers, though it seems he's testing Go vs. Node.js at the same time. I'd like to see performance numbers for HTTP/2, though.
    - diogogmt 11y ago
      HTTP/2 definitely makes it faster for browsers to load assets in a webpage, however, not sure how much it would speed up individual REST requests since most of the time is bound to the request/response round trip. Just clarifying the comment about the benchmark, even though there are both Go and Node.js components the actual bit that was being benchmarked was HTTP and NATS for the inter service communication. All the code is available on github if anybody wants to rerun the benchmarks.
      - lobster_johnson 11y ago
        HTTP/2's main feature is that it's multiplexed, which a client written in a suitably async-friendly language can exploit to pipeline parallel requests, or at least better reuse connections. There's presumably not too much performance gain (though header compression helps) if your client can't exploit the multiplexing.
- manigandham 11y ago
  While a simple HTTP interface is easy to code around, I quite like the AMQP protocol. It's fast, efficient, reliable and powerful. We currently use it to send hundreds of thousands of messages per second, large and small, around the world to different data centers and it always works smoothly. For anyone interested, Microsoft Channel 9 has a great 6 part series on the AMQP 1.0 protocol: https://channel9.msdn.com/Blogs/Subscribe/The-AMQP-10-Protocol-16-Overview https://channel9.msdn.com/Blogs/Subscribe/The-AMQP-10-Protoc...
  - lobster_johnson 11y ago
    Note that NATS is not HTTP, it's its own very simple text-based protocol. AMQP is nice, but I don't think anyone would categorize it as simple. It's binary, for one. And to read and write it, you have to deal with framing, which is not always easy to get right. For example, for a long time RabbitMQ had an issue with dead-letter exchanges (aka DLX), where each bounce would add a header to the message envelope. DLX is great for retries, but after a bunch of retries, a message could get quite large. Some clients (the Node.js client in particular) has a small limit on frame sizes and will throw on such messages rather than grow the buffer. (Fortunately, this header was fixed in a recent RabbitMQ version.)
    - manigandham 11y ago
      Yes, thanks for reminding me, NATS isnt HTTP. Didn't say AMQP is simple though. It's definitely not, but that's where the features and functionality come from. It seems the issues you described are with the broker and clients, not the protocol itself, which I find to be pretty solid.
      - lobster_johnson 11y ago
        AMQP also specifies the data model (exchanges, queues, bindings and so forth), which dictates the implementation of it. I find that data model a bit heavy-handed, but it's not terrible. However, it's not very suitable for RPC.
- lazyant 11y ago
  Setting Rabbitmq's "cluster_partition_handling" as "pause_minority" in theory ameliorates split-brain issues in clusters of 3 or more (odd numbers), where the majority would ignore the minority nodes.
- abrookewood 11y ago
  I spoke to Kyle Kingsbury (Aphyr) about 9 months ago and at that time, the only queue or pub/sub system he thought was relatively safe from partition errors was Kafka. Not sure if his position has changed recently.
  - StreamBright 11y ago
    This is also my personal experience with message queues, even though I haven't had a chance to work with NATS yet. Kafka is just a really solid piece of engineering when you need 5-50 servers. With that many servers you can handle millions of messages per second that usually enough for a mid size company. I am not sure about higher scale but I believe LinkedIN has much larger clusters.
  - lobster_johnson 11y ago
    Indeed, though this is irrelevant to this particular use case. NATS doesn't have replication, sharding or total ordering. Consistency is a challenge for clustered messaging brokers that need this. With NATS, queues are effectively sharded by node. If a node dies, its messages are lost. Incoming messages to the live nodes will still go to connected subscribers, and subscribers are expected to reconnect to the pool of available nodes. Once a previously dead node rejoins, it will start receiving messages. NATS in this case replaces something like HAProxy; a simple in-memory router of requests to backends.
    - abrookewood 11y ago
      Ahh .. OK, that makes it clearer. Thanks.
- deepanchor 11y ago
  NSQ and NATS are my goto tools for messaging, though NSQ seems more flexible to me because it supports message persistence and also provides NATS-like ephemeral channels for when persistence is not a hard requirement. And it comes with a shiny admin-dashboard, which NATS lacks. NATS is useful when raw performance is a priority. One thing I do find lacking in both of these queues is support for per-message TTL though, for pruning time sensitive messages. I'm not sure what the performance overhead would be for supporting something like that.
- StreamBright 11y ago
  Would Kafka be a better option in this case? There might be some properties I am not aware of that makes it impossible.
  - lyha 11y ago
    Kafka stores everything to disk, this may not be what you are looking for for your RPC calls (that you would make usually as a direct service to service HTTP call). Moreover kafka "topics" are statically declared (i.e. by admin scripts instead of a public API), and it's a heavy-weight operation. So it's not the best fit to have micro-services registering themselves and dynamically creating "topics" for each method.
  - lobster_johnson 11y ago
    No, Kafka is completely unsuitable for RPC, for several reasons. First, its data model shards queues into partitions, each of which can be consumed by just a single consumer. Assume we have partititions 1 and 2. P1 is empty, P2 has a ton of messages. You will now have one consumer C1 which is idle, while C2 is doing work. C1 can't take any of C2's work because it can only process its own partition. In other words: A single slow consumer can block a significant portion of the queue. Kafka is designed for fast (or at least evenly performant) consumers. Kafka's queues are also persisted on disk, which is terrible for RPC. Think of Kafka as a linear database that you can append to and read from sequentially. Its main use case is for data that can fan out into multiple parallel processing steps. For example, a log processing system that extracts metrics: You feed the Kafka queue into something like Apache Storm, which churns the data and emits counts into an RDBMS (for example).
olivemonkey 11y ago
IMO this is an emerging anti pattern to use Rabbit to connect "microservices". It often introduces a single point of failure to your "distributed" system and has problems with network partitions. If critical functionality stops working when Rabbit is down, you're probably doing it wrong.
- nugator 11y ago
  Do you have a better solution? Genuinely interested since I'm about to implement webservice data change event notifications using probably RabbitMQ.
  - eropple 11y ago
    It does require some up-front work, but I've generally moved over to Kafka for most of my stuff. Rabbit has some nice aspects, but as noted its HA options start at "dire" and escalate smoothly to "oh dear god no". Throughput with Kafka is very, very good and,in my experience, it's remarkably difficult to kill. If you want fully ephemeral topics, you can write the broker data to a tmpfs. But I really like keeping data around, because I can replay my topics later both for DR and for debugging.
    - slowmovintarget 11y ago
      Kafka makes reliable durable messaging not just possible, but solves all the usual attendant problems you try to design around. * Durable messages cause a slow-down when publishing, but not with Kafka because it uses the Linux kernel's page cache to write. * Durable messages are slow to read if they have to come from disk. Kafka is optimized to load sequential blocks into memory an push them out through a socket with very few copies. This makes for very fast reads. * Slow consumers can bog down the broker. Kafka stores all messages and keeps them on the topic for a time horizon. No back-pressure from slow consumers. * Disconnected but subscribed consumers cause messages to back up on disk and eventually clog the broker. Kafka stores all messages. There's no clogging or backup, that's just how it works. * Brokers must track whether a consumer actually received the message, failures can cause missed messages or clogs. Kafka clients may read from a given point on the topic forward. If they fail during a read, they just back up and read again. The messages will be there for hours/days/weeks as configured. With a rock-steady durable messaging system based on commit logs, all of those problems that arose from attempting to avoid durable messaging go away. Now you build microservices that emit and respond to events. Microservices that can "rehydrate" their state from private checkpoints and topic replays. And all of this with partition tolerance and simple mirroring. Although it isn't usually necessary, if you want, you can make all of that elastic with Mesos, too. Further reading: [1] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... [2] http://mesos.apache.org/ http://mesos.apache.org/
eeZi 11y ago
How does this compare to Autobahn / crossbar.io? http://autobahn.ws/ http://autobahn.ws/ http://crossbar.io/ http://crossbar.io/ I really like the service discovery in Crossbar ("routed RPC").
- grahar64 11y ago
  I had not heard of these before. But they look really interesting. I think they are pretty similar as it is a common pattern, and straight forward solution to many problems.
swang 11y ago
What are you using for general load balancing/availability? A consistent hashring?
- grahar64 11y ago
  I do not understand the question. In production we use an ELB hitting multiple routers, on multiple nodes. Not sure how hashring applies to load balancing.
eddd 11y ago
Looks pretty cool, but doing RPC over RMQ is quite expensive. I would rather see good abstractions over ZMQ - that would be really cool. Also as many people here pointed out, RMQ is a single point of failure which might be acceptable for some cases. To me, the problem is that by design this architecture has a bottleneck which is really hard to get rid off.
mugsie 11y ago
OpenStack has had this as a common design pattern across a lot of the services for a while. It works quite well for us now - there was a period 2-3 years ago where rabbitmq was the bane of my existence, and the cause of many a page but newer versions have been fine. We do assume that everything in the queue can go away, and only in flight calls will fail - and (at least in the service I work on) we do retries if something falls through the cracks. There is a shared library we use for it - http://docs.openstack.org/developer/oslo.messaging/ http://docs.openstack.org/developer/oslo.messaging/ It also has drivers for ZMQ, and QPID afaik.
- pm90 11y ago
  Yes, I have worked with the OpenStack messaging queue, and its really amazing how it just works. Also, easy to grab analytics about service usage by querying the queue directly, is amazing.
ajmurmann 11y ago
I've been working on something similar but decided to go with Kafka as the message bus because I want the messages to persist. This allows for more error recovery solutions and auditing. Still having all that data around also allows me to come up with new ideas to use that days after the fact.
jorgecurio 11y ago
as someone who's used RabbitMQ in a SaaS app to run millions of tasks, heed my warning, RabbitMQ is great if you are low on memory but the overhead was too much to deal with. Celery also contributed to this but if I could travel back in time, I would stop myself from using RabbitMQ. Instead, Redis is a much better alternative for micro-service.
- grahar64 11y ago
  Redis pub/sub (which I assume you are talking about as an alternative) is fine, but you have to be listening when the message is sent otherwise you miss it. Another problem is working out if where you sent it there are any listeners, which RabbitMQ deals with return queues. This is important for 404's when a client is trying to talk to a service that does not exist. In Redis you would just post to a pub/sub queue wait till nothing happens and timeout, rather than immediately knowing through a return queue.
  - lyha 11y ago
    Actually Redis returns the number of consumers that were subscribed during your publish and received the message, so you can detect that no service is available and return your 404 status. The real limitation for RPC over Redis pub/sub to me is that you can only broadcast your messages to all consumers, and not have them load-balanced between the subscribed applications. So you end-up following a discover/unicast call pattern where Redis doesn't bring much to the table for the actual RPC call. Maybe that's an opportunity for a cool addition to redis...
    - grahar64 11y ago
      Cheers, I didn't know that about Redis. I actually really like Redis and it simplicity, it would be cool to get more options for passing messages over Redis.
sunnya 11y ago
Check out http://restbus.org http://restbus.org , a similar project for .NET.
jhgg 11y ago
I feel like this is a bit silly. Feel like one could just speak HTTP or Thrift, and use etcd or zookeeper for discovery. RabbitMQ just complicates the stack. Also, it's HA options are generally... disappointing.
nitrogen 11y ago
This looks cool. I wrote a less developed than this and unreleased RabbitMQ-based microframework that combines RPC and one-way event streams to integrate an ecommerce site in one datacenter with a back office in another. At another company I used ZeroMQ to make a job processing cluster. Comments by others suggesting the use of http are kind of missing the point of message queues. The latency added by http(s) made interactive RPC sluggish, and made batch back office updates take hours instead of minutes with AMQP. If I had Alchemy two years ago I might have used this instead of rolling my own.
philip142au 11y ago
Use AKKA

149 points by grahar64 11y ago

this is helpful. we already use rabbitmq for our golang services. This would ease the task of setting up publishers and subscribers in the node APIs we're building.

atombender 11y ago

As someone who has used RabbitMQ in production for many years, you should rather consider using NATS [1] for RPC. RabbitMQ's high availability support is, frankly, terrible [2]. It's a single point of failure no matter how you turn it, because it cannot merge conflicting queues that result from a split-brain situation. Partitions can happen not just on network outage, but also in high-load situations. NATS is also a lot faster [3], and its client network protocol is so simple that you can implement a client in a couple hundred lines in any language. Compare to AMQP, which is complex, often implemented wrong, and requires a lot of setup (at the very least: declare exchanges, declare queues, then bind them) on the client side. NATS does topic-based pub/sub out of the box, no schema required. (Re performance, relying on ACK/NACK with RPC is a bad idea. The better solution is to move retrying into the client side and rely on timeouts, and of course error replies.) RabbitMQ is one of the better message queue implementations for scenarios where you need the bigger features it provides: durability (on-disk persistence), transactions, cross-data center replication (shovel/federation plugins), hierarchical topologies and so on. [1] http://nats.io http://nats.io [2] https://aphyr.com/posts/315-jepsen-rabbitmq https://aphyr.com/posts/315-jepsen-rabbitmq [3] http://bravenewgeek.com/dissecting-message-queues/ http://bravenewgeek.com/dissecting-message-queues/

sixftmonster 11y ago
Would this library be better off utilizing NATS instead of RabbitMQ baring any requirements a person might have for the persistence and other features you mentioned?
- lobster_johnson 11y ago
  Yes, I think so.
grahar64 11y ago
Initially when developing Alchemy we had a look at NSQ http://nsq.io http://nsq.io and found it a little difficult (that was a year or so ago, so might be better now). Then we started looking at RabbitMQ and thought it fit our requirements better. I have not heard of NATS but will definitely have a look see. At the moment Alchemy is tied to RabbitMQ pretty tightly, but abstracting and supporting many queue solutions would be good. I don't like pushing the retry into the client side. Since microservice by necessity have lots of communication between them that can be quite a bit of code across all the services. I would rather the architecture deal with it and just ensure that endpoints are idempotent so calls can be retried without adding client complexity. This is a personal preference, and in some cases clients do need to deal with retires, but I just like it not to be the default. In the services folder there is this https://github.com/LoyaltyNZ/alchemy-framework/blob/master/services/zglobal_rabbitmq.service https://github.com/LoyaltyNZ/alchemy-framework/blob/master/s... pretty good RabbitMQ HA setup, with: 1. cluster_partition_handling set to autoheal so that it will do its best to recover (a few lost messages is infinitely better than a broken system) 2. queue_master_locator is min-master, so queues are mastered on nodes where the least amount of other queues are mastered. This will balance the queues across the clusters meaning if a node goes down then there will be minimal amount of queues to recreate 3. A mirror policy to mirror every queue (this will only mirror service queues because response queues are exclusive) , this will make the system a bit slower, but makes it much more robust. This is enough to handle split brain (although this is difficult to test) and nodes going down and coming back (much easier to test). Cheers for the comment :)
- spotman 11y ago
  Just curious, what did you find difficult about NSQ? I moved one large project from RabbitMQ to NSQ over a year ago and haven't looked back. It has just been wonderful to work with and build on top of. Anything you didn't like about NSQ or pitfalls to watch out for from your experience?
  - deepanchor 11y ago
    Curious about this too. I've found NSQ to be rock solid and easy to setup and work with.
chuhnk 11y ago
+1 to this. Used rabbit extensively in production but now #10 on the list. Nats is #1 as a transport message bus in my opinion.
bjflanne 11y ago
Also felt this post by Diogo in the NATS community was interesting here: http://www.diogogmt.com/2016/02/08/benchmarking-nats-and-rest/ http://www.diogogmt.com/2016/02/08/benchmarking-nats-and-res...
- lobster_johnson 11y ago
  Impressive numbers, though it seems he's testing Go vs. Node.js at the same time. I'd like to see performance numbers for HTTP/2, though.
  - diogogmt 11y ago
    HTTP/2 definitely makes it faster for browsers to load assets in a webpage, however, not sure how much it would speed up individual REST requests since most of the time is bound to the request/response round trip. Just clarifying the comment about the benchmark, even though there are both Go and Node.js components the actual bit that was being benchmarked was HTTP and NATS for the inter service communication. All the code is available on github if anybody wants to rerun the benchmarks.
    - lobster_johnson 11y ago
      HTTP/2's main feature is that it's multiplexed, which a client written in a suitably async-friendly language can exploit to pipeline parallel requests, or at least better reuse connections. There's presumably not too much performance gain (though header compression helps) if your client can't exploit the multiplexing.
manigandham 11y ago
While a simple HTTP interface is easy to code around, I quite like the AMQP protocol. It's fast, efficient, reliable and powerful. We currently use it to send hundreds of thousands of messages per second, large and small, around the world to different data centers and it always works smoothly. For anyone interested, Microsoft Channel 9 has a great 6 part series on the AMQP 1.0 protocol: https://channel9.msdn.com/Blogs/Subscribe/The-AMQP-10-Protocol-16-Overview https://channel9.msdn.com/Blogs/Subscribe/The-AMQP-10-Protoc...
- lobster_johnson 11y ago
  Note that NATS is not HTTP, it's its own very simple text-based protocol. AMQP is nice, but I don't think anyone would categorize it as simple. It's binary, for one. And to read and write it, you have to deal with framing, which is not always easy to get right. For example, for a long time RabbitMQ had an issue with dead-letter exchanges (aka DLX), where each bounce would add a header to the message envelope. DLX is great for retries, but after a bunch of retries, a message could get quite large. Some clients (the Node.js client in particular) has a small limit on frame sizes and will throw on such messages rather than grow the buffer. (Fortunately, this header was fixed in a recent RabbitMQ version.)
  - manigandham 11y ago
    Yes, thanks for reminding me, NATS isnt HTTP. Didn't say AMQP is simple though. It's definitely not, but that's where the features and functionality come from. It seems the issues you described are with the broker and clients, not the protocol itself, which I find to be pretty solid.
    - lobster_johnson 11y ago
      AMQP also specifies the data model (exchanges, queues, bindings and so forth), which dictates the implementation of it. I find that data model a bit heavy-handed, but it's not terrible. However, it's not very suitable for RPC.
lazyant 11y ago
Setting Rabbitmq's "cluster_partition_handling" as "pause_minority" in theory ameliorates split-brain issues in clusters of 3 or more (odd numbers), where the majority would ignore the minority nodes.
abrookewood 11y ago
I spoke to Kyle Kingsbury (Aphyr) about 9 months ago and at that time, the only queue or pub/sub system he thought was relatively safe from partition errors was Kafka. Not sure if his position has changed recently.
- StreamBright 11y ago
  This is also my personal experience with message queues, even though I haven't had a chance to work with NATS yet. Kafka is just a really solid piece of engineering when you need 5-50 servers. With that many servers you can handle millions of messages per second that usually enough for a mid size company. I am not sure about higher scale but I believe LinkedIN has much larger clusters.
- lobster_johnson 11y ago
  Indeed, though this is irrelevant to this particular use case. NATS doesn't have replication, sharding or total ordering. Consistency is a challenge for clustered messaging brokers that need this. With NATS, queues are effectively sharded by node. If a node dies, its messages are lost. Incoming messages to the live nodes will still go to connected subscribers, and subscribers are expected to reconnect to the pool of available nodes. Once a previously dead node rejoins, it will start receiving messages. NATS in this case replaces something like HAProxy; a simple in-memory router of requests to backends.
  - abrookewood 11y ago
    Ahh .. OK, that makes it clearer. Thanks.
deepanchor 11y ago
NSQ and NATS are my goto tools for messaging, though NSQ seems more flexible to me because it supports message persistence and also provides NATS-like ephemeral channels for when persistence is not a hard requirement. And it comes with a shiny admin-dashboard, which NATS lacks. NATS is useful when raw performance is a priority. One thing I do find lacking in both of these queues is support for per-message TTL though, for pruning time sensitive messages. I'm not sure what the performance overhead would be for supporting something like that.
StreamBright 11y ago
Would Kafka be a better option in this case? There might be some properties I am not aware of that makes it impossible.
- lyha 11y ago
  Kafka stores everything to disk, this may not be what you are looking for for your RPC calls (that you would make usually as a direct service to service HTTP call). Moreover kafka "topics" are statically declared (i.e. by admin scripts instead of a public API), and it's a heavy-weight operation. So it's not the best fit to have micro-services registering themselves and dynamically creating "topics" for each method.
- lobster_johnson 11y ago
  No, Kafka is completely unsuitable for RPC, for several reasons. First, its data model shards queues into partitions, each of which can be consumed by just a single consumer. Assume we have partititions 1 and 2. P1 is empty, P2 has a ton of messages. You will now have one consumer C1 which is idle, while C2 is doing work. C1 can't take any of C2's work because it can only process its own partition. In other words: A single slow consumer can block a significant portion of the queue. Kafka is designed for fast (or at least evenly performant) consumers. Kafka's queues are also persisted on disk, which is terrible for RPC. Think of Kafka as a linear database that you can append to and read from sequentially. Its main use case is for data that can fan out into multiple parallel processing steps. For example, a log processing system that extracts metrics: You feed the Kafka queue into something like Apache Storm, which churns the data and emits counts into an RDBMS (for example).

olivemonkey 11y ago

IMO this is an emerging anti pattern to use Rabbit to connect "microservices". It often introduces a single point of failure to your "distributed" system and has problems with network partitions. If critical functionality stops working when Rabbit is down, you're probably doing it wrong.

nugator 11y ago
Do you have a better solution? Genuinely interested since I'm about to implement webservice data change event notifications using probably RabbitMQ.
- eropple 11y ago
  It does require some up-front work, but I've generally moved over to Kafka for most of my stuff. Rabbit has some nice aspects, but as noted its HA options start at "dire" and escalate smoothly to "oh dear god no". Throughput with Kafka is very, very good and,in my experience, it's remarkably difficult to kill. If you want fully ephemeral topics, you can write the broker data to a tmpfs. But I really like keeping data around, because I can replay my topics later both for DR and for debugging.
  - slowmovintarget 11y ago
    Kafka makes reliable durable messaging not just possible, but solves all the usual attendant problems you try to design around. * Durable messages cause a slow-down when publishing, but not with Kafka because it uses the Linux kernel's page cache to write. * Durable messages are slow to read if they have to come from disk. Kafka is optimized to load sequential blocks into memory an push them out through a socket with very few copies. This makes for very fast reads. * Slow consumers can bog down the broker. Kafka stores all messages and keeps them on the topic for a time horizon. No back-pressure from slow consumers. * Disconnected but subscribed consumers cause messages to back up on disk and eventually clog the broker. Kafka stores all messages. There's no clogging or backup, that's just how it works. * Brokers must track whether a consumer actually received the message, failures can cause missed messages or clogs. Kafka clients may read from a given point on the topic forward. If they fail during a read, they just back up and read again. The messages will be there for hours/days/weeks as configured. With a rock-steady durable messaging system based on commit logs, all of those problems that arose from attempting to avoid durable messaging go away. Now you build microservices that emit and respond to events. Microservices that can "rehydrate" their state from private checkpoints and topic replays. And all of this with partition tolerance and simple mirroring. Although it isn't usually necessary, if you want, you can make all of that elastic with Mesos, too. Further reading: [1] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... [2] http://mesos.apache.org/ http://mesos.apache.org/

eeZi 11y ago

How does this compare to Autobahn / crossbar.io? http://autobahn.ws/ http://autobahn.ws/ http://crossbar.io/ http://crossbar.io/ I really like the service discovery in Crossbar ("routed RPC").

grahar64 11y ago
I had not heard of these before. But they look really interesting. I think they are pretty similar as it is a common pattern, and straight forward solution to many problems.

swang 11y ago

What are you using for general load balancing/availability? A consistent hashring?

grahar64 11y ago
I do not understand the question. In production we use an ELB hitting multiple routers, on multiple nodes. Not sure how hashring applies to load balancing.

eddd 11y ago

Looks pretty cool, but doing RPC over RMQ is quite expensive. I would rather see good abstractions over ZMQ - that would be really cool. Also as many people here pointed out, RMQ is a single point of failure which might be acceptable for some cases. To me, the problem is that by design this architecture has a bottleneck which is really hard to get rid off.

mugsie 11y ago

OpenStack has had this as a common design pattern across a lot of the services for a while. It works quite well for us now - there was a period 2-3 years ago where rabbitmq was the bane of my existence, and the cause of many a page but newer versions have been fine. We do assume that everything in the queue can go away, and only in flight calls will fail - and (at least in the service I work on) we do retries if something falls through the cracks. There is a shared library we use for it - http://docs.openstack.org/developer/oslo.messaging/ http://docs.openstack.org/developer/oslo.messaging/ It also has drivers for ZMQ, and QPID afaik.

pm90 11y ago
Yes, I have worked with the OpenStack messaging queue, and its really amazing how it just works. Also, easy to grab analytics about service usage by querying the queue directly, is amazing.

ajmurmann 11y ago

I've been working on something similar but decided to go with Kafka as the message bus because I want the messages to persist. This allows for more error recovery solutions and auditing. Still having all that data around also allows me to come up with new ideas to use that days after the fact.

jorgecurio 11y ago

as someone who's used RabbitMQ in a SaaS app to run millions of tasks, heed my warning, RabbitMQ is great if you are low on memory but the overhead was too much to deal with. Celery also contributed to this but if I could travel back in time, I would stop myself from using RabbitMQ. Instead, Redis is a much better alternative for micro-service.

grahar64 11y ago
Redis pub/sub (which I assume you are talking about as an alternative) is fine, but you have to be listening when the message is sent otherwise you miss it. Another problem is working out if where you sent it there are any listeners, which RabbitMQ deals with return queues. This is important for 404's when a client is trying to talk to a service that does not exist. In Redis you would just post to a pub/sub queue wait till nothing happens and timeout, rather than immediately knowing through a return queue.
- lyha 11y ago
  Actually Redis returns the number of consumers that were subscribed during your publish and received the message, so you can detect that no service is available and return your 404 status. The real limitation for RPC over Redis pub/sub to me is that you can only broadcast your messages to all consumers, and not have them load-balanced between the subscribed applications. So you end-up following a discover/unicast call pattern where Redis doesn't bring much to the table for the actual RPC call. Maybe that's an opportunity for a cool addition to redis...
  - grahar64 11y ago
    Cheers, I didn't know that about Redis. I actually really like Redis and it simplicity, it would be cool to get more options for passing messages over Redis.

sunnya 11y ago

Check out http://restbus.org http://restbus.org , a similar project for .NET.

jhgg 11y ago

I feel like this is a bit silly. Feel like one could just speak HTTP or Thrift, and use etcd or zookeeper for discovery. RabbitMQ just complicates the stack. Also, it's HA options are generally... disappointing.

nitrogen 11y ago

This looks cool. I wrote a less developed than this and unreleased RabbitMQ-based microframework that combines RPC and one-way event streams to integrate an ecommerce site in one datacenter with a back office in another. At another company I used ZeroMQ to make a job processing cluster. Comments by others suggesting the use of http are kind of missing the point of message queues. The latency added by http(s) made interactive RPC sluggish, and made batch back office updates take hours instead of minutes with AMQP. If I had Alchemy two years ago I might have used this instead of rolling my own.

philip142au 11y ago

Use AKKA