6 ms·

IMO this is an emerging anti pattern to use Rabbit to connect "microservices". It often introduces a single point of failure to your "distributed" system and ha

IMO this is an emerging anti pattern to use Rabbit to connect "microservices". It often introduces a single point of failure to your "distributed" system and has problems with network partitions. If critical functionality stops working when Rabbit is down, you're probably doing it wrong.

nugator 11y ago
Do you have a better solution? Genuinely interested since I'm about to implement webservice data change event notifications using probably RabbitMQ.
- eropple 11y ago
  It does require some up-front work, but I've generally moved over to Kafka for most of my stuff. Rabbit has some nice aspects, but as noted its HA options start at "dire" and escalate smoothly to "oh dear god no". Throughput with Kafka is very, very good and,in my experience, it's remarkably difficult to kill. If you want fully ephemeral topics, you can write the broker data to a tmpfs. But I really like keeping data around, because I can replay my topics later both for DR and for debugging.
  - slowmovintarget 11y ago
    Kafka makes reliable durable messaging not just possible, but solves all the usual attendant problems you try to design around. * Durable messages cause a slow-down when publishing, but not with Kafka because it uses the Linux kernel's page cache to write. * Durable messages are slow to read if they have to come from disk. Kafka is optimized to load sequential blocks into memory an push them out through a socket with very few copies. This makes for very fast reads. * Slow consumers can bog down the broker. Kafka stores all messages and keeps them on the topic for a time horizon. No back-pressure from slow consumers. * Disconnected but subscribed consumers cause messages to back up on disk and eventually clog the broker. Kafka stores all messages. There's no clogging or backup, that's just how it works. * Brokers must track whether a consumer actually received the message, failures can cause missed messages or clogs. Kafka clients may read from a given point on the topic forward. If they fail during a read, they just back up and read again. The messages will be there for hours/days/weeks as configured. With a rock-steady durable messaging system based on commit logs, all of those problems that arose from attempting to avoid durable messaging go away. Now you build microservices that emit and respond to events. Microservices that can "rehydrate" their state from private checkpoints and topic replays. And all of this with partition tolerance and simple mirroring. Although it isn't usually necessary, if you want, you can make all of that elastic with Mesos, too. Further reading: [1] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... [2] http://mesos.apache.org/ http://mesos.apache.org/
  - lobster_johnson 11y ago
    Kafka is very good for synchronizing data streams, but I can't imagine that it's suitable for RPC?
    - slowmovintarget 11y ago
      You don't (or shouldn't) do ordinary RPC over messaging any way. Rather, use an Event Sourcing style: http://martinfowler.com/eaaDev/EventSourcing.html http://martinfowler.com/eaaDev/EventSourcing.html
      - lobster_johnson 11y ago
        I didn't read that web page carefully, but it seems to describe a transaction log, which is what Kafka excels it, but it has precious title to do with RPC. RPC is point-to-point communication based on requests and replies. Kafka's strictly sequential requirement would be terrible for this because a single slow request would hold up its entire partition — no other consumer would be able to process the pending upstream events. Kafka is also persistent (does it have in-memory queues?), which is pointless for RPC.
        eropple 11y ago
        Message queues, period, aren't particularly good for RPC. HTTP, as an online protocol, has such huge advantages that trying to replace it doesn't make sense to me. However, for comms between services where a consumer isn't waiting on the other end, a message queue is plenty appropriate--and Kafka is much, much better at that than RabbitMQ is in terms of throughput and data sanity. I also quite like NATS, but Kafka provides similar performance characteristics in the general case (generally higher latency being the exception, though I have never encountered latency-sensitive processes where a message queue made sense in the first place) and means babysitting fewer systems.
        lobster_johnson 11y ago
        To be clear, NATS is not a heavy messaging broker like RabbitMQ. For one, it's in-memory only, and queues only exist when there are consumers: If you publish and there are no subscribers, the message doesn't go anywhere. NATS is closer to ZeroMQ than RabbitMQ or Kafka. A lot of people use HAProxy to route messages via HTTP to microservices — what's HAProxy if not a glorified message queue? If you don't use an intermediate — meaning you to point-to-point HTTP between one microservice and another — you have to find a way to discover peers, perform health checks, load-balance between them, and so on. Which you can do — services like etcd and Consul exist for this — but using an intermediary such as NATS or Linkerd [1] is also a great, possibly simpler solution. [1] https://linkerd.io/ https://linkerd.io/
  - StreamBright 11y ago
    Kafka saved my "life" few times. We had TTL set to 168h and somebody pushed a change to production that silently ignored a type of message. We realized it few days later. Luckily we could re-play all of the messages after fixing the code. I know there are so many things wrong with this, yet, Kafka is excellent at storing data for medium terms and that can be a real bliss.
officialchicken 11y ago
If it goes down the Rabbit hole, you've engineered it wrong.
grahar64 11y ago
See above comment, in the file https://github.com/LoyaltyNZ/alchemy-framework/blob/master/services/zglobal_rabbitmq.service https://github.com/LoyaltyNZ/alchemy-framework/blob/master/s... is a description of creating a cluster of RabbitMQ nodes that can auto heal if an individual node goes down. We are running CoreOS which will occasionally shutdown a node and update. When this happens we see zero downtime and no error messages.
- sagichmal 11y ago
  Even if you have a cluster of Rabbit nodes, the logical RabbitMQ cluster is still a single point of failure. This is a fundamental problem with message bus architectures that advocates seem to ignore. It's even more problematic in a microservices architecture, where you (presumably) do domain-driven design in order to allow overall forward progress in the face of partial service/component/datastore outages. To throw all that out the window by coupling everything to a message bus... I still don't really understand.
  - luan42 11y ago
    So what is the solution then?
    - hassy 11y ago
      Depends on the application, but as one possible general solution: have microservices talk to each other directly when it makes sense to rather than communicating over RabbitMQ / central message bus out of laziness/convenience.
      - ebiester 11y ago
        So you have a load balancer for every set of microservices? (Presumably, you have more than one instance running.)
        brown9-2 11y ago
calpaterson 11y ago
Most real world microservice projects (I've worked on several) already have many single points of failure. Often there is one service that needs to be up for the system to be up (such as the one that processes your customers orders), you don't realise some VMs are sharing a physical disk or everything is dependent on a single router somewhere you've never heard of that will one day run out of memory and drop TCP connections. This is not to mention the risks posed to availability by third-party tracking software that push changes that break web forms (#1 cause of long outages in my experience). Message brokers like RabbitMQ give you a lot of benefit and introduce only a small number of failure modes. You can obviate tricky service discovery boot orders, do RPC with without caring about whether about your server could be restarted and of course you get a good implementation of pub-sub too. If you stay away from poorly considered high availability schemes I am absolutely fine with recommending it for intra-service communication.
- pm90 11y ago
  Great comment. One thing I don't like very much about microservices is simply that my service often will have some such hard dependency. e.g. if the logging service is down, I lose all logs-based statistics. If the authentication microservice is down, I'm screwed. The parent comment made the excellent point about SPOF, but it seems like for microservices to work correctly, there will always be some SPOFs. Maybe I'm being too pessimistic. I use microservices, but without significant engineering rigor, I think its a recipe for disaster.
  - zeisss 11y ago
    > If the authentication microservice is down, I'm screwed. Well, make sure your sevice is not dependent on those services then. Use signed tokens to only rely on the authenticator for logins. everything afterwards can work it our by themself. logging: keep your stuff in a queue or logfile, until the log service is back up.

nugator 11y ago

Do you have a better solution? Genuinely interested since I'm about to implement webservice data change event notifications using probably RabbitMQ.

eropple 11y ago
It does require some up-front work, but I've generally moved over to Kafka for most of my stuff. Rabbit has some nice aspects, but as noted its HA options start at "dire" and escalate smoothly to "oh dear god no". Throughput with Kafka is very, very good and,in my experience, it's remarkably difficult to kill. If you want fully ephemeral topics, you can write the broker data to a tmpfs. But I really like keeping data around, because I can replay my topics later both for DR and for debugging.
- slowmovintarget 11y ago
  Kafka makes reliable durable messaging not just possible, but solves all the usual attendant problems you try to design around. * Durable messages cause a slow-down when publishing, but not with Kafka because it uses the Linux kernel's page cache to write. * Durable messages are slow to read if they have to come from disk. Kafka is optimized to load sequential blocks into memory an push them out through a socket with very few copies. This makes for very fast reads. * Slow consumers can bog down the broker. Kafka stores all messages and keeps them on the topic for a time horizon. No back-pressure from slow consumers. * Disconnected but subscribed consumers cause messages to back up on disk and eventually clog the broker. Kafka stores all messages. There's no clogging or backup, that's just how it works. * Brokers must track whether a consumer actually received the message, failures can cause missed messages or clogs. Kafka clients may read from a given point on the topic forward. If they fail during a read, they just back up and read again. The messages will be there for hours/days/weeks as configured. With a rock-steady durable messaging system based on commit logs, all of those problems that arose from attempting to avoid durable messaging go away. Now you build microservices that emit and respond to events. Microservices that can "rehydrate" their state from private checkpoints and topic replays. And all of this with partition tolerance and simple mirroring. Although it isn't usually necessary, if you want, you can make all of that elastic with Mesos, too. Further reading: [1] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... [2] http://mesos.apache.org/ http://mesos.apache.org/
- lobster_johnson 11y ago
  Kafka is very good for synchronizing data streams, but I can't imagine that it's suitable for RPC?
  - slowmovintarget 11y ago
    You don't (or shouldn't) do ordinary RPC over messaging any way. Rather, use an Event Sourcing style: http://martinfowler.com/eaaDev/EventSourcing.html http://martinfowler.com/eaaDev/EventSourcing.html
    - lobster_johnson 11y ago
      I didn't read that web page carefully, but it seems to describe a transaction log, which is what Kafka excels it, but it has precious title to do with RPC. RPC is point-to-point communication based on requests and replies. Kafka's strictly sequential requirement would be terrible for this because a single slow request would hold up its entire partition — no other consumer would be able to process the pending upstream events. Kafka is also persistent (does it have in-memory queues?), which is pointless for RPC.
      - eropple 11y ago
        Message queues, period, aren't particularly good for RPC. HTTP, as an online protocol, has such huge advantages that trying to replace it doesn't make sense to me. However, for comms between services where a consumer isn't waiting on the other end, a message queue is plenty appropriate--and Kafka is much, much better at that than RabbitMQ is in terms of throughput and data sanity. I also quite like NATS, but Kafka provides similar performance characteristics in the general case (generally higher latency being the exception, though I have never encountered latency-sensitive processes where a message queue made sense in the first place) and means babysitting fewer systems.
        lobster_johnson 11y ago
        To be clear, NATS is not a heavy messaging broker like RabbitMQ. For one, it's in-memory only, and queues only exist when there are consumers: If you publish and there are no subscribers, the message doesn't go anywhere. NATS is closer to ZeroMQ than RabbitMQ or Kafka. A lot of people use HAProxy to route messages via HTTP to microservices — what's HAProxy if not a glorified message queue? If you don't use an intermediate — meaning you to point-to-point HTTP between one microservice and another — you have to find a way to discover peers, perform health checks, load-balance between them, and so on. Which you can do — services like etcd and Consul exist for this — but using an intermediary such as NATS or Linkerd [1] is also a great, possibly simpler solution. [1] https://linkerd.io/ https://linkerd.io/
- StreamBright 11y ago
  Kafka saved my "life" few times. We had TTL set to 168h and somebody pushed a change to production that silently ignored a type of message. We realized it few days later. Luckily we could re-play all of the messages after fixing the code. I know there are so many things wrong with this, yet, Kafka is excellent at storing data for medium terms and that can be a real bliss.

officialchicken 11y ago

If it goes down the Rabbit hole, you've engineered it wrong.

grahar64 11y ago

See above comment, in the file https://github.com/LoyaltyNZ/alchemy-framework/blob/master/services/zglobal_rabbitmq.service https://github.com/LoyaltyNZ/alchemy-framework/blob/master/s... is a description of creating a cluster of RabbitMQ nodes that can auto heal if an individual node goes down. We are running CoreOS which will occasionally shutdown a node and update. When this happens we see zero downtime and no error messages.

sagichmal 11y ago
Even if you have a cluster of Rabbit nodes, the logical RabbitMQ cluster is still a single point of failure. This is a fundamental problem with message bus architectures that advocates seem to ignore. It's even more problematic in a microservices architecture, where you (presumably) do domain-driven design in order to allow overall forward progress in the face of partial service/component/datastore outages. To throw all that out the window by coupling everything to a message bus... I still don't really understand.
- luan42 11y ago
  So what is the solution then?
  - hassy 11y ago
    Depends on the application, but as one possible general solution: have microservices talk to each other directly when it makes sense to rather than communicating over RabbitMQ / central message bus out of laziness/convenience.
    - ebiester 11y ago
      So you have a load balancer for every set of microservices? (Presumably, you have more than one instance running.)
      - brown9-2 11y ago

calpaterson 11y ago

Most real world microservice projects (I've worked on several) already have many single points of failure. Often there is one service that needs to be up for the system to be up (such as the one that processes your customers orders), you don't realise some VMs are sharing a physical disk or everything is dependent on a single router somewhere you've never heard of that will one day run out of memory and drop TCP connections. This is not to mention the risks posed to availability by third-party tracking software that push changes that break web forms (#1 cause of long outages in my experience). Message brokers like RabbitMQ give you a lot of benefit and introduce only a small number of failure modes. You can obviate tricky service discovery boot orders, do RPC with without caring about whether about your server could be restarted and of course you get a good implementation of pub-sub too. If you stay away from poorly considered high availability schemes I am absolutely fine with recommending it for intra-service communication.

pm90 11y ago
Great comment. One thing I don't like very much about microservices is simply that my service often will have some such hard dependency. e.g. if the logging service is down, I lose all logs-based statistics. If the authentication microservice is down, I'm screwed. The parent comment made the excellent point about SPOF, but it seems like for microservices to work correctly, there will always be some SPOFs. Maybe I'm being too pessimistic. I use microservices, but without significant engineering rigor, I think its a recipe for disaster.
- zeisss 11y ago
  > If the authentication microservice is down, I'm screwed. Well, make sure your sevice is not dependent on those services then. Use signed tokens to only rely on the authenticator for logins. everything afterwards can work it our by themself. logging: keep your stuff in a queue or logfile, until the log service is back up.