7 ms·

See above comment, in the file https://github.com/LoyaltyNZ/alchemy-framework/blob/master/services/zglobal_rabbitmq.service https://github.com/LoyaltyNZ/alchemy

See above comment, in the file https://github.com/LoyaltyNZ/alchemy-framework/blob/master/services/zglobal_rabbitmq.service https://github.com/LoyaltyNZ/alchemy-framework/blob/master/s... is a description of creating a cluster of RabbitMQ nodes that can auto heal if an individual node goes down. We are running CoreOS which will occasionally shutdown a node and update. When this happens we see zero downtime and no error messages.

sagichmal 11y ago
Even if you have a cluster of Rabbit nodes, the logical RabbitMQ cluster is still a single point of failure. This is a fundamental problem with message bus architectures that advocates seem to ignore. It's even more problematic in a microservices architecture, where you (presumably) do domain-driven design in order to allow overall forward progress in the face of partial service/component/datastore outages. To throw all that out the window by coupling everything to a message bus... I still don't really understand.
- luan42 11y ago
  So what is the solution then?
  - hassy 11y ago
    Depends on the application, but as one possible general solution: have microservices talk to each other directly when it makes sense to rather than communicating over RabbitMQ / central message bus out of laziness/convenience.
    - ebiester 11y ago
      So you have a load balancer for every set of microservices? (Presumably, you have more than one instance running.)
      - brown9-2 11y ago
        Service Discovery often removes the need for load balancers. Let the clients discover where all the instances of Service X are and build the clients to handle failures to connect to individual instances.
        grahar64 11y ago
        Service discovery does not remove the need for load balancing. For example, if you had three nodes, each with every service and round robin service discovery to overload the system is just a matter of receiving a difficult request every third query. No matter how good your front end load balancing is in a micro service system, if your intra-service requests are not load balanced you can have problems with overloading one node, while others are idle.
        sagichmal 11y ago
        Service discovery can remove the need for load balancing, if you move load balancing logic into the client services. Have them architect their own load balancing over available instances of their dependent services. Unless you're deploying a Smartstack-esque LB strategy, where each physical node hosts its own load balancer, then the details of what's deployed where are mostly irrelevant. You use your SD system to abstract away the physical dimension of the problem, and address logical clusters of service instances. And you rely on your scheduler to distribute service instances evenly among nodes.
    - pm90 11y ago
      Doesn't this make the architecture a lot more complex though? I mean if every service uses a common messaging broker, number of connections to every other service is O(n). While if every microservice needs to talk to every other microservice, its O(n2). And analysis is much harder, unless all the services send their logs/metrics to a common logging/metrics system.
      - brown9-2 11y ago
        You shouldn't need every single microservice to talk to every single other microservice. If so, you have a design problem.
        grahar64 11y ago
        If you don't have every service talk to every other service then it is either a deployment problem or a load balance problem. If a service only say talks to instances of another service that is local, then every node in a cluster must contain ALL services. If the local service is overloaded but a remote service isn't then the service you are talking to will be slow, regardless of any front end load balancing. Every service must be able to talk to every other service, because if it cannot then you cannot load balance or deploy without it being an n^2 problem. So the question is how to implement it WITHOUT complicating the services
        sagichmal 11y ago
        profilesvc may only need to talk to (depend on) usersvc and accountsvc to do its job. usersvc and accountsvc may only need to talk to (depend on) their data stores. Yes, every instance of every service needs to manage its communication paths to other services. That's N connections, rather than just 1 to the message bus. But this is a pretty well-understood problem. We have connection pools and circuit breakers and so on. And the risks are distributed, isolated, heterogeneous.
- grahar64 11y ago
  In my experience failure occurs more frequently when you use more and more systems in more complex ways. e.g. using HAProxy for load balancing, with Consul for service discovery and Consul template for configuration. Each of these is a single point of failure as they are all required for the system to work. If you define single point of failure, as any computer goes down takes the system with it, then RabbitMQ is not a single point of failure. I am not sure how domain driven design helps solve this.
  - philsnow 11y ago
    > HAProxy for load balancing, with Consul for service discovery and Consul template for configuration. Each of these is a single point of failure as they are all required for the system to work. Not necessarily. I don't know anything about consul, but if you use something like zookeeper to discover services and write those into an HAProxy config, include a failsafe in whatever writes the HAProxy config on ZK updates such that if the delta is "too large" it will refuse to rewrite the config. Then if ZK becomes unavailable, what you lose is the ability to easily _make changes_ to what's in the service list. If your service instances come and go relatively infrequently, this might be fine while the ZK fire get put out.
    - grahar64 11y ago
      Service instances in a continuous deployment environment are coming and going all day. IF your service discovery and config breaks then everything stops, nothing can be developed or deployed until the broken stuff is fixed.
      - sagichmal 11y ago
        If SD or config mgmt dies, you can't deploy new stuff, but the existing services continue to work. When your message bus dies, everything dies. It's a fundamentally different failure.

sagichmal 11y ago

Even if you have a cluster of Rabbit nodes, the logical RabbitMQ cluster is still a single point of failure. This is a fundamental problem with message bus architectures that advocates seem to ignore. It's even more problematic in a microservices architecture, where you (presumably) do domain-driven design in order to allow overall forward progress in the face of partial service/component/datastore outages. To throw all that out the window by coupling everything to a message bus... I still don't really understand.

luan42 11y ago
So what is the solution then?
- hassy 11y ago
  Depends on the application, but as one possible general solution: have microservices talk to each other directly when it makes sense to rather than communicating over RabbitMQ / central message bus out of laziness/convenience.
  - ebiester 11y ago
    So you have a load balancer for every set of microservices? (Presumably, you have more than one instance running.)
    - brown9-2 11y ago
      Service Discovery often removes the need for load balancers. Let the clients discover where all the instances of Service X are and build the clients to handle failures to connect to individual instances.
      - grahar64 11y ago
        Service discovery does not remove the need for load balancing. For example, if you had three nodes, each with every service and round robin service discovery to overload the system is just a matter of receiving a difficult request every third query. No matter how good your front end load balancing is in a micro service system, if your intra-service requests are not load balanced you can have problems with overloading one node, while others are idle.
        sagichmal 11y ago
        Service discovery can remove the need for load balancing, if you move load balancing logic into the client services. Have them architect their own load balancing over available instances of their dependent services. Unless you're deploying a Smartstack-esque LB strategy, where each physical node hosts its own load balancer, then the details of what's deployed where are mostly irrelevant. You use your SD system to abstract away the physical dimension of the problem, and address logical clusters of service instances. And you rely on your scheduler to distribute service instances evenly among nodes.
  - pm90 11y ago
    Doesn't this make the architecture a lot more complex though? I mean if every service uses a common messaging broker, number of connections to every other service is O(n). While if every microservice needs to talk to every other microservice, its O(n2). And analysis is much harder, unless all the services send their logs/metrics to a common logging/metrics system.
    - brown9-2 11y ago
      You shouldn't need every single microservice to talk to every single other microservice. If so, you have a design problem.
      - grahar64 11y ago
        If you don't have every service talk to every other service then it is either a deployment problem or a load balance problem. If a service only say talks to instances of another service that is local, then every node in a cluster must contain ALL services. If the local service is overloaded but a remote service isn't then the service you are talking to will be slow, regardless of any front end load balancing. Every service must be able to talk to every other service, because if it cannot then you cannot load balance or deploy without it being an n^2 problem. So the question is how to implement it WITHOUT complicating the services
        sagichmal 11y ago
        profilesvc may only need to talk to (depend on) usersvc and accountsvc to do its job. usersvc and accountsvc may only need to talk to (depend on) their data stores. Yes, every instance of every service needs to manage its communication paths to other services. That's N connections, rather than just 1 to the message bus. But this is a pretty well-understood problem. We have connection pools and circuit breakers and so on. And the risks are distributed, isolated, heterogeneous.
grahar64 11y ago
In my experience failure occurs more frequently when you use more and more systems in more complex ways. e.g. using HAProxy for load balancing, with Consul for service discovery and Consul template for configuration. Each of these is a single point of failure as they are all required for the system to work. If you define single point of failure, as any computer goes down takes the system with it, then RabbitMQ is not a single point of failure. I am not sure how domain driven design helps solve this.
- philsnow 11y ago
  > HAProxy for load balancing, with Consul for service discovery and Consul template for configuration. Each of these is a single point of failure as they are all required for the system to work. Not necessarily. I don't know anything about consul, but if you use something like zookeeper to discover services and write those into an HAProxy config, include a failsafe in whatever writes the HAProxy config on ZK updates such that if the delta is "too large" it will refuse to rewrite the config. Then if ZK becomes unavailable, what you lose is the ability to easily _make changes_ to what's in the service list. If your service instances come and go relatively infrequently, this might be fine while the ZK fire get put out.
  - grahar64 11y ago
    Service instances in a continuous deployment environment are coming and going all day. IF your service discovery and config breaks then everything stops, nothing can be developed or deployed until the broken stuff is fixed.
    - sagichmal 11y ago
      If SD or config mgmt dies, you can't deploy new stuff, but the existing services continue to work. When your message bus dies, everything dies. It's a fundamentally different failure.