5 ms·

Consul is usually very reliable; when it breaks it can be very painful and mystifying. I've worked on removing it both from internal systems due to outages, and

Consul is usually very reliable; when it breaks it can be very painful and mystifying. I've worked on removing it both from internal systems due to outages, and products, based on feedback.

mijoharas 5y ago
Could you expand a little on your problems with Consul? (I have no experience with it myself).
- kerblang 5y ago
  Not OP, but go look at the consul documentation. In fact, just look at "Configuration" page alone: https://www.consul.io/docs/agent/options https://www.consul.io/docs/agent/options - it goes FOREVER. Maybe you don't need most of that, but whatever you do need, you're going to have to find it somewhere in there. And yes, the error messages can be utterly cryptic. One thing: Don't ever let junior devs try to create a cluster out of their own laptops (yes they will think it sounds like a good idea) as this will be a never-ending nightmare.
- pronoiac 5y ago
  It's been a while, so I'm fuzzy on some details: * in a legacy system, a server, perhaps with the leader node, filled up disk space and became unhealthy, the consul agent kept reporting it and itself as healthy, and failover and gossip generally wedged * in a dev environment, after we replaced some servers in the cluster, the other nodes noted cert changes and refused to work with the new servers * second-hand, in self-hosted installations, it caused a number of hard-to-troubleshoot outages * something about circular dependencies and going by "wait _n_ seconds" rather than by healthiness It was reliable enough that it could gather a really significant blast radius, and it had different gnarly failure modes, so documentation could be irrelevant from case to case.
  - still_grokking 5y ago
    And what is the replacement for it?
    - pronoiac 5y ago
      We rearchitected. At one workplace, we built and distributed our own service. At another, we shifted to semi-automated more static lists of servers for roles; those servers were much less dynamic.
      - still_grokking 5y ago
        There are exceptions but most of the time replacing an off the shelf std. solution with something self made looks like NIH syndrome to me. The exceptions are the few cases where you know you will forever only need some strict set of features and your own solution can provide them by way more simple means than the in comparison "fat" off the shelf solution. I'm not sure service discovery in a cluster is one of those cases.
    - shaklee3 5y ago
      etcd
      - still_grokking 5y ago
        Never heard that someone moved form Consul to etcd. It's always the other way around.
  - proxysna 5y ago
    I don't want to sound bad, but to me all of these sound like a misconfiguration and lack of understanding how consul works tbh. But i don't know the full context so eh, these things happen.

Consul is usually very reliable; when it breaks it can be very painful and mystifying. I've worked on removing it both from internal systems due to outages, and products, based on feedback.

mijoharas 5y ago

Could you expand a little on your problems with Consul? (I have no experience with it myself).

kerblang 5y ago
Not OP, but go look at the consul documentation. In fact, just look at "Configuration" page alone: https://www.consul.io/docs/agent/options https://www.consul.io/docs/agent/options - it goes FOREVER. Maybe you don't need most of that, but whatever you do need, you're going to have to find it somewhere in there. And yes, the error messages can be utterly cryptic. One thing: Don't ever let junior devs try to create a cluster out of their own laptops (yes they will think it sounds like a good idea) as this will be a never-ending nightmare.
pronoiac 5y ago
It's been a while, so I'm fuzzy on some details: * in a legacy system, a server, perhaps with the leader node, filled up disk space and became unhealthy, the consul agent kept reporting it and itself as healthy, and failover and gossip generally wedged * in a dev environment, after we replaced some servers in the cluster, the other nodes noted cert changes and refused to work with the new servers * second-hand, in self-hosted installations, it caused a number of hard-to-troubleshoot outages * something about circular dependencies and going by "wait _n_ seconds" rather than by healthiness It was reliable enough that it could gather a really significant blast radius, and it had different gnarly failure modes, so documentation could be irrelevant from case to case.
- still_grokking 5y ago
  And what is the replacement for it?
  - pronoiac 5y ago
    We rearchitected. At one workplace, we built and distributed our own service. At another, we shifted to semi-automated more static lists of servers for roles; those servers were much less dynamic.
    - still_grokking 5y ago
      There are exceptions but most of the time replacing an off the shelf std. solution with something self made looks like NIH syndrome to me. The exceptions are the few cases where you know you will forever only need some strict set of features and your own solution can provide them by way more simple means than the in comparison "fat" off the shelf solution. I'm not sure service discovery in a cluster is one of those cases.
  - shaklee3 5y ago
    etcd
    - still_grokking 5y ago
      Never heard that someone moved form Consul to etcd. It's always the other way around.
- proxysna 5y ago
  I don't want to sound bad, but to me all of these sound like a misconfiguration and lack of understanding how consul works tbh. But i don't know the full context so eh, these things happen.