4 ms·

My guess is Consul shit itself. It works until it doesn't. For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they

by jpgvm 5y ago

My guess is Consul shit itself. It works until it doesn't. For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against. Consul isn't widely used enough for me to have the same confidence for now, same goes for Vault and Nomad tbh. I really like the design of Vault and it's dynamic secrets system but I am probably just going to implement something similar on k8s secrets so that I don't have to carry around something that might spontaneously combust on me.

faizshah 5y ago
Just curious, what are the common ZK failure modes?
- jpgvm 5y ago
  In high scale situations, somewhat common if you aren't ontop of metrics etc. In most other cases it's just people not turning on auto-purge and having disks fill up with snapshots. So it varies depending on your teams experience with ZK and/or if you are using some sort of community managed deployment primitive that has sensible defaults baked in.
strictfp 5y ago
My stance is: why even have any of those? Are you running in a local datacenter?
- jpgvm 5y ago
  Generally you need them even when running in a "Cloud" of some description. Namely because they provide distributed locking primitives at a speed that can't be matched by any other mechanism. If you only need service discovery you can probably get away with whatever you platform provides (EC2 API, k8s API, hell DNS works, etc), similarly if you only need slow master election there are alternatives there too (DynamoDB w/conditional writes, k8s configmap). However for what these systems do best, i.e very high performance distributed locking and consistent metadata with fast reads w/watches and relative fast writes there is no replacement. You either need one of ZK/etcd/consul/something else based on raft/paxos/zab. EDIT: I realize now you might be talking about Nomad/Vault also. So in this case I think they are running their own DCs on bare metal and they opted for Nomad over k8s. Vault is somewhat special because it has capabilities no other secrets system does out of the box, specifically it has integrations to create short-lived credentials on the fly for clients - so called dynamic secrets have many advantages and is why services like AWS STS are so popular (the magic sauce behind AssumeRole).
  - strictfp 5y ago
    Having built many distributed applications I would hesitate to use ZK/consul directly, mostly since your cloud provider is already providing most type of primitives directly or indirectly in the services they provide.
    - 0xbadcafebee 5y ago
      But most people use them because they're required by something else, like Vault, or Solr, or some other tool built to require a particular distributed key value store.
      - strictfp 5y ago
        There's nothing intrinsically wrong with say ZK, it's very powerful, but my point is that you should avoid these systems if you can. Distributed concensus is not an easy problem, it's very error prone, and if you do this more than you absolutely have to you're doing it wrong.
        jpgvm 5y ago
        You are 100% correct. Avoid at all costs but when you need it, well you need it. Usually if you find yourself in that boat the best thing you can do is just understand exactly why you need consensus and ensure scope of data managed in ZK never expands beyond that point.
- sofixa 5y ago
  Even on a public cloud, Vault is great for secrets and much better ( has more integrations and is more widely supported) than the cloud vendor's equivalents ( not to mention lock-in). Nomad is a great orchestrator, with lots of integrations ( e.g. it can just run JARs or firecracker microVMs). IMHO features wise it's better than AWS ECS ( the only cloud orchestrator bar Kubernetes I've used, can't talk about the others), and gives Kubernetes a run for its money on many fronts (native templating, more flexible networking, no YAML, not restricted to containers, etc.). I wrote about it some time ago, you can take a look if interested: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...
sofixa 5y ago
I don't know, anecdotally it seems to me that Consul is more popular than Zookeeper. But yeah, you should go with the one you know best, especially at any sort of scale. Vault is the most popular secrets manager out there, and has a lot of advantages over something like Kubernetes secrets ( which aren't even encrypted). Nomad is a bit obscure but IMHO it seems to be gaining momentum.
- jpgvm 5y ago
  ZK is vastly more popular, it's just much less sexy so people don't write about it. Every production Kafka, Spark, Hadoop, Pulsar, etc cluster is using ZK. Just the Kafka and Hadoop clusters alone probably allows it to rival etcd in footprint. Consul on the other hand isn't used widely outside of some startups and a few OSS systems like Grafana Cortex/Loki. When it comes to which one has the most production hours under demanding workloads I think ZK comes out miles ahead of everything, then etcd (because of k8s) then in a very distant third is Consul.
  - Thaxll 5y ago
    Lot of people moved away from ZK because it used to be a nightmare to setup and maintain, ZK is not vastly more popular than Consul. In every places I worked in the last 5y we did not even considered ZK because of the setup and all the components it requires.
  - sofixa 5y ago
    ZK isn't sexy, we agree. IIRC Kafka no longer requires it? And IIRC it was one of the main downsides of Kafka, that it required a whole ZK cluster. etcd doesn't compare directly to ZK and Consul, because it's just a distributed KV store. You can build service discovery on top of it with custom tooling, but doesn't come out of the box. Consul does that + service discovery + service mesh + (since last week ) API gateway. On one hand, it's lots of stuff packed together, on the other hand it's useful. I started with it for SD, but when KV was also needed it was easy to add. I'm not too familiar with ZK, but isn't it also mostly a distributed KV store? Does it have things like Consul's health checks, prepared queries, DNS interface, etc? Consul is more widely used than that, like Cloudflare and Criteo at huge scale.
buryat 5y ago
you can simply disable parts of functionality if it's truly consul that's responsible
tjr225 5y ago
> For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against. Consul is insanely easy to fix if you keep backups. It is as simple as deploying a brand new cluster and restoring a known good snapshot. Sadly, a surprising amount of practitioners don't keep backups.
- Fordec 5y ago
  Or backups failing silently in some manner not being a "known" problem until you know.
redwolfb14 5y ago
This is hilarious. Please add a link to what you implement through k8s secrets, i'd be highly interested in its creation.

by jpgvm 5y ago

faizshah 5y ago

Just curious, what are the common ZK failure modes?

jpgvm 5y ago
In high scale situations, somewhat common if you aren't ontop of metrics etc. In most other cases it's just people not turning on auto-purge and having disks fill up with snapshots. So it varies depending on your teams experience with ZK and/or if you are using some sort of community managed deployment primitive that has sensible defaults baked in.

strictfp 5y ago

My stance is: why even have any of those? Are you running in a local datacenter?

jpgvm 5y ago
Generally you need them even when running in a "Cloud" of some description. Namely because they provide distributed locking primitives at a speed that can't be matched by any other mechanism. If you only need service discovery you can probably get away with whatever you platform provides (EC2 API, k8s API, hell DNS works, etc), similarly if you only need slow master election there are alternatives there too (DynamoDB w/conditional writes, k8s configmap). However for what these systems do best, i.e very high performance distributed locking and consistent metadata with fast reads w/watches and relative fast writes there is no replacement. You either need one of ZK/etcd/consul/something else based on raft/paxos/zab. EDIT: I realize now you might be talking about Nomad/Vault also. So in this case I think they are running their own DCs on bare metal and they opted for Nomad over k8s. Vault is somewhat special because it has capabilities no other secrets system does out of the box, specifically it has integrations to create short-lived credentials on the fly for clients - so called dynamic secrets have many advantages and is why services like AWS STS are so popular (the magic sauce behind AssumeRole).
- strictfp 5y ago
  Having built many distributed applications I would hesitate to use ZK/consul directly, mostly since your cloud provider is already providing most type of primitives directly or indirectly in the services they provide.
  - 0xbadcafebee 5y ago
    But most people use them because they're required by something else, like Vault, or Solr, or some other tool built to require a particular distributed key value store.
    - strictfp 5y ago
      There's nothing intrinsically wrong with say ZK, it's very powerful, but my point is that you should avoid these systems if you can. Distributed concensus is not an easy problem, it's very error prone, and if you do this more than you absolutely have to you're doing it wrong.
      - jpgvm 5y ago
        You are 100% correct. Avoid at all costs but when you need it, well you need it. Usually if you find yourself in that boat the best thing you can do is just understand exactly why you need consensus and ensure scope of data managed in ZK never expands beyond that point.
sofixa 5y ago
Even on a public cloud, Vault is great for secrets and much better ( has more integrations and is more widely supported) than the cloud vendor's equivalents ( not to mention lock-in). Nomad is a great orchestrator, with lots of integrations ( e.g. it can just run JARs or firecracker microVMs). IMHO features wise it's better than AWS ECS ( the only cloud orchestrator bar Kubernetes I've used, can't talk about the others), and gives Kubernetes a run for its money on many fronts (native templating, more flexible networking, no YAML, not restricted to containers, etc.). I wrote about it some time ago, you can take a look if interested: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...

sofixa 5y ago

I don't know, anecdotally it seems to me that Consul is more popular than Zookeeper. But yeah, you should go with the one you know best, especially at any sort of scale. Vault is the most popular secrets manager out there, and has a lot of advantages over something like Kubernetes secrets ( which aren't even encrypted). Nomad is a bit obscure but IMHO it seems to be gaining momentum.

jpgvm 5y ago
ZK is vastly more popular, it's just much less sexy so people don't write about it. Every production Kafka, Spark, Hadoop, Pulsar, etc cluster is using ZK. Just the Kafka and Hadoop clusters alone probably allows it to rival etcd in footprint. Consul on the other hand isn't used widely outside of some startups and a few OSS systems like Grafana Cortex/Loki. When it comes to which one has the most production hours under demanding workloads I think ZK comes out miles ahead of everything, then etcd (because of k8s) then in a very distant third is Consul.
- Thaxll 5y ago
  Lot of people moved away from ZK because it used to be a nightmare to setup and maintain, ZK is not vastly more popular than Consul. In every places I worked in the last 5y we did not even considered ZK because of the setup and all the components it requires.
- sofixa 5y ago
  ZK isn't sexy, we agree. IIRC Kafka no longer requires it? And IIRC it was one of the main downsides of Kafka, that it required a whole ZK cluster. etcd doesn't compare directly to ZK and Consul, because it's just a distributed KV store. You can build service discovery on top of it with custom tooling, but doesn't come out of the box. Consul does that + service discovery + service mesh + (since last week ) API gateway. On one hand, it's lots of stuff packed together, on the other hand it's useful. I started with it for SD, but when KV was also needed it was easy to add. I'm not too familiar with ZK, but isn't it also mostly a distributed KV store? Does it have things like Consul's health checks, prepared queries, DNS interface, etc? Consul is more widely used than that, like Cloudflare and Criteo at huge scale.

buryat 5y ago

you can simply disable parts of functionality if it's truly consul that's responsible

tjr225 5y ago

> For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against. Consul is insanely easy to fix if you keep backups. It is as simple as deploying a brand new cluster and restoring a known good snapshot. Sadly, a surprising amount of practitioners don't keep backups.

Fordec 5y ago
Or backups failing silently in some manner not being a "known" problem until you know.

redwolfb14 5y ago

This is hilarious. Please add a link to what you implement through k8s secrets, i'd be highly interested in its creation.