4 ms·

My stance is: why even have any of those? Are you running in a local datacenter?

jpgvm 5y ago
Generally you need them even when running in a "Cloud" of some description. Namely because they provide distributed locking primitives at a speed that can't be matched by any other mechanism. If you only need service discovery you can probably get away with whatever you platform provides (EC2 API, k8s API, hell DNS works, etc), similarly if you only need slow master election there are alternatives there too (DynamoDB w/conditional writes, k8s configmap). However for what these systems do best, i.e very high performance distributed locking and consistent metadata with fast reads w/watches and relative fast writes there is no replacement. You either need one of ZK/etcd/consul/something else based on raft/paxos/zab. EDIT: I realize now you might be talking about Nomad/Vault also. So in this case I think they are running their own DCs on bare metal and they opted for Nomad over k8s. Vault is somewhat special because it has capabilities no other secrets system does out of the box, specifically it has integrations to create short-lived credentials on the fly for clients - so called dynamic secrets have many advantages and is why services like AWS STS are so popular (the magic sauce behind AssumeRole).
- strictfp 5y ago
  Having built many distributed applications I would hesitate to use ZK/consul directly, mostly since your cloud provider is already providing most type of primitives directly or indirectly in the services they provide.
  - 0xbadcafebee 5y ago
    But most people use them because they're required by something else, like Vault, or Solr, or some other tool built to require a particular distributed key value store.
    - strictfp 5y ago
      There's nothing intrinsically wrong with say ZK, it's very powerful, but my point is that you should avoid these systems if you can. Distributed concensus is not an easy problem, it's very error prone, and if you do this more than you absolutely have to you're doing it wrong.
      - jpgvm 5y ago
        You are 100% correct. Avoid at all costs but when you need it, well you need it. Usually if you find yourself in that boat the best thing you can do is just understand exactly why you need consensus and ensure scope of data managed in ZK never expands beyond that point.
sofixa 5y ago
Even on a public cloud, Vault is great for secrets and much better ( has more integrations and is more widely supported) than the cloud vendor's equivalents ( not to mention lock-in). Nomad is a great orchestrator, with lots of integrations ( e.g. it can just run JARs or firecracker microVMs). IMHO features wise it's better than AWS ECS ( the only cloud orchestrator bar Kubernetes I've used, can't talk about the others), and gives Kubernetes a run for its money on many fronts (native templating, more flexible networking, no YAML, not restricted to containers, etc.). I wrote about it some time ago, you can take a look if interested: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...

Generally you need them even when running in a "Cloud" of some description. Namely because they provide distributed locking primitives at a speed that can't be matched by any other mechanism. If you only need service discovery you can probably get away with whatever you platform provides (EC2 API, k8s API, hell DNS works, etc), similarly if you only need slow master election there are alternatives there too (DynamoDB w/conditional writes, k8s configmap). However for what these systems do best, i.e very high performance distributed locking and consistent metadata with fast reads w/watches and relative fast writes there is no replacement. You either need one of ZK/etcd/consul/something else based on raft/paxos/zab. EDIT: I realize now you might be talking about Nomad/Vault also. So in this case I think they are running their own DCs on bare metal and they opted for Nomad over k8s. Vault is somewhat special because it has capabilities no other secrets system does out of the box, specifically it has integrations to create short-lived credentials on the fly for clients - so called dynamic secrets have many advantages and is why services like AWS STS are so popular (the magic sauce behind AssumeRole).

strictfp 5y ago
Having built many distributed applications I would hesitate to use ZK/consul directly, mostly since your cloud provider is already providing most type of primitives directly or indirectly in the services they provide.
- 0xbadcafebee 5y ago
  But most people use them because they're required by something else, like Vault, or Solr, or some other tool built to require a particular distributed key value store.
  - strictfp 5y ago
    There's nothing intrinsically wrong with say ZK, it's very powerful, but my point is that you should avoid these systems if you can. Distributed concensus is not an easy problem, it's very error prone, and if you do this more than you absolutely have to you're doing it wrong.
    - jpgvm 5y ago
      You are 100% correct. Avoid at all costs but when you need it, well you need it. Usually if you find yourself in that boat the best thing you can do is just understand exactly why you need consensus and ensure scope of data managed in ZK never expands beyond that point.

sofixa 5y ago

Even on a public cloud, Vault is great for secrets and much better ( has more integrations and is more widely supported) than the cloud vendor's equivalents ( not to mention lock-in). Nomad is a great orchestrator, with lots of integrations ( e.g. it can just run JARs or firecracker microVMs). IMHO features wise it's better than AWS ECS ( the only cloud orchestrator bar Kubernetes I've used, can't talk about the others), and gives Kubernetes a run for its money on many fronts (native templating, more flexible networking, no YAML, not restricted to containers, etc.). I wrote about it some time ago, you can take a look if interested: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...