5 ms·

> But one of them has 10 times the components than the other I've said this before. Kubernetes gives you a lot more too. For example in Nomad you don't have se

by gizdan 5y ago

> But one of them has 10 times the components than the other I've said this before. Kubernetes gives you a lot more too. For example in Nomad you don't have secrets management, so you need to set up Vault. Both Nomad and Vault need Consul for Enterprise set ups, of which Vault needs 2 Consul clusters for Enterprise setups. So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did you gain really?

raffraffraff 5y ago
Also, Kubernetes can be just a single binary if you use k0s or k3s. And if you don't want to run it yourself you can use a managed k8s from AWS, Google, Digital Ocean, Oracle...
historynops 5y ago
Kubernetes native secrets management is not very good, so you're going to end up using Vault either way.
otterley 5y ago
Kubernetes' secrets management is nominal at best. It's basically just another data type that has K8S' standard ACL management around it. With K8S, the cluster admin has access to everything, including secrets objects. It's not encrypted at rest by default, and putting all the eggs in one basket (namely, etcd) means they're mixed in with all other control plane data. Most security practitioners believe secrets should be stored in a separate system, encrypted at rest, with strong auditing, authorization, and authentication mechanisms.
- alexeldeib 5y ago
  It's "good enough" for most and extension points allow for filling the gaps. This also dodges the crux of GP's argument -- instead of running 1 cluster with 10 components, you now need a half dozen clusters with 1 component each, but oops they all need to talk to each other with all the same fun TLS/authn/authz setup as k8s components.
  - otterley 5y ago
    I'm a little confused. Why does the problem with K8S secrets necessitate having multiple clusters? One could take advantage of a more secure secrets system instead, such as Hashicorp Vault or AWS Secrets Manager.
    - alexeldeib 5y ago
      The point is that once you're talking about comparable setups, you need all of Vault/Nomad/Consul and the complexity of the setup is much more than just "one binary" as hashi likes to put it. > So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did you gain really? GP's point was already talking about running Vault clusters, not sure you realized we aren't only talking about nomad.
      - otterley 5y ago
        The only thing I was trying to say is that although K8S offers secrets "for free," it's not best practice to consider the control plane to be a secure secrets store.
sofixa 5y ago
That's false. Vault has integrated storage and no longer needs Consul. If you want to have the Enterprise versions( which aren't required), you just need 1 each of Nomad, Consul, Vault. Considering many people use Vault with Kubernetes anyway(due to the joke that is Kubernetes "secrets"), and Consul provides some very nice features and is quite popular itself, that's okay IMHO. Unix philosophy and all.
- gizdan 5y ago
  This is just false. I've run Vault in an Enterprise and unless something has changed in the last 12 months, Hashicorp's recommendation for Vaul has been 1 Consul cluster for Vault's data store, and 1 for it's (and other application's) service discovery. Sure Kubernetes's secrets is a joke by default, it's easily substituted by something that one actually considers a secret store.
  - mdekkers 5y ago
    > This is just false. No it isn’t. > I've run Vault in an Enterprise At this point I am starting to doubt that claim.
  - alexeldeib 5y ago
    https://www.vaultproject.io/docs/configuration/storage/raft https://www.vaultproject.io/docs/configuration/storage/raft It's new but I think is quickly becoming preferred. I found trying to setup nomad/consul/vault as described on the hashi docs creates some circular dependencies tbh (e.g. the steps to setup nomad reference a consul setup, the steps for vault mention nomad integration, but there's no clear path outside the dev server examples of getting there without reading ALL the docs/references). There's little good docs in the way of bootstrapping everything 1 shot from scratch in the way most Kubernetes bootstrapping tools do. Setting up an HA Vault/Consul/Nomad setup from scratch isn't crazy, but I'd say it's comparable level to bootstrapping k8s in many ways.
    - sofixa 5y ago
      It's not that circular - you start with Consul, add Vault and then Nomad, clustering them through Consul and configuring Nomad to use Vault and Consul for secrets and KV/SD respectively. And of course it can be done incrementally ( you can deploy Nomad without pointing it to Consul or Vault, and just adding that configuration later).
AlexCoventry 5y ago
That surprises me. Does Google have a more complete secrets-management system for its in-house services?
- rixed 5y ago
  IIUC, despite K8s having started at Google by Go enthusiasts who had good knowledge of borg, the goal has never been to write a borg clone, even less a replacement for borg. And after so many years of independent development, I see no reason to believe that K8s ressemble borg any more than superficially. This seems to be very much assumed by kubernetes authors. Current borg users please correct me if I'm wrong.
  - AlexCoventry 5y ago
    Thanks.
OOPMan 5y ago
You gained the suffering of dealing with split-brains in Consul and Vault ;-)
mdekkers 5y ago
> Both Nomad and Vault need Consul for Enterprise set ups, of which Vault needs 2 Consul clusters for Enterprise setups. So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. This is incorrect. You don’t need consul for enterprise. Vault doesn’t need two consul clusters (it doesn’t need consul at all, if you don’t want it)

by gizdan 5y ago

raffraffraff 5y ago

Also, Kubernetes can be just a single binary if you use k0s or k3s. And if you don't want to run it yourself you can use a managed k8s from AWS, Google, Digital Ocean, Oracle...

historynops 5y ago

Kubernetes native secrets management is not very good, so you're going to end up using Vault either way.

otterley 5y ago

Kubernetes' secrets management is nominal at best. It's basically just another data type that has K8S' standard ACL management around it. With K8S, the cluster admin has access to everything, including secrets objects. It's not encrypted at rest by default, and putting all the eggs in one basket (namely, etcd) means they're mixed in with all other control plane data. Most security practitioners believe secrets should be stored in a separate system, encrypted at rest, with strong auditing, authorization, and authentication mechanisms.

alexeldeib 5y ago
It's "good enough" for most and extension points allow for filling the gaps. This also dodges the crux of GP's argument -- instead of running 1 cluster with 10 components, you now need a half dozen clusters with 1 component each, but oops they all need to talk to each other with all the same fun TLS/authn/authz setup as k8s components.
- otterley 5y ago
  I'm a little confused. Why does the problem with K8S secrets necessitate having multiple clusters? One could take advantage of a more secure secrets system instead, such as Hashicorp Vault or AWS Secrets Manager.
  - alexeldeib 5y ago
    The point is that once you're talking about comparable setups, you need all of Vault/Nomad/Consul and the complexity of the setup is much more than just "one binary" as hashi likes to put it. > So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. So what did you gain really? GP's point was already talking about running Vault clusters, not sure you realized we aren't only talking about nomad.
    - otterley 5y ago
      The only thing I was trying to say is that although K8S offers secrets "for free," it's not best practice to consider the control plane to be a secure secrets store.

sofixa 5y ago

That's false. Vault has integrated storage and no longer needs Consul. If you want to have the Enterprise versions( which aren't required), you just need 1 each of Nomad, Consul, Vault. Considering many people use Vault with Kubernetes anyway(due to the joke that is Kubernetes "secrets"), and Consul provides some very nice features and is quite popular itself, that's okay IMHO. Unix philosophy and all.

gizdan 5y ago
This is just false. I've run Vault in an Enterprise and unless something has changed in the last 12 months, Hashicorp's recommendation for Vaul has been 1 Consul cluster for Vault's data store, and 1 for it's (and other application's) service discovery. Sure Kubernetes's secrets is a joke by default, it's easily substituted by something that one actually considers a secret store.
- mdekkers 5y ago
  > This is just false. No it isn’t. > I've run Vault in an Enterprise At this point I am starting to doubt that claim.
- alexeldeib 5y ago
  https://www.vaultproject.io/docs/configuration/storage/raft https://www.vaultproject.io/docs/configuration/storage/raft It's new but I think is quickly becoming preferred. I found trying to setup nomad/consul/vault as described on the hashi docs creates some circular dependencies tbh (e.g. the steps to setup nomad reference a consul setup, the steps for vault mention nomad integration, but there's no clear path outside the dev server examples of getting there without reading ALL the docs/references). There's little good docs in the way of bootstrapping everything 1 shot from scratch in the way most Kubernetes bootstrapping tools do. Setting up an HA Vault/Consul/Nomad setup from scratch isn't crazy, but I'd say it's comparable level to bootstrapping k8s in many ways.
  - sofixa 5y ago
    It's not that circular - you start with Consul, add Vault and then Nomad, clustering them through Consul and configuring Nomad to use Vault and Consul for secrets and KV/SD respectively. And of course it can be done incrementally ( you can deploy Nomad without pointing it to Consul or Vault, and just adding that configuration later).

AlexCoventry 5y ago

That surprises me. Does Google have a more complete secrets-management system for its in-house services?

rixed 5y ago
IIUC, despite K8s having started at Google by Go enthusiasts who had good knowledge of borg, the goal has never been to write a borg clone, even less a replacement for borg. And after so many years of independent development, I see no reason to believe that K8s ressemble borg any more than superficially. This seems to be very much assumed by kubernetes authors. Current borg users please correct me if I'm wrong.
- AlexCoventry 5y ago
  Thanks.

OOPMan 5y ago

You gained the suffering of dealing with split-brains in Consul and Vault ;-)

mdekkers 5y ago

> Both Nomad and Vault need Consul for Enterprise set ups, of which Vault needs 2 Consul clusters for Enterprise setups. So now you have 3 separate Consul clusters, a Vault cluster, and a Nomad cluster. This is incorrect. You don’t need consul for enterprise. Vault doesn’t need two consul clusters (it doesn’t need consul at all, if you don’t want it)