3 ms·

Kubernetes absolutely eats the HashiStack lunch. I think they’re aware of this. That’s why I think HashiCorp invested so much VC funding into their SaaS offerin

by tjr225 4y ago

Kubernetes absolutely eats the HashiStack lunch. I think they’re aware of this. That’s why I think HashiCorp invested so much VC funding into their SaaS offerings: they know consul, vault, and nomad are a huge pain in the ass to configure and manage, particularly in support of each other. It’s all a little bit more agnostic than k8s but If you actually run consul + vault + nomad in production you know that managing it and repairing it can be a nightmare. GKE or EKS on the other hand feels like I almost shouldn’t even have a job. In fact, two of the most high profile outages in the last two years(Robinhood March 2020 and whatever Roblox outage happened recently) we’re due to consul or vault or something going completely hare brained.

Axsuul 4y ago
As a counterpoint I manage multiple HashiStack clusters in production with little overhead — it's not a nightmare. I am a Kubernetes refugee and Nomad is such a better experience with batteries included. Nomad now has its own service discovery (along with secure variables coming in 1.4.0) so you won't actually need Consul and Vault anymore. This is perfect for smaller deployments like on a Raspberry Pi.
- jdoss 4y ago
  I came to say pretty much the same thing. Nomad is awesome software. I use Nomad at work for internal workloads and automating its deployment with TLS + Consul is wayyyyyyy easier than k8s. I am really looking forward to 1.4.0. Also, I made some automation for setting up Nomad on a single Fedora CoreOS server (still kind of a work in progress) if anyone wants to give it a try. https://github.com/quickvm/butane-nomad https://github.com/quickvm/butane-nomad
- sidlls 4y ago
  Are your clusters at the same scale as Robinhood or Roblox? We’re talking many millions of inbound requests per second generating multiples of that on internal service calls. Nomad, Consul and Vault work find at small to medium scales. Sometimes. If things are working really well and there aren’t any real problems on the network. It’s very fragile, though, and tends to shit itself under the slightest bit of abnormal pressure.
  - tjr225 4y ago
    Exactly this; I would be knocking on all kinds of wood if you’ve had nothing but success with anything Consul/Vault/raft backed. Either that or beefing up your consul server ec2 instance types as soon as possible.
tommaybe 4y ago
So far for me Nomad has been very pleasant to work with. We are a small startup with just three devs - and are running multi-region Nomad in production for a few weeks now. We migrated around 20 services within a couple of days, coming from a mixture of all sorts of AWS services and Hetzner infra. Documentation was fairly good and the community was helpful. (This is without consul or vault, we use native service discovery so far)

by tjr225 4y ago

Axsuul 4y ago

As a counterpoint I manage multiple HashiStack clusters in production with little overhead — it's not a nightmare. I am a Kubernetes refugee and Nomad is such a better experience with batteries included. Nomad now has its own service discovery (along with secure variables coming in 1.4.0) so you won't actually need Consul and Vault anymore. This is perfect for smaller deployments like on a Raspberry Pi.

jdoss 4y ago
I came to say pretty much the same thing. Nomad is awesome software. I use Nomad at work for internal workloads and automating its deployment with TLS + Consul is wayyyyyyy easier than k8s. I am really looking forward to 1.4.0. Also, I made some automation for setting up Nomad on a single Fedora CoreOS server (still kind of a work in progress) if anyone wants to give it a try. https://github.com/quickvm/butane-nomad https://github.com/quickvm/butane-nomad
sidlls 4y ago
Are your clusters at the same scale as Robinhood or Roblox? We’re talking many millions of inbound requests per second generating multiples of that on internal service calls. Nomad, Consul and Vault work find at small to medium scales. Sometimes. If things are working really well and there aren’t any real problems on the network. It’s very fragile, though, and tends to shit itself under the slightest bit of abnormal pressure.
- tjr225 4y ago
  Exactly this; I would be knocking on all kinds of wood if you’ve had nothing but success with anything Consul/Vault/raft backed. Either that or beefing up your consul server ec2 instance types as soon as possible.

tommaybe 4y ago

So far for me Nomad has been very pleasant to work with. We are a small startup with just three devs - and are running multi-region Nomad in production for a few weeks now. We migrated around 20 services within a couple of days, coming from a mixture of all sorts of AWS services and Hetzner infra. Documentation was fairly good and the community was helpful. (This is without consul or vault, we use native service discovery so far)