3 ms·

Disclaimer: I work at Hashicorp but I have been saying what I'm about to say for years before joining (check my blog, linked in bio, for reference). There is a

by sofixa 4y ago

Disclaimer: I work at Hashicorp but I have been saying what I'm about to say for years before joining (check my blog, linked in bio, for reference). There is a way to get all of those necessary things without having the massive complexity of k8s - use Nomad, a vastly simpler orchestrator that does many things better than k8s - e.g. integrated templating instead of go templates over yaml with Helm or other similar abominations; not using yaml for configuration; self-contained configuration and handling of updates (e.g. if your ConfigMap or Secret in k8s change, you have to manually restart/reload your pods; with Nomad you can configure it to handle that directly). It doesn't have nearly the same ecosystem, but there's enough of it through CSI, CNI, various load balancer and autoscaling integrations that you get very far with a production environment.

v3ss0n 4y ago
But Only problem of nomad is all examples are running on Vagrant which is designed for VMs . Everyone who is looking these days are containerization. There may be non-standard ways to run Nomad with containers but , putting Vagrant as main example is not a good idea.
- zozbot234 4y ago
  Vagrant supports containers since v1.6 (released in 2014).
  - v3ss0n 4y ago
    yeah but examples uses vagrant vm, i am just checking nomad and use of vm discourages me / not sure how much compatible if i to placed them with nomad/lxd/dockers. Any good tutorials with Noman + Containers would be nice for CI/CD/Executor setup.
- sofixa 4y ago
  I'm not sure i follow. Nomad is an orchestrator which handles containers and other things (firecracker microVMs, LXC, etc.). The Vagrant examples are to get started and test it out, because running a container orchestrator in containers has some non-obvious pitfalls. You can just as well download the binary anywhere to play with it. That doesn't mean Nomad can't orchestrate containers (in fact that's the main use case for it).
- ViViDboarder 4y ago
  All the examples I’ve found are Docker… Most use of Vagrant I saw was in demoing how to set up and deploy a cluster. So deploying 3 Nomad nodes within 3 VMs created in Vagrant.
thejosh 4y ago
I was amazed how easy it was to setup Nomad. I have a couple of VMs running on a VPS and Nomad works great for this use case. Super easy to maintain and works really well. Networking is my favourite part.
ofrzeta 4y ago
Last week, when that same discussion popped up, "schmichael" (who works on Nomad I understand) said "Nomad supports CSI like Kubernetes. Unfortunately many CSI storage providers assume kubernetes or adhere to kubernetes behavior even when it deviates from the spec, so there are gotchas." https://news.ycombinator.com/item?id=32588931 https://news.ycombinator.com/item?id=32588931 That makes be a bit wary to use it because I would like my cloud storage to be reliable.
- sofixa 4y ago
  Yep, some CSI providers do that, but as far as i know it's more niche ones (like VMware's), not any of the big cloud providers.
  - candiddevmike 4y ago
    I tried using a CSI driver with Nomad (AWS) and AFAIK you have to pre-create the EBS volumes? I couldn't figure out how to spin them up on demand.
    - sofixa 4y ago
      That was probably before the Nomad 1.3 release which added a bunch of new CSI features, including dynamic volume creation IIRC.
  - orthoxerox 4y ago
    I wouldn't call vmware a niche provider, they own a large share of on prem virualization market.
    - sofixa 4y ago
      Which is a dying breed, and furthermore running an orchestrator on top of another one (Nomad or Kubernetes on top of vSphere) is a terrible anti-pattern that wastes CPU and makes performance issues hard to debug due to the two layers.
maccard 4y ago
My only problem with nomad is that I have to manage the infrastructure underneath as well. Give me nomad-on-fargate and I'll happily jump ship.
figassis 4y ago
My issue with nomad is I also have to handle a second cluster, Consul, and that's it's own overhead. I'd rather just stay with k8s if the alternative is Nomad. Id totally jump to Nomad if all I had to do was deploy nomad, and consul was embedded in all nodes. I'm not an expert in consul, and really don't want another time sink.
- schmichael 4y ago
  Nomad 1.3 has builtin service discovery and 1.4 is around the corner with features for secrets so you can get started without vault too. (I work for HashiCorp on Nomad)
- dfinninger 4y ago
  At the core of K8s is an Etcd cluster, so if you’re spinning your own it’s the same thing. However, part of the value-add of cloud providers in the K8s space is that they hide it from you.
  - figassis 4y ago
    I understand that, but I already invested the time to work with etcd and can handle a few of k8s failure modes. Really do not want to throw that away for something that has a similar dependency, and that I do not now know well. Nomad might be simpler to use, but not to setup or manage due to Consul dependency. Also, a lot of these products are missing a lot of documentation. I often find the tutorials are inflexible and focus on only 2 use cases out of the 4 most common. For example, focus on cloud when lots of engineers would rather deploy on bare metal. I would love for there so be one simple Nomad ansible role that would help setup nomad on bare metal with wireguard networking. This would probably be easy if consul was not required and it did not need so much babysitting.
    - sofixa 4y ago
      Just throwing this out there: Since Nomad 1.3, Consul is no longer required for Service Discovery, and since 1.4 Vault will no longer be needed for KV secrets. And for what it's worth, i used to run Nomad+Consul in production on bare metal and it was easy to setup (2x 3 servers with a binary to start with some basic conf) and maintain (an update every month or so, which consisted of upgrading the binary on each node and restarting the service after 5 mins of reading the release notes).
oxfordmale 4y ago
Why are people using Terraform, a relatively horrible and inconsistent tool to deploy their software as code ? As it is better than the alternatives and everyone is using it. It is the same with K8s. Yes, it is complex, however, as everyone is using it it is super easy to find documentation or example for pretty much anything. Finding the same documentation for Nomad is much harder. Career wise all major platforms (Azure, GCP, and AWS) offer K8s, and therefore it is much better to learn the principles of K8s instead of focussing on the relatively obscure Nomad platform.

by sofixa 4y ago

v3ss0n 4y ago

But Only problem of nomad is all examples are running on Vagrant which is designed for VMs . Everyone who is looking these days are containerization. There may be non-standard ways to run Nomad with containers but , putting Vagrant as main example is not a good idea.

zozbot234 4y ago
Vagrant supports containers since v1.6 (released in 2014).
- v3ss0n 4y ago
  yeah but examples uses vagrant vm, i am just checking nomad and use of vm discourages me / not sure how much compatible if i to placed them with nomad/lxd/dockers. Any good tutorials with Noman + Containers would be nice for CI/CD/Executor setup.
sofixa 4y ago
I'm not sure i follow. Nomad is an orchestrator which handles containers and other things (firecracker microVMs, LXC, etc.). The Vagrant examples are to get started and test it out, because running a container orchestrator in containers has some non-obvious pitfalls. You can just as well download the binary anywhere to play with it. That doesn't mean Nomad can't orchestrate containers (in fact that's the main use case for it).
ViViDboarder 4y ago
All the examples I’ve found are Docker… Most use of Vagrant I saw was in demoing how to set up and deploy a cluster. So deploying 3 Nomad nodes within 3 VMs created in Vagrant.

thejosh 4y ago

I was amazed how easy it was to setup Nomad. I have a couple of VMs running on a VPS and Nomad works great for this use case. Super easy to maintain and works really well. Networking is my favourite part.

ofrzeta 4y ago

Last week, when that same discussion popped up, "schmichael" (who works on Nomad I understand) said "Nomad supports CSI like Kubernetes. Unfortunately many CSI storage providers assume kubernetes or adhere to kubernetes behavior even when it deviates from the spec, so there are gotchas." https://news.ycombinator.com/item?id=32588931 https://news.ycombinator.com/item?id=32588931 That makes be a bit wary to use it because I would like my cloud storage to be reliable.

sofixa 4y ago
Yep, some CSI providers do that, but as far as i know it's more niche ones (like VMware's), not any of the big cloud providers.
- candiddevmike 4y ago
  I tried using a CSI driver with Nomad (AWS) and AFAIK you have to pre-create the EBS volumes? I couldn't figure out how to spin them up on demand.
  - sofixa 4y ago
    That was probably before the Nomad 1.3 release which added a bunch of new CSI features, including dynamic volume creation IIRC.
- orthoxerox 4y ago
  I wouldn't call vmware a niche provider, they own a large share of on prem virualization market.
  - sofixa 4y ago
    Which is a dying breed, and furthermore running an orchestrator on top of another one (Nomad or Kubernetes on top of vSphere) is a terrible anti-pattern that wastes CPU and makes performance issues hard to debug due to the two layers.

maccard 4y ago

My only problem with nomad is that I have to manage the infrastructure underneath as well. Give me nomad-on-fargate and I'll happily jump ship.

figassis 4y ago

My issue with nomad is I also have to handle a second cluster, Consul, and that's it's own overhead. I'd rather just stay with k8s if the alternative is Nomad. Id totally jump to Nomad if all I had to do was deploy nomad, and consul was embedded in all nodes. I'm not an expert in consul, and really don't want another time sink.

schmichael 4y ago
Nomad 1.3 has builtin service discovery and 1.4 is around the corner with features for secrets so you can get started without vault too. (I work for HashiCorp on Nomad)
dfinninger 4y ago
At the core of K8s is an Etcd cluster, so if you’re spinning your own it’s the same thing. However, part of the value-add of cloud providers in the K8s space is that they hide it from you.
- figassis 4y ago
  I understand that, but I already invested the time to work with etcd and can handle a few of k8s failure modes. Really do not want to throw that away for something that has a similar dependency, and that I do not now know well. Nomad might be simpler to use, but not to setup or manage due to Consul dependency. Also, a lot of these products are missing a lot of documentation. I often find the tutorials are inflexible and focus on only 2 use cases out of the 4 most common. For example, focus on cloud when lots of engineers would rather deploy on bare metal. I would love for there so be one simple Nomad ansible role that would help setup nomad on bare metal with wireguard networking. This would probably be easy if consul was not required and it did not need so much babysitting.
  - sofixa 4y ago
    Just throwing this out there: Since Nomad 1.3, Consul is no longer required for Service Discovery, and since 1.4 Vault will no longer be needed for KV secrets. And for what it's worth, i used to run Nomad+Consul in production on bare metal and it was easy to setup (2x 3 servers with a binary to start with some basic conf) and maintain (an update every month or so, which consisted of upgrading the binary on each node and restarting the service after 5 mins of reading the release notes).

oxfordmale 4y ago

Why are people using Terraform, a relatively horrible and inconsistent tool to deploy their software as code ? As it is better than the alternatives and everyone is using it. It is the same with K8s. Yes, it is complex, however, as everyone is using it it is super easy to find documentation or example for pretty much anything. Finding the same documentation for Nomad is much harder. Career wise all major platforms (Azure, GCP, and AWS) offer K8s, and therefore it is much better to learn the principles of K8s instead of focussing on the relatively obscure Nomad platform.