3 ms·

Went to nomad, which is working better for my workloads. There's still use-cases where k8s wins; but nomad handles state a bit better and is easier to reason a

by dijit 4y ago

Went to nomad, which is working better for my workloads. There's still use-cases where k8s wins; but nomad handles state a bit better and is easier to reason about from scratch.

bluehatbrit 4y ago
I really like the look of nomad and want to give it a go. The two things holding me back are: 1) I don't really want to manage the installation but there aren't any(?) cloud hosts for nomad that I can see. 2) It doesn't seem as widely used so community support seems thin. There aren't many blog posts about good patterns with it etc, and I'd worry that we'd get stuck and end up reverting back to k8s.
- dradtke 4y ago
  HCP might be what you want, but it doesn't support Nomad yet, and unfortunately it's not clear when it will. https://discuss.hashicorp.com/t/status-of-hcp-nomad/33374 https://discuss.hashicorp.com/t/status-of-hcp-nomad/33374
- rahen 4y ago
  There is no installation needed with Nomad, it's a standalone binary. Just fire it in a small Debian (or Alma) instance on EC2 or GCE and you're done. That should solve point 1. Point 2 is debatable. Lots of people nowadays put Kubernetes on their resume but that doesn't mean they are great architects or technicians, yet a good part of running production on Kubernetes is doing it right. You'll see much fewer people with Nomad on their resume, but on the other hand you know they're not here for the buzz, they're usually more experienced and know what they're talking about.
riadsila 4y ago
Koyeb also moved off Kubernetes and went with Nomad. We started with Kubernetes, thinking it was the right abstraction layer for us to build our platform, but then quickly ran into major limitations. The big ones: as others have mentioned in this thread, its complexity; security (we wanted to explore using Firecracker on Kubernetes, but it was very experimental at that time); we were not interested in keeping up with its release cycles; global and multi-zone deployments was not as straightforward as we needed; and the overhead (10-25% of RAM) was a cost we were not willing to take (we are around 100MB with our new architecture). We wrote about our decision to switch here: https://www.koyeb.com/blog/the-koyeb-serverless-engine-from-kubernetes-to-nomad-firecracker-and-kuma https://www.koyeb.com/blog/the-koyeb-serverless-engine-from-...
AtNightWeCode 4y ago
Nomad replaces parts of K8. It is not a drop-in replacement. If one only want the container orchestration that is fine but then you need Consul for service discovery and so on.
- wlonkly 4y ago
  I've described this as "Nomad is a container orchestrator, while Kubernetes has a container orchestrator". Nomad, Consul and Vault interoperate extremely well and are mostly pleasant to use, but I found myself missing the rest of the ecosystem pretty quickly, especially around ingresses, and I think they made the wrong decision on the networking model compared to Kubernetes. That said, I haven't played with Consul Connect, the Consul+Envoy service mesh, yet. That might address a lot of the problems. But fundamentally I can't help but think that Nomad and Kubernetes both made a run of it and Kubernetes came out the winner of mindshare and ecosystem.
  - chucky_z 4y ago
    Traefik and Nomad play very nice together if ingress is your main concern.
- chucky_z 4y ago
  This is not true anymore. They released service discovery a little bit ago.
  - AtNightWeCode 4y ago
    Thank you, I did not know that. It seems a bit limited in comparison to Consul but it would probably work in many cases.
    - chucky_z 4y ago
      They're also adding in a basic secrets k/v store in the next version. Their intention (I believe) is to target small use-cases and IOT use-cases; while allowing folks to scale to gigantic levels when mixing in Consul/Vault.
bogomipz 4y ago
>"There's still use-cases where k8s wins; but nomad handles state a bit better and is easier to reason about from scratch." Can you elaborate on how Nomad handles state differently than K8S and what makes it better?

by dijit 4y ago

Went to nomad, which is working better for my workloads. There's still use-cases where k8s wins; but nomad handles state a bit better and is easier to reason about from scratch.

bluehatbrit 4y ago

I really like the look of nomad and want to give it a go. The two things holding me back are: 1) I don't really want to manage the installation but there aren't any(?) cloud hosts for nomad that I can see. 2) It doesn't seem as widely used so community support seems thin. There aren't many blog posts about good patterns with it etc, and I'd worry that we'd get stuck and end up reverting back to k8s.

dradtke 4y ago
HCP might be what you want, but it doesn't support Nomad yet, and unfortunately it's not clear when it will. https://discuss.hashicorp.com/t/status-of-hcp-nomad/33374 https://discuss.hashicorp.com/t/status-of-hcp-nomad/33374
rahen 4y ago
There is no installation needed with Nomad, it's a standalone binary. Just fire it in a small Debian (or Alma) instance on EC2 or GCE and you're done. That should solve point 1. Point 2 is debatable. Lots of people nowadays put Kubernetes on their resume but that doesn't mean they are great architects or technicians, yet a good part of running production on Kubernetes is doing it right. You'll see much fewer people with Nomad on their resume, but on the other hand you know they're not here for the buzz, they're usually more experienced and know what they're talking about.

riadsila 4y ago

Koyeb also moved off Kubernetes and went with Nomad. We started with Kubernetes, thinking it was the right abstraction layer for us to build our platform, but then quickly ran into major limitations. The big ones: as others have mentioned in this thread, its complexity; security (we wanted to explore using Firecracker on Kubernetes, but it was very experimental at that time); we were not interested in keeping up with its release cycles; global and multi-zone deployments was not as straightforward as we needed; and the overhead (10-25% of RAM) was a cost we were not willing to take (we are around 100MB with our new architecture). We wrote about our decision to switch here: https://www.koyeb.com/blog/the-koyeb-serverless-engine-from-kubernetes-to-nomad-firecracker-and-kuma https://www.koyeb.com/blog/the-koyeb-serverless-engine-from-...

AtNightWeCode 4y ago

Nomad replaces parts of K8. It is not a drop-in replacement. If one only want the container orchestration that is fine but then you need Consul for service discovery and so on.

wlonkly 4y ago
I've described this as "Nomad is a container orchestrator, while Kubernetes has a container orchestrator". Nomad, Consul and Vault interoperate extremely well and are mostly pleasant to use, but I found myself missing the rest of the ecosystem pretty quickly, especially around ingresses, and I think they made the wrong decision on the networking model compared to Kubernetes. That said, I haven't played with Consul Connect, the Consul+Envoy service mesh, yet. That might address a lot of the problems. But fundamentally I can't help but think that Nomad and Kubernetes both made a run of it and Kubernetes came out the winner of mindshare and ecosystem.
- chucky_z 4y ago
  Traefik and Nomad play very nice together if ingress is your main concern.
chucky_z 4y ago
This is not true anymore. They released service discovery a little bit ago.
- AtNightWeCode 4y ago
  Thank you, I did not know that. It seems a bit limited in comparison to Consul but it would probably work in many cases.
  - chucky_z 4y ago
    They're also adding in a basic secrets k/v store in the next version. Their intention (I believe) is to target small use-cases and IOT use-cases; while allowing folks to scale to gigantic levels when mixing in Consul/Vault.

bogomipz 4y ago

>"There's still use-cases where k8s wins; but nomad handles state a bit better and is easier to reason about from scratch." Can you elaborate on how Nomad handles state differently than K8S and what makes it better?