7 ms·

There seems to be plenty of reasons to run Nomad, compared to Kubernetes, but in what scenarios do Nomad lose out to Kubernetes? Is it simply a matter of Kuber

There seems to be plenty of reasons to run Nomad, compared to Kubernetes, but in what scenarios do Nomad lose out to Kubernetes? Is it simply a matter of Kubernetes being an open source project and Nomad being owned by HashiCorp?

stavros 5y ago
Nomad is an open source project as well though.
nuerow 5y ago
> There seems to be plenty of reasons to run Nomad, compared to Kubernetes, but in what scenarios do Nomad lose out to Kubernetes? It would also be interesting if Docker's Swarm was.also featured in this comparison, as it just works right out of the box and doesn't come with any bells and whistles.
- alvarlagerlof 5y ago
  Does it support multiple nodes?
  - nuerow 5y ago
    > Does it support multiple nodes? Yes it does. Docker Swarm essentially provides an easy way to get a cluster of Docker instances running in multiple nodes that works right out of the box. https://docs.docker.com/engine/swarm/ https://docs.docker.com/engine/swarm/
- zelphirkalt 5y ago
  As far as I remember from trying last time a few years ago, Docker Swarm does not support rebalancing the load of containers and requires either manually issuing a command to do that or merely initially balances where containers are started. It this still true? And what about when you have state, which you would need to move between nodes, to make a container work elsewhere?
- proxysna 5y ago
  Swarm is great for small workloads (hosts =< 10). Networking is slow, but if you need something to just run containers over a set of hosts, you won't find anything simpler than that. Sadly Mirantis (company who now owns docker.inc) is intentionally neglecting it.
WJW 5y ago
I don't think being OSS has much to do with it; it's more that the dominance of k8s leads to more people learning it, which leads to more tooling being written, which makes k8s better, which leads to more companies adopting it and more people learning it. It's a virtuous cycle much like programming languages experience.
q3k 5y ago
Kubernetes is much more than container scheduling. Custom resources, identity and a powerful RBAC system allow you to use it as a general configuration/operational data store in your own code, from implementing operators acting upon kubernetes and the outside world to even moving most of high-level configuration glue to be natively based on Kubernetes. For example, with cert-manager running on Kubernetes you can request a TLS certificate by creating a Certificate resource (like you would any other Kubernetes resource). This is the same regardless of whether you want a self-signed certificate, an ACME-issued certificate (and whether that gets performed via HTTP01 or DNS01 or something else). Oh, and this fully ties into Kubernetes' RBAC system. In Nomad the closest thing is annotating jobs with traefik-specific tags (and allowing Traefik to do cluster-wide discovery of all tags), but that only works for serving certificates that are managed by traefik, not if your application wants eg. to terminate the TLS connection itself, or if it wants some other PKI hierarchy (eg. a self-signed CA which then issues some other certificates for mutual TLS auth between application services). Kubernetes also has better support for organization-wide multi-tenant clusters than Nomad seems to have (eg. nomad's policy engine, audit logging and resource quota system are gated behind their “enterprise” offering).
- mrweasel 5y ago
  > Kubernetes also has better support for organization-wide multi-tenant clusters than Nomad seems to have That one's a little weird, I suppose you're right, but all the clients our Kubernetes team works with all want separate clusters or testing, staging and preproduction. They certainly don't want a multi-tenant cluster and share resources with other clients.
  - q3k 5y ago
    > but all the clients our Kubernetes team works with all want separate clusters or testing, staging and preproduction. And I think that's one of the biggest issues with how people use Kubernetes these days (another candidate being insisting on drive-by deploying a cluster from scratch instead of deferring to cloud providers or a dedicated platform team that can plan for long-term maintenance). Kubernetes thrives in multi-tenant environments: you get huge resource savings and vastly simplified operations. Everyone in your organization gets access to all clusters, and they can just as easily deploy experimental best effort jobs or their development environment as they can deploy and/or inspect production jobs. Well set up quotas and priority classes mean that production jobs never run out of resources, while less important stuff (batch jobs, CI, someone's pet experiment) can continue to run on a best effort basis, just keeps getting preempted when production wants more resources. You can even continue to have hardware separation between highly sensitive and fully untrusted jobs by using taints and tolerations, if you feel that's necessary. You still get one control plane instead of five different ones.
AmazingTurtle 5y ago
Nomad vs kubernetes is like comparing apples with pears. They actually say it in their docs, that kubernetes is a complex solution whereas nomad isn't. For example when building a frontend, you don't go with react or building everything yourself, you use a framework that brings plenty of things at once that play well in an eco system. What's the matter of bringing a ton of hashicorp tools together manually? You can just use kubernetes. I'd say setting up kubernetes is just as complex as putting together a bunch of hashicorp solutions. And ultimately people (the majority) simply love all-in-one solutions
maccard 5y ago
I can't run nomad on aws without managing a nomad cluster myself. I _can_ do that with k8s.
- orthecreedence 5y ago
  How have you found the managed k8s on AWS? I tend to shy away from their managed services until it has gone through a 4-5 year grace period, interested to know what your experience has been.
  - maccard 5y ago
    The short answer is it's less hassle than the local cluster I run locally for development purposes, to the point that were considering spinning up eks clusters for development use rather than running kind or k3s locally.
pid-1 5y ago
I've been using Nomad for about 3 years and now I've setup a K8s cluster as well. I love Nomad and so do our devs, but K8s has a much larger ecosystem and more tooling options. A few examples: - If you want to run some third party software, there's a helm chart for that. In Nomad, you're likely going to figure things for yourself. - K8s has a lot of cool stuff like Knative, KEDA and others for more sophisticated deployments. - K8s has tons of operators to integrate with public clouds and external services. Nomad really lacks something similar. - There's a ton of knowledge and guides related to K8s online, not so much for Nomad. - There are many managed K8s solutions, which makes things easier. To this date Hashicorp still does not have managed Nomad.
film42 5y ago
We tried for multiple years to make Nomad work because it's simple. We're already enterprise customers of Hashicorp, too. We love Hashicorp! Nomad is a great scheduler and the UI is wonderful, but there is no story around network security. Example: You have postgres running on dedicated metal, and you want to restrict which nomad services have access to postgres. Consul Connect allows you to enforce access policies, but these work more like haproxy with automated service discovery. There is no batteries-included way to prevent traffic from leaving your container (or identify which container the traffic is originating from). You can use a custom CNI plugin to give each container an IP address from a unique subnet per service (and restrict traffic at your firewall/ selectively add subnets to iptabels on your postgres server), but now we're adding bespoke complexity. We brought this problem up to the Hashicorp internal teams over many calls, but ultimately they said we'd need to pay for a 3rd party overseas consultant to advise us on implementation. They weren't sure what to do. K8s is certainly more complex out the gate, but you don't feel like you're the first person in the world to implement. That said, I think Nomad is a few years away from being something truly amazing. If they can solve the network security problem (beyond optimistic documentation), I think it'll be amazing. For now, it's just a good scheduler.

stavros 5y ago

Nomad is an open source project as well though.

nuerow 5y ago

> There seems to be plenty of reasons to run Nomad, compared to Kubernetes, but in what scenarios do Nomad lose out to Kubernetes? It would also be interesting if Docker's Swarm was.also featured in this comparison, as it just works right out of the box and doesn't come with any bells and whistles.

alvarlagerlof 5y ago
Does it support multiple nodes?
- nuerow 5y ago
  > Does it support multiple nodes? Yes it does. Docker Swarm essentially provides an easy way to get a cluster of Docker instances running in multiple nodes that works right out of the box. https://docs.docker.com/engine/swarm/ https://docs.docker.com/engine/swarm/
zelphirkalt 5y ago
As far as I remember from trying last time a few years ago, Docker Swarm does not support rebalancing the load of containers and requires either manually issuing a command to do that or merely initially balances where containers are started. It this still true? And what about when you have state, which you would need to move between nodes, to make a container work elsewhere?
proxysna 5y ago
Swarm is great for small workloads (hosts =< 10). Networking is slow, but if you need something to just run containers over a set of hosts, you won't find anything simpler than that. Sadly Mirantis (company who now owns docker.inc) is intentionally neglecting it.

WJW 5y ago

I don't think being OSS has much to do with it; it's more that the dominance of k8s leads to more people learning it, which leads to more tooling being written, which makes k8s better, which leads to more companies adopting it and more people learning it. It's a virtuous cycle much like programming languages experience.

q3k 5y ago

Kubernetes is much more than container scheduling. Custom resources, identity and a powerful RBAC system allow you to use it as a general configuration/operational data store in your own code, from implementing operators acting upon kubernetes and the outside world to even moving most of high-level configuration glue to be natively based on Kubernetes. For example, with cert-manager running on Kubernetes you can request a TLS certificate by creating a Certificate resource (like you would any other Kubernetes resource). This is the same regardless of whether you want a self-signed certificate, an ACME-issued certificate (and whether that gets performed via HTTP01 or DNS01 or something else). Oh, and this fully ties into Kubernetes' RBAC system. In Nomad the closest thing is annotating jobs with traefik-specific tags (and allowing Traefik to do cluster-wide discovery of all tags), but that only works for serving certificates that are managed by traefik, not if your application wants eg. to terminate the TLS connection itself, or if it wants some other PKI hierarchy (eg. a self-signed CA which then issues some other certificates for mutual TLS auth between application services). Kubernetes also has better support for organization-wide multi-tenant clusters than Nomad seems to have (eg. nomad's policy engine, audit logging and resource quota system are gated behind their “enterprise” offering).

mrweasel 5y ago
> Kubernetes also has better support for organization-wide multi-tenant clusters than Nomad seems to have That one's a little weird, I suppose you're right, but all the clients our Kubernetes team works with all want separate clusters or testing, staging and preproduction. They certainly don't want a multi-tenant cluster and share resources with other clients.
- q3k 5y ago
  > but all the clients our Kubernetes team works with all want separate clusters or testing, staging and preproduction. And I think that's one of the biggest issues with how people use Kubernetes these days (another candidate being insisting on drive-by deploying a cluster from scratch instead of deferring to cloud providers or a dedicated platform team that can plan for long-term maintenance). Kubernetes thrives in multi-tenant environments: you get huge resource savings and vastly simplified operations. Everyone in your organization gets access to all clusters, and they can just as easily deploy experimental best effort jobs or their development environment as they can deploy and/or inspect production jobs. Well set up quotas and priority classes mean that production jobs never run out of resources, while less important stuff (batch jobs, CI, someone's pet experiment) can continue to run on a best effort basis, just keeps getting preempted when production wants more resources. You can even continue to have hardware separation between highly sensitive and fully untrusted jobs by using taints and tolerations, if you feel that's necessary. You still get one control plane instead of five different ones.

AmazingTurtle 5y ago

Nomad vs kubernetes is like comparing apples with pears. They actually say it in their docs, that kubernetes is a complex solution whereas nomad isn't. For example when building a frontend, you don't go with react or building everything yourself, you use a framework that brings plenty of things at once that play well in an eco system. What's the matter of bringing a ton of hashicorp tools together manually? You can just use kubernetes. I'd say setting up kubernetes is just as complex as putting together a bunch of hashicorp solutions. And ultimately people (the majority) simply love all-in-one solutions

maccard 5y ago

I can't run nomad on aws without managing a nomad cluster myself. I _can_ do that with k8s.

orthecreedence 5y ago
How have you found the managed k8s on AWS? I tend to shy away from their managed services until it has gone through a 4-5 year grace period, interested to know what your experience has been.
- maccard 5y ago
  The short answer is it's less hassle than the local cluster I run locally for development purposes, to the point that were considering spinning up eks clusters for development use rather than running kind or k3s locally.

pid-1 5y ago

I've been using Nomad for about 3 years and now I've setup a K8s cluster as well. I love Nomad and so do our devs, but K8s has a much larger ecosystem and more tooling options. A few examples: - If you want to run some third party software, there's a helm chart for that. In Nomad, you're likely going to figure things for yourself. - K8s has a lot of cool stuff like Knative, KEDA and others for more sophisticated deployments. - K8s has tons of operators to integrate with public clouds and external services. Nomad really lacks something similar. - There's a ton of knowledge and guides related to K8s online, not so much for Nomad. - There are many managed K8s solutions, which makes things easier. To this date Hashicorp still does not have managed Nomad.

film42 5y ago

We tried for multiple years to make Nomad work because it's simple. We're already enterprise customers of Hashicorp, too. We love Hashicorp! Nomad is a great scheduler and the UI is wonderful, but there is no story around network security. Example: You have postgres running on dedicated metal, and you want to restrict which nomad services have access to postgres. Consul Connect allows you to enforce access policies, but these work more like haproxy with automated service discovery. There is no batteries-included way to prevent traffic from leaving your container (or identify which container the traffic is originating from). You can use a custom CNI plugin to give each container an IP address from a unique subnet per service (and restrict traffic at your firewall/ selectively add subnets to iptabels on your postgres server), but now we're adding bespoke complexity. We brought this problem up to the Hashicorp internal teams over many calls, but ultimately they said we'd need to pay for a 3rd party overseas consultant to advise us on implementation. They weren't sure what to do. K8s is certainly more complex out the gate, but you don't feel like you're the first person in the world to implement. That said, I think Nomad is a few years away from being something truly amazing. If they can solve the network security problem (beyond optimistic documentation), I think it'll be amazing. For now, it's just a good scheduler.