4 ms·

To most people, it's not Nomad vs Kubernetes - it's a choice between Nomad vs Managed Kubernetes. All major cloud providers offer a managed kubernetes service

by shrumm 5y ago

To most people, it's not Nomad vs Kubernetes - it's a choice between Nomad vs Managed Kubernetes. All major cloud providers offer a managed kubernetes service at minimal added cost to running the worker VMs yourself. With managed Kubernetes, the simplicity question is no longer obviously in Nomad's favour. As other comments allude to, Kubernetes as a user is pretty easy to master once you get used to its mental model.

pid-1 5y ago
Agreed. Managing a Nomad cluster alongside a Consul cluster does not require a PHD, but it's also not a walk in the park. Hopefully Hashicorp will have managed Nomad soon.
mac-chaffee 5y ago
I'm currently trying to convince people that a managed k8s service is not that "simple", and that we can't "just spin up another cluster" without a great deal of operational overhead. Some of the things that might still be needed in managed k8s instances: better ingress with ingress-nginx, cert-manager, monitoring/logging/alerting, tuning the alerts, integration with company SSO, security hardening. If it's a multi-tenant cluster: LimitRanges/ResourceQuotas, NetworkPolicies, scripts to manage namespaces and roles, PodSecurityPolicies (or equivalent), onboarding offboarding procedures. I'm sure you'd need similar things to have a proper production Nomad cluster too, so your point still stands. But at least for EKS/GKE clusters, they're pretty bare-bones.
- darkwater 5y ago
  Thanks for this, I was really starting to think I was the madman.
- busterarm 5y ago
  As someone with 7-digit spend in GKE/EKS, I will agree with you that it is _anything but simple_. Your developers aren't going to say that it's simple when Google force upgrades their cluster to a version that deprecates APIs in their yamls for a job they worked on 2 years ago and swiftly forgot about. Then when you explain to them that Google insists on putting everyone on a force-upgrade treadmill, you can literally watch as the panic sets in on the faces of your engineering team managers/leads. Nomad is a breeze in comparison to managed K8s. Everyone that I've talked to that thinks Kubernetes is simple is barely using the thing and could likely save a lot of money and development effort using something like Nomad instead.
  - jpgvm 5y ago
    So instead of using higher level primitives that are widely used and tested and have many simple high quality integrations (external-dns, cert-manager, et al) you would recommend reinventing all of that on Nomad and then calling that "saving a lot of money and development effort"? Yeah no thanks. At this point k8s has "won" for all intents and purposes. It has gained critical mass, succeeding where other infrastructure management tools both open and closed source have failed. Also API upgrades shouldn't be a problem, even if you were using beta APIs. They are only really troublesome if you decided to indulge yourself with some alpha APIs before they were fully baked. If you don't do that then you won't run into any problems.
    - busterarm 5y ago
      Thanks, Mr. Well Ackshually, First of all, those things you're talking about are installables in K8s. They don't come by default. Many people with (managed) kubernetes installations aren't even using them and get by just fine. Having them certainly isn't free and work certainly had to be done to build those for Kubernetes and likely are (or will be) trivial to implement elsewhere. I certainly was able to automate my certificate management infrastructure before I had Kubernetes. The reality is that there are many companies out there that want like 10% of the features of an orchestrator like Kubernetes and don't need all of those features that make you think that it's a zero-sum game that Kubernetes has won. The reality is that there are competing offers like Nomad that more-easily accomplish our technology goals and thus more attractive to enterprises with big budgets like mine. Kuberenetes having the "critical-mass" that you speak of isn't to the exclusion of other competing tools having "critical-mass". Just like Oracle and DB2 aren't the dominant RDBMS of today. As for calling people indulgent for using beta APIs, let's not forget that Deployment, StatefulSet, DaemonSet, ReplicaSet, NetworkPolicy and PodSecurityPolicy all started as beta APIs and were only removed in 1.16. And I imagine you wanted to use ingresses, but the old ingress.class annotation was deprecated and replaced with IngressClass in 1.18 and that was firm cutover for _everyone_. I can't imagine Kubernetes being very useful to many people without any of these...
      - jpgvm 5y ago
still_grokking 5y ago
> managed kubernetes service at minimal added cost I dare to disagree. The costs are in fact horrific! There are orders of magnitude in costs between running your stuff yourself vs the highest level of managed services on the cloud which is usually managed k8s. (That also explains why there is so much marketing fuss, and push of management, to use k8s: It's by far the most profitable offering for the cloud providers).
- jpgvm 5y ago
  It's not. You generally only pay for the master clusters with no per-node costs over whatever instance type you are using and scheduling class (ondemand, reserved, spot, etc). So not only are the masters relatively cheap, they cost is amortized over the cluster size. AWS is probably very unhappy about this but they were forced into this by GKE pricing model. Google pushed people to adopt k8s not because they can make a lot of money from people using it but because they can take away a lot of money from AWS and increase workload portability, which benefits Google far more than AWS as they are in second place. The most expensive managed services are things like RDS, MSK, Elasticache, etc. MSK in particular is pretty egregious with a 80% premium considering they haven't done any additional engineering like RDS.

by shrumm 5y ago

pid-1 5y ago

Agreed. Managing a Nomad cluster alongside a Consul cluster does not require a PHD, but it's also not a walk in the park. Hopefully Hashicorp will have managed Nomad soon.

mac-chaffee 5y ago

I'm currently trying to convince people that a managed k8s service is not that "simple", and that we can't "just spin up another cluster" without a great deal of operational overhead. Some of the things that might still be needed in managed k8s instances: better ingress with ingress-nginx, cert-manager, monitoring/logging/alerting, tuning the alerts, integration with company SSO, security hardening. If it's a multi-tenant cluster: LimitRanges/ResourceQuotas, NetworkPolicies, scripts to manage namespaces and roles, PodSecurityPolicies (or equivalent), onboarding offboarding procedures. I'm sure you'd need similar things to have a proper production Nomad cluster too, so your point still stands. But at least for EKS/GKE clusters, they're pretty bare-bones.

darkwater 5y ago
Thanks for this, I was really starting to think I was the madman.
busterarm 5y ago
As someone with 7-digit spend in GKE/EKS, I will agree with you that it is _anything but simple_. Your developers aren't going to say that it's simple when Google force upgrades their cluster to a version that deprecates APIs in their yamls for a job they worked on 2 years ago and swiftly forgot about. Then when you explain to them that Google insists on putting everyone on a force-upgrade treadmill, you can literally watch as the panic sets in on the faces of your engineering team managers/leads. Nomad is a breeze in comparison to managed K8s. Everyone that I've talked to that thinks Kubernetes is simple is barely using the thing and could likely save a lot of money and development effort using something like Nomad instead.
- jpgvm 5y ago
  So instead of using higher level primitives that are widely used and tested and have many simple high quality integrations (external-dns, cert-manager, et al) you would recommend reinventing all of that on Nomad and then calling that "saving a lot of money and development effort"? Yeah no thanks. At this point k8s has "won" for all intents and purposes. It has gained critical mass, succeeding where other infrastructure management tools both open and closed source have failed. Also API upgrades shouldn't be a problem, even if you were using beta APIs. They are only really troublesome if you decided to indulge yourself with some alpha APIs before they were fully baked. If you don't do that then you won't run into any problems.
  - busterarm 5y ago
    Thanks, Mr. Well Ackshually, First of all, those things you're talking about are installables in K8s. They don't come by default. Many people with (managed) kubernetes installations aren't even using them and get by just fine. Having them certainly isn't free and work certainly had to be done to build those for Kubernetes and likely are (or will be) trivial to implement elsewhere. I certainly was able to automate my certificate management infrastructure before I had Kubernetes. The reality is that there are many companies out there that want like 10% of the features of an orchestrator like Kubernetes and don't need all of those features that make you think that it's a zero-sum game that Kubernetes has won. The reality is that there are competing offers like Nomad that more-easily accomplish our technology goals and thus more attractive to enterprises with big budgets like mine. Kuberenetes having the "critical-mass" that you speak of isn't to the exclusion of other competing tools having "critical-mass". Just like Oracle and DB2 aren't the dominant RDBMS of today. As for calling people indulgent for using beta APIs, let's not forget that Deployment, StatefulSet, DaemonSet, ReplicaSet, NetworkPolicy and PodSecurityPolicy all started as beta APIs and were only removed in 1.16. And I imagine you wanted to use ingresses, but the old ingress.class annotation was deprecated and replaced with IngressClass in 1.18 and that was firm cutover for _everyone_. I can't imagine Kubernetes being very useful to many people without any of these...
    - jpgvm 5y ago

still_grokking 5y ago

> managed kubernetes service at minimal added cost I dare to disagree. The costs are in fact horrific! There are orders of magnitude in costs between running your stuff yourself vs the highest level of managed services on the cloud which is usually managed k8s. (That also explains why there is so much marketing fuss, and push of management, to use k8s: It's by far the most profitable offering for the cloud providers).

jpgvm 5y ago
It's not. You generally only pay for the master clusters with no per-node costs over whatever instance type you are using and scheduling class (ondemand, reserved, spot, etc). So not only are the masters relatively cheap, they cost is amortized over the cluster size. AWS is probably very unhappy about this but they were forced into this by GKE pricing model. Google pushed people to adopt k8s not because they can make a lot of money from people using it but because they can take away a lot of money from AWS and increase workload portability, which benefits Google far more than AWS as they are in second place. The most expensive managed services are things like RDS, MSK, Elasticache, etc. MSK in particular is pretty egregious with a 80% premium considering they haven't done any additional engineering like RDS.