4 ms·

The main advantage of Kubernetes isn't really Kubernetes anymore — it's the ecosystem of stuff around it and the innumerable vendors you can pay to make your pr

by jxf 6y ago

The main advantage of Kubernetes isn't really Kubernetes anymore — it's the ecosystem of stuff around it and the innumerable vendors you can pay to make your problems someone else's problems.

ozim 6y ago
Up vote because that really is the main thing to consider when evaluating actual needs of a project. My actual need is that I deploy my application and I don't care where or how it is running. Most of people should not even think about running k8s on their own. That should be job of a service provider and there should be service provider for running "Nomad".
orthecreedence 6y ago
The main advantage of Nomad is Nomad. It strikes the perfect balance between simple and capable, enough so that I don't have to pay someone else. You can read the bulk of the docs in a day or two and deploy it that afternoon. Consul/Nomad deployment is easy and operation is a breeze. As the sole ops person at my company, Nomad has made it possible for me to scale our infrastructure without an entire team dedicated to it. I'm not saying that it's better than K8S, but for a small team who wants to focus on building things, it's an amazing time saver. Obviously, read the docs and make your own decisions, but there is a definitely an infrastructure void that Nomad has filled perfectly.
- peter_l_downs 6y ago
  As a counterpoint, everything you just said about Consul/Nomad applies to using hosted K8s like GKE. I moved my company's infra to it last year and now we barely think about it. We're a small team, I'm the only person doing devops, and there are very, very few problems requiring my attention. I'm skeptical that anything allowing flexibility in this space can ever be that simple because the domain is actually pretty complicated. Networking, secrets, scaling, healthchecking, deployment strategies, containerization -- you need to know something about all of these before you can use Nomad or Kubernetes.
  - AnxiousCoward 6y ago
    K8s is complicated because it does many things, not because each of the things it does, individually, is complicated. If your orchestrator only runs stuff, if your service discovery is independent of the orchestrator, and just does the wiring and routing, if your secrets management is simply a thing that provides secrets to authorized clients, a lot of the complexity associated with kubernetes goes away. The whole is always more than the sum of its parts, and in the case of kubernetes this adds up in an unfavorable way, complexity-wise. IME, a lot of complexity, in kubernetes land, also comes from kubernetes being highly opinionated. Whenever your needs don't match kubernetes' opinions, you have to fight it. To give you an example, service versioning is not something kubernetes knows about. If you need it, you need to add it manually. You can easily automate this with consul, and make applications unaware of service versions. Reciprocal trust verification between ... things that communicate is another. Kubernetes provides nothing. If someone manages to inject a pod into your namespace, and randomly starts replaying requests, there's no automatic way to verify the caller's identity - you have to do this manually, at the application level, using a custom sidecar (if you're lucky, your service mesh has something to help you). If you use consul with nomad, you get these things out of the box. I believe your life is simple now because what you manage isn't overly complicated. Once you have thousands of distinct deployments, versioned, across several clusters, and thousands or even tens of thousands of pods running in tens of data centers, life with kubernetes becomes a lot more difficult. Of course, you can still manage, by automating everything, but automation, at that level, with kubernetes, is not an easy thing. Or at least that's my experience.
- jxf 6y ago
  I think that is the point, though. If you're small and you want to focus on building things, it doesn't matter very much what it's running on; it's almost always going to be a better use of your time to make it someone else's problem. If you're doing your own bespoke ops or you need to own more of the stack, then Nomad is a powerful choice.
- rualca 6y ago
  > The main advantage of Nomad is Nomad. Is it, really? It feels like it is it's main disadvantage, in the sense that Kubernetes is a first-class citizen offered by practically all major cloud providers, while Nomad... Is anyone actually offering Nomad at all?
  - jamesog 6y ago
    Managed Kubernetes exists because Kubernetes is so mind-bendingly complex. Nomad, on the other hand, is pretty simple and easy to run, and a single binary. There's probably no benefit to having a managed offering. That said, I wouldn't be surprised if HashiCorp add Nomad to HashiCorp Cloud Platform, which currently lets you deploy Consul and Vault to cloud providers via their system.
    - jpgvm 6y ago
      Managed Kubernetes exists for the same reason managed PostgreSQL, MySQL, Redis, ElasticSearch so on and so forth exist. Which is to say it's useful enough to a large amount of people to make it viable as a product offering. Nomad would fail as a managed offering here like like Deis, Flynn, Convox and probably 100 other container management platforms that came before. As a niche tool to manage sub-1000 boxen it's probably ok. But k8s has won the greater war. Disclaimer: Worked on Flynn, still run it for my own personal stuff by $DAY_JOB is all k8s.
      - AnxiousCoward 6y ago
        You got that sub-1000 boxen wrong. Nomad is usually employed where the infrastructure is huge. IMO, K8s is driven by hype and marketing, not by technical merit. Your enumeration of the managed things is very particular, in that it includes only stores. There's a reason for things that have keeping persistent state as their reason to exist are managed: reliably maintaining persistent state in the cloud is a lot more difficult than reliably orchestrating things that just have to run.
    - tutfbhuf 6y ago
      If public cloud providers would offer Nomad, then it would only be a question of time until managed Nomad would be a thing. Managed _something_ does not mean it's "mind-bendingly complex", but rather that people don't want to take care about it and focus on their own stuff like building applications.
KaiserPro 6y ago
or, just hear me out, not use k8s and avoid 90% of the problems in the first place. There are some specific places that k8s can shine, but easy scaling(1000 nodes +) and low management overhead isn't one of them.
AnxiousCoward 6y ago
In reality, this is always an illusion, IME. You always have to deal with the platform yourself. You can hire a team to do this dealing with you, but that's not cost effective.

by jxf 6y ago

The main advantage of Kubernetes isn't really Kubernetes anymore — it's the ecosystem of stuff around it and the innumerable vendors you can pay to make your problems someone else's problems.

ozim 6y ago

Up vote because that really is the main thing to consider when evaluating actual needs of a project. My actual need is that I deploy my application and I don't care where or how it is running. Most of people should not even think about running k8s on their own. That should be job of a service provider and there should be service provider for running "Nomad".

orthecreedence 6y ago

The main advantage of Nomad is Nomad. It strikes the perfect balance between simple and capable, enough so that I don't have to pay someone else. You can read the bulk of the docs in a day or two and deploy it that afternoon. Consul/Nomad deployment is easy and operation is a breeze. As the sole ops person at my company, Nomad has made it possible for me to scale our infrastructure without an entire team dedicated to it. I'm not saying that it's better than K8S, but for a small team who wants to focus on building things, it's an amazing time saver. Obviously, read the docs and make your own decisions, but there is a definitely an infrastructure void that Nomad has filled perfectly.

peter_l_downs 6y ago
As a counterpoint, everything you just said about Consul/Nomad applies to using hosted K8s like GKE. I moved my company's infra to it last year and now we barely think about it. We're a small team, I'm the only person doing devops, and there are very, very few problems requiring my attention. I'm skeptical that anything allowing flexibility in this space can ever be that simple because the domain is actually pretty complicated. Networking, secrets, scaling, healthchecking, deployment strategies, containerization -- you need to know something about all of these before you can use Nomad or Kubernetes.
- AnxiousCoward 6y ago
  K8s is complicated because it does many things, not because each of the things it does, individually, is complicated. If your orchestrator only runs stuff, if your service discovery is independent of the orchestrator, and just does the wiring and routing, if your secrets management is simply a thing that provides secrets to authorized clients, a lot of the complexity associated with kubernetes goes away. The whole is always more than the sum of its parts, and in the case of kubernetes this adds up in an unfavorable way, complexity-wise. IME, a lot of complexity, in kubernetes land, also comes from kubernetes being highly opinionated. Whenever your needs don't match kubernetes' opinions, you have to fight it. To give you an example, service versioning is not something kubernetes knows about. If you need it, you need to add it manually. You can easily automate this with consul, and make applications unaware of service versions. Reciprocal trust verification between ... things that communicate is another. Kubernetes provides nothing. If someone manages to inject a pod into your namespace, and randomly starts replaying requests, there's no automatic way to verify the caller's identity - you have to do this manually, at the application level, using a custom sidecar (if you're lucky, your service mesh has something to help you). If you use consul with nomad, you get these things out of the box. I believe your life is simple now because what you manage isn't overly complicated. Once you have thousands of distinct deployments, versioned, across several clusters, and thousands or even tens of thousands of pods running in tens of data centers, life with kubernetes becomes a lot more difficult. Of course, you can still manage, by automating everything, but automation, at that level, with kubernetes, is not an easy thing. Or at least that's my experience.
jxf 6y ago
I think that is the point, though. If you're small and you want to focus on building things, it doesn't matter very much what it's running on; it's almost always going to be a better use of your time to make it someone else's problem. If you're doing your own bespoke ops or you need to own more of the stack, then Nomad is a powerful choice.
rualca 6y ago
> The main advantage of Nomad is Nomad. Is it, really? It feels like it is it's main disadvantage, in the sense that Kubernetes is a first-class citizen offered by practically all major cloud providers, while Nomad... Is anyone actually offering Nomad at all?
- jamesog 6y ago
  Managed Kubernetes exists because Kubernetes is so mind-bendingly complex. Nomad, on the other hand, is pretty simple and easy to run, and a single binary. There's probably no benefit to having a managed offering. That said, I wouldn't be surprised if HashiCorp add Nomad to HashiCorp Cloud Platform, which currently lets you deploy Consul and Vault to cloud providers via their system.
  - jpgvm 6y ago
    Managed Kubernetes exists for the same reason managed PostgreSQL, MySQL, Redis, ElasticSearch so on and so forth exist. Which is to say it's useful enough to a large amount of people to make it viable as a product offering. Nomad would fail as a managed offering here like like Deis, Flynn, Convox and probably 100 other container management platforms that came before. As a niche tool to manage sub-1000 boxen it's probably ok. But k8s has won the greater war. Disclaimer: Worked on Flynn, still run it for my own personal stuff by $DAY_JOB is all k8s.
    - AnxiousCoward 6y ago
      You got that sub-1000 boxen wrong. Nomad is usually employed where the infrastructure is huge. IMO, K8s is driven by hype and marketing, not by technical merit. Your enumeration of the managed things is very particular, in that it includes only stores. There's a reason for things that have keeping persistent state as their reason to exist are managed: reliably maintaining persistent state in the cloud is a lot more difficult than reliably orchestrating things that just have to run.
  - tutfbhuf 6y ago
    If public cloud providers would offer Nomad, then it would only be a question of time until managed Nomad would be a thing. Managed _something_ does not mean it's "mind-bendingly complex", but rather that people don't want to take care about it and focus on their own stuff like building applications.

KaiserPro 6y ago

or, just hear me out, not use k8s and avoid 90% of the problems in the first place. There are some specific places that k8s can shine, but easy scaling(1000 nodes +) and low management overhead isn't one of them.

AnxiousCoward 6y ago

In reality, this is always an illusion, IME. You always have to deal with the platform yourself. You can hire a team to do this dealing with you, but that's not cost effective.