9 ms·
Take a look at Nomad before jumping on Kubernetes
- justinko 6y agoOr ECS for that matter.
- sofixa 6y agoIf you're on AWS and don't intend to move, yep, it's a decent option.
- gravypod 6y agoI personally feel HCL (or any specific thing) is a large disadvantage for any orchestration platform. Kubernetes is an API for infrastructure. This is why it uses json/yaml. It doesn't mean your configuration should use json/yaml natively. You can easily write tools that generate json/yaml for you based on your needs. For most applications you're deploying you'll need: 1. A Deployment 2. A Service that points to that Deployment 3. An Ingress or similar In some cases you'll want to manage persistent state on disk and you'll need a StatefulSet. A good example of this can be seen https://youtu.be/keT8ixRS6Fk?t=1317 https://youtu.be/keT8ixRS6Fk?t=1317 and https://youtu.be/muvU1DYrY0w?t=581 https://youtu.be/muvU1DYrY0w?t=581
- okso 6y agoHCL is JSON compatible, and Nomad can load JSON natively.
- mrkurt 6y agoIt can only load json with the api, the nomad CLI can't launch json jobs.
- core-questions 6y agoBut who would use the Nomad CLI for serious work? I use the Terraform provider for Nomad to launch all my jobs.
- mrkurt 6y agoMe? It's handy to be able to dump -> modify -> load jobs when I'm poking at things. No one said anything about serious work, though, they said "load json natively".
- AnxiousCoward 6y agoYou may not use it for automation. But you usually do much exploratory work before doing the automation. That's where the various CLIs for all kinds of things come in really handy.
- hedwall 6y agoYes it can. "For machine-friendliness, Nomad can also read JSON-equivalent configurations. In general, we recommend using the HCL syntax." https://www.nomadproject.io/docs/job-specification#job-specification https://www.nomadproject.io/docs/job-specification#job-speci...
- gravypod 6y agoI assumed that was a possibility but that still means "HCL" isn't a benefit since kubectl can load json from the CLI and from apiserver.
- xyzzy_plugh 6y agoA thousand times this. With Kubernetes, you can even use one of the many native clients (and/or protobuf) which makes converting from something else a breeze, avoiding the JSON/YAML entirely. You get static typing, you can write tests, etc.
- sofixa 6y agoWell, it's same thing with Nomad - there are native SDKs, protobuf and a classical JSON REST API for everything.
- sofixa 6y ago> You can easily write tools that generate json/yaml for you based on your needs. For most applications you're deploying you'll need: Can you with yaml? Helm tries, but isn't that great, jinja ( depending on how its done ( jinja in yaml as in ansible or jinja to yaml as in SaltStack)) does an ok job, but fundamentally managing a language which uses spaces for logic with code is hard, and certainly YAML wasn't created to be managed that way. JSON i agree, you can easily convert your language of choice's data structures to it. But do you have to? It adds another layer of complexity, and you can have unexpected bugs ( since you have a logic layer that creates JSON, which could be wrong due to a logical error). HCL is a DSL that needs learning and few use outside of Hashicorp products, but IMHO it's not that hard ( docs are great, tutorials are plenty) and it's worthwhile to have data+logic in the same human and machine readable layer. In any case, Nomad accepts JSON for everything ( mostly for when non-humans interact with it).
- gravypod 6y agoHelm, in my personal opinion, is one of the biggest mistakes of the kubernetes ecosystem. Not embracing Jsonnet, Cue, Dhall, or even an in-language DSL like Pulumi has set the usability of kube back quite far.
- jpgvm 6y ago$DAY_JOB replaced helm with custom tool that does equivalent of kubectl apply by uses Jsonnet as templating engine. I intend to write an OSS version of it because Jsonnet really is probably the best fit here (though Cue/Dhall are interesting I think Jsonnet is easiest to get others to adopt).
- gravypod 6y agoI've done the same in the past. The important thing is making it so it's easy to build up abstractions and to work in a sensible deployment flow. kubectl is close but doesn't understand secrets/variables, the concept of clusters, or changing image tags programmatically. It's difficult to get right.
- nunez 6y agoUnless you start messing with Operators (which are hot right now) for desired state configuration within your cluster, which can, theoretically, be written in anything, but realistically tend to be written in Golang.
- gravypod 6y agoYep, exactly. KubeDB is a great example of this.
- tutfbhuf 6y agoI think operators are great and true to the k8s spirit. You describe your desired state (e.g. a postgres database) according to a CRD. And the operator will make your yaml definition become reality. I think the declarative approach is a big improvement over the imperative step-by-step setup. I rather like to describe how I would like things to be, instead of having to create the desired state myself.
- nunez 6y agoTo be clear: I like operators! I was just responding to the parent post; JSON/YAML aren’t the only ways to express things in the Kubeverse.
- AnxiousCoward 6y agoMay be. But operators are a relative pain to develop. I prefer the HCL approach. You can have modules instead of operators, and while HCL overall might be painful, it's less painful, IMO, than constantly changing operators. My team spends a significant amount of time chasing deprecations of operators and associated helm charts.
- darthrupert 6y agoHCL alone is a good reason to prefer Nomad over any YAML mess that exists.
- francis-io 6y agoAs a cloud engineer in the UK, it's almost required these days to know K8s in one form or another. I wish it wasn't the case, but I think the battle is over.
- sgt 6y agoLearning k8s is not a problem, but if you are faced with the decision to implement either k8s or Nomad at a company, it's worth evaluating both and what fits better for your team. Recruitment should not be an issue since anyone who understands k8s should be able to grasp Nomad, and vice versa (although in the latter case, with much more effort).
- AnxiousCoward 6y agoOnce you get to a place where you need to run really-really huge workloads, like thousands or tens of thousands of machines, kubernetes, hosted or not, will take the will to live away from you. It's not about the learning curve, or the convenience, it's that kubernetes gets slow to catch up on large workloads that need to change often. That kind of setup is where nomad + consul + vault + terraform shine.
- kashif 6y agoMuch better and easier to run/maintain than Kubernetes. I say this after using it for 5 years.
- leetrout 6y agoI love nomad so much. It never really gets a fair shake IMO. The world needs both k8s and nomad and I hope it continues to grow and see more adoption. And many years ago we used it as a feature in a product written in Go. We could just tap in to the APIs directly. It is amazing for batch jobs.
- jxf 6y agoThe main advantage of Kubernetes isn't really Kubernetes anymore — it's the ecosystem of stuff around it and the innumerable vendors you can pay to make your problems someone else's problems.
- ozim 6y agoUp vote because that really is the main thing to consider when evaluating actual needs of a project. My actual need is that I deploy my application and I don't care where or how it is running. Most of people should not even think about running k8s on their own. That should be job of a service provider and there should be service provider for running "Nomad".
- orthecreedence 6y agoThe main advantage of Nomad is Nomad. It strikes the perfect balance between simple and capable, enough so that I don't have to pay someone else. You can read the bulk of the docs in a day or two and deploy it that afternoon. Consul/Nomad deployment is easy and operation is a breeze. As the sole ops person at my company, Nomad has made it possible for me to scale our infrastructure without an entire team dedicated to it. I'm not saying that it's better than K8S, but for a small team who wants to focus on building things, it's an amazing time saver. Obviously, read the docs and make your own decisions, but there is a definitely an infrastructure void that Nomad has filled perfectly.
- peter_l_downs 6y agoAs a counterpoint, everything you just said about Consul/Nomad applies to using hosted K8s like GKE. I moved my company's infra to it last year and now we barely think about it. We're a small team, I'm the only person doing devops, and there are very, very few problems requiring my attention. I'm skeptical that anything allowing flexibility in this space can ever be that simple because the domain is actually pretty complicated. Networking, secrets, scaling, healthchecking, deployment strategies, containerization -- you need to know something about all of these before you can use Nomad or Kubernetes.
- 6y ago
- sdotsen 6y agoI looked into Nomad and my problem is having to run a minimum of six machines in a cluster for a few microservices. The argument could be made that we wouldn't need container orchestration for a few, but if we plan on scaling I don't want to have to deal with a fleet of servers. With ECS or EKS, I would only have to worry about the worker nodes so it's less to manage
- core-questions 6y agoYou can totally do it on three machines, each running triple duty as consul, nomad, and worker machines.
- sofixa 6y agoOr if reduced redundancy is accepted, on one/two ( either one with everything running as server and client, or two with one running Nomad, Vault Consul servers and the other clients)
- core-questions 6y agoRight, and it's not like there's actually a hard requirement for perfect uptime in a lot of settings.
- jiveturkey 6y agowhy do you have a microservices environment for “just a few”?
- sdotsen 6y agoWe'll have more but I can't justify running six from the get go.
- markbnj 6y agoWe're all-in on kubernetes and cri where I work, but Nomad is cool and if we were starting out now we'd certainly take a close look at it. We use other Hashicorp tools, terraform for example, so we're already familiar with HCL (which I don't necessarily prefer over yaml or json for declarative tasks but that's another debate). Most of the simplicity argument comes from the single server/worker binary and ease of deployment on the ops side, which if you use managed k8s like GKE might not mean all that much. If you're going to implement and run your own clusters that's a bigger deal. Looking at it from the consumer side and comparing like to like I don't feel its a _lot_ simpler. Task groups and pods, tasks and containers, config, networking, storage, it is all different on nomad but not, imo, tremendously simpler. That makes sense given the nature of the problem the systems are addressing. Over the last five years kubernetes has grown horizontally a great deal to encompass more and more enterprise features and use cases, and I feel in general this is a repeating pattern: a thing is created, it works well and becomes popular, more people use it and join the community bringing their own use cases, which drives the addition of features to the thing, which eventually gets complicated enough that people are attracted to a new thing that does largely the same stuff but hasn't had a lot of the new features added yet. Rinse and repeat. On a somewhat unrelated note: I'm not very attracted to server-side features for tracking and manipulating versions of deployments and their history. I'm a fan of git ops, and what I want out of the server-side runtime is reliable, fast reconciliation of the infrastructure state with the release branch in my repo. If I want to roll back I would much rather revert in git and redeploy. It seems troublesome and error prone to rely on a cli command to force the server side state to a previous iteration that is not in sync with what master in the repo says should be the canonical release version. Interested in others' thoughts, but maybe its a separate topic.
- AnxiousCoward 6y agoHaving a rollback feature at the orchestrator level is useful when you want the rollback to be fast. Sometimes that's critical - when you deploy something buggy in production. One reason I'd prefer to use Nomad over K8s is scalability. K8s becomes slow and occasionally acts weird with just a few thousand pods. (Not that it is particularly fast with small numbers of pods.) Nomad is known to run reliably in production with many thousands of mixed workloads, not just containers. Another reason is its specialization. I'd rather deal with a handful of independent, well documented components (consul, vault, nomad, basically) than with one that does all things, but in a way I have to occasionally fight, and which, by necessity, given its breadth, is awfully documented. We do use k8s but still run vault for secrets - k8s secrets are a joke. We don't use consul because our discovery needs are simple, but the service abstraction in k8s is weak at best. While I'm not a big fan of HCL, any version, either, I do think that being able to manage everything, from cluster configuration to application deployment, via the same tool (terraform), in git, is more convenient than still having to use terraform for infrastructure but being forced to use helm on top of it - you might be able to maintain applications as terraform HCL scripts, but it would be unreasonable, given that for many applications that you'll want to deploy there are readymade charts available. From a strictly theoretical point of view, designing any kind of software the way kubernetes is designed is bad practice. It consists of very few components - but which do many different things. Some people work hard to maintain the system usable, but IMO the bad structure shows in how users need to interact with the system. Which is why, if the choice was mine, and not hype-driven, I'd take managed nomad with consul, vault, terraform and some git repo (my choice: gitlab, because it comes with a nice integrated CI) already integrated any time over managed kubernetes. But that's just me.
- joana035 6y agoI don't get why people are so afraid of kubernetes, the thing is pretty much hands off.
- corobo 6y agoHonestly my turn off was seeing Google Cloud and later DigitalOcean providing managed versions of it. If they see value in doing that I assume it's gonna be a pain in the arse to solo sysadmin it. I later realised my blog and little side projects do not need a cluster but that's another story. I was mainly looking at it to learn than to properly utilize Completely off topic, sorry. Whatever happened to Flynn.io? That was my favorite of the clustery things but it just sort of fell off the internet e: oh. Answered by visiting the site just now. "Flynn is no longer being developed." that is a shame.
- hiimnate 6y agoI just started using CapRover (https://caprover.com/ https://caprover.com/) for hosting some side projects, it was extremely easy to set up and appears to work similar to how Flynn.io used to.
- rualca 6y ago> I don't get why people are so afraid of kubernetes, the thing is pretty much hands off. To be fair, this time around the ones afraid of Kubernetes are the companies trying to sell a Kubernetes competitor.
- sofixa 6y agoTo be clear, i ( the author of the article) am not affiliated with Hashicorp in any way besides using some of their ( free, open source versions) products. I just used Kubernetes at work, inherited a poorly maintained cluster, and when time came to change, due to very limited time available, we opted for Nomad. Few years in, i'm telling what i've learned ( while still running a k8s cluster for home use, so i'm not entirely disconnected from the k8s ecosystem)
- haolez 6y agoI'd consider Nomad if I'd need to manage my own orchestration infrastructure somehow. My first option would be to hire a managed Kubernetes service. If that proves to be unreliable or not cost-effective in the long run, I'd then consider making this in-house and use Nomad instead.
- justinsaccount 6y agoI think nomad really needs a k3s equivalent. The equivalent of k8s is not nomad, it's nomad+consul+vault If you read through something like kubernetes-the-hard-way most of the really tricky parts are setting up the PKI infrastructure and bootstrapping etcd. https://learn.hashicorp.com/tutorials/nomad/security-enable-tls?in=nomad/transport-security https://learn.hashicorp.com/tutorials/nomad/security-enable-... starts to look a lot like https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/04-certificate-authority.md https://github.com/kelseyhightower/kubernetes-the-hard-way/b... Vault is better than k8s secrets, but getting nomad working doesn't magically give you a working vault cluster.
- steeleduncan 6y ago> The equivalent of k8s is not nomad, it's nomad+consul+vault This was my stumbling block. Nomad alone is clearly easier than k8s, and if it alone will suit your needs then you are in luck, but I didn't find installing and configuring 3 separate services any easier than k8s alone.
- insanejudge 6y agoI've never seen a reliable, scaled and coherent k8s operation without at least vault, and often consul as well, so that argument never held any water for me
- AnxiousCoward 6y agoWe had to add vault to our setup even if we use kubernetes - kubernetes secrets are a joke. We don't use consul, but managing istio, or any other service mesh, on top of kubernetes, isn't any simpler than running your own consul. (In fact, I have found istio to be a lot more difficult to get to do what you need than consul's service mesh, and with less functionality.)
- nunez 6y agoI'm surprised that Nomad is gaining traction _now_, five years later. Nomad has always been "container orchestration, the easy HashiCorp way," but finding case studies from others using it has been challenging
- jokethrowaway 6y agoFrom my experience nomad and other hashicorp tools required more learning how to use all the pieces together / figuring out edge cases / debugging. K8s was definitely easier to setup. I feel like nomad is appealing to people who don't really need orchestration but feel like they do. It's perfectly fine to run apps with just docker. I have small projects on the side which are just a couple of servers with docker: one nginx container pointing at a few docker containers containing some services. Once my needs grow, I'll just roll them over to k8s.
- sofixa 6y ago> From my experience nomad and other hashicorp tools required more learning how to use all the pieces together / figuring out edge cases / debugging. > K8s was definitely easier to setup How so? I can't imagine any production setup of Kubernetes from scratch ( not managed by a cloud provider) being easier, initially or on day 2, than Nomad+Consul+Vault, but maybe i'm missing something.
- flamemyst 6y agoSimpler option for k8s are available: - Charmed Kubernetes (Juju) https://jaas.ai/canonical-kubernetes https://jaas.ai/canonical-kubernetes - Mikrok8s https://microk8s.io/ https://microk8s.io/
- AnxiousCoward 6y agoI wouldn't consider any of these for production.
- mrkurt 6y agoWe use (and abuse) Nomad for Fly.io workloads. My favorite thing about it is that it's simple to understand. You can read the whole project and pretty much figure out what's going on . We've written: * A custom task driver to launch and manage Firecracker VMs * A device plugin for managing persistent disks The second is interesting – we looked hard at CSI, but the complexity cost of CSI is very high. The benefit is pluggable storage providers. We don't need pluggable storage providers, though. Likewise, we don't need pluggable networking. Device plugins are a really nice way to extend Nomad.
- pchiusano 6y agoCool. Related question - how has been your experience working with firecracker VMs, and what led you to that decision? (I’ve been eyeing them for a while to get faster startup times for autoscaling).
- junon 6y agoNeither have ever really made me happy as a systems designer. They both have some massive drawbacks.
- jasfi 6y agoCould you list their main drawbacks as you see them?
- rileymichael 6y agoIf multi-region deployments weren’t an enterprise only feature, Nomad would be a lot more appealing for me since Kubefed is still nowhere near ready.
- ahachete 6y agoThe absence of CRDs is a significant lacking. They enable abstraction, composability and reusability. Even if Kubernetes may be more complex, deploying complex jobs on Nomad may become (much) more complex than CRD-powered complex equivalent on Kubernetes. I'm also not sure if Nomad has an equivalent of the listeners that you have on Kubernetes to implement reconciliation loops, i.e. the controller part of operators.
- 42wim 6y agoOne of the reasons we choose nomad was because it had decent ipv6 support way before kubernetes (and I'm not sure if it's already decent nowadays). See my blog for more info: https://blog.42.be/2020/11/using-nomad-to-deploymanage-containers_29.html https://blog.42.be/2020/11/using-nomad-to-deploymanage-conta...