28 ms·

Nomad vs. Kubernetes

547 points by capableweb 5y ago

vegai_ 5y ago
Nomad could use an official or semi-official distribution. Something that you could throw into any VM with minimal configuration and it would create a new or join an existing nomad cluster. I've been thinking about building such a thing on Arch (btw) but haven't acquired enough time-energy to do it.
- proxysna 5y ago
  Nomad + Consul does auto-clustering. All you have to do is specify in which mode your nomad binary is working and where your consul client is listening. For local development -dev mode exists for both Consul and Nomad.
  - vegai_ 5y ago
    Yeah, I figure this shouldn't be too difficult to implement, but will be at least laborious to consistently keep up-to-date and stable.
- ghuntley 5y ago
  What you are seeking is called NixOS w/consul configured.
streetcat1 5y ago
I assume that Nomad is simpler since the other features are provided by other hashi corp tools (e.g. secrets, auth, etc).
- capableweb 5y ago
  Indeed, the focus is different for the two tools, something that is outlined with more words in the opening paragraph of this very submission: > Kubernetes aims to provide all the features needed to run Linux container-based applications including cluster management, scheduling, service discovery, monitoring, secrets management and more. > Nomad only aims to focus on cluster management and scheduling and is designed with the Unix philosophy of having a small scope while composing with tools like Consul for service discovery/service mesh and Vault for secret management.
- sjtindell 5y ago
  In my experience you end up needing additional tools for everything in Kubernetes such as secrets, auth, etc. that’s the pluggable API selling point.
  - streetcat1 5y ago
    Might. But most of them are open source with healthy competition. With Nomad, the ecosystem is much smaller(basically one company), and can always be closed under a paywall (e.g. terraform enterprise)
    - nirn 5y ago
      Do you mean Nomad Enterprise? Which actually exist https://www.nomadproject.io/docs/enterprise https://www.nomadproject.io/docs/enterprise :)
Fiahil 5y ago
> Kubernetes is an orchestration system for containers Yes, but No. Kubernetes is a portability platform, that happen to -also- orchestrate containers. Using kubernetes means you have complete reproducibility of the network setup, the deployment and the operation of any workload -no matter how complex-, on any kubernenetes cluster and cloud provider. Nomad, is -well- just a glorified Airflow.
- capableweb 5y ago
  Nomad, just like Unix, prefers composition of simple tools in favor of one-big-tool-for-everything that Kubernetes is going for. So for achieving those things, you'd use Terraform or something similar, and then you have a reproducible environment for the hardware/software setup outside of Nomad. > Yes, but No. Kubernetes is a portability platform, that happen to -also- orchestrate containers. The homepage of Kubernetes seems to disagree with you. Their headline there is "Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications." and also "Production-Grade Container Orchestration" so it feels safe to assume that Kubernetes is for orchestrating containers. > Nomad, is -well- just a glorified Airflow. I never used Airflow, but looking at the website it seems to be geared towards automating workflows, something like Zapier but self-hosted and open source? That's very different from what Nomad is.
  - Fiahil 5y ago
    > The homepage of Kubernetes seems to disagree with you. People have trouble understanding what k8s is, and what to use it for. That's fine, it'll take a while, but they will eventually understand what "Production-Grade Container Orchestration" really means when they start working with it.
    - acdha 5y ago
      Can you explain this? Having read it I have no idea what you think the differences are — it just sounds smug.
tazjin 5y ago
As a Xoogler it's always seemed weird to me how Kubernetes was compared to Borg. Kubernetes covers a much larger set of things than Borg alone, and I don't necessarily think that's for the better. Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. Nowadays I don't have use-cases for either, but from playing around with Nomad it felt a lot more "Borg-y" than Kubernetes ever did to me. I rarely hear anything about the Hashicorp alternative for service discovery and such though, so it would be interesting how that compares.
- _hzrk 5y ago
  > Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. I know that Borg was written in Java and Kubernetes in Go. Though the latter had a reputation in the beginning as a systems programming language, its purpose was actually to build large-scale cloud infrastructure projects with it and it proved formidably well suited for the task. It compiles fast, anyone can read it, good tooling and it is efficient for the layer on which it is meant to be deployed. Go, as Java is one of the most productive languages in use today, judging by the ecosystems they have spawned.
  - tazjin 5y ago
    Borg is C++.
    - johnnycerberus 5y ago
      There you Go (pun intended). Go even replaced a low-level language like C++ and achieved the same result in the end. I don't know why I thought it's Java, probably the first Kube was initially in Java. It's even better that they managed to pull that off.
      - tazjin 5y ago
        But it did not replace it. Google runs on Borg. Go lets you build things fast, but the lack of strong typing and the vast amount of language pitfalls make maintenance hard in the long run. The community also has the attitude to pretend that these pitfalls don't actually exist, which is very different from C++ where most peculiar behaviours are well-understood and controlled.
DeathArrow 5y ago
Nomad seems much simpler to use and manage if you need to do simpler things, but Kubernetes allows you to do more. We use Kubernetes instead of Nomad at work but we are also using Consul in the Kubernetes cluster.
- capableweb 5y ago
  > Nomad seems much simpler to use and manage Agree, Nomad is so easy to get started and because of the simplicity of the architecture, very easy to maintain as well. > but Kubernetes allows you to do more ... We use Kubernetes instead of Nomad at work Same here, Kubernetes at work, Nomad for personal projects. But I have yet to find anything I cannot do in Nomad that you normally do in Kubernetes. Could you provide some specific things you've tried to get working in Nomad but couldn't (or simply doesn't exists in Nomad), while it worked in Kubernetes? Would be useful to know before I get too deep into Nomad.
  - proxysna 5y ago
    > But I have yet to find anything I cannot do in Nomad that you normally do in Kubernetes. VM live migration. I was surprised that people use kubevirt for that, but apparently this is a valid usecase. Otherwise nomad can do relatively complex vm configurations.
  - candiddevmike 5y ago
    The entire operator paradigm is kubernetes centric. You're missing out on all of the innovations around operations automation if you use nomad. Same with GitOps to an extent, HC pushes people to use terraform or waypoint for deployments while everyone else uses argo and tekton (or some other k8s runner).
- proxysna 5y ago
  Nomad will still get you where you need to be. Aside from vm live migration i am yet to find a good example of a workload where nomad is straight up is unable to do what you need.
  - 5y ago
sofixa 5y ago
If someone's interested, i wrote a deeper dive into the matter a few months back, with concrete examples: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...
- capableweb 5y ago
  Thank you for that post, I found it a couple of months ago and helped a lot in making my decision to go with Nomad over Kubernetes!
- egberts1 5y ago
  nicely covered.
proxysna 5y ago
Been running Nomad for a while now at work and home, and it is such a fun project to work and tinker with. Great community, lots of plugins and other great stuff. After running k8s in prod and home Nomad felt like a breath of fresh air in all aspects.
- jbotz 5y ago
  What do you do with Nomad at home? What comprises your hardware? Just curious...
  - proxysna 5y ago
    Hardware: A few droplets in DO, a bunch of Orangepi's/Raspberrypi's and 3 HP Elitedesk G1 with some i7 and 32 gigs of ram each. Software: bunch of solaris/omnios vm's, torrent client, owncloud, mtproto proxies for family and friends, my site, minecraft, quake, openttd, trss, gitea, drone.io, jenkins. There is other stuff, but it is usually shortlived, i check it out, test it and most often discard it.
  - omneity 5y ago
    I'm also using Nomad at home. It's a single node "cluster" in my desktop Ubuntu-based machine, really using it to deploy some self hosted services more easily (the usual suspects for home automation and the likes).
mrweasel 5y ago
There seems to be plenty of reasons to run Nomad, compared to Kubernetes, but in what scenarios do Nomad lose out to Kubernetes? Is it simply a matter of Kubernetes being an open source project and Nomad being owned by HashiCorp?
- stavros 5y ago
  Nomad is an open source project as well though.
- nuerow 5y ago
  > There seems to be plenty of reasons to run Nomad, compared to Kubernetes, but in what scenarios do Nomad lose out to Kubernetes? It would also be interesting if Docker's Swarm was.also featured in this comparison, as it just works right out of the box and doesn't come with any bells and whistles.
  - alvarlagerlof 5y ago
    Does it support multiple nodes?
    - nuerow 5y ago
      > Does it support multiple nodes? Yes it does. Docker Swarm essentially provides an easy way to get a cluster of Docker instances running in multiple nodes that works right out of the box. https://docs.docker.com/engine/swarm/ https://docs.docker.com/engine/swarm/
  - zelphirkalt 5y ago
    As far as I remember from trying last time a few years ago, Docker Swarm does not support rebalancing the load of containers and requires either manually issuing a command to do that or merely initially balances where containers are started. It this still true? And what about when you have state, which you would need to move between nodes, to make a container work elsewhere?
  - proxysna 5y ago
marvinblum 5y ago
I've been running a production-grade Nomad cluster on Hetzner for the past 1 1/2 years and it's fantastic. It was amazingly easy to set up compared to Kubernetes (which I also did), the UI is awesome, updates haven't broken anything yet (as long as you follow the changelog) and it's stable. I really like the separation of concerns the HashiStack offers. You can start out just using Consul for your service meshing, and then add Nomad + Vault to get a similar experience to Kubernetes. Yes, it doesn't cover as many features as Kubernetes, but it should be good enough for most software and you can still make the switch later. I would never go back. You can read more on our blog if you're interested: https://pirsch.io/blog/techstack/ https://pirsch.io/blog/techstack/
- kalev 5y ago
  Great read, thank you.
  - marvinblum 5y ago
    I'm glad you like it :)
- Uberphallus 5y ago
  Hi, I see you mention the tiniest nodes in Hetzner there, whereas the Nomad documentation [0] talks about 3-5 server nodes in the 2-digit GiB memory range, which is what has kept me from trying Nomad as I find it insane. How much truth is there in the docs? [0] https://www.nomadproject.io/docs/install/production/requirements https://www.nomadproject.io/docs/install/production/requirem...
  - tetha 5y ago
    This very much depends on your workload, number of jobs and complexity of scheduling. Our Nomad servers have 4GB of memory in the VM and are using about 0.5 - 1G at a low three-digit number of jobs. Hashicorp is doing a smart but normal thing for on-prem sizing there - they are recommending specs which ensure you have no problems for a lot of workload sizes. And "workload" can grow very large there, since a single DC can handle up to 5k clients and such.
jnsaff2 5y ago
I liked Mesos when I worked on it, and it's been replaced by more modern tools like Nomad, but every time I have to work on k8s it's .. well it's being promoted like a cult, has a cult following and I think the whole thing is set up to suck up complexity and layering. How do you deploy a thing to run on k8s? One would think you deploy a manifest to it and that's it. Like yaml or json or hcl or whatever. No. The built in thing is not good so someone wrote Helm. So you deploy helm manifests on there? No. Helm is not good either, you need helmsman or helmfile to further abstract things. How do you do networking? Layer a service mesh with custom overlay networking that abstract upon k8s clusterip. jeesh. why?
- jjeaff 5y ago
  You don't need any of that stuff. When I started with k8s, all I used were Yaml manifest files and kubectl apply. As I started using it more, i eventually moved up to using helm. I've been running production k8s for a few years now and haven't used helmsman or anything but helm yet.
  - musingsole 5y ago
    > When I started with k8s But that's the problem right there. Shop after shop has invested in kubernetes with a team that was just learning it. And then they layered tool after tool to help massage pain points -- of which kubernetes has plenty. That leaves us at today where every shop effectively has a custom stack of kubernetes infrastructure cobbled together of various tools and philosophies. It's the same problem as JavaScript. There's nothing terrible about the technology -- on the contrary it's amazing stuff! But the context of how it gets used leads to a type of gridlock of opinionated tooling published just last week.
  - smoyer 5y ago
    At its core, Kubernetes is a controller manager. It's true that most Kubernetes systems include a lot of the Hashicorp equivalents but you could in theory remove the controllers and associated CRDs. Kubernetes has gradually moved to what I think is a relatively sensible set of defaults which you can get in different flavors (k0s, k3s, microkube, kubespray, etc.) The comment about development and packaging tools can indeed present a problem. I tend to favor Helm for applications that need to be deployed more than once and Kustomize to differentiate between environments but I've definitely seen what I would consider horror stories. Even if you add a bit of GitOps, it's not too hard to understand. The horror stories seem to occur when teams try to map their previous workflows directly to k8s. Many times the worst k8s development workflows are created to enable teams that had broken workflows without k8s.
kakoni 5y ago
Also worth noting when running your own k8s(or k3s) cluster; > Running Kubernetes is not free. It is expected that it will take about 0.5 to 0.75 of a single core on an embedded system like a Raspberry Pi just as a baseline. [1] [1] https://github.com/k3s-io/k3s/issues/2278 https://github.com/k3s-io/k3s/issues/2278
- aequitas 5y ago
  Ran into this recently with a RPi 3b K3s cluster. Before I could easily allow running workloads on the master node, but since a recent update that node will just get overloaded until unresponsive. Yay for automated hands off updates I guess :).
francis-io 5y ago
As much as i think k8s is the most complex option that exists, which is bad, its being pushed by the big boys, which means tools exist for it. I can get so much value by plugging in other tools to this generic platform. Feature flags? just plug it in. On cluster builds? Just plug it in. Monitoring? Plug it in.
glynnforrest 5y ago
> Flexible Workload Support This is Nomad's most underrated feature, IMHO. You don't have to use containers for everything if you don't want to. For example, if you're a golang shop you can run everything as native binaries and cut out docker completely. Nomad has much simpler networking, i.e. no web of iptables rules to figure out. You can add Consul connect as a service mesh if you need it, but if you don't, you can keep things very simple. Simple = easy to understand, run, and debug. The main downside for me is a lack of plug and play pieces, e.g. a helm chart for Grafana or Prometheus. You'll have to write your own job for those, though it's very easy to learn. I'd love to see a series of Nomad recipes that people could use. I think it's the ideal choice for on-prem, bare-metal, or 'weird' deployments where you need a bit more control. You can build the exact stack you need with the different HashiCorp projects with minimal overhead. I can't recommend it enough! I help people move to Nomad, my email is in my profile if you want to chat :)
- Jedd 5y ago
  This has been my biggest pain point - world+dog shares their helm charts, but blog posts or git*.com repos with non-trivial cluster recipes are rare. f.e. We're exploring cortexmetrics currently, and spinning the full stack version up on k8s (openshift) was straightforward. Porting all that spaghetti onto nomad would be a huge job, though part of the frustration is knowing someone, tucked away on a private network, has already done this.
  - glynnforrest 5y ago
    > the frustration is knowing someone, tucked away on a private network, has already done this. Hard agree. I know this person and have been this person before. I've toyed with the idea of writing a book of Nomad recipes and tips, I wonder if anyone would read it? Also, watch this space, helm for Nomad may be coming soon: https://github.com/hashicorp/nomad-pack https://github.com/hashicorp/nomad-pack
    - Jedd 5y ago
      A book - probably not so much. The big value of sites (mostly github master repos) that offer recipes for saltstack/chef/ansible/puppet, or the helm collective, etc - are that they're continually being tweaked as new versions of (upstream) software are released. They usually all require a fair bit of localisation before they 'just work', at least in my experience, but the template taken from a proven functioning system, and then abstracted & shared, is worth its weight in gold. I've set up a few nomad jobs, but nothing anywhere as complex as, say, this cortexmetrics monstrosity. Even our k8s & nomad guru baulks at such an undertaking.
_wldu 5y ago
I have never deployed or built a K8s cluster, but recently I moved about a dozen AWS ECS Fargate workloads into a K8s cluster that a colleague of mine has setup. I was surprised. I really like it (from the perspective of a user/developer). I deploy my apps by updating a simple yaml file in a git repo. All my other practices (vim, go code with standard Makefiles, docker container image registry) are unchanged. I also think K8s is a reasonable cloud independent abstraction when you need to move your workloads to another provider. It prevents cloud lock-in. And I suppose Nomad would do that too. So far my K8s experience has been very good. However, if I ever have to do more with it (on the administrative side) I may have a less positive experience. Not sure.
- tekkk 5y ago
  What was wrong with ECS Fargate if you don't mind me asking? Too expensive? Too vendor locked-in?
  - _wldu 5y ago
    We want to apply consistent policies and technical controls across multiple dev groups that have multiple AWS accounts. K8s seems to be a good solution for that.
    - throwaway923840 5y ago
      Was it not possible in ECS? Or did your teams just want to use K8s, and decided they would use the opportunity of wanting to organize things better as a reason to switch? It seems to me that with SSO and IAM you could create virtually any kind of access controls around ECS. K8s doesn't solve the multi-account problem, and federation in K8s can be quite difficult.
    - tekkk 5y ago
      All right, thanks! Just wondered as I have a ECS setup and I really prefer not having to switch unless I have to.
    - still_grokking 5y ago
megamix 5y ago
Oh God another one
- jaaron 5y ago
  Another one? Nomad and kubernetes were released within months of each other in 2015. This isn't new.
deleted 5y ago
[deleted]
aequitas 5y ago
Looking forward to Nomad Packs, they where announced with their 1.2 beta[0]. I've really been missing something like k8s-at-home[1] for homelab Nomad setups. Don't know if they will become as versatile as Helm charts since Nomad is less of an abstraction than Kubernetes. [0] https://www.hashicorp.com/blog/announcing-hashicorp-nomad-1-2 https://www.hashicorp.com/blog/announcing-hashicorp-nomad-1-... [1] https://github.com/k8s-at-home/charts https://github.com/k8s-at-home/charts
sharmin123 5y ago
Facebook Safety Tips: Take Steps Now and Avoid Hacking: https://www.hackerslist.co/facebook-safety-tips-take-steps-now-and-avoid-hacking/ https://www.hackerslist.co/facebook-safety-tips-take-steps-n...
hacker_newz 5y ago
I've run into so many issues with Nomad that it really doesn't make sense to compare the two. Many are well documented in Github issues and occur when utilizing other Hashicorp products.
- numbsafari 5y ago
  I'm curious how responsive you have found them to issues you've identified. Any particular problems you've experienced that have been long standing an unresolved? I occasionally like to kick the tires on Nomad, but so far haven't found it compelling enough to switch away from my current solution, GKE (mostly because of how integrated it is with everything else I am doing and, at this point, familiarity). So, curious to hear from others where the rough edges are.
  - 3np 5y ago
    Not the person you’re replying to, but I’m a couple of years in on Nomad, coming from kubernetes. I’ve reported and participated in a few issues. Most got fixed within the next couple of minor versions, some went neglected. I think the somewhat recent 1.0-release was pretty well-timed. There are still a few rough edges (though it’s gotten a lot better continuously over than last year). Specifically, Consul Connect and federation (especially together with ACLs and mTLS) are just about coming together now. If you don’t rely on Connect and federation across regions, I’d say it’s reliable for production use. Other than that maybe wait 6-12 months and reassess. I’d also advise to put ACLs and TLS in place before you start putting production loads there; it’s not great to introduce in a running cluster.
omneity 5y ago
Nomad is amazing. We've been using it alongside Consul for close to 2 years at this point at Monitoro.co[0]. We started by using it as a "systemd with a REST API" on a single server, and gradually evolved into multiple clusters with dozens of nodes each. It has been mostly a smooth ride, even if bumpy at moments, but operationally speaking Nomad is closer to Docker Swarm in simplicity and Kubernetes in terms of the feature set. We didn't find ourselves needing K8s as we're also leveraging components from the cloud provider to complete our infrastructure. [0]: https://monitoro.co https://monitoro.co
- oneweekwonder 5y ago
  > Nomad is closer to Docker Swarm in simplicity and Kubernetes in terms of the feature set. This a question I still need to google but what features does Kubernetes have that Docker Swarm needs? Because the perceived complexity of Kubernetes just blows my mind, where Docker Swarm seems a lot more simpler for the same benefits, but my its just abstracted away? I will say upfront im naive when it comes to container tech.
  - proxysna 5y ago
    Swarm has a lot of issues. Some that are on the surface like bad networking and stemming from it scaling issues. Others are related directly to Mirantis. Company that owns docker.inc now. It neglects the swarm part of the docker, even was planning to straight up sunset swarm and move everyone to k8s. They do maintenance, and add feature or two a year which is not enough. Swarm is great for small deployments, as it only requires a docker daemon present. Otherwise you should look towards nomad/k8s/whatewer the cloud solution is.
robertlagrant 5y ago
> In contrast to Kubernetes' fragmented distributions This article just seems like "All of Nomad's best features phrased kindly vs all of K8s' worst features phrased critically".
jeswin 5y ago
Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. The fundamental abstractions are as simple as they can be, representing concepts that you'd already be familiar with in a datacenter environment. A cluster has nodes (machines), and you can run multiple pods (which is the smallest deployable unit on the cluster) on each node. A pod runs various types of workloads such as web services, daemons, jobs and recurring jobs which are made available (to the cluster) as docker/container images. You can attach various types of storage to pods, front your services with load-balancers etc. All of the nouns in the previous paragraph are available as building blocks in Kubernetes. You build your complex system declaratively from these simpler parts. When I look at Nomad's documentation, I see complexity. It mixes these basic blocks with those from HashiCorp's product suite.
- andrewstuart2 5y ago
  From a system architecture perspective, kubernetes is very complex since it handles a multitude of complexities, but that's why it _can_ be very simple from a user's perspective. Most of the complexity can be ignored (until you want it). It's the same reason I still like to use postgres when I can versus NoSQL until I know I need the one feature I may not be able to achieve with postgres: automatic sharding for massive global scale. The rest of the features postgres (and friends) give easily (ACID etc) are very tricky to get right in a distributed system. It's also the same reason bash is great for tasks and small automated programs, but kind of terrible for more complex needs. The primitives are super simple, easy to grok, and crazy productive at that scale, but other languages give tools to more easily handle the complexities that start to emerge when things get more complicated.
  - xtracto 5y ago
    > The rest of the features postgres (and friends) give easily (ACID etc) are very tricky to get right in a distributed system. But that's just basically a calculated tradeoff of Postgres (and several CP databases) trading Availability for Consistency.
    - 5y ago
mkopinsky 5y ago
So far I've kept things simple, avoided k8s/HashiStack/etc by using docker compose with a simple docker-compose.yml for each server. This has been working well, but I'm starting to feel the pain points - HA requires copy-pasting yaml, I need to specify which services are on each server, and so on. What's the simplest next step I can take? I'd like something with (close to) the simplicity of docker compose, but ideally being able to specify something like `instances: 2` and being able to route/load-balance to those instances from another service.
- ledneb 5y ago
  Assuming serverless is out if the question for your use case, have you tried spending a couple of days investigating a managed Kubernetes cluster with node autoscaling enabled? EKS, AKS, GKE... Honestly it sounds like you could be at the point where K8s is worthwhile.
  - mkopinsky 5y ago
    I'm considering k8s, but that also means moving services from on-prem to AKS, getting INF to open up the necessary firewall rules to make the services reachable from on-prem, and so on. And as you said, it's definitely days of investigation. I'm not closed to the option.
    - p_l 5y ago
      You might also want to consider one of the simpler pre-packaged solutions for small k8s on premise clusters (like k3s)
- proxysna 5y ago
  You should be ok with consul, ansible and templating your docker-compose files. Might take some time to set it all up, but should be ok.
  - mkopinsky 5y ago
    Thanks for the suggestion. Any suggested starting points for my research? DDG search for "ansible docker-compose" brings up some suggestions like [1] and [2] but I'm curious if you have other suggestions. And just so I understand how these work together - I'd use Jenkins+ansible to push containers to my servers, I'd run consul within docker as well, and ... would Ansible register services with consul as it pushes them? Do the services need to be modified to speak with consul directly? [1] https://docs.ansible.com/ansible/latest/collections/community/docker/docker_compose_module.html https://docs.ansible.com/ansible/latest/collections/communit... [2] https://www.ansible.com/blog/six-ways-ansible-makes-docker-compose-better https://www.ansible.com/blog/six-ways-ansible-makes-docker-c...
deleted 5y ago
[deleted]
ascendantlogic 5y ago
Nomad gets the job done. What you end up missing is the huge ecosystem of tooling and knowledge that comes "for free" with Kubernetes.
- tetha 5y ago
  There has been a great announcement today with the beta of nomad 1.2 - nomad-pack: https://github.com/hashicorp/nomad-pack https://github.com/hashicorp/nomad-pack . Which, in my opinion, is aiming to be helm for nomad. Which is really great, because we've basically built nomad-pack in terraform and we would really like nomad-pack to replace that, because .. it works, but it could be better than terraform wrangling for templating :)
desktopninja 5y ago
My personal win for Nomad is its ability to run non containerized apps.
InternetPerson 5y ago
Wow, there's so much technical detail here! Like the part about how it's "simpler" and "flexible" and "consistent" I'm totally convinced by this product's marketing page!!
- bonetruck 5y ago
  I know you're being sarcastic. But the marketing page is not wrong. Try nomad yourself. Install: $ brew tap hashicorp/tap $ brew install hashicorp/tap/nomad Run: $ nomad agent -dev Start a job: $ nomad job run example.nomad
daenney 5y ago
These kinds of posts, by the vendor comparing to a competitor, always leave such a bad taste in my mouth. They decrease my confidence in both the product and the vendor. Stand on your own merits. There's a saying in Dutch: "Wij van Wc-eend adviseren Wc-eend"[1]. It basically boils down to pretending to give advice or information but you're just promoting your own interests. [1]: https://untranslatable.co/p/amarens/wij-van-wc-eend https://untranslatable.co/p/amarens/wij-van-wc-eend
- xfz 5y ago
  The Hashicorp ones tend to be well written and informative. I have found them useful in the past, even though I usually loathe sales BS. In this case the bias is a bit more obvious but we know it's written by Hashicorp and can make allowances.
- rkangel 5y ago
  I know what mean, but it's not like they're pretending that much if it's an article hosted under nomadproject.io. The first question everyone is going to ask "why should I use this instead of K8s" so you might as well have a good answer. On the other hand, the Rust project has purposefully avoided "Rust vs X" comparisons on its website. I can't find the HN comments to back this up, but people like steveklabnik have indicated that they don't find that sort of adversarial comparison to be useful. Rust has done an excellent job at community building, so I give a lot of credence to their approach.
  - devit 5y ago
    Well, Rust is currently the best programming language so it's the other languages that need to come up with a "X vs Rust".
  - steveklabnik 5y ago
    Yes. This topic is long and complicated; maybe I'll write it up or give a talk or something someday. There is a lot of nuance.
    - 5y ago
ketanmaheshwari 5y ago
Title should be "Nomad vs Kubernetes from Nomad's point of view".
- capableweb 5y ago
  The domain you see here on HN is nomadproject.io and the blog doesn't hide who the blogpost is from, it's very apparent already it's from the perspective of the people that maintain Nomad.
exdsq 5y ago
Does anyone know if Nomad can run on ChromeOS? I’ve had issues open with the Chrome and K8 teams to fix an error preventing it from working for months but I don’t think a fix will ever see the light of day, as it just gets passed around various repos.
hikerclimber1 5y ago
Everything is subjective. Especially laws.
zeroz 5y ago
For my solo developer self-hosted PaaS I really like CapRover [1]. Nice GUI, scalable Docker Swarm , with integrated route management, load balancing and one-click SSL. With available CLI you can use Gitlab CI to directly build and docker images and leverage CcapRover API to directly deploy on a dev stage. Interessting discussion about CapRover vs. Flynn, Dokku, Nomad on HN one year ago. [2] [1] https://caprover.com/ https://caprover.com/ [2] https://news.ycombinator.com/item?id=23465087 https://news.ycombinator.com/item?id=23465087
foobarbecue 5y ago
Do any of these systems come with a container image registry setup?
jaaron 5y ago
We're using Nomad for our backend at Singularity 6 (developing the mmo Palia). In my experience (having worked with Nomad, k8s, mesos, and homegrown schedulers), k8s is fantastic from an early developer experience, but rough from an operations perspective. It's definitely gotten better, but there's a reason lots of folks want to have someone else host and operate their cluster. Nomad is the opposite. Less tooling and bit more complexity for the app developer, significantly easier to operate at the cluster level. Nomad is also less "all-in" in it's architecture, which gives operators more flexibility in how they design and support the cluster. The fact that we can run on anything and don't _have_ to use containers is a serious advantage, particularly for game development.
awinter-py 5y ago
wonder what the 'orchestration feature set' of 2 years in the future will be speaking just for myself, feels like I want a system that runs a mix of serverless (like knative), normal containers, raw linux binaries (easy w/ fat binary systems like golang), language-specific packages like heroku does, maybe firecracker VMs. hard to get knative right if you don't have a reverse proxy tightly integrated with your tool
- omneity 5y ago
  Wild guess, probably more than 2 years ahead (also kinda what I'm hoping for) A natively distributed runtime providing similar capabilities to Docker + Nomad but running on "bare" WASM, providing a compilation target, distributed and clustering primitives right in your application code. Imagine the nodejs cluster module, targeting nomad directly, or talking to an instance of a service running on another node, the communication and addressing would be handled by the runtime. Similarly, cluster or instance level ports could be opened directly from your code and the distributed runtime will do the external/internal load balancing. Maybe taking it one step further and advertising services directly abstracting ports away. Service mesh + Scheduling + Capability-based Runtime in one binary you could run on Linux. One can dream.
shrumm 5y ago
To most people, it's not Nomad vs Kubernetes - it's a choice between Nomad vs Managed Kubernetes. All major cloud providers offer a managed kubernetes service at minimal added cost to running the worker VMs yourself. With managed Kubernetes, the simplicity question is no longer obviously in Nomad's favour. As other comments allude to, Kubernetes as a user is pretty easy to master once you get used to its mental model.
- pid-1 5y ago
  Agreed. Managing a Nomad cluster alongside a Consul cluster does not require a PHD, but it's also not a walk in the park. Hopefully Hashicorp will have managed Nomad soon.
- mac-chaffee 5y ago
  I'm currently trying to convince people that a managed k8s service is not that "simple", and that we can't "just spin up another cluster" without a great deal of operational overhead. Some of the things that might still be needed in managed k8s instances: better ingress with ingress-nginx, cert-manager, monitoring/logging/alerting, tuning the alerts, integration with company SSO, security hardening. If it's a multi-tenant cluster: LimitRanges/ResourceQuotas, NetworkPolicies, scripts to manage namespaces and roles, PodSecurityPolicies (or equivalent), onboarding offboarding procedures. I'm sure you'd need similar things to have a proper production Nomad cluster too, so your point still stands. But at least for EKS/GKE clusters, they're pretty bare-bones.
  - darkwater 5y ago
    Thanks for this, I was really starting to think I was the madman.
  - busterarm 5y ago
    As someone with 7-digit spend in GKE/EKS, I will agree with you that it is _anything but simple_. Your developers aren't going to say that it's simple when Google force upgrades their cluster to a version that deprecates APIs in their yamls for a job they worked on 2 years ago and swiftly forgot about. Then when you explain to them that Google insists on putting everyone on a force-upgrade treadmill, you can literally watch as the panic sets in on the faces of your engineering team managers/leads. Nomad is a breeze in comparison to managed K8s. Everyone that I've talked to that thinks Kubernetes is simple is barely using the thing and could likely save a lot of money and development effort using something like Nomad instead.
siliconc0w 5y ago
What is nice about using the same tech as the 'big players' is you get the benefits of massive armies of engineers building integrated products and features to sell to other big users of the product. This means there are options to add on distributed databases, storage, monitoring, tracing, CD/CI, etc. So it can be worth swallowing 'big player' complexity if it means you can play in the same ecosystem. If you are already on a cloud then your infrastructure provider will handle a lot of that complexity for you anyway and it's often better integrated into their other products (i.e networking, access control, etc).
chucky_z 5y ago
If you have a mixed Windows/Linux environment I cannot suggest Nomad strongly enough. It truly shines in this environment. Especially if you're dealing with legacy stuff. Even better if you're running Mac + BSD as well, it just runs everywhere.
rubenhak 5y ago
I don't agree with "Kubernetes is an orchestration system for containers originally designed by Google" statement. White it is not false, it creates a wrong impression of what K8s is. Kubernetes is a cloud operating system which lets you run modern apps in any environment. One important component of course container orchestration, but it went far beyond just orchestrator. Kubernetes has a very powerful ecosystem, and it managed to unite almost all infrastructure vendors and projects. Its amazing to see how so many competing companies could agree on something. And that become K8s. Nomad is great when you're working with VMs, but I don't see it is much relevant in the modern era of K8s and cloud-native.
deleted 5y ago
[deleted]
toddnni 5y ago
Is it so that the fragmentation (the long list of distributions), or variety of configuration options, is a bad quality? This seems to be used as a counter argument to Kubernetes. Probably, it could be end result of unconsistent design or bad technical choises. However, most likely it just means that there are multiple organizations and interest groups pushing changes and ideas to the project. This should be seen as a good thing. The downside is that there is no single source of best practices and this is confusing to newcomers. You just need to pick one distribution and trust the choises, or develop the competence and understanding. And we could imagine that the userbase or the number of developers in single distribution (take OpenShift or Rancher) could be bigger that in Nomad itself. Having said that, I still would like to see more stable Kubernetes landscape, and that has to happen eventually. The light distributions k3s and k0s are pushing things to nice direction. OpenStack had similar, or maybe even worse, fragmentation and complexity issue when the hype was high. There were probably technically better alternatives (Eucalyptus?), but people (and companies) gathered around OpenStack and it won the round. However, comparing OpenStack to Kubernetes feels bad, as Kubernetes is technically far superior.
ookblah 5y ago
i would say the "simplest" setup for us has been ECS fargate (even thought there's some major lock in). Very easy to spin up a cluster and you don't have to manage the underlying structure. If you use docker-compose or equivalent for local dev you just define your tasks/services and launch. Even pulling in ENVs is easy with SSM.
- politelemon 5y ago
  What kind of lock-in? One of the reasons I started using it was due to the minimal lock-in, so we've arrived at different conclusions. A task definition is required, which is sort of a docker-compose.yml analogue, but it's also possible to use docker compose directly with ECS Fargate. So my 'get out of AWS' is pretty simple, I already have the docker-compose.yml ready.
  - ookblah 5y ago
    i suppose what i mean is you'd never be able to take a ECS task definition/service and move it over to some other provider. whereas if u did some kubernetes setup u could conceivably do that, moving form AWS to GCP.
AlexCoventry 5y ago
Why has this post dropped 20- odd ranks in roughly an hour?
kosolam 5y ago
Any nice step by step tutorial how to get nomad vault and consul architecture running quickly? I’m especially interested in non container job.
xtat 5y ago
I was into it until I saw there is an enterprise version link. If it's not FLOSS I'm out.
- dstrickland 5y ago
  It's mostly OSS. There are pay to play enterprise features, but you can most definitely run high end production clusters without needing enterprise.

547 points by capableweb 5y ago

Nomad could use an official or semi-official distribution. Something that you could throw into any VM with minimal configuration and it would create a new or join an existing nomad cluster. I've been thinking about building such a thing on Arch (btw) but haven't acquired enough time-energy to do it.

proxysna 5y ago
Nomad + Consul does auto-clustering. All you have to do is specify in which mode your nomad binary is working and where your consul client is listening. For local development -dev mode exists for both Consul and Nomad.
- vegai_ 5y ago
  Yeah, I figure this shouldn't be too difficult to implement, but will be at least laborious to consistently keep up-to-date and stable.
ghuntley 5y ago
What you are seeking is called NixOS w/consul configured.

streetcat1 5y ago

I assume that Nomad is simpler since the other features are provided by other hashi corp tools (e.g. secrets, auth, etc).

capableweb 5y ago
Indeed, the focus is different for the two tools, something that is outlined with more words in the opening paragraph of this very submission: > Kubernetes aims to provide all the features needed to run Linux container-based applications including cluster management, scheduling, service discovery, monitoring, secrets management and more. > Nomad only aims to focus on cluster management and scheduling and is designed with the Unix philosophy of having a small scope while composing with tools like Consul for service discovery/service mesh and Vault for secret management.
sjtindell 5y ago
In my experience you end up needing additional tools for everything in Kubernetes such as secrets, auth, etc. that’s the pluggable API selling point.
- streetcat1 5y ago
  Might. But most of them are open source with healthy competition. With Nomad, the ecosystem is much smaller(basically one company), and can always be closed under a paywall (e.g. terraform enterprise)
  - nirn 5y ago
    Do you mean Nomad Enterprise? Which actually exist https://www.nomadproject.io/docs/enterprise https://www.nomadproject.io/docs/enterprise :)

Fiahil 5y ago

> Kubernetes is an orchestration system for containers Yes, but No. Kubernetes is a portability platform, that happen to -also- orchestrate containers. Using kubernetes means you have complete reproducibility of the network setup, the deployment and the operation of any workload -no matter how complex-, on any kubernenetes cluster and cloud provider. Nomad, is -well- just a glorified Airflow.

capableweb 5y ago
Nomad, just like Unix, prefers composition of simple tools in favor of one-big-tool-for-everything that Kubernetes is going for. So for achieving those things, you'd use Terraform or something similar, and then you have a reproducible environment for the hardware/software setup outside of Nomad. > Yes, but No. Kubernetes is a portability platform, that happen to -also- orchestrate containers. The homepage of Kubernetes seems to disagree with you. Their headline there is "Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications." and also "Production-Grade Container Orchestration" so it feels safe to assume that Kubernetes is for orchestrating containers. > Nomad, is -well- just a glorified Airflow. I never used Airflow, but looking at the website it seems to be geared towards automating workflows, something like Zapier but self-hosted and open source? That's very different from what Nomad is.
- Fiahil 5y ago
  > The homepage of Kubernetes seems to disagree with you. People have trouble understanding what k8s is, and what to use it for. That's fine, it'll take a while, but they will eventually understand what "Production-Grade Container Orchestration" really means when they start working with it.
  - acdha 5y ago
    Can you explain this? Having read it I have no idea what you think the differences are — it just sounds smug.

tazjin 5y ago

As a Xoogler it's always seemed weird to me how Kubernetes was compared to Borg. Kubernetes covers a much larger set of things than Borg alone, and I don't necessarily think that's for the better. Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. Nowadays I don't have use-cases for either, but from playing around with Nomad it felt a lot more "Borg-y" than Kubernetes ever did to me. I rarely hear anything about the Hashicorp alternative for service discovery and such though, so it would be interesting how that compares.

_hzrk 5y ago
> Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either. I know that Borg was written in Java and Kubernetes in Go. Though the latter had a reputation in the beginning as a systems programming language, its purpose was actually to build large-scale cloud infrastructure projects with it and it proved formidably well suited for the task. It compiles fast, anyone can read it, good tooling and it is efficient for the layer on which it is meant to be deployed. Go, as Java is one of the most productive languages in use today, judging by the ecosystems they have spawned.
- tazjin 5y ago
  Borg is C++.
  - johnnycerberus 5y ago
    There you Go (pun intended). Go even replaced a low-level language like C++ and achieved the same result in the end. I don't know why I thought it's Java, probably the first Kube was initially in Java. It's even better that they managed to pull that off.
    - tazjin 5y ago
      But it did not replace it. Google runs on Borg. Go lets you build things fast, but the lack of strong typing and the vast amount of language pitfalls make maintenance hard in the long run. The community also has the attitude to pretend that these pitfalls don't actually exist, which is very different from C++ where most peculiar behaviours are well-understood and controlled.

DeathArrow 5y ago

Nomad seems much simpler to use and manage if you need to do simpler things, but Kubernetes allows you to do more. We use Kubernetes instead of Nomad at work but we are also using Consul in the Kubernetes cluster.

capableweb 5y ago
> Nomad seems much simpler to use and manage Agree, Nomad is so easy to get started and because of the simplicity of the architecture, very easy to maintain as well. > but Kubernetes allows you to do more ... We use Kubernetes instead of Nomad at work Same here, Kubernetes at work, Nomad for personal projects. But I have yet to find anything I cannot do in Nomad that you normally do in Kubernetes. Could you provide some specific things you've tried to get working in Nomad but couldn't (or simply doesn't exists in Nomad), while it worked in Kubernetes? Would be useful to know before I get too deep into Nomad.
- proxysna 5y ago
  > But I have yet to find anything I cannot do in Nomad that you normally do in Kubernetes. VM live migration. I was surprised that people use kubevirt for that, but apparently this is a valid usecase. Otherwise nomad can do relatively complex vm configurations.
- candiddevmike 5y ago
  The entire operator paradigm is kubernetes centric. You're missing out on all of the innovations around operations automation if you use nomad. Same with GitOps to an extent, HC pushes people to use terraform or waypoint for deployments while everyone else uses argo and tekton (or some other k8s runner).
proxysna 5y ago
Nomad will still get you where you need to be. Aside from vm live migration i am yet to find a good example of a workload where nomad is straight up is unable to do what you need.
- 5y ago

sofixa 5y ago

If someone's interested, i wrote a deeper dive into the matter a few months back, with concrete examples: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...

capableweb 5y ago
Thank you for that post, I found it a couple of months ago and helped a lot in making my decision to go with Nomad over Kubernetes!
egberts1 5y ago
nicely covered.

proxysna 5y ago

Been running Nomad for a while now at work and home, and it is such a fun project to work and tinker with. Great community, lots of plugins and other great stuff. After running k8s in prod and home Nomad felt like a breath of fresh air in all aspects.

jbotz 5y ago
What do you do with Nomad at home? What comprises your hardware? Just curious...
- proxysna 5y ago
  Hardware: A few droplets in DO, a bunch of Orangepi's/Raspberrypi's and 3 HP Elitedesk G1 with some i7 and 32 gigs of ram each. Software: bunch of solaris/omnios vm's, torrent client, owncloud, mtproto proxies for family and friends, my site, minecraft, quake, openttd, trss, gitea, drone.io, jenkins. There is other stuff, but it is usually shortlived, i check it out, test it and most often discard it.
- omneity 5y ago
  I'm also using Nomad at home. It's a single node "cluster" in my desktop Ubuntu-based machine, really using it to deploy some self hosted services more easily (the usual suspects for home automation and the likes).

mrweasel 5y ago

There seems to be plenty of reasons to run Nomad, compared to Kubernetes, but in what scenarios do Nomad lose out to Kubernetes? Is it simply a matter of Kubernetes being an open source project and Nomad being owned by HashiCorp?

stavros 5y ago
Nomad is an open source project as well though.
nuerow 5y ago
> There seems to be plenty of reasons to run Nomad, compared to Kubernetes, but in what scenarios do Nomad lose out to Kubernetes? It would also be interesting if Docker's Swarm was.also featured in this comparison, as it just works right out of the box and doesn't come with any bells and whistles.
- alvarlagerlof 5y ago
  Does it support multiple nodes?
  - nuerow 5y ago
    > Does it support multiple nodes? Yes it does. Docker Swarm essentially provides an easy way to get a cluster of Docker instances running in multiple nodes that works right out of the box. https://docs.docker.com/engine/swarm/ https://docs.docker.com/engine/swarm/
- zelphirkalt 5y ago
  As far as I remember from trying last time a few years ago, Docker Swarm does not support rebalancing the load of containers and requires either manually issuing a command to do that or merely initially balances where containers are started. It this still true? And what about when you have state, which you would need to move between nodes, to make a container work elsewhere?
- proxysna 5y ago

marvinblum 5y ago

I've been running a production-grade Nomad cluster on Hetzner for the past 1 1/2 years and it's fantastic. It was amazingly easy to set up compared to Kubernetes (which I also did), the UI is awesome, updates haven't broken anything yet (as long as you follow the changelog) and it's stable. I really like the separation of concerns the HashiStack offers. You can start out just using Consul for your service meshing, and then add Nomad + Vault to get a similar experience to Kubernetes. Yes, it doesn't cover as many features as Kubernetes, but it should be good enough for most software and you can still make the switch later. I would never go back. You can read more on our blog if you're interested: https://pirsch.io/blog/techstack/ https://pirsch.io/blog/techstack/

kalev 5y ago
Great read, thank you.
- marvinblum 5y ago
  I'm glad you like it :)
Uberphallus 5y ago
Hi, I see you mention the tiniest nodes in Hetzner there, whereas the Nomad documentation [0] talks about 3-5 server nodes in the 2-digit GiB memory range, which is what has kept me from trying Nomad as I find it insane. How much truth is there in the docs? [0] https://www.nomadproject.io/docs/install/production/requirements https://www.nomadproject.io/docs/install/production/requirem...
- tetha 5y ago
  This very much depends on your workload, number of jobs and complexity of scheduling. Our Nomad servers have 4GB of memory in the VM and are using about 0.5 - 1G at a low three-digit number of jobs. Hashicorp is doing a smart but normal thing for on-prem sizing there - they are recommending specs which ensure you have no problems for a lot of workload sizes. And "workload" can grow very large there, since a single DC can handle up to 5k clients and such.

jnsaff2 5y ago

I liked Mesos when I worked on it, and it's been replaced by more modern tools like Nomad, but every time I have to work on k8s it's .. well it's being promoted like a cult, has a cult following and I think the whole thing is set up to suck up complexity and layering. How do you deploy a thing to run on k8s? One would think you deploy a manifest to it and that's it. Like yaml or json or hcl or whatever. No. The built in thing is not good so someone wrote Helm. So you deploy helm manifests on there? No. Helm is not good either, you need helmsman or helmfile to further abstract things. How do you do networking? Layer a service mesh with custom overlay networking that abstract upon k8s clusterip. jeesh. why?

jjeaff 5y ago
You don't need any of that stuff. When I started with k8s, all I used were Yaml manifest files and kubectl apply. As I started using it more, i eventually moved up to using helm. I've been running production k8s for a few years now and haven't used helmsman or anything but helm yet.
- musingsole 5y ago
  > When I started with k8s But that's the problem right there. Shop after shop has invested in kubernetes with a team that was just learning it. And then they layered tool after tool to help massage pain points -- of which kubernetes has plenty. That leaves us at today where every shop effectively has a custom stack of kubernetes infrastructure cobbled together of various tools and philosophies. It's the same problem as JavaScript. There's nothing terrible about the technology -- on the contrary it's amazing stuff! But the context of how it gets used leads to a type of gridlock of opinionated tooling published just last week.
- smoyer 5y ago
  At its core, Kubernetes is a controller manager. It's true that most Kubernetes systems include a lot of the Hashicorp equivalents but you could in theory remove the controllers and associated CRDs. Kubernetes has gradually moved to what I think is a relatively sensible set of defaults which you can get in different flavors (k0s, k3s, microkube, kubespray, etc.) The comment about development and packaging tools can indeed present a problem. I tend to favor Helm for applications that need to be deployed more than once and Kustomize to differentiate between environments but I've definitely seen what I would consider horror stories. Even if you add a bit of GitOps, it's not too hard to understand. The horror stories seem to occur when teams try to map their previous workflows directly to k8s. Many times the worst k8s development workflows are created to enable teams that had broken workflows without k8s.

kakoni 5y ago

Also worth noting when running your own k8s(or k3s) cluster; > Running Kubernetes is not free. It is expected that it will take about 0.5 to 0.75 of a single core on an embedded system like a Raspberry Pi just as a baseline. [1] [1] https://github.com/k3s-io/k3s/issues/2278 https://github.com/k3s-io/k3s/issues/2278

aequitas 5y ago
Ran into this recently with a RPi 3b K3s cluster. Before I could easily allow running workloads on the master node, but since a recent update that node will just get overloaded until unresponsive. Yay for automated hands off updates I guess :).

francis-io 5y ago

As much as i think k8s is the most complex option that exists, which is bad, its being pushed by the big boys, which means tools exist for it. I can get so much value by plugging in other tools to this generic platform. Feature flags? just plug it in. On cluster builds? Just plug it in. Monitoring? Plug it in.

glynnforrest 5y ago

> Flexible Workload Support This is Nomad's most underrated feature, IMHO. You don't have to use containers for everything if you don't want to. For example, if you're a golang shop you can run everything as native binaries and cut out docker completely. Nomad has much simpler networking, i.e. no web of iptables rules to figure out. You can add Consul connect as a service mesh if you need it, but if you don't, you can keep things very simple. Simple = easy to understand, run, and debug. The main downside for me is a lack of plug and play pieces, e.g. a helm chart for Grafana or Prometheus. You'll have to write your own job for those, though it's very easy to learn. I'd love to see a series of Nomad recipes that people could use. I think it's the ideal choice for on-prem, bare-metal, or 'weird' deployments where you need a bit more control. You can build the exact stack you need with the different HashiCorp projects with minimal overhead. I can't recommend it enough! I help people move to Nomad, my email is in my profile if you want to chat :)

Jedd 5y ago
This has been my biggest pain point - world+dog shares their helm charts, but blog posts or git*.com repos with non-trivial cluster recipes are rare. f.e. We're exploring cortexmetrics currently, and spinning the full stack version up on k8s (openshift) was straightforward. Porting all that spaghetti onto nomad would be a huge job, though part of the frustration is knowing someone, tucked away on a private network, has already done this.
- glynnforrest 5y ago
  > the frustration is knowing someone, tucked away on a private network, has already done this. Hard agree. I know this person and have been this person before. I've toyed with the idea of writing a book of Nomad recipes and tips, I wonder if anyone would read it? Also, watch this space, helm for Nomad may be coming soon: https://github.com/hashicorp/nomad-pack https://github.com/hashicorp/nomad-pack
  - Jedd 5y ago
    A book - probably not so much. The big value of sites (mostly github master repos) that offer recipes for saltstack/chef/ansible/puppet, or the helm collective, etc - are that they're continually being tweaked as new versions of (upstream) software are released. They usually all require a fair bit of localisation before they 'just work', at least in my experience, but the template taken from a proven functioning system, and then abstracted & shared, is worth its weight in gold. I've set up a few nomad jobs, but nothing anywhere as complex as, say, this cortexmetrics monstrosity. Even our k8s & nomad guru baulks at such an undertaking.

_wldu 5y ago

I have never deployed or built a K8s cluster, but recently I moved about a dozen AWS ECS Fargate workloads into a K8s cluster that a colleague of mine has setup. I was surprised. I really like it (from the perspective of a user/developer). I deploy my apps by updating a simple yaml file in a git repo. All my other practices (vim, go code with standard Makefiles, docker container image registry) are unchanged. I also think K8s is a reasonable cloud independent abstraction when you need to move your workloads to another provider. It prevents cloud lock-in. And I suppose Nomad would do that too. So far my K8s experience has been very good. However, if I ever have to do more with it (on the administrative side) I may have a less positive experience. Not sure.

tekkk 5y ago
What was wrong with ECS Fargate if you don't mind me asking? Too expensive? Too vendor locked-in?
- _wldu 5y ago
  We want to apply consistent policies and technical controls across multiple dev groups that have multiple AWS accounts. K8s seems to be a good solution for that.
  - throwaway923840 5y ago
    Was it not possible in ECS? Or did your teams just want to use K8s, and decided they would use the opportunity of wanting to organize things better as a reason to switch? It seems to me that with SSO and IAM you could create virtually any kind of access controls around ECS. K8s doesn't solve the multi-account problem, and federation in K8s can be quite difficult.
  - tekkk 5y ago
    All right, thanks! Just wondered as I have a ECS setup and I really prefer not having to switch unless I have to.
  - still_grokking 5y ago

megamix 5y ago

Oh God another one

jaaron 5y ago
Another one? Nomad and kubernetes were released within months of each other in 2015. This isn't new.

deleted 5y ago

[deleted]

aequitas 5y ago

Looking forward to Nomad Packs, they where announced with their 1.2 beta[0]. I've really been missing something like k8s-at-home[1] for homelab Nomad setups. Don't know if they will become as versatile as Helm charts since Nomad is less of an abstraction than Kubernetes. [0] https://www.hashicorp.com/blog/announcing-hashicorp-nomad-1-2 https://www.hashicorp.com/blog/announcing-hashicorp-nomad-1-... [1] https://github.com/k8s-at-home/charts https://github.com/k8s-at-home/charts

sharmin123 5y ago

Facebook Safety Tips: Take Steps Now and Avoid Hacking: https://www.hackerslist.co/facebook-safety-tips-take-steps-now-and-avoid-hacking/ https://www.hackerslist.co/facebook-safety-tips-take-steps-n...

hacker_newz 5y ago

I've run into so many issues with Nomad that it really doesn't make sense to compare the two. Many are well documented in Github issues and occur when utilizing other Hashicorp products.

numbsafari 5y ago
I'm curious how responsive you have found them to issues you've identified. Any particular problems you've experienced that have been long standing an unresolved? I occasionally like to kick the tires on Nomad, but so far haven't found it compelling enough to switch away from my current solution, GKE (mostly because of how integrated it is with everything else I am doing and, at this point, familiarity). So, curious to hear from others where the rough edges are.
- 3np 5y ago
  Not the person you’re replying to, but I’m a couple of years in on Nomad, coming from kubernetes. I’ve reported and participated in a few issues. Most got fixed within the next couple of minor versions, some went neglected. I think the somewhat recent 1.0-release was pretty well-timed. There are still a few rough edges (though it’s gotten a lot better continuously over than last year). Specifically, Consul Connect and federation (especially together with ACLs and mTLS) are just about coming together now. If you don’t rely on Connect and federation across regions, I’d say it’s reliable for production use. Other than that maybe wait 6-12 months and reassess. I’d also advise to put ACLs and TLS in place before you start putting production loads there; it’s not great to introduce in a running cluster.

omneity 5y ago

Nomad is amazing. We've been using it alongside Consul for close to 2 years at this point at Monitoro.co[0]. We started by using it as a "systemd with a REST API" on a single server, and gradually evolved into multiple clusters with dozens of nodes each. It has been mostly a smooth ride, even if bumpy at moments, but operationally speaking Nomad is closer to Docker Swarm in simplicity and Kubernetes in terms of the feature set. We didn't find ourselves needing K8s as we're also leveraging components from the cloud provider to complete our infrastructure. [0]: https://monitoro.co https://monitoro.co

oneweekwonder 5y ago
> Nomad is closer to Docker Swarm in simplicity and Kubernetes in terms of the feature set. This a question I still need to google but what features does Kubernetes have that Docker Swarm needs? Because the perceived complexity of Kubernetes just blows my mind, where Docker Swarm seems a lot more simpler for the same benefits, but my its just abstracted away? I will say upfront im naive when it comes to container tech.
- proxysna 5y ago
  Swarm has a lot of issues. Some that are on the surface like bad networking and stemming from it scaling issues. Others are related directly to Mirantis. Company that owns docker.inc now. It neglects the swarm part of the docker, even was planning to straight up sunset swarm and move everyone to k8s. They do maintenance, and add feature or two a year which is not enough. Swarm is great for small deployments, as it only requires a docker daemon present. Otherwise you should look towards nomad/k8s/whatewer the cloud solution is.

robertlagrant 5y ago

> In contrast to Kubernetes' fragmented distributions This article just seems like "All of Nomad's best features phrased kindly vs all of K8s' worst features phrased critically".

jeswin 5y ago

Despite its reputation, Kubernetes is actually quite easy to master for simple use cases. And affordable enough for more complex ones. The fundamental abstractions are as simple as they can be, representing concepts that you'd already be familiar with in a datacenter environment. A cluster has nodes (machines), and you can run multiple pods (which is the smallest deployable unit on the cluster) on each node. A pod runs various types of workloads such as web services, daemons, jobs and recurring jobs which are made available (to the cluster) as docker/container images. You can attach various types of storage to pods, front your services with load-balancers etc. All of the nouns in the previous paragraph are available as building blocks in Kubernetes. You build your complex system declaratively from these simpler parts. When I look at Nomad's documentation, I see complexity. It mixes these basic blocks with those from HashiCorp's product suite.

andrewstuart2 5y ago
From a system architecture perspective, kubernetes is very complex since it handles a multitude of complexities, but that's why it _can_ be very simple from a user's perspective. Most of the complexity can be ignored (until you want it). It's the same reason I still like to use postgres when I can versus NoSQL until I know I need the one feature I may not be able to achieve with postgres: automatic sharding for massive global scale. The rest of the features postgres (and friends) give easily (ACID etc) are very tricky to get right in a distributed system. It's also the same reason bash is great for tasks and small automated programs, but kind of terrible for more complex needs. The primitives are super simple, easy to grok, and crazy productive at that scale, but other languages give tools to more easily handle the complexities that start to emerge when things get more complicated.
- xtracto 5y ago
  > The rest of the features postgres (and friends) give easily (ACID etc) are very tricky to get right in a distributed system. But that's just basically a calculated tradeoff of Postgres (and several CP databases) trading Availability for Consistency.
  - 5y ago

mkopinsky 5y ago

So far I've kept things simple, avoided k8s/HashiStack/etc by using docker compose with a simple docker-compose.yml for each server. This has been working well, but I'm starting to feel the pain points - HA requires copy-pasting yaml, I need to specify which services are on each server, and so on. What's the simplest next step I can take? I'd like something with (close to) the simplicity of docker compose, but ideally being able to specify something like `instances: 2` and being able to route/load-balance to those instances from another service.

ledneb 5y ago
Assuming serverless is out if the question for your use case, have you tried spending a couple of days investigating a managed Kubernetes cluster with node autoscaling enabled? EKS, AKS, GKE... Honestly it sounds like you could be at the point where K8s is worthwhile.
- mkopinsky 5y ago
  I'm considering k8s, but that also means moving services from on-prem to AKS, getting INF to open up the necessary firewall rules to make the services reachable from on-prem, and so on. And as you said, it's definitely days of investigation. I'm not closed to the option.
  - p_l 5y ago
    You might also want to consider one of the simpler pre-packaged solutions for small k8s on premise clusters (like k3s)
proxysna 5y ago
You should be ok with consul, ansible and templating your docker-compose files. Might take some time to set it all up, but should be ok.
- mkopinsky 5y ago
  Thanks for the suggestion. Any suggested starting points for my research? DDG search for "ansible docker-compose" brings up some suggestions like [1] and [2] but I'm curious if you have other suggestions. And just so I understand how these work together - I'd use Jenkins+ansible to push containers to my servers, I'd run consul within docker as well, and ... would Ansible register services with consul as it pushes them? Do the services need to be modified to speak with consul directly? [1] https://docs.ansible.com/ansible/latest/collections/community/docker/docker_compose_module.html https://docs.ansible.com/ansible/latest/collections/communit... [2] https://www.ansible.com/blog/six-ways-ansible-makes-docker-compose-better https://www.ansible.com/blog/six-ways-ansible-makes-docker-c...

deleted 5y ago

[deleted]

ascendantlogic 5y ago

Nomad gets the job done. What you end up missing is the huge ecosystem of tooling and knowledge that comes "for free" with Kubernetes.

tetha 5y ago
There has been a great announcement today with the beta of nomad 1.2 - nomad-pack: https://github.com/hashicorp/nomad-pack https://github.com/hashicorp/nomad-pack . Which, in my opinion, is aiming to be helm for nomad. Which is really great, because we've basically built nomad-pack in terraform and we would really like nomad-pack to replace that, because .. it works, but it could be better than terraform wrangling for templating :)

desktopninja 5y ago

My personal win for Nomad is its ability to run non containerized apps.

InternetPerson 5y ago

Wow, there's so much technical detail here! Like the part about how it's "simpler" and "flexible" and "consistent" I'm totally convinced by this product's marketing page!!

bonetruck 5y ago
I know you're being sarcastic. But the marketing page is not wrong. Try nomad yourself. Install: $ brew tap hashicorp/tap $ brew install hashicorp/tap/nomad Run: $ nomad agent -dev Start a job: $ nomad job run example.nomad

daenney 5y ago

These kinds of posts, by the vendor comparing to a competitor, always leave such a bad taste in my mouth. They decrease my confidence in both the product and the vendor. Stand on your own merits. There's a saying in Dutch: "Wij van Wc-eend adviseren Wc-eend"[1]. It basically boils down to pretending to give advice or information but you're just promoting your own interests. [1]: https://untranslatable.co/p/amarens/wij-van-wc-eend https://untranslatable.co/p/amarens/wij-van-wc-eend

xfz 5y ago
The Hashicorp ones tend to be well written and informative. I have found them useful in the past, even though I usually loathe sales BS. In this case the bias is a bit more obvious but we know it's written by Hashicorp and can make allowances.
rkangel 5y ago
I know what mean, but it's not like they're pretending that much if it's an article hosted under nomadproject.io. The first question everyone is going to ask "why should I use this instead of K8s" so you might as well have a good answer. On the other hand, the Rust project has purposefully avoided "Rust vs X" comparisons on its website. I can't find the HN comments to back this up, but people like steveklabnik have indicated that they don't find that sort of adversarial comparison to be useful. Rust has done an excellent job at community building, so I give a lot of credence to their approach.
- devit 5y ago
  Well, Rust is currently the best programming language so it's the other languages that need to come up with a "X vs Rust".
- steveklabnik 5y ago
  Yes. This topic is long and complicated; maybe I'll write it up or give a talk or something someday. There is a lot of nuance.
  - 5y ago

ketanmaheshwari 5y ago

Title should be "Nomad vs Kubernetes from Nomad's point of view".

capableweb 5y ago
The domain you see here on HN is nomadproject.io and the blog doesn't hide who the blogpost is from, it's very apparent already it's from the perspective of the people that maintain Nomad.

exdsq 5y ago

Does anyone know if Nomad can run on ChromeOS? I’ve had issues open with the Chrome and K8 teams to fix an error preventing it from working for months but I don’t think a fix will ever see the light of day, as it just gets passed around various repos.

hikerclimber1 5y ago

Everything is subjective. Especially laws.

zeroz 5y ago

For my solo developer self-hosted PaaS I really like CapRover [1]. Nice GUI, scalable Docker Swarm , with integrated route management, load balancing and one-click SSL. With available CLI you can use Gitlab CI to directly build and docker images and leverage CcapRover API to directly deploy on a dev stage. Interessting discussion about CapRover vs. Flynn, Dokku, Nomad on HN one year ago. [2] [1] https://caprover.com/ https://caprover.com/ [2] https://news.ycombinator.com/item?id=23465087 https://news.ycombinator.com/item?id=23465087

foobarbecue 5y ago

Do any of these systems come with a container image registry setup?

jaaron 5y ago

We're using Nomad for our backend at Singularity 6 (developing the mmo Palia). In my experience (having worked with Nomad, k8s, mesos, and homegrown schedulers), k8s is fantastic from an early developer experience, but rough from an operations perspective. It's definitely gotten better, but there's a reason lots of folks want to have someone else host and operate their cluster. Nomad is the opposite. Less tooling and bit more complexity for the app developer, significantly easier to operate at the cluster level. Nomad is also less "all-in" in it's architecture, which gives operators more flexibility in how they design and support the cluster. The fact that we can run on anything and don't _have_ to use containers is a serious advantage, particularly for game development.

awinter-py 5y ago

wonder what the 'orchestration feature set' of 2 years in the future will be speaking just for myself, feels like I want a system that runs a mix of serverless (like knative), normal containers, raw linux binaries (easy w/ fat binary systems like golang), language-specific packages like heroku does, maybe firecracker VMs. hard to get knative right if you don't have a reverse proxy tightly integrated with your tool

omneity 5y ago
Wild guess, probably more than 2 years ahead (also kinda what I'm hoping for) A natively distributed runtime providing similar capabilities to Docker + Nomad but running on "bare" WASM, providing a compilation target, distributed and clustering primitives right in your application code. Imagine the nodejs cluster module, targeting nomad directly, or talking to an instance of a service running on another node, the communication and addressing would be handled by the runtime. Similarly, cluster or instance level ports could be opened directly from your code and the distributed runtime will do the external/internal load balancing. Maybe taking it one step further and advertising services directly abstracting ports away. Service mesh + Scheduling + Capability-based Runtime in one binary you could run on Linux. One can dream.

shrumm 5y ago

To most people, it's not Nomad vs Kubernetes - it's a choice between Nomad vs Managed Kubernetes. All major cloud providers offer a managed kubernetes service at minimal added cost to running the worker VMs yourself. With managed Kubernetes, the simplicity question is no longer obviously in Nomad's favour. As other comments allude to, Kubernetes as a user is pretty easy to master once you get used to its mental model.

pid-1 5y ago
Agreed. Managing a Nomad cluster alongside a Consul cluster does not require a PHD, but it's also not a walk in the park. Hopefully Hashicorp will have managed Nomad soon.
mac-chaffee 5y ago
I'm currently trying to convince people that a managed k8s service is not that "simple", and that we can't "just spin up another cluster" without a great deal of operational overhead. Some of the things that might still be needed in managed k8s instances: better ingress with ingress-nginx, cert-manager, monitoring/logging/alerting, tuning the alerts, integration with company SSO, security hardening. If it's a multi-tenant cluster: LimitRanges/ResourceQuotas, NetworkPolicies, scripts to manage namespaces and roles, PodSecurityPolicies (or equivalent), onboarding offboarding procedures. I'm sure you'd need similar things to have a proper production Nomad cluster too, so your point still stands. But at least for EKS/GKE clusters, they're pretty bare-bones.
- darkwater 5y ago
  Thanks for this, I was really starting to think I was the madman.
- busterarm 5y ago
  As someone with 7-digit spend in GKE/EKS, I will agree with you that it is _anything but simple_. Your developers aren't going to say that it's simple when Google force upgrades their cluster to a version that deprecates APIs in their yamls for a job they worked on 2 years ago and swiftly forgot about. Then when you explain to them that Google insists on putting everyone on a force-upgrade treadmill, you can literally watch as the panic sets in on the faces of your engineering team managers/leads. Nomad is a breeze in comparison to managed K8s. Everyone that I've talked to that thinks Kubernetes is simple is barely using the thing and could likely save a lot of money and development effort using something like Nomad instead.

siliconc0w 5y ago

What is nice about using the same tech as the 'big players' is you get the benefits of massive armies of engineers building integrated products and features to sell to other big users of the product. This means there are options to add on distributed databases, storage, monitoring, tracing, CD/CI, etc. So it can be worth swallowing 'big player' complexity if it means you can play in the same ecosystem. If you are already on a cloud then your infrastructure provider will handle a lot of that complexity for you anyway and it's often better integrated into their other products (i.e networking, access control, etc).

chucky_z 5y ago

If you have a mixed Windows/Linux environment I cannot suggest Nomad strongly enough. It truly shines in this environment. Especially if you're dealing with legacy stuff. Even better if you're running Mac + BSD as well, it just runs everywhere.

rubenhak 5y ago

I don't agree with "Kubernetes is an orchestration system for containers originally designed by Google" statement. White it is not false, it creates a wrong impression of what K8s is. Kubernetes is a cloud operating system which lets you run modern apps in any environment. One important component of course container orchestration, but it went far beyond just orchestrator. Kubernetes has a very powerful ecosystem, and it managed to unite almost all infrastructure vendors and projects. Its amazing to see how so many competing companies could agree on something. And that become K8s. Nomad is great when you're working with VMs, but I don't see it is much relevant in the modern era of K8s and cloud-native.

deleted 5y ago

[deleted]

toddnni 5y ago

Is it so that the fragmentation (the long list of distributions), or variety of configuration options, is a bad quality? This seems to be used as a counter argument to Kubernetes. Probably, it could be end result of unconsistent design or bad technical choises. However, most likely it just means that there are multiple organizations and interest groups pushing changes and ideas to the project. This should be seen as a good thing. The downside is that there is no single source of best practices and this is confusing to newcomers. You just need to pick one distribution and trust the choises, or develop the competence and understanding. And we could imagine that the userbase or the number of developers in single distribution (take OpenShift or Rancher) could be bigger that in Nomad itself. Having said that, I still would like to see more stable Kubernetes landscape, and that has to happen eventually. The light distributions k3s and k0s are pushing things to nice direction. OpenStack had similar, or maybe even worse, fragmentation and complexity issue when the hype was high. There were probably technically better alternatives (Eucalyptus?), but people (and companies) gathered around OpenStack and it won the round. However, comparing OpenStack to Kubernetes feels bad, as Kubernetes is technically far superior.

ookblah 5y ago

i would say the "simplest" setup for us has been ECS fargate (even thought there's some major lock in). Very easy to spin up a cluster and you don't have to manage the underlying structure. If you use docker-compose or equivalent for local dev you just define your tasks/services and launch. Even pulling in ENVs is easy with SSM.

politelemon 5y ago
What kind of lock-in? One of the reasons I started using it was due to the minimal lock-in, so we've arrived at different conclusions. A task definition is required, which is sort of a docker-compose.yml analogue, but it's also possible to use docker compose directly with ECS Fargate. So my 'get out of AWS' is pretty simple, I already have the docker-compose.yml ready.
- ookblah 5y ago
  i suppose what i mean is you'd never be able to take a ECS task definition/service and move it over to some other provider. whereas if u did some kubernetes setup u could conceivably do that, moving form AWS to GCP.

AlexCoventry 5y ago

Why has this post dropped 20- odd ranks in roughly an hour?

kosolam 5y ago

Any nice step by step tutorial how to get nomad vault and consul architecture running quickly? I’m especially interested in non container job.

xtat 5y ago

I was into it until I saw there is an enterprise version link. If it's not FLOSS I'm out.

dstrickland 5y ago
It's mostly OSS. There are pay to play enterprise features, but you can most definitely run high end production clusters without needing enterprise.