18 ms·

How we use HashiCorp Nomad

318 points by jen20 6y ago

rcar 6y ago
Article is missing the title's leading "How"
4636760295 6y ago
I worked at a hedge fund that also used nomad. The problem, however, is not how well it scales or whatever, but the fact that all the accompanying deployment info and literature is for kubernetes, and k8s has far more features. I like the quality of products from HashiCorp, but k8s is far, far, ahead of where nomad is. What I really want is better integration for Terraform and kubernetes. The current TF k8s support leaves much to be desired, too many things are missing or broken and I find there are several bugs that result in deployment flapping (i.e., constantly re-deploying the same thing when there are no changes).
- api 6y ago
  Which features is Nomad missing? Feature count comparisons are meaningless unless the features are tied to actual important use cases. Lots of software is encrusted with rarely used features that just add complexity.
  - 4636760295 6y ago
    The main missing feature is that it's not kubernetes.
    - aganame 6y ago
      You have an extra word there.
  - SteveNuts 6y ago
    For us it's not a feature missing, but the mindshare is low, and there's not really any prebuilt examples of how to run and maintain services long term reliably.
  - DavyJone 6y ago
    Does it have "easy" side-cars/init/post? Last time I checked those were missing for example.
    - adadgar 6y ago
      It does! Here is the relevant documentation and examples: https://www.nomadproject.io/docs/job-specification/lifecycle/ https://www.nomadproject.io/docs/job-specification/lifecycle... https://learn.hashicorp.com/nomad/task-deps/interjob https://learn.hashicorp.com/nomad/task-deps/interjob
      - DavyJone 6y ago
        Ill check that out, I really think K8s is becoming a bit too complex for many customers. Thanks!
  - q3k 6y ago
    The free version of Nomad is missing, IIUC: - no quotas for teams/projects/organizations - no preemption (ie higher priority job preempts lower priority job) - no namespacing So generally it's somewhat useless in organizations where there are multiple different teams that should be able to coexist on a cluster without stepping on eachothers' toes, or even where you want a CI system to access the cluster in a safe manner.
    - chucky_z 6y ago
      Preemption is coming in the next release in OSS :)
      - deleted 6y ago
        [deleted]
    - BurritoAlPastor 6y ago
      We run everything in Nomad for all our teams, but we don’t use any of the features you mention, and it’s not causing issues. We wrote some metrics for how much memory all the allocations are claiming, and we throw another EC2 instance in the cluster when we’re running out. Works pretty well. Preemption seems like a nice feature, but I imagine getting teams to coordinate their preemption values would be a political nightmare.
      - q3k 6y ago
        For me the ability to squeeze in some jobs in between the cracks at best effort priority is crucial (ie. any sort of high latency batch processing / experiments). I wouldn't want these extremely low priority jobs to compete with, I don't know, an actual customer facing service. Also, we run on bare metal, so there really isn't a way to request extra capacity within seconds.
    - schmichael 6y ago
      Preemption is going OSS! https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md We fully intend to migrate more features to OSS in the future -- especially as we build out more enterprise features. As you can imagine building a sustainable business is quite the balancing act, and there's constant internal discussion. (I'm the Nomad Team Lead at HashiCorp)
  - syllogism 6y ago
    A few long-standing feature requests I've noticed a lot: * No autoscaling * Can't reserve entire CPU core: https://github.com/hashicorp/nomad/issues/289 https://github.com/hashicorp/nomad/issues/289 * No way to run jobs sequentially: https://github.com/hashicorp/nomad/issues/419 https://github.com/hashicorp/nomad/issues/419
    - jambay 6y ago
      Autoscaling is in Tech Preview as of March: https://www.hashicorp.com/blog/hashicorp-nomad-autoscaling-tech-preview/ https://www.hashicorp.com/blog/hashicorp-nomad-autoscaling-t... Disclaimer: I work at HashiCorp in Product Management.
      - syllogism 6y ago
        Nice! It's definitely been one of the bigger holes. I hope you'll also consider sequential tasks in a single jobfile. Without it it's kind of awkward because you have to be passing shell scripts around just to run a setup step before your actual workload.
        nickethier 6y ago
        We’ve also started to implement this with a new task dependencies feature in the latest release. Check out: https://www.hashicorp.com/resources/preview-of-nomad-0-11-task-dependencies/ https://www.hashicorp.com/resources/preview-of-nomad-0-11-ta... https://learn.hashicorp.com/nomad/task-deps/interjob https://learn.hashicorp.com/nomad/task-deps/interjob There’s more work going on there to further improve the feature in upcoming releases. Disclaimer: I’m an engineer on the Nomad team.
- monus 6y ago
  Crossplane https://crossplane.io https://crossplane.io might be what you’re looking for with its bunch of controllers and a nice composition API. One of the best features is that you can bundle a CR to request a MySQL database and it will be satisfied with whatever config is in your cluster so that app only declares the need but not care how it’s done. Disclaimer: I’m one of the maintainers of Crossplane.
  - mleonard 6y ago
    Any chance you plan to integrate with Google's Config Connector? It's very similar to crossplane (but gcp specific). https://cloud.google.com/config-connector/docs/overview https://cloud.google.com/config-connector/docs/overview
    - monus 6y ago
      I think that'd be a bit challenging because Config Connector is highly opinionated, for example, Kubernetes Namespace corresponds to GCP project. Though it might be enabled to be used as part of a Composition when we support namespaced CRs to be used as composition member.
      - rossmohax 6y ago
        Every config connector resource can be annotated with the project for GCP resource to be created in. Namespece to GCP project mapping is encouraged, but not enforced, you are still free to create resources in multiple projects from a single namespace as well as in a single project from multiple namespaces.
        monus 6y ago
        Oh, thanks for letting me know!
- closeparen 6y ago
  "Kubernetes but with less stuff" is a valuable niche that I'm glad someone is targeting.
  - 4636760295 6y ago
    That's a fair point, I guess it depends on your use case. The risk, however, is that the powers that be at HashiCorp one day decide to abandon Nomad once they realize it will never be a profit centre for them.
    - otterley 6y ago
      Nomad is open source (or, at least, a significant subset of it is). Anyone is able to continue to improve it, even if Hashicorp is no longer paying people to work on it.
      - 4636760295 6y ago
        I used to do sales for an enterprise "open source" software product, so I get it, but the truth is as soon as someone stops paying people to keep the project going it will die.
        toomuchtodo 6y ago
        Who pays for k8s development?
        closeparen 6y ago
        Google. Commoditizing the cloud is their strategy to make GCP relevant.
        toomuchtodo 6y ago
        Question was rhetorical, I still appreciate the effort though.
- sciurus 6y ago
  Looked at https://www.hashicorp.com/blog/deploy-any-resource-with-the-new-kubernetes-provider-for-hashicorp-terraform https://www.hashicorp.com/blog/deploy-any-resource-with-the-... ?
- halbritt 6y ago
  Why would you want TF in k8s? That hurts my soul. Isn't exactly declarative, nor really imperative, mostly but not entirely idempotent, and worst of all, it has faux-state that may or may not reflect the actual state of the services. I used to love Terraform, but it broke my heart. Now we're done.
chucky_z 6y ago
Cloudflare, make sure you upgrade to 0.11.3, the new scheduling behavior is awesome for large clusters. Also a massive warning to anyone wanting to use hard cpu limits and cgroups (they do work in nomad it’s just not trivial), they don’t work like anyone expects and need to be heavily tested.
- waynesonfire 6y ago
  are you serious? you've got a major company touting a technology and a key component of it is broken?
  - rbjorklin 6y ago
    I haven’t tested hard CPU quotas with Nomad but I suspect the issue mentioned above is due to cgroups/CFS and is also applicable to Kubernetes. See these slides for more details: https://www.slideshare.net/mobile/try_except_/optimizing-kubernetes-resource-requestslimits-for-costefficiency-and-latency-highload https://www.slideshare.net/mobile/try_except_/optimizing-kub...
    - Pirate-of-SV 6y ago
      I recall that there's been some improvements to CFS to fix this since 2018.
      - chucky_z 6y ago
        There's been a ton of improvements over time. That doesn't make it any less foot-gun-y, unfortunately. I had the exact same problem with Kubernetes (and even just straight up containers), I just want to make sure folks are extremely aware of how big of a footgun it is, and to really, really, really test it well.
  - chucky_z 6y ago
    Yes, you know that small organization, Linux; who manages cgroups that everything else calls through with. I have only tested it with Docker, again, that company Moby with this very little used software. How dare their core systems have any systems that are not extremely simple to understand and have caveats.
ianlevesque 6y ago
After trying out most of the kubernetes ecosystem in the pursuit of a declarative language to describe and provision services, Nomad was a breath of fresh air. It is so much easier to administer and the nomad job specs were exactly what I was looking for. I also noticed a lot of k8s apps encourage the use of Helm or even shell scripts to set up key pieces, which defeats the purpose if you are trying to be declarative about your deployment.
- ognyankulev 6y ago
  Helm charts are declarative way of deploying app(s) and their accompanying resources.
  - q3k 6y ago
    Helm, however, is objectively terrible with its yaml-based templating language and zero practical modularity.
    - speedgoose 6y ago
      Indeed. Helm offers great features but it suffers from the kubernetes unnecessary complexity and by using golang templates in YAML. When I started with kubernetes I converted my small Docker compose files to kubernetes files. Later I rewrote everything in helm charts. Now it's almost more YAML and golang templates lines than business logic lines in my applications. I'm considering to go back to Docker compose files. It's simple, readable, and easy to maintain.
      - q3k 6y ago
        Highly recommend trying Jsonnet (via https://github.com/bitnami/kubecfg https://github.com/bitnami/kubecfg and https://github.com/bitnami-labs/kube-libsonnet https://github.com/bitnami-labs/kube-libsonnet) as an alternative. It makes writing Kubernetes manifests much more expressive and integrates better with Git/VCS based workflows. Another language like Dhall or CUE might also work, but I'm not aware of a kubecfg equivalent for them. Jsonnet in general is a pretty damn good configuration language, not only for Kubernetes. And much more powerful than HCL.
pier25 6y ago
Off topic but... I've always wondered how Cloudflare can not charge for the bandwidth. Even the free plan is super generous (CDN, SSL, etc). How are they making a profit when AFAIK all other CDNs charge you for the bandwidth (and I assume they have to pay for to their providers)?
- nielsole 6y ago
  See the second answer for a statement from cf https://webmasters.stackexchange.com/questions/88659/how-can-cloudflare-offer-a-free-cdn-with-unlimited-bandwidth https://webmasters.stackexchange.com/questions/88659/how-can...
  - pier25 6y ago
    I'm familiar with those terms but still. There are people with free accounts moving GBs every month through their network and I imagine those free users must account for a very large percentage of their traffic.
    - fach 6y ago
      I’d imagine at this point they are heavily peered in most markets, driven by said free users, so there isn’t a significant opex hit bandwidth -wise. Space/power opex plus network/compute hardware capex probably dominates their spend.
  - ksec 6y ago
    Thank You! The answer was from 2016, and yet this is the first time anyone has posted this every time a similar question was raised on HN over the past few years.
- jlgaddis 6y ago
  One reason is "settlement-free peering".
  - pier25 6y ago
candiddevmike 6y ago
The biggest hurdle with adopting nomad is the kubernetes ecosystem and related network effects. Things like operators only run on Kubernetes, and they're driving an entirely new paradigm of infrastructure and application management. HashiCorp is doing their best to keep up while supporting standard kube interfaces like CSI/CNI/CRI, but I don't know how they can possibly stay relevant with Kubernetes momentum. In my opinion, HashiCorp should look at what Rancher did with K3s and offer something like that, integrated with the entire Hashi stack. The only reason most people choose nomad is the (initial) simplicity of it (which quickly goes away once you realize how "on an island" you are with solutions for ingress etc). Deliver kube with that simplicity and Integration and it's a much more compelling story than what Nomad delivers today.
- chucky_z 6y ago
  This is what Nomad is though... It works without their other products and also natively integrates into them. Deploying Vault, Consul, and Nomad gives you a very nice experience. Also with Consul 1.8 and Nomad 0.11 you’ll get Consul Connect with Ingress gateways which solves some of those problems you mentioned.
  - shaklee3 6y ago
    Nomad uses consul as the kv store, so it doesn't work without their other products.
    - chucky_z 6y ago
      Please make an attempt to understand a technology before making comments like this. Nomad has no requirement to store kv's, nor secrets. It's an entirely separate thing from Consul and Vault. It simply integrates with Consul, and Vault. It even runs its own raft store, so each product's backend is totally separate (Vault dropped it's reliance on other backends as of 1.4 and can run it's own raft store now). Nomad template stanzas are simply consul-template (https://github.com/hashicorp/consul-template#plugins https://github.com/hashicorp/consul-template#plugins), you could use `{{ with secret "" }}` and never touch Consul. You also have every function beyond service and the key* set of functions. You could build pretty static, or dynamic configurations using these blocks without ever touching Consul. On top of that, both Terraform templates and Levant work well for templating job specs out themselves, which contain template stanzas in them. An example of something helpful would be if you wanted to drop a very small binary that changes with each deploy, you could use https://github.com/hashicorp/consul-template#base64decode https://github.com/hashicorp/consul-template#base64decode and just change the contents each time you deploy the job. If you wanted to use redis keys in your consul-template, simply drop a binary on each server as a plugin to consul-template, then for example: `{{ "my-key" | plugin "redis" "get" }}`. Why not try running it yourself? `nomad agent -dev` will get you a server running, then `nomad job init` will give you a full spec which you can run with `nomad job run example.job`.
jeffbee 6y ago
"... here is the CPU usage over a day in one of our data centers where each time series represents one machine and the different colors represent different generations of hardware. Unimog keeps all machines processing traffic and at roughly the same CPU utilization." Still a mystery to me why "balancing" has SO MUCH mindshare. This is almost certainly not the optimal strategy for user experience. It is going to be much better to drain traffic away from older machines while newer machines stay fully loaded, rather than running every machine at equal utilization factor.
- dpw 6y ago
  I'm an engineer at Cloudflare, and I work on Unimog (the system in question). You are right that even balancing of utilization across servers with different hardware is not necessarily the optimal strategy. But keeping faster machines busy while slower machines are idle would not be better. This is because the time to service a request is only partly determined by the time it takes while being processed on a CPU somewhere. It's also determined by the time that the request has to wait to get hold of a CPU (which can happen at many points in the processing of a request). As the utilization of a server gets higher, it becomes more likely that requests on that server will end up waiting in a queue at some point (queuing theory comes into play, so the effects are very non-linear). Furthermore, most of the increase in server performance in the last 10 years has been due to adding more cores, and non-core improvements (e.g. cache sizes). Single thread performance has increased, but more modestly. Putting those things together, if you have an old server that is almost idle, and a new server that is busy, then a connection to the old server will actually see better performance. There are other factors to consider. The most important duty of Unimog is to ensure that when the demand on a data center approaches its capacity, no server becomes overloaded (i.e. its utilization goes above some threshold where response latency starts to degrade rapidly). Most of the time, our data centers have a good margin of spare capacity, and so it would be possible to avoid overloading servers without needing to balance the load evenly. But we still need to be confident that if there is a sudden burst of demand on one of our data centers, it will be balanced evenly. The easiest way to demonstrate that is to balance the load evenly long before it becomes strictly necessary. That way, if the ongoing evolution of our hardware and software stack introduces some new challenge to balancing the load evenly, it will be relatively easy to diagnose it and get it addressed. So, even load balancing might not be the optimal strategy, but it is a good and simple one. It's the approach we use today, but we've discussed more sophisticated approaches, and at some point we might revisit this.
justincormack 6y ago
Nice that they did a kernel patch to fix the pivot root on initramfs problem, but sadly looking at replies it won't get merged https://lore.kernel.org/linux-fsdevel/20200305193511.28621-1-ignat@cloudflare.com/ https://lore.kernel.org/linux-fsdevel/20200305193511.28621-1...
heipei 6y ago
Great writeup, even if I'm longing for more details about things like Unimog and their configuration management tool. Pay close attention to the section where they described why they went with Nomad (Simple, Single-Binary, can schedule different workloads and not just docker, Integration with Consul). Nomad is so simple to run that you can run a cluster in dev mode with one command on your laptop (even MacOS native where you can try scheduling non-Docker workloads). I'd go so far as saying that it would even pay off to use Nomad as a job scheduler when you only have one machine and might have used systemd instead. You can wget the binary, write a small systemd unit for Nomad, then deploy any additional workloads with Nomad itself. By the time you have to scale to multiple machines you just add them to the cluster and don't have to rewrite your job definitions from systemd.
schoolornot 6y ago
I'm surprised Hashicorp hasn't repositioned this product. Terraform was a huge breath of fresh air after Cloudformation. If you ask me, deploying apps via k8s is even better. Everyone wants a free lunch and for me, momentum + cloud vendor support + ultimately the Nomad Enterprise features that come free with K8s made the choice easy.
- rossmohax 6y ago
  TF can be often suffocating still. Pulumi runs circles around it, when it comes to user experience in writing complex, modular, composable and reusable configurations.
  - tetha 6y ago
    Except now you have to deal with javascript. It's hard enough to turn some operators towards an IaC approach, but throwing Javascript at them isn't going to help.
    - memexy 6y ago
      What language would be good for operators if not JavaScript?
      - macintux 6y ago
        Speaking as a 25-year sys/netadmin who writes a fair bit of code too: nearly anything. I'd rather learn Go than deal with JavaScript. Python would be fine. Elixir would be great (I'd much prefer Erlang but there are only so many miracles I'm allotted in this lifetime). Perl, please.
        memexy 6y ago
        Pulumi has support both for Python and Go along with TypeScript and JavaScript. I never understand when people say operators don't like programming languages. Always seems weird to me that a person that constantly works with computers would prefer a static markup language instead of full programming language for managing the complexity of operating software in production environments. My original comment was about this attitude. I personally think programming languages are much better than markup languages for almost everything. If you are presenting static content then markup languages are fine but if you're doing anything more than presenting static content then YAML just makes no sense.
jFriedensreich 6y ago
One point i never get for companies operating their own hardware: If your problem was having a number of well known servers for the internal management services and you then move to a nomad cluster or kubernetes to schedule them dynamically, you end up with the same problem as before to schedule the well known nomad servers or kubernetes masters. So is the only advantage here that the nomad server images update less often than the images of the management services?
- dmwilcox 6y ago
  Not associated with CloudFlare but I've built similar stuff with Nomad. The cost/maintenance trade-off works when you have more SPOF management hosts than Nomad servers (5). You decrease host images down to 2, Nomad server and client, versus N management images. Though it does sound a bit like they're using config management rather than pre-build images. Bonus, Nomad servers are more failure resistant using Raft consensus versus any N management hosts. And for discovery I found the optimal pattern is to put all of the Nomad servers in a "cluster" A record for clients to easily join (pattern works well for Consul too)
JaggerFoo 6y ago
Thanks for sharing this excellent article.
technological 6y ago
Hope they allow users to access documentation for past versions of nomad. Currently there is no easy way to find if particular configuration option mentioned in the documentation is available for not.
fideloper 6y ago
What Grafana dashboards do you all use/like for Prometheus + Nomad?
jugg1es 6y ago
I love nomad and architected a large system that used it for several years. We are finally moving away from it, however, because of its lack of native support for autoscaling. I know there are third party solutions for it, but that doesn't work for us. I suspect a large number of k8s users could use nomad instead with way less overhead.
- jambay 6y ago
  I'm curious if you tried the new autoscaler [1] that is in Tech Preview? Is there something it's missing for your scenarios? [1] https://www.hashicorp.com/blog/hashicorp-nomad-autoscaling-tech-preview/ https://www.hashicorp.com/blog/hashicorp-nomad-autoscaling-t...
  - jugg1es 6y ago
    As you probably know, decisions like changing infrastructure architecture is a team decision and are made with the information available at the time. The nomad autoscaler just wasn't announced/released in time for us to seriously consider it. That coupled with the uncertainty around the future of Fabio made us look elsewhere. We remain staunch supporters of Hashicorp products overall. We will continue to use Terraform, Consul and a little Vault.
aprdm 6y ago
Do people that aren't cloudflare scale really see the need for kubernetes and/or Nomad? Of the two Nomad seems much more sane because it does one thing only and is much simpler to manage and deploy. That said, having have used it, we are mostly moving away from it. Consul + Docker/Docker-compose with systemd in "a service per vm" model has proved much easier to administrate to our scale (couple of datacenters, ~1k VM mark). It is actually so simple that is boring. Instead of fiddling with infrastructure the developers spend time solving business problems... Our stack is really boring, developers can find their way around ansible and share galaxy roles to configure / provision infrastructure... Other cloud native projects like Prometheus and Fluentd give us all the visibility we need in a very straight forward and boring way. p.s: Great article!
- shanemhansen 6y ago
  Imo kubernetes isn't about "scale". Folks like cloudflare often run a very limited number of services (proxy plus control plane), which can make features like quotas/affinities/limits less useful because 100% of the machine installed in the pop is dedicated to one process. I see kubernetes as more useful when organizationally you have more heterogeneous services and separate dev/sre groups. This allows them to divide up responsibilities more easily.
  - halbritt 6y ago
    Increases velocity as well. I have a set of developers that can focus on their code and not have to worry about the infrastructure, TLS certs, DNS, storage or whatever. That all gets abstracted away. For SRE, that gives one a common set of orchestration tooling one can develop against.
- stitched2gethr 6y ago
  From my personal experience, yes, but not just for scale. I've used Kubernetes in a 300k person organization and now in a 35 person organization. Kubernetes isn't the simplest solution, but it's just reliability in a box. You can control the network, and the servers, and the load balancing all through the API which means for most things I can install a fully clustered solution with a handful of commands. Set the number of instances I want to run and forget the whole thing exists. Suddenly I'm cloud agnostic by default and the underlying K8s management is handled by the cloud provider. In the 300k person org it was for scale and reliability. In the 35 person org it's so that 1 guy can manage everything without trying.
MuffinFlavored 6y ago
What can Nomad do that Terraform can't? I'm pretty sure you can blue-green rolling deploy services inside of Docker containers with Terraform too.
- jen20 6y ago
  They’re completely different classes of tool - Nomad is a runtime component, Terraform is not.
  - MuffinFlavored 6y ago
    but terraform can talk to the docker daemon and orchestrate containers. what can you achieve with nomad that you can’t achieve with terraform?
    - jen20 6y ago
      - run a non-dockerised job - have a restart policy outside of what the docker daemon offers - automatically schedule across multiple machines based on constraints including affinity and anti-affinity, bin packing etc - “dispatch” jobs The Terraform Docker provider effectively provides a tiny subset of the functionality of Nomad, which provides similar functionality across many drivers, not just Docker.
solatic 6y ago
So basically, when CloudFlare was making the decision to adopt Nomad, they had already adopted Consul and had already built in-house a custom scheduler (unimog) for their customer traffic. It's rather disingenuous to compare to Kubernetes-based installations by pointing out that Nomad is a single Go binary that's easy to install, because they're also running Consul and unimog. For what it's worth, kubelet (which schedules jobs, like Nomad) is also a single Go binary that's easy to install, as is kube-proxy (which helps services running on a node to send traffic to the right node, so, roughly analogous to Consul). If anything, the author practically makes the case against Nomad. Practically speaking, the only reason why CloudFlare adopted Nomad is because the stars aligned on their stack. Most companies will actually reduce complexity by adopting Kubernetes instead, compared to running two different schedulers for jobs and externally-managed service discovery.
- shaklee3 6y ago
  The single binary thing is a tired trope. Kubernetes has had hyperkube for many years, which is a single binary for many years. But in really, a single binary should be at the bottom of your list of concerns.
  - jordanbeiber 6y ago
    Hyperkube is several binaries bundled in one container, no?
    - mdaniel 6y ago
      It's often _distributed_ as a container, but it ~~is~~ was a single monolithic go binary. However, while I was looking for supporting evidence to that claim, I discovered that it recently[1] was evicted from the github.com/k/k repo and the docker image that is now built uses a shell script named hyperkube and the Dockerfile just untars the underlying binaries into /usr/local/bin I wasn't able to immediately track down the alleged new repo for hyperkube, though 1 = https://github.com/kubernetes/kubernetes/pull/83454 https://github.com/kubernetes/kubernetes/pull/83454
- 6y ago
deleted 6y ago
[deleted]
ospider 6y ago
Actually, I think K8S is much simpler to operate than nomad. With nomad, you have too many options, you can run your service/job as a process or within containers, but with Kubernetes, there is only one way to do things -- container, which is really a simple choice to make. Besides, Kubernetes got etcd builtin, so I don't have to use deploy and maintain consul. Last but not least, I still see containers mysteriously gone, and have no idea how nomad did that. With kubernetes, such thing never happened.

318 points by jen20 6y ago

rcar 6y ago

Article is missing the title's leading "How"

I worked at a hedge fund that also used nomad. The problem, however, is not how well it scales or whatever, but the fact that all the accompanying deployment info and literature is for kubernetes, and k8s has far more features. I like the quality of products from HashiCorp, but k8s is far, far, ahead of where nomad is. What I really want is better integration for Terraform and kubernetes. The current TF k8s support leaves much to be desired, too many things are missing or broken and I find there are several bugs that result in deployment flapping (i.e., constantly re-deploying the same thing when there are no changes).

api 6y ago
Which features is Nomad missing? Feature count comparisons are meaningless unless the features are tied to actual important use cases. Lots of software is encrusted with rarely used features that just add complexity.
- 4636760295 6y ago
  The main missing feature is that it's not kubernetes.
  - aganame 6y ago
    You have an extra word there.
- SteveNuts 6y ago
  For us it's not a feature missing, but the mindshare is low, and there's not really any prebuilt examples of how to run and maintain services long term reliably.
- DavyJone 6y ago
  Does it have "easy" side-cars/init/post? Last time I checked those were missing for example.
  - adadgar 6y ago
    It does! Here is the relevant documentation and examples: https://www.nomadproject.io/docs/job-specification/lifecycle/ https://www.nomadproject.io/docs/job-specification/lifecycle... https://learn.hashicorp.com/nomad/task-deps/interjob https://learn.hashicorp.com/nomad/task-deps/interjob
    - DavyJone 6y ago
      Ill check that out, I really think K8s is becoming a bit too complex for many customers. Thanks!
- q3k 6y ago
  The free version of Nomad is missing, IIUC: - no quotas for teams/projects/organizations - no preemption (ie higher priority job preempts lower priority job) - no namespacing So generally it's somewhat useless in organizations where there are multiple different teams that should be able to coexist on a cluster without stepping on eachothers' toes, or even where you want a CI system to access the cluster in a safe manner.
  - chucky_z 6y ago
    Preemption is coming in the next release in OSS :)
    - deleted 6y ago
      [deleted]
  - BurritoAlPastor 6y ago
    We run everything in Nomad for all our teams, but we don’t use any of the features you mention, and it’s not causing issues. We wrote some metrics for how much memory all the allocations are claiming, and we throw another EC2 instance in the cluster when we’re running out. Works pretty well. Preemption seems like a nice feature, but I imagine getting teams to coordinate their preemption values would be a political nightmare.
    - q3k 6y ago
      For me the ability to squeeze in some jobs in between the cracks at best effort priority is crucial (ie. any sort of high latency batch processing / experiments). I wouldn't want these extremely low priority jobs to compete with, I don't know, an actual customer facing service. Also, we run on bare metal, so there really isn't a way to request extra capacity within seconds.
  - schmichael 6y ago
    Preemption is going OSS! https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md We fully intend to migrate more features to OSS in the future -- especially as we build out more enterprise features. As you can imagine building a sustainable business is quite the balancing act, and there's constant internal discussion. (I'm the Nomad Team Lead at HashiCorp)
- syllogism 6y ago
  A few long-standing feature requests I've noticed a lot: * No autoscaling * Can't reserve entire CPU core: https://github.com/hashicorp/nomad/issues/289 https://github.com/hashicorp/nomad/issues/289 * No way to run jobs sequentially: https://github.com/hashicorp/nomad/issues/419 https://github.com/hashicorp/nomad/issues/419
  - jambay 6y ago
    Autoscaling is in Tech Preview as of March: https://www.hashicorp.com/blog/hashicorp-nomad-autoscaling-tech-preview/ https://www.hashicorp.com/blog/hashicorp-nomad-autoscaling-t... Disclaimer: I work at HashiCorp in Product Management.
    - syllogism 6y ago
      Nice! It's definitely been one of the bigger holes. I hope you'll also consider sequential tasks in a single jobfile. Without it it's kind of awkward because you have to be passing shell scripts around just to run a setup step before your actual workload.
      - nickethier 6y ago
        We’ve also started to implement this with a new task dependencies feature in the latest release. Check out: https://www.hashicorp.com/resources/preview-of-nomad-0-11-task-dependencies/ https://www.hashicorp.com/resources/preview-of-nomad-0-11-ta... https://learn.hashicorp.com/nomad/task-deps/interjob https://learn.hashicorp.com/nomad/task-deps/interjob There’s more work going on there to further improve the feature in upcoming releases. Disclaimer: I’m an engineer on the Nomad team.
monus 6y ago
Crossplane https://crossplane.io https://crossplane.io might be what you’re looking for with its bunch of controllers and a nice composition API. One of the best features is that you can bundle a CR to request a MySQL database and it will be satisfied with whatever config is in your cluster so that app only declares the need but not care how it’s done. Disclaimer: I’m one of the maintainers of Crossplane.
- mleonard 6y ago
  Any chance you plan to integrate with Google's Config Connector? It's very similar to crossplane (but gcp specific). https://cloud.google.com/config-connector/docs/overview https://cloud.google.com/config-connector/docs/overview
  - monus 6y ago
    I think that'd be a bit challenging because Config Connector is highly opinionated, for example, Kubernetes Namespace corresponds to GCP project. Though it might be enabled to be used as part of a Composition when we support namespaced CRs to be used as composition member.
    - rossmohax 6y ago
      Every config connector resource can be annotated with the project for GCP resource to be created in. Namespece to GCP project mapping is encouraged, but not enforced, you are still free to create resources in multiple projects from a single namespace as well as in a single project from multiple namespaces.
      - monus 6y ago
        Oh, thanks for letting me know!
closeparen 6y ago
"Kubernetes but with less stuff" is a valuable niche that I'm glad someone is targeting.
- 4636760295 6y ago
  That's a fair point, I guess it depends on your use case. The risk, however, is that the powers that be at HashiCorp one day decide to abandon Nomad once they realize it will never be a profit centre for them.
  - otterley 6y ago
    Nomad is open source (or, at least, a significant subset of it is). Anyone is able to continue to improve it, even if Hashicorp is no longer paying people to work on it.
    - 4636760295 6y ago
      I used to do sales for an enterprise "open source" software product, so I get it, but the truth is as soon as someone stops paying people to keep the project going it will die.
      - toomuchtodo 6y ago
        Who pays for k8s development?
        closeparen 6y ago
        Google. Commoditizing the cloud is their strategy to make GCP relevant.
        toomuchtodo 6y ago
        Question was rhetorical, I still appreciate the effort though.
sciurus 6y ago
Looked at https://www.hashicorp.com/blog/deploy-any-resource-with-the-new-kubernetes-provider-for-hashicorp-terraform https://www.hashicorp.com/blog/deploy-any-resource-with-the-... ?
halbritt 6y ago
Why would you want TF in k8s? That hurts my soul. Isn't exactly declarative, nor really imperative, mostly but not entirely idempotent, and worst of all, it has faux-state that may or may not reflect the actual state of the services. I used to love Terraform, but it broke my heart. Now we're done.

chucky_z 6y ago

Cloudflare, make sure you upgrade to 0.11.3, the new scheduling behavior is awesome for large clusters. Also a massive warning to anyone wanting to use hard cpu limits and cgroups (they do work in nomad it’s just not trivial), they don’t work like anyone expects and need to be heavily tested.

waynesonfire 6y ago
are you serious? you've got a major company touting a technology and a key component of it is broken?
- rbjorklin 6y ago
  I haven’t tested hard CPU quotas with Nomad but I suspect the issue mentioned above is due to cgroups/CFS and is also applicable to Kubernetes. See these slides for more details: https://www.slideshare.net/mobile/try_except_/optimizing-kubernetes-resource-requestslimits-for-costefficiency-and-latency-highload https://www.slideshare.net/mobile/try_except_/optimizing-kub...
  - Pirate-of-SV 6y ago
    I recall that there's been some improvements to CFS to fix this since 2018.
    - chucky_z 6y ago
      There's been a ton of improvements over time. That doesn't make it any less foot-gun-y, unfortunately. I had the exact same problem with Kubernetes (and even just straight up containers), I just want to make sure folks are extremely aware of how big of a footgun it is, and to really, really, really test it well.
- chucky_z 6y ago
  Yes, you know that small organization, Linux; who manages cgroups that everything else calls through with. I have only tested it with Docker, again, that company Moby with this very little used software. How dare their core systems have any systems that are not extremely simple to understand and have caveats.

ianlevesque 6y ago

After trying out most of the kubernetes ecosystem in the pursuit of a declarative language to describe and provision services, Nomad was a breath of fresh air. It is so much easier to administer and the nomad job specs were exactly what I was looking for. I also noticed a lot of k8s apps encourage the use of Helm or even shell scripts to set up key pieces, which defeats the purpose if you are trying to be declarative about your deployment.

ognyankulev 6y ago
Helm charts are declarative way of deploying app(s) and their accompanying resources.
- q3k 6y ago
  Helm, however, is objectively terrible with its yaml-based templating language and zero practical modularity.
  - speedgoose 6y ago
    Indeed. Helm offers great features but it suffers from the kubernetes unnecessary complexity and by using golang templates in YAML. When I started with kubernetes I converted my small Docker compose files to kubernetes files. Later I rewrote everything in helm charts. Now it's almost more YAML and golang templates lines than business logic lines in my applications. I'm considering to go back to Docker compose files. It's simple, readable, and easy to maintain.
    - q3k 6y ago
      Highly recommend trying Jsonnet (via https://github.com/bitnami/kubecfg https://github.com/bitnami/kubecfg and https://github.com/bitnami-labs/kube-libsonnet https://github.com/bitnami-labs/kube-libsonnet) as an alternative. It makes writing Kubernetes manifests much more expressive and integrates better with Git/VCS based workflows. Another language like Dhall or CUE might also work, but I'm not aware of a kubecfg equivalent for them. Jsonnet in general is a pretty damn good configuration language, not only for Kubernetes. And much more powerful than HCL.

pier25 6y ago

Off topic but... I've always wondered how Cloudflare can not charge for the bandwidth. Even the free plan is super generous (CDN, SSL, etc). How are they making a profit when AFAIK all other CDNs charge you for the bandwidth (and I assume they have to pay for to their providers)?

nielsole 6y ago
See the second answer for a statement from cf https://webmasters.stackexchange.com/questions/88659/how-can-cloudflare-offer-a-free-cdn-with-unlimited-bandwidth https://webmasters.stackexchange.com/questions/88659/how-can...
- pier25 6y ago
  I'm familiar with those terms but still. There are people with free accounts moving GBs every month through their network and I imagine those free users must account for a very large percentage of their traffic.
  - fach 6y ago
    I’d imagine at this point they are heavily peered in most markets, driven by said free users, so there isn’t a significant opex hit bandwidth -wise. Space/power opex plus network/compute hardware capex probably dominates their spend.
- ksec 6y ago
  Thank You! The answer was from 2016, and yet this is the first time anyone has posted this every time a similar question was raised on HN over the past few years.
jlgaddis 6y ago
One reason is "settlement-free peering".
- pier25 6y ago

candiddevmike 6y ago

The biggest hurdle with adopting nomad is the kubernetes ecosystem and related network effects. Things like operators only run on Kubernetes, and they're driving an entirely new paradigm of infrastructure and application management. HashiCorp is doing their best to keep up while supporting standard kube interfaces like CSI/CNI/CRI, but I don't know how they can possibly stay relevant with Kubernetes momentum. In my opinion, HashiCorp should look at what Rancher did with K3s and offer something like that, integrated with the entire Hashi stack. The only reason most people choose nomad is the (initial) simplicity of it (which quickly goes away once you realize how "on an island" you are with solutions for ingress etc). Deliver kube with that simplicity and Integration and it's a much more compelling story than what Nomad delivers today.

chucky_z 6y ago
This is what Nomad is though... It works without their other products and also natively integrates into them. Deploying Vault, Consul, and Nomad gives you a very nice experience. Also with Consul 1.8 and Nomad 0.11 you’ll get Consul Connect with Ingress gateways which solves some of those problems you mentioned.
- shaklee3 6y ago
  Nomad uses consul as the kv store, so it doesn't work without their other products.
  - chucky_z 6y ago
    Please make an attempt to understand a technology before making comments like this. Nomad has no requirement to store kv's, nor secrets. It's an entirely separate thing from Consul and Vault. It simply integrates with Consul, and Vault. It even runs its own raft store, so each product's backend is totally separate (Vault dropped it's reliance on other backends as of 1.4 and can run it's own raft store now). Nomad template stanzas are simply consul-template (https://github.com/hashicorp/consul-template#plugins https://github.com/hashicorp/consul-template#plugins), you could use `{{ with secret "" }}` and never touch Consul. You also have every function beyond service and the key* set of functions. You could build pretty static, or dynamic configurations using these blocks without ever touching Consul. On top of that, both Terraform templates and Levant work well for templating job specs out themselves, which contain template stanzas in them. An example of something helpful would be if you wanted to drop a very small binary that changes with each deploy, you could use https://github.com/hashicorp/consul-template#base64decode https://github.com/hashicorp/consul-template#base64decode and just change the contents each time you deploy the job. If you wanted to use redis keys in your consul-template, simply drop a binary on each server as a plugin to consul-template, then for example: `{{ "my-key" | plugin "redis" "get" }}`. Why not try running it yourself? `nomad agent -dev` will get you a server running, then `nomad job init` will give you a full spec which you can run with `nomad job run example.job`.

jeffbee 6y ago

"... here is the CPU usage over a day in one of our data centers where each time series represents one machine and the different colors represent different generations of hardware. Unimog keeps all machines processing traffic and at roughly the same CPU utilization." Still a mystery to me why "balancing" has SO MUCH mindshare. This is almost certainly not the optimal strategy for user experience. It is going to be much better to drain traffic away from older machines while newer machines stay fully loaded, rather than running every machine at equal utilization factor.

dpw 6y ago
I'm an engineer at Cloudflare, and I work on Unimog (the system in question). You are right that even balancing of utilization across servers with different hardware is not necessarily the optimal strategy. But keeping faster machines busy while slower machines are idle would not be better. This is because the time to service a request is only partly determined by the time it takes while being processed on a CPU somewhere. It's also determined by the time that the request has to wait to get hold of a CPU (which can happen at many points in the processing of a request). As the utilization of a server gets higher, it becomes more likely that requests on that server will end up waiting in a queue at some point (queuing theory comes into play, so the effects are very non-linear). Furthermore, most of the increase in server performance in the last 10 years has been due to adding more cores, and non-core improvements (e.g. cache sizes). Single thread performance has increased, but more modestly. Putting those things together, if you have an old server that is almost idle, and a new server that is busy, then a connection to the old server will actually see better performance. There are other factors to consider. The most important duty of Unimog is to ensure that when the demand on a data center approaches its capacity, no server becomes overloaded (i.e. its utilization goes above some threshold where response latency starts to degrade rapidly). Most of the time, our data centers have a good margin of spare capacity, and so it would be possible to avoid overloading servers without needing to balance the load evenly. But we still need to be confident that if there is a sudden burst of demand on one of our data centers, it will be balanced evenly. The easiest way to demonstrate that is to balance the load evenly long before it becomes strictly necessary. That way, if the ongoing evolution of our hardware and software stack introduces some new challenge to balancing the load evenly, it will be relatively easy to diagnose it and get it addressed. So, even load balancing might not be the optimal strategy, but it is a good and simple one. It's the approach we use today, but we've discussed more sophisticated approaches, and at some point we might revisit this.

justincormack 6y ago

Nice that they did a kernel patch to fix the pivot root on initramfs problem, but sadly looking at replies it won't get merged https://lore.kernel.org/linux-fsdevel/20200305193511.28621-1-ignat@cloudflare.com/ https://lore.kernel.org/linux-fsdevel/20200305193511.28621-1...

heipei 6y ago

Great writeup, even if I'm longing for more details about things like Unimog and their configuration management tool. Pay close attention to the section where they described why they went with Nomad (Simple, Single-Binary, can schedule different workloads and not just docker, Integration with Consul). Nomad is so simple to run that you can run a cluster in dev mode with one command on your laptop (even MacOS native where you can try scheduling non-Docker workloads). I'd go so far as saying that it would even pay off to use Nomad as a job scheduler when you only have one machine and might have used systemd instead. You can wget the binary, write a small systemd unit for Nomad, then deploy any additional workloads with Nomad itself. By the time you have to scale to multiple machines you just add them to the cluster and don't have to rewrite your job definitions from systemd.

schoolornot 6y ago

I'm surprised Hashicorp hasn't repositioned this product. Terraform was a huge breath of fresh air after Cloudformation. If you ask me, deploying apps via k8s is even better. Everyone wants a free lunch and for me, momentum + cloud vendor support + ultimately the Nomad Enterprise features that come free with K8s made the choice easy.

rossmohax 6y ago
TF can be often suffocating still. Pulumi runs circles around it, when it comes to user experience in writing complex, modular, composable and reusable configurations.
- tetha 6y ago
  Except now you have to deal with javascript. It's hard enough to turn some operators towards an IaC approach, but throwing Javascript at them isn't going to help.
  - memexy 6y ago
    What language would be good for operators if not JavaScript?
    - macintux 6y ago
      Speaking as a 25-year sys/netadmin who writes a fair bit of code too: nearly anything. I'd rather learn Go than deal with JavaScript. Python would be fine. Elixir would be great (I'd much prefer Erlang but there are only so many miracles I'm allotted in this lifetime). Perl, please.
      - memexy 6y ago
        Pulumi has support both for Python and Go along with TypeScript and JavaScript. I never understand when people say operators don't like programming languages. Always seems weird to me that a person that constantly works with computers would prefer a static markup language instead of full programming language for managing the complexity of operating software in production environments. My original comment was about this attitude. I personally think programming languages are much better than markup languages for almost everything. If you are presenting static content then markup languages are fine but if you're doing anything more than presenting static content then YAML just makes no sense.

jFriedensreich 6y ago

One point i never get for companies operating their own hardware: If your problem was having a number of well known servers for the internal management services and you then move to a nomad cluster or kubernetes to schedule them dynamically, you end up with the same problem as before to schedule the well known nomad servers or kubernetes masters. So is the only advantage here that the nomad server images update less often than the images of the management services?

dmwilcox 6y ago
Not associated with CloudFlare but I've built similar stuff with Nomad. The cost/maintenance trade-off works when you have more SPOF management hosts than Nomad servers (5). You decrease host images down to 2, Nomad server and client, versus N management images. Though it does sound a bit like they're using config management rather than pre-build images. Bonus, Nomad servers are more failure resistant using Raft consensus versus any N management hosts. And for discovery I found the optimal pattern is to put all of the Nomad servers in a "cluster" A record for clients to easily join (pattern works well for Consul too)

JaggerFoo 6y ago

Thanks for sharing this excellent article.

technological 6y ago

Hope they allow users to access documentation for past versions of nomad. Currently there is no easy way to find if particular configuration option mentioned in the documentation is available for not.

fideloper 6y ago

What Grafana dashboards do you all use/like for Prometheus + Nomad?

jugg1es 6y ago

I love nomad and architected a large system that used it for several years. We are finally moving away from it, however, because of its lack of native support for autoscaling. I know there are third party solutions for it, but that doesn't work for us. I suspect a large number of k8s users could use nomad instead with way less overhead.

jambay 6y ago
I'm curious if you tried the new autoscaler [1] that is in Tech Preview? Is there something it's missing for your scenarios? [1] https://www.hashicorp.com/blog/hashicorp-nomad-autoscaling-tech-preview/ https://www.hashicorp.com/blog/hashicorp-nomad-autoscaling-t...
- jugg1es 6y ago
  As you probably know, decisions like changing infrastructure architecture is a team decision and are made with the information available at the time. The nomad autoscaler just wasn't announced/released in time for us to seriously consider it. That coupled with the uncertainty around the future of Fabio made us look elsewhere. We remain staunch supporters of Hashicorp products overall. We will continue to use Terraform, Consul and a little Vault.

aprdm 6y ago

Do people that aren't cloudflare scale really see the need for kubernetes and/or Nomad? Of the two Nomad seems much more sane because it does one thing only and is much simpler to manage and deploy. That said, having have used it, we are mostly moving away from it. Consul + Docker/Docker-compose with systemd in "a service per vm" model has proved much easier to administrate to our scale (couple of datacenters, ~1k VM mark). It is actually so simple that is boring. Instead of fiddling with infrastructure the developers spend time solving business problems... Our stack is really boring, developers can find their way around ansible and share galaxy roles to configure / provision infrastructure... Other cloud native projects like Prometheus and Fluentd give us all the visibility we need in a very straight forward and boring way. p.s: Great article!

shanemhansen 6y ago
Imo kubernetes isn't about "scale". Folks like cloudflare often run a very limited number of services (proxy plus control plane), which can make features like quotas/affinities/limits less useful because 100% of the machine installed in the pop is dedicated to one process. I see kubernetes as more useful when organizationally you have more heterogeneous services and separate dev/sre groups. This allows them to divide up responsibilities more easily.
- halbritt 6y ago
  Increases velocity as well. I have a set of developers that can focus on their code and not have to worry about the infrastructure, TLS certs, DNS, storage or whatever. That all gets abstracted away. For SRE, that gives one a common set of orchestration tooling one can develop against.
stitched2gethr 6y ago
From my personal experience, yes, but not just for scale. I've used Kubernetes in a 300k person organization and now in a 35 person organization. Kubernetes isn't the simplest solution, but it's just reliability in a box. You can control the network, and the servers, and the load balancing all through the API which means for most things I can install a fully clustered solution with a handful of commands. Set the number of instances I want to run and forget the whole thing exists. Suddenly I'm cloud agnostic by default and the underlying K8s management is handled by the cloud provider. In the 300k person org it was for scale and reliability. In the 35 person org it's so that 1 guy can manage everything without trying.

MuffinFlavored 6y ago

What can Nomad do that Terraform can't? I'm pretty sure you can blue-green rolling deploy services inside of Docker containers with Terraform too.

jen20 6y ago
They’re completely different classes of tool - Nomad is a runtime component, Terraform is not.
- MuffinFlavored 6y ago
  but terraform can talk to the docker daemon and orchestrate containers. what can you achieve with nomad that you can’t achieve with terraform?
  - jen20 6y ago
    - run a non-dockerised job - have a restart policy outside of what the docker daemon offers - automatically schedule across multiple machines based on constraints including affinity and anti-affinity, bin packing etc - “dispatch” jobs The Terraform Docker provider effectively provides a tiny subset of the functionality of Nomad, which provides similar functionality across many drivers, not just Docker.

solatic 6y ago

So basically, when CloudFlare was making the decision to adopt Nomad, they had already adopted Consul and had already built in-house a custom scheduler (unimog) for their customer traffic. It's rather disingenuous to compare to Kubernetes-based installations by pointing out that Nomad is a single Go binary that's easy to install, because they're also running Consul and unimog. For what it's worth, kubelet (which schedules jobs, like Nomad) is also a single Go binary that's easy to install, as is kube-proxy (which helps services running on a node to send traffic to the right node, so, roughly analogous to Consul). If anything, the author practically makes the case against Nomad. Practically speaking, the only reason why CloudFlare adopted Nomad is because the stars aligned on their stack. Most companies will actually reduce complexity by adopting Kubernetes instead, compared to running two different schedulers for jobs and externally-managed service discovery.

shaklee3 6y ago
The single binary thing is a tired trope. Kubernetes has had hyperkube for many years, which is a single binary for many years. But in really, a single binary should be at the bottom of your list of concerns.
- jordanbeiber 6y ago
  Hyperkube is several binaries bundled in one container, no?
  - mdaniel 6y ago
    It's often _distributed_ as a container, but it ~~is~~ was a single monolithic go binary. However, while I was looking for supporting evidence to that claim, I discovered that it recently[1] was evicted from the github.com/k/k repo and the docker image that is now built uses a shell script named hyperkube and the Dockerfile just untars the underlying binaries into /usr/local/bin I wasn't able to immediately track down the alleged new repo for hyperkube, though 1 = https://github.com/kubernetes/kubernetes/pull/83454 https://github.com/kubernetes/kubernetes/pull/83454
6y ago

deleted 6y ago

[deleted]

ospider 6y ago

Actually, I think K8S is much simpler to operate than nomad. With nomad, you have too many options, you can run your service/job as a process or within containers, but with Kubernetes, there is only one way to do things -- container, which is really a simple choice to make. Besides, Kubernetes got etcd builtin, so I don't have to use deploy and maintain consul. Last but not least, I still see containers mysteriously gone, and have no idea how nomad did that. With kubernetes, such thing never happened.