3 ms·

So basically, when CloudFlare was making the decision to adopt Nomad, they had already adopted Consul and had already built in-house a custom scheduler (unimog)

So basically, when CloudFlare was making the decision to adopt Nomad, they had already adopted Consul and had already built in-house a custom scheduler (unimog) for their customer traffic. It's rather disingenuous to compare to Kubernetes-based installations by pointing out that Nomad is a single Go binary that's easy to install, because they're also running Consul and unimog. For what it's worth, kubelet (which schedules jobs, like Nomad) is also a single Go binary that's easy to install, as is kube-proxy (which helps services running on a node to send traffic to the right node, so, roughly analogous to Consul). If anything, the author practically makes the case against Nomad. Practically speaking, the only reason why CloudFlare adopted Nomad is because the stars aligned on their stack. Most companies will actually reduce complexity by adopting Kubernetes instead, compared to running two different schedulers for jobs and externally-managed service discovery.

shaklee3 6y ago
The single binary thing is a tired trope. Kubernetes has had hyperkube for many years, which is a single binary for many years. But in really, a single binary should be at the bottom of your list of concerns.
- jordanbeiber 6y ago
  Hyperkube is several binaries bundled in one container, no?
  - mdaniel 6y ago
    It's often _distributed_ as a container, but it ~~is~~ was a single monolithic go binary. However, while I was looking for supporting evidence to that claim, I discovered that it recently[1] was evicted from the github.com/k/k repo and the docker image that is now built uses a shell script named hyperkube and the Dockerfile just untars the underlying binaries into /usr/local/bin I wasn't able to immediately track down the alleged new repo for hyperkube, though 1 = https://github.com/kubernetes/kubernetes/pull/83454 https://github.com/kubernetes/kubernetes/pull/83454
gen220 6y ago
I agree with everything you said, but it’s important to note that the blogpost isn’t comparing nomad to K8s, but describing why nomad makes sense for cloudflare. From cloudflare’s perspective, K8s is a riskier choice, precisely because it’s large and multifunctional. If Nomad (or Consul) upstream went south, CF would probably be able to maintain it without seriously growing their headcount. And they’d probably be playing the lead role in that ensemble. If kubernetes upstream went south, the above statements would almost certainly not hold. For them, picking a stack means picking something they need to stick to, potentially for the next decade or two. When your decisions are made with that long term in focus, the biggest, shiniest, and most exciting community is not always the right choice. Like you said, it seems like this is the right choice for CF. It’s probably not the right choice for $TECH_STARTUP, for whom pinning the future of business critical infrastructure to Google and RedHat’s stewardship of K8s is the least of their concerns, and that’s OK.
jordanbeiber 6y ago
Having worked extensively with both the k8s and hashicorp stack let me tell you that there is something to be said regarding the simplicity of the former. First - companies will want service discovery for all managed resources. Consul is just outstanding, so there’s that. On-prem, nomad is simple to manage and it gives you a form of freedom that k8s locks up in a black box. After all, it’s a process scheduler, go ahead and schedule what you want. A container, raw exec or batch job. Add parameters to the batch job and dispatch them at will. If you start from scratch or can move absolutely everything to k8s? Well sure, but not a lot of places you’d call “enterprise” isn’t in that position.

shaklee3 6y ago

The single binary thing is a tired trope. Kubernetes has had hyperkube for many years, which is a single binary for many years. But in really, a single binary should be at the bottom of your list of concerns.

jordanbeiber 6y ago
Hyperkube is several binaries bundled in one container, no?
- mdaniel 6y ago
  It's often _distributed_ as a container, but it ~~is~~ was a single monolithic go binary. However, while I was looking for supporting evidence to that claim, I discovered that it recently[1] was evicted from the github.com/k/k repo and the docker image that is now built uses a shell script named hyperkube and the Dockerfile just untars the underlying binaries into /usr/local/bin I wasn't able to immediately track down the alleged new repo for hyperkube, though 1 = https://github.com/kubernetes/kubernetes/pull/83454 https://github.com/kubernetes/kubernetes/pull/83454

gen220 6y ago

I agree with everything you said, but it’s important to note that the blogpost isn’t comparing nomad to K8s, but describing why nomad makes sense for cloudflare. From cloudflare’s perspective, K8s is a riskier choice, precisely because it’s large and multifunctional. If Nomad (or Consul) upstream went south, CF would probably be able to maintain it without seriously growing their headcount. And they’d probably be playing the lead role in that ensemble. If kubernetes upstream went south, the above statements would almost certainly not hold. For them, picking a stack means picking something they need to stick to, potentially for the next decade or two. When your decisions are made with that long term in focus, the biggest, shiniest, and most exciting community is not always the right choice. Like you said, it seems like this is the right choice for CF. It’s probably not the right choice for $TECH_STARTUP, for whom pinning the future of business critical infrastructure to Google and RedHat’s stewardship of K8s is the least of their concerns, and that’s OK.

jordanbeiber 6y ago

Having worked extensively with both the k8s and hashicorp stack let me tell you that there is something to be said regarding the simplicity of the former. First - companies will want service discovery for all managed resources. Consul is just outstanding, so there’s that. On-prem, nomad is simple to manage and it gives you a form of freedom that k8s locks up in a black box. After all, it’s a process scheduler, go ahead and schedule what you want. A container, raw exec or batch job. Add parameters to the batch job and dispatch them at will. If you start from scratch or can move absolutely everything to k8s? Well sure, but not a lot of places you’d call “enterprise” isn’t in that position.