4 ms·

My question is: Why is only k8s so popular when there are better alternatives for a large swath of users? I believe the answer is "Manufactured Hype". k8s is fr

by heipei 6y ago

My question is: Why is only k8s so popular when there are better alternatives for a large swath of users? I believe the answer is "Manufactured Hype". k8s is from a purely architectural standpoint the way to go, even for smaller setups, but the concrete project is still complex enough that it requires dozens of different setup tools and will keep hordes of consultants as well as many hosted solutions from Google/AWS/etc in business for some time to come, so there's a vested interest in continuing to push it. Everyone wins, users get a solid tool (even if it's not the best for the job) and cloud providers retain their unique selling point over people setting up their own servers. I still believe 90% of users would be better served by Nomad. And if someone says "developers want to use the most widely used tech", then I'm here to call bullshit, because the concepts between workload schedulers and orchestrators like k8s and nomad are easy enough to carry over from one side to the other. Learning either even if you end up using the other one is not a waste of time. Heck, I started out using CoreOS with fleetctl and even that taught me many valuable lessons.

verdverm 6y ago
What are these alternatives with more users? Where is the momentum? Hosted GKE costs the same per month as an hour of DevOps time, what's wrong with paid management for k8s?
- heipei 6y ago
  I didn't say more users, I said appropriate for more users. The alternative I mentioned is Nomad and I wish more people would give it a try and decide for themselves. The momentum behind it is Hashicorp, makers of Vault, Consul, Terraform, Vagrant, all battle-proven tools. The fact that there's one big player behind it really shows in how polished the tool, UI and documentation is. The issue that I have with managed k8s is that these products will decrease the pressure to improve k8s documentation, tooling and setup itself. And then there's folks (like me) who want or need to run something like k8s on bare metal hardware outside of a cloud where the cloud-managed solution isn't available.
ravenstine 6y ago
> I still believe 90% of users would be better served by Nomad. Well sure, but if the story just ended with "everyone use the least exciting tool", then there'd be few articles for tech journals to write. But Kubernetes promises so much, and deep down everyone subtly thinks "what if I have to scale my project?" Why settle for good enough when you could settle for "awesome"? It's just human nature to choose the most exciting thing. And given that I do agree that there's some manufactured hype around Kubernetes, it isn't surprising to me why few are talking about Nomad.
dnautics 6y ago
I'm resisting kubernetes and might go with nomad (currently I'm "just using systemd" and I get HA from the BEAM VM)... But I do also get the argument that the difference between kubernetes and nomad is that increasingly kubernetes is supported by the cloud vendors, and nomad supports the cloud vendors.
jsmith12673 6y ago
I got a bit disillusioned with k8s and looked at Nomad as an alternative. As a relatively noob sysadmin, I liked it a lot. Easy to deploy and easy to maintain. We've got a lot of mixed rented hardware + cloud VPS, and having one layer to unify them all seemed great. Unfortunately I had a hard convincing the org to give it a serious shot. At the crux of it, it wasn't clear what 'production ready' Nomad should look like. It seemed like Nomad is useless without Consul, and you really should use Vault to do the PKI for all of it. It's a bit frustrating how so many of the HashiCorp products are 'in for penny, in for a pound' type deals. I know there's _technically_ ways for you use Nomad without Consul, but it didn't seem like the happy path, and the community support was non-existent. Please tell me why I'm wrong lol, I really wanted to love Nomad. We are running a mix of everything and its a nightmare
- kelnos 6y ago
  I'm sympathetic toward the idea of a system made of interchangeable parts, but I also kinda feel like it's a bit unrealistic, maybe? Even with well-defined interfaces, there will always be interop problems due to bugs or just people interpreting the interface specs differently. Every new piece to the puzzle adds another line (or several) to a testing matrix, and most projects just don't have the time and resources to do that kind of testing. It's unfortunate, but IMO understandable that there's often a well-tested happy path that everyone should use, even when theoretically things are modular and replaceable.
- chucky_z 6y ago
  Nomad + Consul is the happy path. Adding Vault into the mix is nice, but not required. Consul by itself is the game-changer. Even in k8s it's a game-changer. It solves so many questions in an elegant way. "How do I find and reach the things running in (orchestrator) with (unknown ip/random port) from (legacy)?" being the most important. You run 5 servers, and a relatively lightweight client on everything (which isn't even outright required, but it sure is useful!), and you get a _lot_ with that. Consul provides multiple interfaces and ingress points to find everything. It also is super easy to operate, and has a pretty big community. If you absolutely cannot have Consul, Nomad is still a really good batch job engine, and makes a very great "distributed cron," which is more extensible, scalable, and easy to use than something like Jenkins for the same task. My team is pretty small (was 4 people, now 6) and we manage one of the worlds largest nomad and consul clusters (there are some truly staggeringly large users of Vault so I won't make that claim). Even when shit really hits the fan, everything is designed in a way that stuff mostly works; and there's enough operator friendly entry points that we can always figure out the problem.
torvald 6y ago
I second this.
peterwwillis 6y ago
Nomad isn't really feature mature or user friendly enough, you still eventually need 100 bolt-ons. I think a Distributed OS is the only sane solution. Build the features we need into the kernel and stop futzing around with 15 abstractions to just run an isolated process on multiple hosts.
- schmichael 6y ago
  As the Nomad Team Lead I sympathize with your first statement, but I hope our continued efforts will dissuade you from the second. Linux (and the BSDs) are remarkably stable, festureful, and resilient operating systems. I would hate to give up such a strong foundation. Nomad can crash without affecting your running services. Nomad can be upgraded or reconfigured without affecting your running services. Nomad can be observed, developed, and debugged as a unit often without having to consider the abstractions that sit above or below it. The right number of abstractions is a beautiful thing. Just no more and no less. :)

by heipei 6y ago

verdverm 6y ago

What are these alternatives with more users? Where is the momentum? Hosted GKE costs the same per month as an hour of DevOps time, what's wrong with paid management for k8s?

heipei 6y ago
I didn't say more users, I said appropriate for more users. The alternative I mentioned is Nomad and I wish more people would give it a try and decide for themselves. The momentum behind it is Hashicorp, makers of Vault, Consul, Terraform, Vagrant, all battle-proven tools. The fact that there's one big player behind it really shows in how polished the tool, UI and documentation is. The issue that I have with managed k8s is that these products will decrease the pressure to improve k8s documentation, tooling and setup itself. And then there's folks (like me) who want or need to run something like k8s on bare metal hardware outside of a cloud where the cloud-managed solution isn't available.

ravenstine 6y ago

> I still believe 90% of users would be better served by Nomad. Well sure, but if the story just ended with "everyone use the least exciting tool", then there'd be few articles for tech journals to write. But Kubernetes promises so much, and deep down everyone subtly thinks "what if I have to scale my project?" Why settle for good enough when you could settle for "awesome"? It's just human nature to choose the most exciting thing. And given that I do agree that there's some manufactured hype around Kubernetes, it isn't surprising to me why few are talking about Nomad.

dnautics 6y ago

I'm resisting kubernetes and might go with nomad (currently I'm "just using systemd" and I get HA from the BEAM VM)... But I do also get the argument that the difference between kubernetes and nomad is that increasingly kubernetes is supported by the cloud vendors, and nomad supports the cloud vendors.

jsmith12673 6y ago

I got a bit disillusioned with k8s and looked at Nomad as an alternative. As a relatively noob sysadmin, I liked it a lot. Easy to deploy and easy to maintain. We've got a lot of mixed rented hardware + cloud VPS, and having one layer to unify them all seemed great. Unfortunately I had a hard convincing the org to give it a serious shot. At the crux of it, it wasn't clear what 'production ready' Nomad should look like. It seemed like Nomad is useless without Consul, and you really should use Vault to do the PKI for all of it. It's a bit frustrating how so many of the HashiCorp products are 'in for penny, in for a pound' type deals. I know there's _technically_ ways for you use Nomad without Consul, but it didn't seem like the happy path, and the community support was non-existent. Please tell me why I'm wrong lol, I really wanted to love Nomad. We are running a mix of everything and its a nightmare

kelnos 6y ago
I'm sympathetic toward the idea of a system made of interchangeable parts, but I also kinda feel like it's a bit unrealistic, maybe? Even with well-defined interfaces, there will always be interop problems due to bugs or just people interpreting the interface specs differently. Every new piece to the puzzle adds another line (or several) to a testing matrix, and most projects just don't have the time and resources to do that kind of testing. It's unfortunate, but IMO understandable that there's often a well-tested happy path that everyone should use, even when theoretically things are modular and replaceable.
chucky_z 6y ago
Nomad + Consul is the happy path. Adding Vault into the mix is nice, but not required. Consul by itself is the game-changer. Even in k8s it's a game-changer. It solves so many questions in an elegant way. "How do I find and reach the things running in (orchestrator) with (unknown ip/random port) from (legacy)?" being the most important. You run 5 servers, and a relatively lightweight client on everything (which isn't even outright required, but it sure is useful!), and you get a _lot_ with that. Consul provides multiple interfaces and ingress points to find everything. It also is super easy to operate, and has a pretty big community. If you absolutely cannot have Consul, Nomad is still a really good batch job engine, and makes a very great "distributed cron," which is more extensible, scalable, and easy to use than something like Jenkins for the same task. My team is pretty small (was 4 people, now 6) and we manage one of the worlds largest nomad and consul clusters (there are some truly staggeringly large users of Vault so I won't make that claim). Even when shit really hits the fan, everything is designed in a way that stuff mostly works; and there's enough operator friendly entry points that we can always figure out the problem.

torvald 6y ago

I second this.

peterwwillis 6y ago

Nomad isn't really feature mature or user friendly enough, you still eventually need 100 bolt-ons. I think a Distributed OS is the only sane solution. Build the features we need into the kernel and stop futzing around with 15 abstractions to just run an isolated process on multiple hosts.

schmichael 6y ago
As the Nomad Team Lead I sympathize with your first statement, but I hope our continued efforts will dissuade you from the second. Linux (and the BSDs) are remarkably stable, festureful, and resilient operating systems. I would hate to give up such a strong foundation. Nomad can crash without affecting your running services. Nomad can be upgraded or reconfigured without affecting your running services. Nomad can be observed, developed, and debugged as a unit often without having to consider the abstractions that sit above or below it. The right number of abstractions is a beautiful thing. Just no more and no less. :)