5 ms·
Kubernetes isn't the only thing around. Kubernetes and Mesos are kinda the heavyweight solutions, but there are smaller things around like Hashicorps Nomad and
by tetha 8y ago
Kubernetes isn't the only thing around. Kubernetes and Mesos are kinda the heavyweight solutions, but there are smaller things around like Hashicorps Nomad and Swarm, and probably a lot more I don't know.
We're currently evaluating nomad, and it's surprisingly pleasant. Nomad doesn't solve every problem every application in every situation might have. Nomad schedules containers, VMs or whatever else on hosts. This reduces complexity a lot.
It took us like 1 - 2 man-weeks to have an almost arbitrarily scalable nomad setup which allows you to submit a bunch of jobs and mark some public ports for a loadbalancer, be it mysql, http, whatever. And it's easy to understand and operate. There's 3 nodes of consul, 3 nodes of nomad-server and 2 hosts of nomad-client, some certs in the middle, consul-template + haproxy with a config almost from a blogpost. That's it. It has very few moving parts and it's easy to understand and troubleshoot with 2-3 main guys in our ops team. (EDIT: this doesn't read clear. We have 2-3 guys on our ops team. They are not working on nomad alone. Nomad atm is a low-maintenance system and we're mostly dealing with other crap /EDIT)
And now we're just going with it for now. Our CI needs resources for on-demand test-systems, so let's figure out how to make that happen. Our self-service test system for demos / manual acceptance testing needs resources for systems so let's figure that out. We might need to use gluster or something for persistent storage if we want to migrate internal tooling to this. A sister company might want to tinker around with windows VMs scheduled by nomad, or windows containers, so why not?
But the good thing: It took us 2 weeks to start delivering business value. That's a relatively small up-front payment for an established company, even a small one. Now we can leave it alone for some time, or we can invest some more well defined packages of time to make it better in concrete, requested ways. That's easy to schedule and prioritize.
- heipei 8y agoJust came here to second this. We evaluated both Kubernetes as well as Nomad for a relatively small cluster of some worker nodes and web services. In the end, the ease of standing up a Nomad cluster and the whole feel of the thing won us over. Nomad is a single golang binary that you can run on your laptop and have a fully working working Nomad client and server, along with a builtin UI and command line tools (same binary). The story for production is the same: Throw the binary on your server, setup a systemd unit to run it and you have another Nomad node. If you're evaluating container schedulers and are not sure what you need, take an afternoon or so and just run it locally and play with it. If there aren't any specific features about Kubernetes you could point to that Nomad does not meet my suggestion would be to get started with it first.
- shaklee3 8y agoKubernetes is a single binary, too (hyperkube).
- lmb 8y agoCan you give an idea how big your cluster is? 10, 10s, 100s? I'm curious how well Normand works in reality, but it's hard to find ppl running it.
- tetha 8y agoRight now we're 4 weeks in and IT and 2 guys want it. We're 3 server nodes + 2 client nodes big, 128G RAM, 16 cores, 32 threads. Please go ahead and call this tiny and irrelevant. I'm aware and under the same assumption. I've dealt with individual servers around 4x that size. But, go ahead and ask me again in 3 or 6 month. We'll migrate around 10 HW servers as nomad clients, we'll probably get HW capacity from our sister company and then we'll start interacting with the 2 other sister companies. If we get all of their resources into nomad, we'd be up to 60 - 80 beefy metal servers. And those 2 other sister companies are in pain for easy windows resources. If nomad can do that properly, I'll be sold for it quite hard. It'll be fun.
- freehunter 8y agoI'm not the person you're directly replying to but I'm the guy who originally asked the question at the top level, and I can say your needs right now are far bigger than mine. Right now my 5-container Dokku host has 4GB of RAM with 2 CPUs, but we're expecting to ramp up quickly. It's good to hear your input as I expect to be a similar size to you in a year or two. Far from irrelevant. I'll have to look into it.
- shaklee3 8y agoThe problem is if you come on here and talk about how great nomad is, then only later say you are only running five nodes, that gives people false impressions. Kubernetes is tested regularly with 5000 nodes. To get to that level takes an entirely different level of software to be able to run a cluster that large reliably without issues.
- freehunter 8y agoIsn't the entire purpose of this threat that Kubernetes is a big solution made for big problems and smaller problems might not need that big of a solution? The comment was in response to a question I asked about "do I really need something as big as Kubernetes". Not sure why people are picking apart this answer when it seems to answer my question quite well.