5 ms·
Interesting, what do you think should be used to manage a cluster of less than 25 machines?
by znpy 9y ago
Interesting, what do you think should be used to manage a cluster of less than 25 machines?
- hacknat 9y agoIt wasn’t a hard rule. It depend on the app and the use case, but probably just docker on all the machines with some static cloud loadbalancers in between everything. Or just straight config management.
- akvadrako 9y agoHosted k8s. all the benefits with almost no extra maintenance.
- Filligree 9y agoOkay, how about for a tiny budget?
- zie 9y agoNomad (www.nomadproject.io) is what I would recommend. Nomad is ONLY a task scheduler across a cluster of machines. Which is why it’s not rocket science, nor is it operationally complex. You say I need X cpu and X memory and I need these files out on disk(or this docker image) and run this command. It will enforce your task gets exactly X memory, X cpu and X disk, so you can’t over-provision at the task level, Docker and friends don't do this. It handles batch(i.e. cron) and spark workloads, system jobs(run on every node) and services (any long-running task). For instance with nomad batch jobs you can almost entirely replace Celery and other distributed task queues, in a platform and language agnostic way! You can do load balancing and all the other things k8s does, but you use specialized tools for these things: * For HTTPS traffic you can use Fabio, Traefik, HAProxy, Nginx, etc. * For TCP traffic you can use Fabio, Relayd, etc. These are outside of Nomad’s scope, except that you can run those jobs inside of Nomad just fine. and it’s all declarative, a total win.
- justinsaccount 9y ago> You can do load balancing and all the other things k8s does Nomad can do many of the things that k8s does, and even some things that k8s can't, but it does not do all the things that k8s does. For example, nomad doesn't have network policies. Also, just to approach the base level of functionality and security in k8s you need to setup consul as well and bootstrap all the tls stuff, and at that point it's not much more work to get k8s going. Nomad is pretty cool, and is rapidly catching up to k8s, it will be interesting to see what happens in the long run.
- zie 9y agoYes, True enough. Arguably, network policies are outside of Nomad's scope. It's a resource scheduler. Nomad doesn't turn up interfaces for you, or do routing of IP traffic. Nomad is not a one-stop shop, like k8s tries to be, it does resource scheduling in a nice declarative manner, that's about it. It's much more in the unix toolset philosophy, let 1 tool do 1 thing well, and make integration as painless as possible. k8s is much more in-line with the systemd way of thinking, it owns all the things and can be the only shiny in the room. I'm not sure I'd agree that turning up k8s is as easy as turning up consul, nomad and vault. K8s in my experience tends to require lots of babysitting. Consul can require some babysitting, but vault and nomad require basically no babysitting, except the occasional, mostly painless upgrade. For me it's really about operational simplicity. I have to keep this stuff running tomorrow and next year. Shiny, super-fast moving be all and do all stuff like k8s, Openstack, etc tend to be a giant pain that requires loads of babysitting(most notably at upgrade time), as the edge cases tend to hurt a lot, when you hit them.
- justinsaccount 9y agoI don't really see how nomad is operationally simpler than k8s. To run a service behind something like traefik on k8s I would: bootstrap a CA for cluster tls run etcd run k8s (apiserver,etc,kubelets) run your application run the traefik ingress controller To run a service behind something like traefik on nomad you would: bootstrap a CA for cluster tls run consul run nomad (servers, clients) run your application run traefik somehow I think the only thing that is really more complicated in k8s is the networking stuff, but that's only because it has more features, like having cluster dns automatically configured and being able to give every pod its own ip address which means every service can bind to port 80 without conflicts, and policy enforcement.
- akvadrako 9y agoHosted k8s on google can be as little as $50/month. If your project has really little load, I think serverless with docker containers is the way to go, which might be under $1/month. However that's not supported yet by Google or AWS, so you have to find another provider.
- merb 9y agowhat if you can't use hosted k8s. what if you are on vmware 3 node (no vsphere). what if you at least need consul/etcd, postgresql, haproxy + two nodes (should be as highly available as possible). how do you keep everything up to date, how do you deploy (no downtime) without a complex ansible config. for myself I use k8s either bootkubed or kubeadm, I create ignition configs via cloud-config (coreos) and create coreos nodes via the vmware gui. after that I ssh once into either one node (bootkube) or in all nodes and call kubeadm or bootkube after that I have a working cluster. Maintance is done via coreos update operator. and I can upgrade k8s via either kubeadm or via kubectl on bootkube. (so basically the most complex thing on k8s is updating k8s, everything else isn't as hard as people thing. p.s. I do not have a bach/major in tech).
- pm90 9y agoWhat did you find complex about updating k8s? Serious question since I work closely with k8s and they have a pretty fast release cycle.
- merb 9y agorunning into bugs and that I need to ssh into the nodes ;) (serious answer) I can basically automate nearly everything, os updates (coreos), deployment updates (ci/cd), however updating etcd or k8s is a manual operation on bare metal, either I'm on bootkube than I need to login and update kubectl, than I need to update all daemonsets/deployments in kube-system (it's not always easy and bootkube has strange bugs, https://github.com/kubernetes-incubator/bootkube/issues/977 https://github.com/kubernetes-incubator/bootkube/issues/977) So one should actually use kubeadm, once again I still need to update kubelet, so I need to ssh into the machine (anyway) I also need to update kubeadm and run it, again a manual operation. *ssh means I need to do it on my own, for small clusters this can be done manually for bigger clusters it can be scripted via ansible or even better create an own software that does it. My point is, that's the only thing which is not "just works" on k8s (yet). Edit: I also run into a bug that I could not upgrade k8s with kubeadm (but thats gonna be fixed soon: https://github.com/kubernetes/kubeadm/issues/727 https://github.com/kubernetes/kubeadm/issues/727) Edit2: the good thing is that reloading kubelet won't kill anything.
- nathan_f77 9y agoIf you're on AWS, I highly recommend Convox [1]. They use AWS ECS instead of k8s, but EKS support will be coming soon. Their rack project is free and open-source [2]. They also have a hosted console that can manage your racks, deploy from GitHub webhooks, and send slack notifications. But you can use convox without that. [1] https://convox.com https://convox.com [2] https://github.com/convox/rack https://github.com/convox/rack
- jon-wood 9y agoAt less than 25 machines throw it on Heroku. Yes, it’ll be more expensive hardware wise, but you’ll probably save that much in not operating you’re own servers.