8 ms·

I've been thinking about setting up a small Kubernetes cluster for hosting some smaller client projects (read websites, shopping carts, API's, admin panels). M

I've been thinking about setting up a small Kubernetes cluster for hosting some smaller client projects (read websites, shopping carts, API's, admin panels). My current setup uses a couple of Hetzner dedicated machines and services are deployed with ansible playbooks. The playbooks install and configure nginx, install the right version of ruby/java/php/postgres, configure and start systemd services. These playbooks end up copied and slightly modified for each project, and sometimes they interfere with one another in sublte ways (different ruby versions, conflicting ports, etc) With my future Kubernetes setup I would just package each project into its own self-contained container image, write a kubernetes deployment/pod spec, update the ingress and I'm done.

grillorafael 8y ago
I recommend having a look at DCOS
- p_l 8y ago
  I recently started working with DCOS, and so far it seems that while it might have been more mature and ready platform 2-3 years ago, today I have to deal with issues I don't have to care about on K8s
  - grillorafael 8y ago
    Curious to know what. We run a DCOS cluster with 40+ machines and we have to deal with pretty much nothing
hardwaresofton 8y ago
If you can find the time to get up to speed on Kubernetes, I would say do it. I actually have a weirdly similar setup to you (I run on Hetzner and used and still use ansible), and I've written about it, most recently when I switched my single node cluster to Ubuntu 18.04 [0]. In the past I've also run a single node kubernetes clusters on CoreOS Container Linux, Arch, and back to CoreOS Container Linux in that order, from versions 1.7~1.11. [0]: https://vadosware.io/post/hetzner-fresh-ubuntu-install-to-single-node-kubernetes-cluster-with-ansible/ https://vadosware.io/post/hetzner-fresh-ubuntu-install-to-si...
- vbsteven 8y ago
  Thanks, I'll check it out. I have quite some experience working with Kubernetes clusters for my larger clients. Usually for clients that are big enough to have their own AWS account. The thing I am still on the fence about is whether I should go for a DIY Kubernetes setup on one or more Hetzner dedicated machines (cheap, more work, less scalable) or if I should just shell out for AWS and run an easily scalable cluster with Kops (which is what I use for some clients) and take advantage of all the AWS goodies like load balancing and RDS.
  - hardwaresofton 8y ago
    Well I think that's more of a cost question -- AWS can get expensive pretty quickly. Three t2.micros (one coordinator, 2 nodes) are absolutely pitiful in terms of processing power but that's already ~$30/month when you could get a way beefier machine on Hetzner whether dedicated or cloud (also Scaleway[0]). I'm a fan of Hetzner because I think their cheap dedicated machines are worth the operational costs for me, and the issues with upkeep I'll face are good for me because that knowledge has value. Also, I want to note that if you actually start subscribing to the immutable infrastructure movement that's going on right now, once you look past all the buzzwords it's a fantastic way to run servers stress free -- as long as your data is backed up/properly managed, just shoot servers left and right, spend a lot of time to get them into the right state ONCE, and never worry about it again. You can even use terraform to provision hetzner. Again, this kind of thinking and the related tooling is catchy/popular right now because it's useful at larger scales, but it can also free you of a lot of worry at lower scale. For example I have a post on using Container Linux to go from brand new machine to single node k8s cluster, with one file. To be honest though, setting up a Hetzner dedicated machine is very very easy -- they've got great utilities. You could even go with Hetzner Cloud and things will be more managed. I would say go with AWS if you want to experiment with AWS technology as well -- and want to use their value added services. If you run kubernetes on a dedicated machine on hetzner you're definitely not going to get the rest of that, of course. BTW, kubeadm is better/less complex than kops -- It's almost impossible to fuck it up, but there are subtle things in kops due to the AWS integration that make things a little things ever so slightly more difficult. [0]: https://www.scaleway.com/pricing/ https://www.scaleway.com/pricing/ [1]: https://vadosware.io/post/k8s-container-linux-ignition-with-rkt-and-kube-router https://vadosware.io/post/k8s-container-linux-ignition-with-...
matthewmacleod 8y ago
This is the sort of thing that I think a containerised deployment platform can really help with, and I'm doing much the same – specifically because it completely avoids interactions between applications' environments. But I would recommend looking at the Hashicorp stack as a possible alternative, which might be entirely suitable for your use-case without the complexity of Kubernetes. This involves running Nomad and Consul to provide cluster scheduling and service discovery respectively - these are both single binaries with minimal configuration. Then you'd need some kind of front-end load-balancer like nginx or traefik which uses Consul to decide where to route requests. It doesn't cover all the use-cases and features that Kubernetes does, but it does have the benefit of being much more straightforward to work with, so definitely worth considering!
erikb 8y ago
Your current setup is probably already quite stable. Why switch to another? You probably underestimate the learning effort and the required effort to keep your stuff running in a long-term maintainable way on k8s.
malgregator 8y ago
we are using similar setup and ended up using ansible for installing and setuping kubernetes cluster itself together with some other configurations like ipsec between the nodes etc. Learning curve for a new employee is small as every internal project is using the same directory structure and is completely gitops. Actual services are then deployed to the kubernetes and it works great.
mockingbirdy 8y ago
I will just link to my recommendation of CaptainDuckDuck: https://news.ycombinator.com/item?id=18128575 https://news.ycombinator.com/item?id=18128575
sciurus 8y ago
One thing to consider is packaging projects as container images, but continuing to deploy them via ansible. That could solve some of your problems without requiring the time investment and complexity of running kubernetes.

grillorafael 8y ago

I recommend having a look at DCOS

p_l 8y ago
I recently started working with DCOS, and so far it seems that while it might have been more mature and ready platform 2-3 years ago, today I have to deal with issues I don't have to care about on K8s
- grillorafael 8y ago
  Curious to know what. We run a DCOS cluster with 40+ machines and we have to deal with pretty much nothing

hardwaresofton 8y ago

If you can find the time to get up to speed on Kubernetes, I would say do it. I actually have a weirdly similar setup to you (I run on Hetzner and used and still use ansible), and I've written about it, most recently when I switched my single node cluster to Ubuntu 18.04 [0]. In the past I've also run a single node kubernetes clusters on CoreOS Container Linux, Arch, and back to CoreOS Container Linux in that order, from versions 1.7~1.11. [0]: https://vadosware.io/post/hetzner-fresh-ubuntu-install-to-single-node-kubernetes-cluster-with-ansible/ https://vadosware.io/post/hetzner-fresh-ubuntu-install-to-si...

vbsteven 8y ago
Thanks, I'll check it out. I have quite some experience working with Kubernetes clusters for my larger clients. Usually for clients that are big enough to have their own AWS account. The thing I am still on the fence about is whether I should go for a DIY Kubernetes setup on one or more Hetzner dedicated machines (cheap, more work, less scalable) or if I should just shell out for AWS and run an easily scalable cluster with Kops (which is what I use for some clients) and take advantage of all the AWS goodies like load balancing and RDS.
- hardwaresofton 8y ago
  Well I think that's more of a cost question -- AWS can get expensive pretty quickly. Three t2.micros (one coordinator, 2 nodes) are absolutely pitiful in terms of processing power but that's already ~$30/month when you could get a way beefier machine on Hetzner whether dedicated or cloud (also Scaleway[0]). I'm a fan of Hetzner because I think their cheap dedicated machines are worth the operational costs for me, and the issues with upkeep I'll face are good for me because that knowledge has value. Also, I want to note that if you actually start subscribing to the immutable infrastructure movement that's going on right now, once you look past all the buzzwords it's a fantastic way to run servers stress free -- as long as your data is backed up/properly managed, just shoot servers left and right, spend a lot of time to get them into the right state ONCE, and never worry about it again. You can even use terraform to provision hetzner. Again, this kind of thinking and the related tooling is catchy/popular right now because it's useful at larger scales, but it can also free you of a lot of worry at lower scale. For example I have a post on using Container Linux to go from brand new machine to single node k8s cluster, with one file. To be honest though, setting up a Hetzner dedicated machine is very very easy -- they've got great utilities. You could even go with Hetzner Cloud and things will be more managed. I would say go with AWS if you want to experiment with AWS technology as well -- and want to use their value added services. If you run kubernetes on a dedicated machine on hetzner you're definitely not going to get the rest of that, of course. BTW, kubeadm is better/less complex than kops -- It's almost impossible to fuck it up, but there are subtle things in kops due to the AWS integration that make things a little things ever so slightly more difficult. [0]: https://www.scaleway.com/pricing/ https://www.scaleway.com/pricing/ [1]: https://vadosware.io/post/k8s-container-linux-ignition-with-rkt-and-kube-router https://vadosware.io/post/k8s-container-linux-ignition-with-...

matthewmacleod 8y ago

This is the sort of thing that I think a containerised deployment platform can really help with, and I'm doing much the same – specifically because it completely avoids interactions between applications' environments. But I would recommend looking at the Hashicorp stack as a possible alternative, which might be entirely suitable for your use-case without the complexity of Kubernetes. This involves running Nomad and Consul to provide cluster scheduling and service discovery respectively - these are both single binaries with minimal configuration. Then you'd need some kind of front-end load-balancer like nginx or traefik which uses Consul to decide where to route requests. It doesn't cover all the use-cases and features that Kubernetes does, but it does have the benefit of being much more straightforward to work with, so definitely worth considering!

erikb 8y ago

Your current setup is probably already quite stable. Why switch to another? You probably underestimate the learning effort and the required effort to keep your stuff running in a long-term maintainable way on k8s.

malgregator 8y ago

we are using similar setup and ended up using ansible for installing and setuping kubernetes cluster itself together with some other configurations like ipsec between the nodes etc. Learning curve for a new employee is small as every internal project is using the same directory structure and is completely gitops. Actual services are then deployed to the kubernetes and it works great.

mockingbirdy 8y ago

I will just link to my recommendation of CaptainDuckDuck: https://news.ycombinator.com/item?id=18128575 https://news.ycombinator.com/item?id=18128575

sciurus 8y ago

One thing to consider is packaging projects as container images, but continuing to deploy them via ansible. That could solve some of your problems without requiring the time investment and complexity of running kubernetes.