4 ms·

> We'd just snapshot AMIs to keep OS dependencies fixed This is a good solution, but I would not call it easier. Using docker container feels like installing

> We'd just snapshot AMIs to keep OS dependencies fixed This is a good solution, but I would not call it easier. Using docker container feels like installing an app on my smartphone. I choose the version and it will always work like I build it at date x without an additional system. Works for every programming language with every dependency out of the box. Python, Java, Javascript, GO, Ocaml, C, ... > How containers are deployed varies wildly between prod and the local machine I just brought a product of my company to Kubernetes. Run helm upgrade --install . -f dev-values.yaml for dev Run helm upgrade --install . -f prod-values.yaml for prod (of course you need the secrets there. Jenkins has them). My laptop does run an environment with all components of the prod env. Something like email and sap services are of course mocked, but everything else? All on my machine. Why not? I can spin up a new test environment for customers with new settings on the same day. > Both your apps now have different "ubuntu" layers We use a base image that does change not that often. Even if: no problem, the registry is connected via 1000 MBit/s and zero-downtime deployment does its magic so I don't even notice if it takes one or two minutes. Another thing: my node (or VM) libs and the libs of my software should not be connected in any way (at least for me). I want to patch my nodes and my software independently. Different software should also not be bound to libs of another software. > All the items you listed under k8s are things I had before it - How do you easily scale up? Including starting new machines and spinning down machines that are no longer needed - How are multiple software parts executed on one host? - How do you do fail over? I know that everything can be done without Kubernetes. With enough time and money one can create large systems that do this. I spun up a new Kubernetes cluster and ported our product (already containerized) on the cluster in about three months. Really: I also love the classic dev ops and have a proxmox server at home, but Kubernetes just solves many problems at once in a short time.

TheDong 5y ago
> How do you easily scale up? Including starting new machines and spinning down machines that are no longer needed AWS autoscaling groups + cloudwatch for adding and removing machines + checking them into load balancers is something that has worked for longer than K8s has been a thing. > How are multiple software parts executed on one host? systemd units, or for more resource hungry things, multiple autoscaling groups. The overhead of running the kubelet on each host + etcd cluster + apiserver means that I still end up with fewer hosts if I just run each component on every single host vs scaling different deployments independently. It is true that kubernetes might be more resource efficient in some combination of nodes and software, but at under 10 servers, I've always found the overhead of the etcd cluster + apiserver + kubelet to dwarf any savings from not just running 10 copies of my software. > How do you do fail over? The AWS-managed load balancer can fail over based on health checks failing, metrics, or I can add/remove servers from it manually. You can also do DNS health checks, or add a layer of haproxy/nginx/whatever if you want. It's not like k8s has some magic ability to fail over under the hood. It's just using k8s service objects (probably LoadBalancer type), which does the same thing.
- secondcoming 5y ago
  Correct. We use the pretty much the same setup on GCP. All scaling is automatic. When we deploy new code we just run a jenkins job that creates an image from custom debian packages. Push that to GCP and it rolls it out automatically to all our DCs.
- mixermachine 5y ago
  ... not to be disrespectful but this seems to contain quite a high amount of vendor specific implementations. May very well be because I don't know the solution in detail, just seems like it to me. In the end you are bound to AWS and have to reinvent many things when moving to another cloud provider. How long did it take to set this up? > AWS autoscaling groups + cloudwatch for adding and removing machines Does this also work for multiple applications on one host? > means that I still end up with fewer hosts if I just run each component I don't know how your environments looks like but we used one VM for every environment. One environment needs about 20 GB max so we have to use 32 GB RAM VMs and waste quite some resources. With k8s I have two beefy 64 GB nodes that host 6 environments. This also speeds up the execution as 70% of time the environments have a low load and the beefy nodes have more CPU cores available than smaller VMs. Most of the time we can also throw some Jenkins and other test jobs on the cluster for free (low priority deployments). > Overhead Yes there is definitely some overhead. 2 GB RAM for the master and 700 MB RAM on every node. We choose larger nodes (8 CPUs and 64 GB RAM) so the overhead is not that great in comparison. We do gain savings from k8s (about 15 to 30 %). > k8s magic You are right, there is no magic sauce. k8s just packs a nice package of things I otherwise need to do externally. Sure the external logic worked well in the past, but now I get many features for a rather low price without committing my company to a provider to hard.
  - TheDong 5y ago
    > seems to contain quite a high amount of vendor specific implementations The "vendor specific implementation" is the loadbalancer and the autoscaling group. If you want to switch to baremetal servers, or some other cloud, getting a loadbalancer setup with similar properties is easy. Every major offering (GCE, Azure, HAProxy on baremetal, etc) will handle healthchecks and dynamically adding/removing servers. Autoscaling servers based on metrics is admittedly more provider specific, but kubernetes doesn't solve that either. If I have k8s on 5 bare metal hosts, there's no cloud-agnostic way for k8s to magically launch a 6th host. You need to solve the same problem in either case. The k8s cloud autoscaler exists, and so do similar features for most clouds. Kubernetes feels like even more lock-in than I have. If you were unlucky enough to run your services on docker swarm, mesos, triton, or various other container-running abstractions, then you'll have had to deal with a painful migration to kubernetes (since each of those technologies clearly 'lost'). Moving off kubernetes to another abstraction that runs containers (like mesos or such) is much more painful than moving from an AWS load balancer to a GCE load balancer or a bare-metal L5 load balancer. In that sense, I feel like I have less lock-in than the typical K8s setup. > [the rest of your comment] It sounds like in your situation, k8s is working decently well for you. Congrats. My point is not that k8s is always bad or using plain VMs + a load balancer is inherently superior, but that neither is clearly better, and both have benefits in different situations. In yours, k8s seems fine. In some other situations, k8s is a net negative.

TheDong 5y ago

> How do you easily scale up? Including starting new machines and spinning down machines that are no longer needed AWS autoscaling groups + cloudwatch for adding and removing machines + checking them into load balancers is something that has worked for longer than K8s has been a thing. > How are multiple software parts executed on one host? systemd units, or for more resource hungry things, multiple autoscaling groups. The overhead of running the kubelet on each host + etcd cluster + apiserver means that I still end up with fewer hosts if I just run each component on every single host vs scaling different deployments independently. It is true that kubernetes might be more resource efficient in some combination of nodes and software, but at under 10 servers, I've always found the overhead of the etcd cluster + apiserver + kubelet to dwarf any savings from not just running 10 copies of my software. > How do you do fail over? The AWS-managed load balancer can fail over based on health checks failing, metrics, or I can add/remove servers from it manually. You can also do DNS health checks, or add a layer of haproxy/nginx/whatever if you want. It's not like k8s has some magic ability to fail over under the hood. It's just using k8s service objects (probably LoadBalancer type), which does the same thing.

secondcoming 5y ago
Correct. We use the pretty much the same setup on GCP. All scaling is automatic. When we deploy new code we just run a jenkins job that creates an image from custom debian packages. Push that to GCP and it rolls it out automatically to all our DCs.
mixermachine 5y ago
... not to be disrespectful but this seems to contain quite a high amount of vendor specific implementations. May very well be because I don't know the solution in detail, just seems like it to me. In the end you are bound to AWS and have to reinvent many things when moving to another cloud provider. How long did it take to set this up? > AWS autoscaling groups + cloudwatch for adding and removing machines Does this also work for multiple applications on one host? > means that I still end up with fewer hosts if I just run each component I don't know how your environments looks like but we used one VM for every environment. One environment needs about 20 GB max so we have to use 32 GB RAM VMs and waste quite some resources. With k8s I have two beefy 64 GB nodes that host 6 environments. This also speeds up the execution as 70% of time the environments have a low load and the beefy nodes have more CPU cores available than smaller VMs. Most of the time we can also throw some Jenkins and other test jobs on the cluster for free (low priority deployments). > Overhead Yes there is definitely some overhead. 2 GB RAM for the master and 700 MB RAM on every node. We choose larger nodes (8 CPUs and 64 GB RAM) so the overhead is not that great in comparison. We do gain savings from k8s (about 15 to 30 %). > k8s magic You are right, there is no magic sauce. k8s just packs a nice package of things I otherwise need to do externally. Sure the external logic worked well in the past, but now I get many features for a rather low price without committing my company to a provider to hard.
- TheDong 5y ago
  > seems to contain quite a high amount of vendor specific implementations The "vendor specific implementation" is the loadbalancer and the autoscaling group. If you want to switch to baremetal servers, or some other cloud, getting a loadbalancer setup with similar properties is easy. Every major offering (GCE, Azure, HAProxy on baremetal, etc) will handle healthchecks and dynamically adding/removing servers. Autoscaling servers based on metrics is admittedly more provider specific, but kubernetes doesn't solve that either. If I have k8s on 5 bare metal hosts, there's no cloud-agnostic way for k8s to magically launch a 6th host. You need to solve the same problem in either case. The k8s cloud autoscaler exists, and so do similar features for most clouds. Kubernetes feels like even more lock-in than I have. If you were unlucky enough to run your services on docker swarm, mesos, triton, or various other container-running abstractions, then you'll have had to deal with a painful migration to kubernetes (since each of those technologies clearly 'lost'). Moving off kubernetes to another abstraction that runs containers (like mesos or such) is much more painful than moving from an AWS load balancer to a GCE load balancer or a bare-metal L5 load balancer. In that sense, I feel like I have less lock-in than the typical K8s setup. > [the rest of your comment] It sounds like in your situation, k8s is working decently well for you. Congrats. My point is not that k8s is always bad or using plain VMs + a load balancer is inherently superior, but that neither is clearly better, and both have benefits in different situations. In yours, k8s seems fine. In some other situations, k8s is a net negative.