4 ms·

... not to be disrespectful but this seems to contain quite a high amount of vendor specific implementations. May very well be because I don't know the solution

... not to be disrespectful but this seems to contain quite a high amount of vendor specific implementations. May very well be because I don't know the solution in detail, just seems like it to me. In the end you are bound to AWS and have to reinvent many things when moving to another cloud provider. How long did it take to set this up? > AWS autoscaling groups + cloudwatch for adding and removing machines Does this also work for multiple applications on one host? > means that I still end up with fewer hosts if I just run each component I don't know how your environments looks like but we used one VM for every environment. One environment needs about 20 GB max so we have to use 32 GB RAM VMs and waste quite some resources. With k8s I have two beefy 64 GB nodes that host 6 environments. This also speeds up the execution as 70% of time the environments have a low load and the beefy nodes have more CPU cores available than smaller VMs. Most of the time we can also throw some Jenkins and other test jobs on the cluster for free (low priority deployments). > Overhead Yes there is definitely some overhead. 2 GB RAM for the master and 700 MB RAM on every node. We choose larger nodes (8 CPUs and 64 GB RAM) so the overhead is not that great in comparison. We do gain savings from k8s (about 15 to 30 %). > k8s magic You are right, there is no magic sauce. k8s just packs a nice package of things I otherwise need to do externally. Sure the external logic worked well in the past, but now I get many features for a rather low price without committing my company to a provider to hard.

TheDong 5y ago
> seems to contain quite a high amount of vendor specific implementations The "vendor specific implementation" is the loadbalancer and the autoscaling group. If you want to switch to baremetal servers, or some other cloud, getting a loadbalancer setup with similar properties is easy. Every major offering (GCE, Azure, HAProxy on baremetal, etc) will handle healthchecks and dynamically adding/removing servers. Autoscaling servers based on metrics is admittedly more provider specific, but kubernetes doesn't solve that either. If I have k8s on 5 bare metal hosts, there's no cloud-agnostic way for k8s to magically launch a 6th host. You need to solve the same problem in either case. The k8s cloud autoscaler exists, and so do similar features for most clouds. Kubernetes feels like even more lock-in than I have. If you were unlucky enough to run your services on docker swarm, mesos, triton, or various other container-running abstractions, then you'll have had to deal with a painful migration to kubernetes (since each of those technologies clearly 'lost'). Moving off kubernetes to another abstraction that runs containers (like mesos or such) is much more painful than moving from an AWS load balancer to a GCE load balancer or a bare-metal L5 load balancer. In that sense, I feel like I have less lock-in than the typical K8s setup. > [the rest of your comment] It sounds like in your situation, k8s is working decently well for you. Congrats. My point is not that k8s is always bad or using plain VMs + a load balancer is inherently superior, but that neither is clearly better, and both have benefits in different situations. In yours, k8s seems fine. In some other situations, k8s is a net negative.

TheDong 5y ago

> seems to contain quite a high amount of vendor specific implementations The "vendor specific implementation" is the loadbalancer and the autoscaling group. If you want to switch to baremetal servers, or some other cloud, getting a loadbalancer setup with similar properties is easy. Every major offering (GCE, Azure, HAProxy on baremetal, etc) will handle healthchecks and dynamically adding/removing servers. Autoscaling servers based on metrics is admittedly more provider specific, but kubernetes doesn't solve that either. If I have k8s on 5 bare metal hosts, there's no cloud-agnostic way for k8s to magically launch a 6th host. You need to solve the same problem in either case. The k8s cloud autoscaler exists, and so do similar features for most clouds. Kubernetes feels like even more lock-in than I have. If you were unlucky enough to run your services on docker swarm, mesos, triton, or various other container-running abstractions, then you'll have had to deal with a painful migration to kubernetes (since each of those technologies clearly 'lost'). Moving off kubernetes to another abstraction that runs containers (like mesos or such) is much more painful than moving from an AWS load balancer to a GCE load balancer or a bare-metal L5 load balancer. In that sense, I feel like I have less lock-in than the typical K8s setup. > [the rest of your comment] It sounds like in your situation, k8s is working decently well for you. Congrats. My point is not that k8s is always bad or using plain VMs + a load balancer is inherently superior, but that neither is clearly better, and both have benefits in different situations. In yours, k8s seems fine. In some other situations, k8s is a net negative.