9 ms·

The idea with Kubernetes isn't necessarily that it can do resource allocation more carefully and more intelligently than a human ops person, but it can do so at

The idea with Kubernetes isn't necessarily that it can do resource allocation more carefully and more intelligently than a human ops person, but it can do so at a level of automation and speed that a human can't. Pods die, Kubernetes spawns new ones. Server goes full, Kubernetes spawns new node. Server runs out of disk space, Kubernetes evicts nodes until everything is good again. Server goes down, Kubernetes shuffles all the pods over to a different node. And so on. It does this at a scale of hundreds or thousands of machines. The utility it brings also applies to small clusters. You don't remove ops, you just flatten it into Kubernetes' operational model. Equally importantly, Kubernetes also brings a declarative deployment model. You have the same with Systemd, but at the node level. Kubernetes is a bit like Systemd at the cluster level; its "units" are pods declared in YAML/JSON files, but unlike Systemd they run on any node the cluster, and are subject to a resource allocation model that includes things like persistent storage (e.g. AWS EBS volumes). You don't get anything close to that with Systemd, which is much more analogous to Docker. I haven't heard anyone ever claim that Docker brings reproducible builds -- why would anyone assume that? What Docker give you is reproducible deploys; build once, deploy many times from the exact same image. Maybe that's what you're thinking of?

erulabs 10y ago
I don't dislike Kubernetes, in fact im a (very minor) contributor and regular user and am a member of the federated kubernetes workgroup. A small group of microservices very happily lives on Kube in production. But, for the sake of the argument: > Pods die, Kubernetes spawns new ones. AWS ASGs also solve for this, as does every other cloud provider at the infrastructure level. For applications, systemd and any init system can restart an application when it crashes... > Server goes full, Kubernetes spawns new node. I assuming you mean utilization - again, ASGs/CloudWatch solve for this as do every other cloud provider (if they didn't a few minutes worth of Sensu plugins and good-to-go). > Server runs out of disk space, Kubernetes evicts nodes until everything is good again Right but this can and has lead to disaster. I got paged for bad response time and found out the issue was that containers were being evicted and pods were going critical because _all_ hosts were filling up on diskspace and there were no healthy nodes on which to put pods. This due to one bad service writing out tons and tons of log data). Had that been dedicated, I would have just gotten paged for one full disk and fixed the problem when it showed up instead of waiting for a bigger issue. This is super fixable in Kube (and was - that service now has its own volume and still lives in kube) - but my point is that it's a problem for everyone. > Server goes down, Kubernetes shuffles all the pods over to a different node. This effects response time of user facing applications as new 1GiB+ containers are shuffled around (I know, thats an unfair example). ASGs and Cloudwatch solve for this in a saner / safer way. As for the reproducible builds, it was in response to the article which says: > The ‘repeatable builds’ argument is the most interesting to me, in part because this is the same promise that java made in the 90s (write once run anywhere). You may be correct that Docker doesn't promise this explicitly, but it is a big misconception. Additionally, `tar` solves for "reproducible deploys"... I'm not an AWS employee, just figured I'd use a common cloud as an example.
- moderation 10y ago
  Your responses assume that I'm running in the cloud and AWS specifically. What if I need to run on-premise?
  - erulabs 10y ago
    Kube is actually fantastic for on-prem/dedicated/colo. I still wind up needing to manage my _non_ Kube stuff - and fall back to Ansible + systemd for the data layer for example. Additionally, one big complexity re: Kube on dedicated is dealing with custom hardware. Kube and Docker really require modern and specialized kernels - This actually applies to AWS as well; the current Kube AMI for AWS doesn't install the `linux-aws` kernel for example, so you have no access to the ENA driver or the ixgbefv driver, and the t* and i3 instance performance is awful (the most common and the most vital machines respectively). There is a PR for that open somewhere and I'm sure it will be fixed soon, but you see my point. After the hardware/kernel build (which you'll need to manage outside of Kube, so you'll probably choose something like Ansible), you'll probably want auth management at the host level (assuming you run on-prem), which you'll most likely fall back to Ansible for again. I should probably also admit somewhere in this thread I actually use and prefer Chef :)
- perlpimp 10y ago
  tar does reproduce a deploy but onto c clean slate machine. docker is just a clean space where you can take an issue from there and easier reproduce elsewhere because there is no state left over from installations of previous packages and/or other software.
- amazingman 10y ago
  What makes containers in general, and Kubernetes specifically, compelling to me are the following: - the deployment artifact is a "fresh" instance that's "booted" in a fraction of the time—as in a couple of seconds vs a few minutes—than a new EC2 instance would be - since the deployment artifact is a fresh instance, there is no need to manipulate an existing instance already running your application as part of your deploy process (e.g. scp, ssh, unpack, run commands, restart/reload application) - if you design your build pipeline appropriately, the actual data needed to be shipped is effectively just a diff from the previous image. In my org, that could mean the difference between shipping a 300MB vehicle to ~300 instances and shipping a 300KB vehicle to almost certainly less than 300 Kubernetes nodes There are other things as well, but I wanted to stay roughly within the parameters of your examples. If your organization doesn't already have a continuous integration story, I'd say it probably isn't ready for containers in production. If your cloud footprint isn't costing you let's say >= $50k/mo or so, I might argue that the engineering effort to Kubernetize your things is probably better spent in more direct ways on your product. If you do have a CI/CD story, and you do have a very wide, expansive, and expensive cloud footprint, containers and Kubernetes are extremely compelling possible solutions.
hueving 10y ago
Doesn't auto rolling evacuation like that give the ability for one malicious user that found a flaw in an app to DoS an entire kubernetes cluster by filling disk space?

erulabs 10y ago

I don't dislike Kubernetes, in fact im a (very minor) contributor and regular user and am a member of the federated kubernetes workgroup. A small group of microservices very happily lives on Kube in production. But, for the sake of the argument: > Pods die, Kubernetes spawns new ones. AWS ASGs also solve for this, as does every other cloud provider at the infrastructure level. For applications, systemd and any init system can restart an application when it crashes... > Server goes full, Kubernetes spawns new node. I assuming you mean utilization - again, ASGs/CloudWatch solve for this as do every other cloud provider (if they didn't a few minutes worth of Sensu plugins and good-to-go). > Server runs out of disk space, Kubernetes evicts nodes until everything is good again Right but this can and has lead to disaster. I got paged for bad response time and found out the issue was that containers were being evicted and pods were going critical because _all_ hosts were filling up on diskspace and there were no healthy nodes on which to put pods. This due to one bad service writing out tons and tons of log data). Had that been dedicated, I would have just gotten paged for one full disk and fixed the problem when it showed up instead of waiting for a bigger issue. This is super fixable in Kube (and was - that service now has its own volume and still lives in kube) - but my point is that it's a problem for everyone. > Server goes down, Kubernetes shuffles all the pods over to a different node. This effects response time of user facing applications as new 1GiB+ containers are shuffled around (I know, thats an unfair example). ASGs and Cloudwatch solve for this in a saner / safer way. As for the reproducible builds, it was in response to the article which says: > The ‘repeatable builds’ argument is the most interesting to me, in part because this is the same promise that java made in the 90s (write once run anywhere). You may be correct that Docker doesn't promise this explicitly, but it is a big misconception. Additionally, `tar` solves for "reproducible deploys"... I'm not an AWS employee, just figured I'd use a common cloud as an example.

moderation 10y ago
Your responses assume that I'm running in the cloud and AWS specifically. What if I need to run on-premise?
- erulabs 10y ago
  Kube is actually fantastic for on-prem/dedicated/colo. I still wind up needing to manage my _non_ Kube stuff - and fall back to Ansible + systemd for the data layer for example. Additionally, one big complexity re: Kube on dedicated is dealing with custom hardware. Kube and Docker really require modern and specialized kernels - This actually applies to AWS as well; the current Kube AMI for AWS doesn't install the `linux-aws` kernel for example, so you have no access to the ENA driver or the ixgbefv driver, and the t* and i3 instance performance is awful (the most common and the most vital machines respectively). There is a PR for that open somewhere and I'm sure it will be fixed soon, but you see my point. After the hardware/kernel build (which you'll need to manage outside of Kube, so you'll probably choose something like Ansible), you'll probably want auth management at the host level (assuming you run on-prem), which you'll most likely fall back to Ansible for again. I should probably also admit somewhere in this thread I actually use and prefer Chef :)
perlpimp 10y ago
tar does reproduce a deploy but onto c clean slate machine. docker is just a clean space where you can take an issue from there and easier reproduce elsewhere because there is no state left over from installations of previous packages and/or other software.
amazingman 10y ago
What makes containers in general, and Kubernetes specifically, compelling to me are the following: - the deployment artifact is a "fresh" instance that's "booted" in a fraction of the time—as in a couple of seconds vs a few minutes—than a new EC2 instance would be - since the deployment artifact is a fresh instance, there is no need to manipulate an existing instance already running your application as part of your deploy process (e.g. scp, ssh, unpack, run commands, restart/reload application) - if you design your build pipeline appropriately, the actual data needed to be shipped is effectively just a diff from the previous image. In my org, that could mean the difference between shipping a 300MB vehicle to ~300 instances and shipping a 300KB vehicle to almost certainly less than 300 Kubernetes nodes There are other things as well, but I wanted to stay roughly within the parameters of your examples. If your organization doesn't already have a continuous integration story, I'd say it probably isn't ready for containers in production. If your cloud footprint isn't costing you let's say >= $50k/mo or so, I might argue that the engineering effort to Kubernetize your things is probably better spent in more direct ways on your product. If you do have a CI/CD story, and you do have a very wide, expansive, and expensive cloud footprint, containers and Kubernetes are extremely compelling possible solutions.

hueving 10y ago

Doesn't auto rolling evacuation like that give the ability for one malicious user that found a flaw in an app to DoS an entire kubernetes cluster by filling disk space?