6 ms·

I don't dislike Kubernetes, in fact im a (very minor) contributor and regular user and am a member of the federated kubernetes workgroup. A small group of micro

by erulabs 10y ago

I don't dislike Kubernetes, in fact im a (very minor) contributor and regular user and am a member of the federated kubernetes workgroup. A small group of microservices very happily lives on Kube in production. But, for the sake of the argument: > Pods die, Kubernetes spawns new ones. AWS ASGs also solve for this, as does every other cloud provider at the infrastructure level. For applications, systemd and any init system can restart an application when it crashes... > Server goes full, Kubernetes spawns new node. I assuming you mean utilization - again, ASGs/CloudWatch solve for this as do every other cloud provider (if they didn't a few minutes worth of Sensu plugins and good-to-go). > Server runs out of disk space, Kubernetes evicts nodes until everything is good again Right but this can and has lead to disaster. I got paged for bad response time and found out the issue was that containers were being evicted and pods were going critical because _all_ hosts were filling up on diskspace and there were no healthy nodes on which to put pods. This due to one bad service writing out tons and tons of log data). Had that been dedicated, I would have just gotten paged for one full disk and fixed the problem when it showed up instead of waiting for a bigger issue. This is super fixable in Kube (and was - that service now has its own volume and still lives in kube) - but my point is that it's a problem for everyone. > Server goes down, Kubernetes shuffles all the pods over to a different node. This effects response time of user facing applications as new 1GiB+ containers are shuffled around (I know, thats an unfair example). ASGs and Cloudwatch solve for this in a saner / safer way. As for the reproducible builds, it was in response to the article which says: > The ‘repeatable builds’ argument is the most interesting to me, in part because this is the same promise that java made in the 90s (write once run anywhere). You may be correct that Docker doesn't promise this explicitly, but it is a big misconception. Additionally, `tar` solves for "reproducible deploys"... I'm not an AWS employee, just figured I'd use a common cloud as an example.

moderation 10y ago
Your responses assume that I'm running in the cloud and AWS specifically. What if I need to run on-premise?
- erulabs 10y ago
  Kube is actually fantastic for on-prem/dedicated/colo. I still wind up needing to manage my _non_ Kube stuff - and fall back to Ansible + systemd for the data layer for example. Additionally, one big complexity re: Kube on dedicated is dealing with custom hardware. Kube and Docker really require modern and specialized kernels - This actually applies to AWS as well; the current Kube AMI for AWS doesn't install the `linux-aws` kernel for example, so you have no access to the ENA driver or the ixgbefv driver, and the t* and i3 instance performance is awful (the most common and the most vital machines respectively). There is a PR for that open somewhere and I'm sure it will be fixed soon, but you see my point. After the hardware/kernel build (which you'll need to manage outside of Kube, so you'll probably choose something like Ansible), you'll probably want auth management at the host level (assuming you run on-prem), which you'll most likely fall back to Ansible for again. I should probably also admit somewhere in this thread I actually use and prefer Chef :)
perlpimp 10y ago
tar does reproduce a deploy but onto c clean slate machine. docker is just a clean space where you can take an issue from there and easier reproduce elsewhere because there is no state left over from installations of previous packages and/or other software.
amazingman 10y ago
What makes containers in general, and Kubernetes specifically, compelling to me are the following: - the deployment artifact is a "fresh" instance that's "booted" in a fraction of the time—as in a couple of seconds vs a few minutes—than a new EC2 instance would be - since the deployment artifact is a fresh instance, there is no need to manipulate an existing instance already running your application as part of your deploy process (e.g. scp, ssh, unpack, run commands, restart/reload application) - if you design your build pipeline appropriately, the actual data needed to be shipped is effectively just a diff from the previous image. In my org, that could mean the difference between shipping a 300MB vehicle to ~300 instances and shipping a 300KB vehicle to almost certainly less than 300 Kubernetes nodes There are other things as well, but I wanted to stay roughly within the parameters of your examples. If your organization doesn't already have a continuous integration story, I'd say it probably isn't ready for containers in production. If your cloud footprint isn't costing you let's say >= $50k/mo or so, I might argue that the engineering effort to Kubernetize your things is probably better spent in more direct ways on your product. If you do have a CI/CD story, and you do have a very wide, expansive, and expensive cloud footprint, containers and Kubernetes are extremely compelling possible solutions.
- cryptarch 10y ago
  What if you're in the ground-zero scenario: there is an application but practically no infrastructure? I'm in that situation now (with a SaaS product), and my approach so far has been: * set up tests * set up Docker images * set up docker-compose for easy local deployments * do lots of work to make the app behave inside Docker * take control of AWS and IAM using CloudFormation, making it easy to set up and roll back permissions * bootstrap KOPS/K8s with CloudFormation Future work: * set up Jenkins CI on top of K8s * K8s deployments of the app for CI * staging K8s deployments * K8s in production This has been my first venture in operations and it's been very interesting and rewarding, but at times it gets very intimidating to "be" operations with no senior guidance. I enjoy the freedom and responsibility and I feel like I at least have a significantly better grasp on this than anyone else in the company, but I wish I knew of a way to feel like my work was good in an absolute sense, like "definitely good enough". Has anyone here gone through a similar scenario, bootstrapping ops infrastructure in a small business? Would you have any remarks regarding architecture, or the politics required to make the importance of solid ops and the improvement of software reliability and security clear?
PeterisP 10y ago
Tar definitely doesn't solve the issue of reproducible deploys, a tar deploy has a high chance of working differently or not working at all unless the target environment magically happens to be totally identical to yours. We use docker to solve the distribution/deployment headaches where we don't know and definitely cannot influence the (different) environments where the package will be deployed. Docker made a headache go away for us - we can make a deployment available to others, they can run it, and it generally will just work properly the first time. The earlier alternative for that style of "reproducible deploys" was full virtual machine images, not tar. The alternative in the years before shipping VMs was easy was configuring it manually on site or (if it made financial sense) shipping pre-configured hardware. Something like tar was never a reasonable alternative for deploys - even decades ago when big unix vendors controlled/owned/managed your whole system, them 'simply deploying' a tar package worked usually, but didn't work reliably.
sleepydog 10y ago
Dunno if this has been mentioned, but on the reproducible builds front, you can use bazel (Google's build system) to produce docker images: https://bazel.build/blog/2015/07/28/docker_build.html https://bazel.build/blog/2015/07/28/docker_build.html . It's a little more difficult and explicit to introduce non-determinism into a bazel build.

by erulabs 10y ago

moderation 10y ago

Your responses assume that I'm running in the cloud and AWS specifically. What if I need to run on-premise?

erulabs 10y ago
Kube is actually fantastic for on-prem/dedicated/colo. I still wind up needing to manage my _non_ Kube stuff - and fall back to Ansible + systemd for the data layer for example. Additionally, one big complexity re: Kube on dedicated is dealing with custom hardware. Kube and Docker really require modern and specialized kernels - This actually applies to AWS as well; the current Kube AMI for AWS doesn't install the `linux-aws` kernel for example, so you have no access to the ENA driver or the ixgbefv driver, and the t* and i3 instance performance is awful (the most common and the most vital machines respectively). There is a PR for that open somewhere and I'm sure it will be fixed soon, but you see my point. After the hardware/kernel build (which you'll need to manage outside of Kube, so you'll probably choose something like Ansible), you'll probably want auth management at the host level (assuming you run on-prem), which you'll most likely fall back to Ansible for again. I should probably also admit somewhere in this thread I actually use and prefer Chef :)

perlpimp 10y ago

tar does reproduce a deploy but onto c clean slate machine. docker is just a clean space where you can take an issue from there and easier reproduce elsewhere because there is no state left over from installations of previous packages and/or other software.

amazingman 10y ago

What makes containers in general, and Kubernetes specifically, compelling to me are the following: - the deployment artifact is a "fresh" instance that's "booted" in a fraction of the time—as in a couple of seconds vs a few minutes—than a new EC2 instance would be - since the deployment artifact is a fresh instance, there is no need to manipulate an existing instance already running your application as part of your deploy process (e.g. scp, ssh, unpack, run commands, restart/reload application) - if you design your build pipeline appropriately, the actual data needed to be shipped is effectively just a diff from the previous image. In my org, that could mean the difference between shipping a 300MB vehicle to ~300 instances and shipping a 300KB vehicle to almost certainly less than 300 Kubernetes nodes There are other things as well, but I wanted to stay roughly within the parameters of your examples. If your organization doesn't already have a continuous integration story, I'd say it probably isn't ready for containers in production. If your cloud footprint isn't costing you let's say >= $50k/mo or so, I might argue that the engineering effort to Kubernetize your things is probably better spent in more direct ways on your product. If you do have a CI/CD story, and you do have a very wide, expansive, and expensive cloud footprint, containers and Kubernetes are extremely compelling possible solutions.

cryptarch 10y ago
What if you're in the ground-zero scenario: there is an application but practically no infrastructure? I'm in that situation now (with a SaaS product), and my approach so far has been: * set up tests * set up Docker images * set up docker-compose for easy local deployments * do lots of work to make the app behave inside Docker * take control of AWS and IAM using CloudFormation, making it easy to set up and roll back permissions * bootstrap KOPS/K8s with CloudFormation Future work: * set up Jenkins CI on top of K8s * K8s deployments of the app for CI * staging K8s deployments * K8s in production This has been my first venture in operations and it's been very interesting and rewarding, but at times it gets very intimidating to "be" operations with no senior guidance. I enjoy the freedom and responsibility and I feel like I at least have a significantly better grasp on this than anyone else in the company, but I wish I knew of a way to feel like my work was good in an absolute sense, like "definitely good enough". Has anyone here gone through a similar scenario, bootstrapping ops infrastructure in a small business? Would you have any remarks regarding architecture, or the politics required to make the importance of solid ops and the improvement of software reliability and security clear?

PeterisP 10y ago

Tar definitely doesn't solve the issue of reproducible deploys, a tar deploy has a high chance of working differently or not working at all unless the target environment magically happens to be totally identical to yours. We use docker to solve the distribution/deployment headaches where we don't know and definitely cannot influence the (different) environments where the package will be deployed. Docker made a headache go away for us - we can make a deployment available to others, they can run it, and it generally will just work properly the first time. The earlier alternative for that style of "reproducible deploys" was full virtual machine images, not tar. The alternative in the years before shipping VMs was easy was configuring it manually on site or (if it made financial sense) shipping pre-configured hardware. Something like tar was never a reasonable alternative for deploys - even decades ago when big unix vendors controlled/owned/managed your whole system, them 'simply deploying' a tar package worked usually, but didn't work reliably.

sleepydog 10y ago

Dunno if this has been mentioned, but on the reproducible builds front, you can use bazel (Google's build system) to produce docker images: https://bazel.build/blog/2015/07/28/docker_build.html https://bazel.build/blog/2015/07/28/docker_build.html . It's a little more difficult and explicit to introduce non-determinism into a bazel build.