10 ms·
Convergence to Kubernetes
- ordinaryperson 8y agoDumb question from someone who doesn't use Kubernetes (or Docker) in production: don't routine security updates mean you're constantly rebuilding and redeploying these images? And if so, how is that more efficient than just using Puppet / Chef / Ansible and a 'real' server?
- number6 8y agoYou can automate the build process and the deploy process. Given that you can do that with any orchestration software it gives you the ability to roll back changes in your environment. Is it more efficent than a real server? I guess thats only the case if you automate almost everything and use CI and CD. It sure shows a diffrent viewpoint on servers focusing more on services or containers than on servers. Comparable with Functional Programming and Object Orientated Programming. While you can archive the same functionality with both it gives you another mindset to solve problems.
- tyingq 8y agoBasically just less rolling your own solutions. It's more political than technical. "Devops" for me is just shorthand for we want the sysadmins on our team (or on a platform our team controls) versus a separate sysadmin org we can't control. K8S is one of many things that makes that more achievable. That, of course, has it's obvious benefits and shortfalls.
- orf 8y agoIf your referring to security updates in the docker images rather than Kubernetes itself, then not really. The rebuild should be automated for you, so it's as simple as triggering a Jenkins master build. Kubernetes handles the deployment for you, bringing down the old pods and upping the new ones without any connection loss. It makes it a lot simpler to deploy these updates.
- brazzledazzle 8y agoI think one of the powerful things that something like containers offer is a happy path that's also the easy path path and it meshes well with the principles you need for a scalable or micro-service infrastructure. With "real" servers you can build infrastructure that operates and scales in similar ways but the happy path isn't as clear or enforced. Containers make you think about persistence and state as being distinct from your compute at/near the beginning. You're punished for not doing so fairly early, even on your local machine. Your AWS instance can go away but months down the line you're not wondering what you did wrong, you're wondering what Amazon or your ops team did wrong. So the happy path that was easier to deviate from before becomes much harder with containers. It's also easier and faster to update a container image than servers. With something like Kubernetes you can even do it in stages. Yes, you still have to keep the underlying servers updated but they just need to run containers. No testing dependencies and prerequisites. And that decoupling of the app runtime makes updating the servers themselves easier. You can create a new machine image and replace outdated ones trivially.
- jacques_chester 8y ago> don't routine security updates mean you're constantly rebuilding and redeploying these images? In my experience, no. Dockerfiles tend to obfuscate deeper dependencies. Either there's stale dependencies or uncontrolled versioning. In both cases you're running mystery meat in production. > And if so, how is that more efficient than just using Puppet / Chef / Ansible and a 'real' server? The fundamental DNA of configuration managers is to take a closed world - a server - as given and try to make it into the world as desired. But this assumes you can come up with a safe, terminating and not-too-long plan for doing so. Very frequently that is not the case. What gets done in such cases? The server gets backed up, wiped and rebuilt from a clean state. And that's more or less what you get from a container-centric substrate over a server-centric substrate. Just as I don't care about how the JVM allocates or collects my objects, I don't care about how a container orchestrator builds my processes. The idea predates containers, but many ideas predate their economical realisation. I am most familiar with BOSH, which does this at a whole-VM-centric level (and which, like Kubernetes, was inspired by Borg). Or the 12 Factor App, which focused on this idea from a devops perspective. But I would be unsurprised to find carvings from the 4th Dynasty outlining a similar idea.
- keypusher 8y agoI don't think people primarily choose Docker because it's more efficient(?) to deploy than a real server. The benefits of Docker, to me, are having a single artifact and reproducible builds where a developer can run the docker image locally, it then gets built once on Jenkins, and the same image is deployed to staging, production, etc. It eliminates an entire class of problems related to the operating system, installed dependencies, what language the application was written in, etc. That being said, those are also benefits when deploying, because everything is homogeneous at the deployment layer. As far as constantly rebuilding and deploying things, we would be doing that anyway. We do dozens of deploys a day to push new code, that means building new images every time, if one of those deploys picks up a security fix that was recently merged upstream, great.
- hinkley 8y ago> having a single artifact and reproducible builds where a developer can run the docker image locally This is the key thing here. As a team grows it’s easy to get various kinds of learned helplessness. Docker, for its faults, is mostly simple enough that you can expect/insist that the team use it. Which means fewer kinds of surprises at deployment time.
- thawkins 8y agoWe enforced that by giving our devs linux workstations with no sudo rights (iso27001 requirement), but they have access to the docker daemon, and MUST install everything they need in docker images, its massively increased tools sharing between teams, and forced people to learn docker. We have some devs starting to use kubernetes locally using minikube .
- rehemiau 8y agoSorry for being off-topic but do you have any links or tips on how to achieve this? My understand was that adding a user to the 'docker' group gives him 'sudo'-equivalent rights.
- 8y ago
- nh2 8y ago> don't routine security updates mean you're constantly rebuilding and redeploying these images? It means you should be constantly rebuilding and redeploying these images. The fewest people I've seen use Docker actually do that. The answer I've heard most commonly so far is "uh hmm ... right, given that I see new CVEs fixed every day, scrolling by in the `apt-get dist-upgrade` I do daily on my desktop, we should probably be doing that for our Docker as well ..." Many then refer to per-push CI/CD, not realising that holes need to be fixed regularly and not when you push. The fewest have automation set up that rebuilds and redeploys when the upstream docker image changes. Plain Debian/Ubuntu servers have the benefit of `unattended-upgrades`, but with Docker you have to take on that task yourself and build automation for it. Also have a look at https://hub.docker.com/r/library/ubuntu/tags/ https://hub.docker.com/r/library/ubuntu/tags/ At the time of writing, every single image is labelled with "This image has vulnerabilities".
- jes5199 8y agowho has an app that they don’t deploy daily?
- kawsper 8y agoMany - and our deploys on weekends are almost zero.
- majewsky 8y agoI highly doubt that anyone would deploy everything daily. Your own code, maybe. But what about dependencies? Do you upgrade your Postgres container daily?
- barrkel 8y agoSome of us have contractual obligations not to deploy during the week, and require us to notify and receive permission from our customers for out of band deployments.
- jrs95 8y agoYou probably shouldn’t be using Ubuntu base images anyways. The closer you can get to “scratch” the better, and the fewer security related issues you’ll have. For most use cases I think Alpine is a much better base image.
- lkrubner 8y agoDocker is a dangerous gamble and you can get more of an automated build system, with less devops effort, from Terraform and Packer. Avoid containers and stick with real servers “baked” by Packer: http://www.smashcompany.com/technology/docker-is-a-dangerous-gamble-which-we-will-regret http://www.smashcompany.com/technology/docker-is-a-dangerous...
- lsh 8y agothat was a good read, thankyou
- MPSimmons 8y agoThere's no reason you can build containers in an automated, routine fashion and use those to run your applications and services on. You don't have to run containers like joeblow/randomservice - start with Alpine from the Alpine maintainers (or CentOS or whatever) and write a custom dockerfile to build your stuff.
- RVuRnvbM2e 8y agoWhy would I want the overhead of a hypervisor when I can just use the process isolation features built into the kernel? Re: your essay I couldn't quite tell but is it correct to say that you consider Docker a "dangerous gamble" because Docker Inc. may cease to exist in its current form at some point?
- icebraining 8y agoDocker aside, your arguments regarding fat binaries are kinda off mark. Python has been able to package an app and all of its dependencies into a single zip file - which is what a jar file really is - since before Go and Clojure even appeared (see PEP 273). Single-file packaging in JS is also common - very much used for browser deployment. But "let's change languages because we don't have a reliable way to copy more than one file" sounds a bit insane to me. My solution was Debian packages, not Docker, though.
- barrkel 8y agoI've not been impressed by docker outside of the local development experience (primarily, creating sandboxes and avoiding installing stuff on the main OS, and not having the latency of starting up a VM). This discussion is about Kubernetes though. Docker is mostly only incidental to Kubernetes. It's a much better thought through system than anything I've seen come out of the Docker world. Docker images are adequate; Dockerfile as a build mechanism for images is deeply flawed, requiring so many kludges and duct tape to accommodate shared dependencies, rebuild on upstream change, shrinking after building, etc. Hopefully a better system for building images to run on Kubernetes will crop up.
- oso2k 8y agoFull-disclosure: I’m a Consulting Architect for Red Hat focused on OpenShift. No. In Kubernetes and OpenShift you can control whether builds and deployments are automatic or manual, & which events triggers them [0][1]. Combined with fact that each application’s config is an independent object, this allows an admin to host hundreds of apps on a single cluster node. Usually you’re using IaC practice and storing each app’s config in a git repo or next to the app in git. [0] https://docs.openshift.com/container-platform/3.9/dev_guide/builds/triggering_builds.html https://docs.openshift.com/container-platform/3.9/dev_guide/... [1] https://docs.openshift.com/container-platform/3.9/dev_guide/deployments/basic_deployment_operations.html https://docs.openshift.com/container-platform/3.9/dev_guide/...
- snuxoll 8y agoOpenShift has an answer to what GP was asking, but build triggers are not it. Security vulnerabilities in images is the concern, OpenShift handles this through ImageStream('s) - parent images are tracked in the integrated registry and when one is updated all dependent images are updated. Good example, the dotnet image in our OpenShift cluster was updated late last week - all of our .Net Core projects were automatically rebuilt with the latest image with no intervention from the developers. It doesn't handle your application INSIDE the container, or if you build the image yourself via Dockerfile so you'll still need some dependency scanning tools and/or release notifications to keep your own stuff up-to-date and secure. To be honest ImageStream is one of the best value-add features OpenShift has over vanilla Kubernetes, I don't have to worry if developers are keeping their images up-to-date when some vulnerability in a CentOS package or application runtime gets patched.
- erikb 8y agoKubernetes itself is not a PaaS though. It won't do all of that for you, which has the advantage of being more flexible but the disadvantage that nothing happens if you don't code it.
- raesene9 8y agoOne point that's not been addressed in your other replies is that Docker images should be a lot smaller than a "real" server so there should be fewer issues to patch. If you base off a very minimal image (and make use of things like multi-stage builds to remove dev. tools from the image) you can get the package numbers down a lot. For the ultimate in low dependencies of course you can build off scratch and put a single statically linked binary into the image.
- mmt 8y ago> One point that's not been addressed in your other replies is that Docker images should be a lot smaller than a "real" server so there should be fewer issues to patch. Could you quantify this? It's possible to strip down a "real" server much farther than even what distros' "minimal" packages specify, which is why I wonder just how much smaller "a lot smaller" really is. I commented on a short sub-thread [1] discussing dependency bloat/radii of distros. One commentor quantified it in bytes, though perhaps a more meaningful number in this context would be number of packages. [1] https://news.ycombinator.com/item?id=17345982 https://news.ycombinator.com/item?id=17345982
- KaiserPro 8y agoYes, but its less of an issue mainly because there is an orchestration layer over the top of the deployment layer. Basically there is a thing that spins up machines, connects them to a thing that places programs on them to run. The scheduler then starts your program and maps in storage, outside world connections (HTTP reverse proxy, or a bit more advance and some level 7 routing [ie /v1 goes to progam a /v2 goes to b]) when it works it means that the developer really doesn't need to think tomuch about _where_ a program runs. Basically its a feature incomplete mainframe clone.
- bsenftner 8y agoFrom Saturday's posts: http://catern.com/posts/docker.html http://catern.com/posts/docker.html An assessment of what Docker and similar actually are, and why they are unneeded redundant, insecure technology.
- solatic 8y ago> don't routine security updates mean you're constantly rebuilding and redeploying these images? Yes and no; yes, because you should be rebuilding upon upstream changes, but no, because there will be fewer upstream changes. You can reduce the amount of software that is baked into a Docker image, compared to a full traditional distribution, thus reducing your attack surface and reducing the frequency with which the image needs to be rebuilt. > And if so, how is that more efficient than just using Puppet / Chef / Ansible and a 'real' server? "Real" servers are always (exception: NixOS) stateful, even if they are only used to host stateless services. Uninstalled packages leave behind old service users, buggy packages depend on specific user/group IDs that conflict in production but not where they were built, conflicting ports that are already bound. Designing for immutable infrastructure makes it easier to reason about state in production and therefore makes production easier to manage.
- polskibus 8y agoIs there a recommended way to handle database migrations in kubernetes? Is there a best practice or a tool for that?
- cutety 8y agoThis is also something I've been wondering about. I've been trying to nudge my workplace to start moving to a containerized infrastructure, but this has been one of my nagging questions I've had. Admittedly I haven't done much research on the subject though. I currently have a swarm cluster on DigitalOcean that I use for self hosting apps. And, only one of those required migrating the database before it could be deployed, so to get it running I just spun it up on one of the nodes and only connecting the database container to the data volume I was going to have it connect to in the swarm, migrated the db, then killed the containers and then deployed it to the swarm. But, as I was doing it I realized this would hardly be maintainable/enjoyable method of handling this in any non-personal/production environment.
- marcc 8y agoWe've been packaging migrations into a container and just shipping it as a Kubernetes Job. It executes once, and if doesn't success (non-zero exit code), Kubernetes will reschedule it.
- polskibus 8y agoSo if you want to migrate a particular env from schema version 3 to version 6, do you deploying 3 different init containers manually, or is there a single migration container that always contains all migrations and it by itself understands from which level to which level should the migration be run? Schema migration failure is a stop the world a scenario for our entire stack.
- majewsky 8y ago> Schema migration failure is a stop the world a scenario for our entire stack. That's the big problem, and I don't believe that Kubernetes can help with that. It's too intimately tied to your application internals. The recommended practice for migrations is to do them in multiple zero-downtime steps, e.g. first deploy code that handles both old and new schema, then migrate, then deploy to remove transitional code.
- 101km 8y ago"The result was a system composed of many wavefronts of change: some systems were automated with Puppet, some with Terraform, some used ECS and others used straight EC2. In 2012 we were proud to have an architecture that could evolve so frequently, letting us experiment continually, discovering what worked and doing more of it. In 2017, however, we finally recognised that things had changed. AWS is significantly more complex today than when we started using it. It provides an incredible amount of choice and power but not without cost. Any team that interacts with EC2 today must now navigate decisions on VPCs , networking and many, many more." Of course, this time, it is different.
- mmt 8y agoI do certainly wonder if the ever-increasing levels of complexity in the layers of abstraction will backfire in some way soon. It seems the trend has accelerated recently.
- bonesss 8y agoLuckily there's an easy fix for that: adding more layers abstraction :D
- mmt 8y agoI'll bet you have an "easy" fix for Social Security, too :) Joking aside, I certainly understand the benefits of abstraction. As someone always points out in any discussion about ORMs, for example, abstractions are leaky. Whenever one has to learn about the inner workings of what the abstraction is hiding, some of that ease evaporates.
- draw_down 8y agoAhhh man, you quoted all that just to say “this time it’s different”?!
- erikb 8y ago> [...] this time, it is different. Literally a quote from my managers (not related to k8s or aws alone, though). They also made sure to repeat that phrase a few times so it becomes more believable.
- ben0x539 8y agoHeads up, this article is from the future, apparently: > In late 2017 all teams ran all their own AWS infrastructure. [...] In a little over a year that’s changed for all teams.
- ajross 8y ago> We have close to 30 teams that run some or all of their workloads on our clusters. Approximately 70% of all HTTP traffic we serve is generated from applications within our Kubernetes clusters. Sounds big. But then per wikipedia: > uSwitch.com [...] allows consumers to compare prices for a range of energy, personal finance, insurance and communications services. And: > On 30 April 2015, the property website firm Zoopla agreed to purchase uSwitch from LDC for £160 million So... a low bandwidth business (we're hardly talking Netflix here!) doing maybe, what, $10M in revenue annually and not growing fast enough to justify venture investment or IPO funding (they were a private acquisition!)... Seriously, I'm sure they like it. But do they really, truly need Kubernetes? This really sounds like the kind of scale that can be achieved with 2-3 hand-managed servers, or maybe twice that number of AWS boxen.
- marcc 8y agoShould any business that has SLAs and needs to be reliable ever rely on “2-3 hand-managed servers”? No. Kubernetes isn’t only about scale. It also provides rolling upgrades and rollbacks. And failover. And DNS based service discovery. And there’s more. You can find solutions to these without Kubernetes but a lot of Kubernetes use is to get these, not simply for scale issues.
- oso2k 8y agoFull-disclosure: I’m a Consulting Architect at Red Hat focused on OpenShift. I like to say Kubernetes & OpenShift focus on availability of the cluster and applications as their primary concern. Many other concerns my customers want to impose are actually detrimental to the goal(s) they’re trying to achieve.
- bonesss 8y ago> And there’s more Being able to reliably create your entire infrastructure on another platform in 'minutes', for example. Not to mention applications architected from the ground up around cloud-friendly and scale-friendly primitives... 2-3 hand-managed servers are great, but will absolutely warp your application and will slowly accrue configuration cruft. That's not terrible, but for many Real World issues portability and freedom to fire up wholly valid test-environments are game-changers. Even the acquisition stories are nicer.
- hardwaresofton 8y agoCan anyone who used Docker Swarm/Mesos/Nomad and then switched to Kubernetes comment on anything that was done better by Swarm/Mesos/Nomad? I invested in Kubernetes early and always meant to give the others a try (so I could at least know the differences), but never got a chance to.
- minieggs 8y agoI would love to see a comparison between Docker Swarm and Kubernetes. From talking to peeps I've got that Kubernetes is better for a large number of nodes? I self host all my side projects with Docker Swarm and it's been so good I haven't needed to look into other container management solutions (but I've only got eight nodes).
- piva00 8y agoAt the moment I'd say that Kubernetes is only worthy the effort if you have a bunch of idle capacity in your nodes and some tens of machines at least. The setup can get a bit complex quite early and won't be worth the effort to manage 8 nodes, when you begin to scale to around 20 nodes running a bunch of different workloads (batch jobs, web services, etc.), can avoid provisioning on the application side, etc., then k8s begins to shine more and pay back the investment.
- brianwawok 8y agoWhy should node count matter? Two clicks to get 1-1000 nodes on GKE. The work is to learn the yaml syntax and the way to deploy to GKE... but most apps need to learn something about how it will be deployed (be it how to use ansible to deploy vs how to setup on k8s vs how to use serverless). But you need to do this for 1-1000 nodes, may as well just do it once...
- raesene9 8y agoI'd say that if you're on swarm and it's doing what you want, you probably want to stick with it. Kubernetes is more flexible and more powerful, but it's also a lot more complex. You either go with one of the managed distro's which take on some of that complexity but also reduce your flexibility, or you manage the whole thing yourself.
- eecc 8y agothe steady-state architecture reminds me of this hn story about the Lava Layer anti-pattern https://news.ycombinator.com/item?id=8772641 https://news.ycombinator.com/item?id=8772641