8 ms·

Dumb question from someone who doesn't use Kubernetes (or Docker) in production: don't routine security updates mean you're constantly rebuilding and redeployin

Dumb question from someone who doesn't use Kubernetes (or Docker) in production: don't routine security updates mean you're constantly rebuilding and redeploying these images? And if so, how is that more efficient than just using Puppet / Chef / Ansible and a 'real' server?

number6 8y ago
You can automate the build process and the deploy process. Given that you can do that with any orchestration software it gives you the ability to roll back changes in your environment. Is it more efficent than a real server? I guess thats only the case if you automate almost everything and use CI and CD. It sure shows a diffrent viewpoint on servers focusing more on services or containers than on servers. Comparable with Functional Programming and Object Orientated Programming. While you can archive the same functionality with both it gives you another mindset to solve problems.
tyingq 8y ago
Basically just less rolling your own solutions. It's more political than technical. "Devops" for me is just shorthand for we want the sysadmins on our team (or on a platform our team controls) versus a separate sysadmin org we can't control. K8S is one of many things that makes that more achievable. That, of course, has it's obvious benefits and shortfalls.
orf 8y ago
If your referring to security updates in the docker images rather than Kubernetes itself, then not really. The rebuild should be automated for you, so it's as simple as triggering a Jenkins master build. Kubernetes handles the deployment for you, bringing down the old pods and upping the new ones without any connection loss. It makes it a lot simpler to deploy these updates.
brazzledazzle 8y ago
I think one of the powerful things that something like containers offer is a happy path that's also the easy path path and it meshes well with the principles you need for a scalable or micro-service infrastructure. With "real" servers you can build infrastructure that operates and scales in similar ways but the happy path isn't as clear or enforced. Containers make you think about persistence and state as being distinct from your compute at/near the beginning. You're punished for not doing so fairly early, even on your local machine. Your AWS instance can go away but months down the line you're not wondering what you did wrong, you're wondering what Amazon or your ops team did wrong. So the happy path that was easier to deviate from before becomes much harder with containers. It's also easier and faster to update a container image than servers. With something like Kubernetes you can even do it in stages. Yes, you still have to keep the underlying servers updated but they just need to run containers. No testing dependencies and prerequisites. And that decoupling of the app runtime makes updating the servers themselves easier. You can create a new machine image and replace outdated ones trivially.
jacques_chester 8y ago
> don't routine security updates mean you're constantly rebuilding and redeploying these images? In my experience, no. Dockerfiles tend to obfuscate deeper dependencies. Either there's stale dependencies or uncontrolled versioning. In both cases you're running mystery meat in production. > And if so, how is that more efficient than just using Puppet / Chef / Ansible and a 'real' server? The fundamental DNA of configuration managers is to take a closed world - a server - as given and try to make it into the world as desired. But this assumes you can come up with a safe, terminating and not-too-long plan for doing so. Very frequently that is not the case. What gets done in such cases? The server gets backed up, wiped and rebuilt from a clean state. And that's more or less what you get from a container-centric substrate over a server-centric substrate. Just as I don't care about how the JVM allocates or collects my objects, I don't care about how a container orchestrator builds my processes. The idea predates containers, but many ideas predate their economical realisation. I am most familiar with BOSH, which does this at a whole-VM-centric level (and which, like Kubernetes, was inspired by Borg). Or the 12 Factor App, which focused on this idea from a devops perspective. But I would be unsurprised to find carvings from the 4th Dynasty outlining a similar idea.
keypusher 8y ago
I don't think people primarily choose Docker because it's more efficient(?) to deploy than a real server. The benefits of Docker, to me, are having a single artifact and reproducible builds where a developer can run the docker image locally, it then gets built once on Jenkins, and the same image is deployed to staging, production, etc. It eliminates an entire class of problems related to the operating system, installed dependencies, what language the application was written in, etc. That being said, those are also benefits when deploying, because everything is homogeneous at the deployment layer. As far as constantly rebuilding and deploying things, we would be doing that anyway. We do dozens of deploys a day to push new code, that means building new images every time, if one of those deploys picks up a security fix that was recently merged upstream, great.
- hinkley 8y ago
  > having a single artifact and reproducible builds where a developer can run the docker image locally This is the key thing here. As a team grows it’s easy to get various kinds of learned helplessness. Docker, for its faults, is mostly simple enough that you can expect/insist that the team use it. Which means fewer kinds of surprises at deployment time.
  - thawkins 8y ago
    We enforced that by giving our devs linux workstations with no sudo rights (iso27001 requirement), but they have access to the docker daemon, and MUST install everything they need in docker images, its massively increased tools sharing between teams, and forced people to learn docker. We have some devs starting to use kubernetes locally using minikube .
    - rehemiau 8y ago
      Sorry for being off-topic but do you have any links or tips on how to achieve this? My understand was that adding a user to the 'docker' group gives him 'sudo'-equivalent rights.
      - 8y ago
nh2 8y ago
> don't routine security updates mean you're constantly rebuilding and redeploying these images? It means you should be constantly rebuilding and redeploying these images. The fewest people I've seen use Docker actually do that. The answer I've heard most commonly so far is "uh hmm ... right, given that I see new CVEs fixed every day, scrolling by in the `apt-get dist-upgrade` I do daily on my desktop, we should probably be doing that for our Docker as well ..." Many then refer to per-push CI/CD, not realising that holes need to be fixed regularly and not when you push. The fewest have automation set up that rebuilds and redeploys when the upstream docker image changes. Plain Debian/Ubuntu servers have the benefit of `unattended-upgrades`, but with Docker you have to take on that task yourself and build automation for it. Also have a look at https://hub.docker.com/r/library/ubuntu/tags/ https://hub.docker.com/r/library/ubuntu/tags/ At the time of writing, every single image is labelled with "This image has vulnerabilities".
- jes5199 8y ago
  who has an app that they don’t deploy daily?
  - kawsper 8y ago
    Many - and our deploys on weekends are almost zero.
  - majewsky 8y ago
    I highly doubt that anyone would deploy everything daily. Your own code, maybe. But what about dependencies? Do you upgrade your Postgres container daily?
  - barrkel 8y ago
    Some of us have contractual obligations not to deploy during the week, and require us to notify and receive permission from our customers for out of band deployments.
- jrs95 8y ago
  You probably shouldn’t be using Ubuntu base images anyways. The closer you can get to “scratch” the better, and the fewer security related issues you’ll have. For most use cases I think Alpine is a much better base image.
lkrubner 8y ago
Docker is a dangerous gamble and you can get more of an automated build system, with less devops effort, from Terraform and Packer. Avoid containers and stick with real servers “baked” by Packer: http://www.smashcompany.com/technology/docker-is-a-dangerous-gamble-which-we-will-regret http://www.smashcompany.com/technology/docker-is-a-dangerous...
- lsh 8y ago
  that was a good read, thankyou
- MPSimmons 8y ago
  There's no reason you can build containers in an automated, routine fashion and use those to run your applications and services on. You don't have to run containers like joeblow/randomservice - start with Alpine from the Alpine maintainers (or CentOS or whatever) and write a custom dockerfile to build your stuff.
- RVuRnvbM2e 8y ago
  Why would I want the overhead of a hypervisor when I can just use the process isolation features built into the kernel? Re: your essay I couldn't quite tell but is it correct to say that you consider Docker a "dangerous gamble" because Docker Inc. may cease to exist in its current form at some point?
- icebraining 8y ago
  Docker aside, your arguments regarding fat binaries are kinda off mark. Python has been able to package an app and all of its dependencies into a single zip file - which is what a jar file really is - since before Go and Clojure even appeared (see PEP 273). Single-file packaging in JS is also common - very much used for browser deployment. But "let's change languages because we don't have a reliable way to copy more than one file" sounds a bit insane to me. My solution was Debian packages, not Docker, though.
- barrkel 8y ago
  I've not been impressed by docker outside of the local development experience (primarily, creating sandboxes and avoiding installing stuff on the main OS, and not having the latency of starting up a VM). This discussion is about Kubernetes though. Docker is mostly only incidental to Kubernetes. It's a much better thought through system than anything I've seen come out of the Docker world. Docker images are adequate; Dockerfile as a build mechanism for images is deeply flawed, requiring so many kludges and duct tape to accommodate shared dependencies, rebuild on upstream change, shrinking after building, etc. Hopefully a better system for building images to run on Kubernetes will crop up.
oso2k 8y ago
Full-disclosure: I’m a Consulting Architect for Red Hat focused on OpenShift. No. In Kubernetes and OpenShift you can control whether builds and deployments are automatic or manual, & which events triggers them [0][1]. Combined with fact that each application’s config is an independent object, this allows an admin to host hundreds of apps on a single cluster node. Usually you’re using IaC practice and storing each app’s config in a git repo or next to the app in git. [0] https://docs.openshift.com/container-platform/3.9/dev_guide/builds/triggering_builds.html https://docs.openshift.com/container-platform/3.9/dev_guide/... [1] https://docs.openshift.com/container-platform/3.9/dev_guide/deployments/basic_deployment_operations.html https://docs.openshift.com/container-platform/3.9/dev_guide/...
- snuxoll 8y ago
  OpenShift has an answer to what GP was asking, but build triggers are not it. Security vulnerabilities in images is the concern, OpenShift handles this through ImageStream('s) - parent images are tracked in the integrated registry and when one is updated all dependent images are updated. Good example, the dotnet image in our OpenShift cluster was updated late last week - all of our .Net Core projects were automatically rebuilt with the latest image with no intervention from the developers. It doesn't handle your application INSIDE the container, or if you build the image yourself via Dockerfile so you'll still need some dependency scanning tools and/or release notifications to keep your own stuff up-to-date and secure. To be honest ImageStream is one of the best value-add features OpenShift has over vanilla Kubernetes, I don't have to worry if developers are keeping their images up-to-date when some vulnerability in a CentOS package or application runtime gets patched.
  - erikb 8y ago
    Kubernetes itself is not a PaaS though. It won't do all of that for you, which has the advantage of being more flexible but the disadvantage that nothing happens if you don't code it.
raesene9 8y ago
One point that's not been addressed in your other replies is that Docker images should be a lot smaller than a "real" server so there should be fewer issues to patch. If you base off a very minimal image (and make use of things like multi-stage builds to remove dev. tools from the image) you can get the package numbers down a lot. For the ultimate in low dependencies of course you can build off scratch and put a single statically linked binary into the image.
- mmt 8y ago
  > One point that's not been addressed in your other replies is that Docker images should be a lot smaller than a "real" server so there should be fewer issues to patch. Could you quantify this? It's possible to strip down a "real" server much farther than even what distros' "minimal" packages specify, which is why I wonder just how much smaller "a lot smaller" really is. I commented on a short sub-thread [1] discussing dependency bloat/radii of distros. One commentor quantified it in bytes, though perhaps a more meaningful number in this context would be number of packages. [1] https://news.ycombinator.com/item?id=17345982 https://news.ycombinator.com/item?id=17345982
KaiserPro 8y ago
Yes, but its less of an issue mainly because there is an orchestration layer over the top of the deployment layer. Basically there is a thing that spins up machines, connects them to a thing that places programs on them to run. The scheduler then starts your program and maps in storage, outside world connections (HTTP reverse proxy, or a bit more advance and some level 7 routing [ie /v1 goes to progam a /v2 goes to b]) when it works it means that the developer really doesn't need to think tomuch about _where_ a program runs. Basically its a feature incomplete mainframe clone.
bsenftner 8y ago
From Saturday's posts: http://catern.com/posts/docker.html http://catern.com/posts/docker.html An assessment of what Docker and similar actually are, and why they are unneeded redundant, insecure technology.
solatic 8y ago
> don't routine security updates mean you're constantly rebuilding and redeploying these images? Yes and no; yes, because you should be rebuilding upon upstream changes, but no, because there will be fewer upstream changes. You can reduce the amount of software that is baked into a Docker image, compared to a full traditional distribution, thus reducing your attack surface and reducing the frequency with which the image needs to be rebuilt. > And if so, how is that more efficient than just using Puppet / Chef / Ansible and a 'real' server? "Real" servers are always (exception: NixOS) stateful, even if they are only used to host stateless services. Uninstalled packages leave behind old service users, buggy packages depend on specific user/group IDs that conflict in production but not where they were built, conflicting ports that are already bound. Designing for immutable infrastructure makes it easier to reason about state in production and therefore makes production easier to manage.

number6 8y ago

You can automate the build process and the deploy process. Given that you can do that with any orchestration software it gives you the ability to roll back changes in your environment. Is it more efficent than a real server? I guess thats only the case if you automate almost everything and use CI and CD. It sure shows a diffrent viewpoint on servers focusing more on services or containers than on servers. Comparable with Functional Programming and Object Orientated Programming. While you can archive the same functionality with both it gives you another mindset to solve problems.

tyingq 8y ago

Basically just less rolling your own solutions. It's more political than technical. "Devops" for me is just shorthand for we want the sysadmins on our team (or on a platform our team controls) versus a separate sysadmin org we can't control. K8S is one of many things that makes that more achievable. That, of course, has it's obvious benefits and shortfalls.

orf 8y ago

If your referring to security updates in the docker images rather than Kubernetes itself, then not really. The rebuild should be automated for you, so it's as simple as triggering a Jenkins master build. Kubernetes handles the deployment for you, bringing down the old pods and upping the new ones without any connection loss. It makes it a lot simpler to deploy these updates.

brazzledazzle 8y ago

I think one of the powerful things that something like containers offer is a happy path that's also the easy path path and it meshes well with the principles you need for a scalable or micro-service infrastructure. With "real" servers you can build infrastructure that operates and scales in similar ways but the happy path isn't as clear or enforced. Containers make you think about persistence and state as being distinct from your compute at/near the beginning. You're punished for not doing so fairly early, even on your local machine. Your AWS instance can go away but months down the line you're not wondering what you did wrong, you're wondering what Amazon or your ops team did wrong. So the happy path that was easier to deviate from before becomes much harder with containers. It's also easier and faster to update a container image than servers. With something like Kubernetes you can even do it in stages. Yes, you still have to keep the underlying servers updated but they just need to run containers. No testing dependencies and prerequisites. And that decoupling of the app runtime makes updating the servers themselves easier. You can create a new machine image and replace outdated ones trivially.

jacques_chester 8y ago

> don't routine security updates mean you're constantly rebuilding and redeploying these images? In my experience, no. Dockerfiles tend to obfuscate deeper dependencies. Either there's stale dependencies or uncontrolled versioning. In both cases you're running mystery meat in production. > And if so, how is that more efficient than just using Puppet / Chef / Ansible and a 'real' server? The fundamental DNA of configuration managers is to take a closed world - a server - as given and try to make it into the world as desired. But this assumes you can come up with a safe, terminating and not-too-long plan for doing so. Very frequently that is not the case. What gets done in such cases? The server gets backed up, wiped and rebuilt from a clean state. And that's more or less what you get from a container-centric substrate over a server-centric substrate. Just as I don't care about how the JVM allocates or collects my objects, I don't care about how a container orchestrator builds my processes. The idea predates containers, but many ideas predate their economical realisation. I am most familiar with BOSH, which does this at a whole-VM-centric level (and which, like Kubernetes, was inspired by Borg). Or the 12 Factor App, which focused on this idea from a devops perspective. But I would be unsurprised to find carvings from the 4th Dynasty outlining a similar idea.

keypusher 8y ago

I don't think people primarily choose Docker because it's more efficient(?) to deploy than a real server. The benefits of Docker, to me, are having a single artifact and reproducible builds where a developer can run the docker image locally, it then gets built once on Jenkins, and the same image is deployed to staging, production, etc. It eliminates an entire class of problems related to the operating system, installed dependencies, what language the application was written in, etc. That being said, those are also benefits when deploying, because everything is homogeneous at the deployment layer. As far as constantly rebuilding and deploying things, we would be doing that anyway. We do dozens of deploys a day to push new code, that means building new images every time, if one of those deploys picks up a security fix that was recently merged upstream, great.

hinkley 8y ago
> having a single artifact and reproducible builds where a developer can run the docker image locally This is the key thing here. As a team grows it’s easy to get various kinds of learned helplessness. Docker, for its faults, is mostly simple enough that you can expect/insist that the team use it. Which means fewer kinds of surprises at deployment time.
- thawkins 8y ago
  We enforced that by giving our devs linux workstations with no sudo rights (iso27001 requirement), but they have access to the docker daemon, and MUST install everything they need in docker images, its massively increased tools sharing between teams, and forced people to learn docker. We have some devs starting to use kubernetes locally using minikube .
  - rehemiau 8y ago
    Sorry for being off-topic but do you have any links or tips on how to achieve this? My understand was that adding a user to the 'docker' group gives him 'sudo'-equivalent rights.
    - 8y ago

nh2 8y ago

> don't routine security updates mean you're constantly rebuilding and redeploying these images? It means you should be constantly rebuilding and redeploying these images. The fewest people I've seen use Docker actually do that. The answer I've heard most commonly so far is "uh hmm ... right, given that I see new CVEs fixed every day, scrolling by in the `apt-get dist-upgrade` I do daily on my desktop, we should probably be doing that for our Docker as well ..." Many then refer to per-push CI/CD, not realising that holes need to be fixed regularly and not when you push. The fewest have automation set up that rebuilds and redeploys when the upstream docker image changes. Plain Debian/Ubuntu servers have the benefit of `unattended-upgrades`, but with Docker you have to take on that task yourself and build automation for it. Also have a look at https://hub.docker.com/r/library/ubuntu/tags/ https://hub.docker.com/r/library/ubuntu/tags/ At the time of writing, every single image is labelled with "This image has vulnerabilities".

jes5199 8y ago
who has an app that they don’t deploy daily?
- kawsper 8y ago
  Many - and our deploys on weekends are almost zero.
- majewsky 8y ago
  I highly doubt that anyone would deploy everything daily. Your own code, maybe. But what about dependencies? Do you upgrade your Postgres container daily?
- barrkel 8y ago
  Some of us have contractual obligations not to deploy during the week, and require us to notify and receive permission from our customers for out of band deployments.
jrs95 8y ago
You probably shouldn’t be using Ubuntu base images anyways. The closer you can get to “scratch” the better, and the fewer security related issues you’ll have. For most use cases I think Alpine is a much better base image.

lkrubner 8y ago

Docker is a dangerous gamble and you can get more of an automated build system, with less devops effort, from Terraform and Packer. Avoid containers and stick with real servers “baked” by Packer: http://www.smashcompany.com/technology/docker-is-a-dangerous-gamble-which-we-will-regret http://www.smashcompany.com/technology/docker-is-a-dangerous...

lsh 8y ago
that was a good read, thankyou
MPSimmons 8y ago
There's no reason you can build containers in an automated, routine fashion and use those to run your applications and services on. You don't have to run containers like joeblow/randomservice - start with Alpine from the Alpine maintainers (or CentOS or whatever) and write a custom dockerfile to build your stuff.
RVuRnvbM2e 8y ago
Why would I want the overhead of a hypervisor when I can just use the process isolation features built into the kernel? Re: your essay I couldn't quite tell but is it correct to say that you consider Docker a "dangerous gamble" because Docker Inc. may cease to exist in its current form at some point?
icebraining 8y ago
Docker aside, your arguments regarding fat binaries are kinda off mark. Python has been able to package an app and all of its dependencies into a single zip file - which is what a jar file really is - since before Go and Clojure even appeared (see PEP 273). Single-file packaging in JS is also common - very much used for browser deployment. But "let's change languages because we don't have a reliable way to copy more than one file" sounds a bit insane to me. My solution was Debian packages, not Docker, though.
barrkel 8y ago
I've not been impressed by docker outside of the local development experience (primarily, creating sandboxes and avoiding installing stuff on the main OS, and not having the latency of starting up a VM). This discussion is about Kubernetes though. Docker is mostly only incidental to Kubernetes. It's a much better thought through system than anything I've seen come out of the Docker world. Docker images are adequate; Dockerfile as a build mechanism for images is deeply flawed, requiring so many kludges and duct tape to accommodate shared dependencies, rebuild on upstream change, shrinking after building, etc. Hopefully a better system for building images to run on Kubernetes will crop up.

oso2k 8y ago

Full-disclosure: I’m a Consulting Architect for Red Hat focused on OpenShift. No. In Kubernetes and OpenShift you can control whether builds and deployments are automatic or manual, & which events triggers them [0][1]. Combined with fact that each application’s config is an independent object, this allows an admin to host hundreds of apps on a single cluster node. Usually you’re using IaC practice and storing each app’s config in a git repo or next to the app in git. [0] https://docs.openshift.com/container-platform/3.9/dev_guide/builds/triggering_builds.html https://docs.openshift.com/container-platform/3.9/dev_guide/... [1] https://docs.openshift.com/container-platform/3.9/dev_guide/deployments/basic_deployment_operations.html https://docs.openshift.com/container-platform/3.9/dev_guide/...

snuxoll 8y ago
OpenShift has an answer to what GP was asking, but build triggers are not it. Security vulnerabilities in images is the concern, OpenShift handles this through ImageStream('s) - parent images are tracked in the integrated registry and when one is updated all dependent images are updated. Good example, the dotnet image in our OpenShift cluster was updated late last week - all of our .Net Core projects were automatically rebuilt with the latest image with no intervention from the developers. It doesn't handle your application INSIDE the container, or if you build the image yourself via Dockerfile so you'll still need some dependency scanning tools and/or release notifications to keep your own stuff up-to-date and secure. To be honest ImageStream is one of the best value-add features OpenShift has over vanilla Kubernetes, I don't have to worry if developers are keeping their images up-to-date when some vulnerability in a CentOS package or application runtime gets patched.
- erikb 8y ago
  Kubernetes itself is not a PaaS though. It won't do all of that for you, which has the advantage of being more flexible but the disadvantage that nothing happens if you don't code it.

raesene9 8y ago

One point that's not been addressed in your other replies is that Docker images should be a lot smaller than a "real" server so there should be fewer issues to patch. If you base off a very minimal image (and make use of things like multi-stage builds to remove dev. tools from the image) you can get the package numbers down a lot. For the ultimate in low dependencies of course you can build off scratch and put a single statically linked binary into the image.

mmt 8y ago
> One point that's not been addressed in your other replies is that Docker images should be a lot smaller than a "real" server so there should be fewer issues to patch. Could you quantify this? It's possible to strip down a "real" server much farther than even what distros' "minimal" packages specify, which is why I wonder just how much smaller "a lot smaller" really is. I commented on a short sub-thread [1] discussing dependency bloat/radii of distros. One commentor quantified it in bytes, though perhaps a more meaningful number in this context would be number of packages. [1] https://news.ycombinator.com/item?id=17345982 https://news.ycombinator.com/item?id=17345982

KaiserPro 8y ago

Yes, but its less of an issue mainly because there is an orchestration layer over the top of the deployment layer. Basically there is a thing that spins up machines, connects them to a thing that places programs on them to run. The scheduler then starts your program and maps in storage, outside world connections (HTTP reverse proxy, or a bit more advance and some level 7 routing [ie /v1 goes to progam a /v2 goes to b]) when it works it means that the developer really doesn't need to think tomuch about _where_ a program runs. Basically its a feature incomplete mainframe clone.

bsenftner 8y ago

From Saturday's posts: http://catern.com/posts/docker.html http://catern.com/posts/docker.html An assessment of what Docker and similar actually are, and why they are unneeded redundant, insecure technology.

solatic 8y ago

> don't routine security updates mean you're constantly rebuilding and redeploying these images? Yes and no; yes, because you should be rebuilding upon upstream changes, but no, because there will be fewer upstream changes. You can reduce the amount of software that is baked into a Docker image, compared to a full traditional distribution, thus reducing your attack surface and reducing the frequency with which the image needs to be rebuilt. > And if so, how is that more efficient than just using Puppet / Chef / Ansible and a 'real' server? "Real" servers are always (exception: NixOS) stateful, even if they are only used to host stateless services. Uninstalled packages leave behind old service users, buggy packages depend on specific user/group IDs that conflict in production but not where they were built, conflicting ports that are already bound. Designing for immutable infrastructure makes it easier to reason about state in production and therefore makes production easier to manage.