8 ms·

The Need for Slimmer Containers

80 points by r4um 5y ago

willvarfar 5y ago
Luckily most of those vulnerabilities will be dormant, impossible to reach and exploit. Hopefully. I don't think there's going to be any change in how people package containers. So perhaps what is needed is a dependency-walking Link-Time-Optimization-like tool that can trim down, perhaps by masking rather than removing, the dead code?
- iximiuz 5y ago
  I still have a hope that increasing awareness of the problem may eventually lead to a reduction in the number of Dockerfiles starting `FROM <something_huge_and_not_really_needed>`.
- dspillett 5y ago
  > Luckily most of those vulnerabilities will be dormant, impossible to reach and exploit. Hopefully. Our people who use docker for some of our infrastructure have taken to using locally rolled images (from scratch or by updating official ones) for this reason. I don't think any of the issues found when we scanned were actually exploitable in our config, but we need to be a bit more careful than just hoping (due diligence & security-in-depth and all that - an unknown flaw elsewhere could expose one of the issues and that in turn allow some form of DoS, data exfiltration, or arbitrary code execution). We were surprised how much excess stuff, much of it out of date, is in some official images. Sometimes even the required dependencies seemed slower to be updated than we would prefer.
ulzeraj 5y ago
FROM scratch CMD ['/some_golang_or_rust_program_without_dynlibs'] There you go. Never actually tried it but people here told it should work.
- iximiuz 5y ago
  It does work! Actually, it's my favorite type of containers so far https://iximiuz.com/en/posts/not-every-container-has-an-operating-system-inside/ https://iximiuz.com/en/posts/not-every-container-has-an-oper...
- eru 5y ago
  Compare https://en.wikipedia.org/wiki/Unikernel https://en.wikipedia.org/wiki/Unikernel and http://unikernel.org/ http://unikernel.org/
- edflsafoiewq 5y ago
  What's the difference with just using some_golang_or_rust_program_without_dynlibs directly?
  - ulzeraj 5y ago
    I agree but on the other hand I can't really deny how containery stuff offers a standard way of deploying and automation.
  - mrweasel 5y ago
    Maybe you want to run your application on Kubernetes? Nomad can just run an application directly, but I don't believe Kubernetes can.
  - zaphar 5y ago
    In isolation there is no difference. In fact if rust or go apps are the only apps you deploy containers are probably overkill for you. But if you also deploy java/node/python/ruby then containers provide not just reproducible builds but a common deployment method that is language/framework agnostic. And the consistency is often worth it even if some of the deployments don't strictly need docker themselves. Consistency is sometimes an underrated value but it can provide teams a lot of speedup when it's there.
calpaterson 5y ago
Using alpine with Python in production is usually a mistake. Python is considerably slower with musl libc than with glibc and on top of that no one is providing built wheels for musl-based platforms so you need to drag in the entire buildchain to package most projects. Who wants to build numpy from scratch as part of their build pipeline? And having a C compiler present is not great as it allows exploits to escalate their seriousness considerably. Anyway most of these security scanners are bunkum: https://pythonspeed.com/articles/docker-security-scanner/ https://pythonspeed.com/articles/docker-security-scanner/ There are not, in fact, a huge number of known and unpatched vulnerabilities in debian stable. If your tool is finding numerous problems, it is time to look more closely at your tool.
- nickjj 5y ago
  > Python is considerably slower with musl libc than with glibc I've noticed this years ago too. Interacting with the pg package (PostgreSQL) which has a C dependency was noticeably slower on Alpine vs Debian when performing common database operations like selecting data. Since around 2018'ish I've been using Debian (Slim) and haven't looked back. Haven't had a single issue.
  - tyingq 5y ago
    It seems like it's usually down to musl's malloc being slower than glibc's. They are working on a new malloc. https://github.com/richfelker/mallocng-draft https://github.com/richfelker/mallocng-draft
    - megous 5y ago
      New malloc is already in released version of musl.
      - tyingq 5y ago
        Ah, yep, as of 1.2.1 it seems: https://git.musl-libc.org/cgit/musl/commit/?id=ea6d7847ac128f02d1f01ffd01e68341df34ac6e https://git.musl-libc.org/cgit/musl/commit/?id=ea6d7847ac128...
nickjj 5y ago
I'm all for smaller images but his final distroless Python image to run Flask is going to fall apart as soon as his Flask app needs to connect to a database. That's because the official Python PostgreSQL DB package requires C dependencies which get built and referenced when you install the pg package. You also need certain system libraries to exist on your system in order to build them, such as libpq-dev on a Debian based system.
- otabdeveloper4 5y ago
  Just use nix, it solves this problem with minimal disruption for the end-user.
- KptMarchewa 5y ago
  You can use psycopg2-binary package. Not sure about dynamic dependencies though.
  - nickjj 5y ago
    > You can use psycopg2-binary package. Yep but the author of psycopg2 recommends not using it in production. That's mentioned in the package's docs with "The binary package is a practical choice for development and testing but in production it is advised to use the package built from sources." But more generally, this is only 1 package of many that has a C dependency and requires certain system lib files to run.
raesene9 5y ago
One thing to watch out for, when using container scanning tools, is how they handle "unfixed" vulnerabilities in images based on Debian/Ubuntu. Both those distros maintain a list of CVEs that they know of but don't have a patch for. Traditional VA tools (e.g. Nessus) default to not flagging those, but a lot of container scanning tools will default to showing them, so you end up seeing wildly different results. (some more details https://raesene.github.io/blog/2020/11/22/When_Is_A_Vulnerability_Not_A_Vulnerability/ https://raesene.github.io/blog/2020/11/22/When_Is_A_Vulnerab...) Whether you consider this a problem is ofc, dependent on your threat model, but it's one to consider. (full disclosure, I work for a company that makes Trivy , but not on that project :) )
tkiolp4 5y ago
I don’t need slimmer containers, I need slimmer VMs. Honest question: I’m using Vagrant and virtualbox to locally reproduce my cloud infrastructure. I can’t do this just with containers (Some of my servers do not run containers, so I install packages via Ansible and apt, I use systemd, etc.). How do you reproduce locally infrastructure with just containers?
- iximiuz 5y ago
  Containers for sure aren't a replacement for VMs. And indeed, VMs still have and will have legitimate use cases. There is actually a relatively simple way to turn a container into a VM. Maybe you'll find it useful https://iximiuz.com/en/posts/from-docker-container-to-bootable-linux-disk-image/ https://iximiuz.com/en/posts/from-docker-container-to-bootab...
- zozbot234 5y ago
  You can't reproduce your entire configuration with containers, since kernel version might be different. But everything else is fair game given proper namespacing, and AFAICT modern Linux aims for namespacing of all possible resources.
- k__ 5y ago
  Have you looked into Firecracker?
- raesene9 5y ago
  For testing purposes, you can create container environments that are similar to VMs. The way I do it, is to use ansible playbooks, where the individual tasks can be run either on a VM or in a container, depending on my needs (one example https://github.com/raesene/container_sec_workstation https://github.com/raesene/container_sec_workstation)
- ashishbijlani 5y ago
  We've been working on a new hypervisor https://kwarantine.xyz https://kwarantine.xyz that can run strongly isolated containers. It’s still a WIP though.
- 5y ago
sonthonax 5y ago
FROM scratch COPY hello / CMD ["/hello"] This just raises the question of what the point of containerisation for applications with no dependancies.
- raesene9 5y ago
  There's a couple that spring to mind - Consistent packaging/execution environment. containers are delivered and executed in a standard way regardless of target env. - A level of sandboxing from the underlying host. Things like automatically applied Linux namespaces and dropped capabilities if you use Docker/runc, and then more sandboxing automatically applied if you use things like gVsior or firecracker.
  - megous 5y ago
    1) Is not needed if your app has no dependencies (otherwise it would not work in an empty "container") 2) Sandboxing can be done using a systemd service configuration to a large extent.
    - raesene9 5y ago
      1) I was more thinking of the delivery mechanism via centralized registries (i.e. Docker pull works for anything) 2) Sure in theory, you can totally replicate docker's sandboxing with Linux security principles, however containerized deployment gives you an easy to use sensible starting point :)
- dimes 5y ago
  Genuinely curious about alternatives. I have a simple node application that all I really need to do is run npm install and node index.js. Even with such a simple setup, provisioning hosts was a huge pain. Each one needed node installed on it, and I had to deploy the app to each new host. With k8s, I just change the number of hosts I want in a config file and everything just works.
- majewsky 5y ago
  The main appeal of containerisation is not security, the main appeal is having a deployment process that's consistent across the entire team/org/company. Sure, if all you have are Go binaries, you can just run those directly, but if you have some Go binaries and also some Python applications and also some Java beans, you stuff them all into containers and then they can be deployed with the same tooling.
mrweasel 5y ago
Sure slimmer fixes part of the problem, because it reduces the attack surface, but that's not really the issue. Somehow everyone just assume that because something is in a container, then it won't need patching. What really happened is that we moved the patch management responsibility from operations to development. The developers just didn't notice. One issue could be that it breaks many people mental model of containers. Container images are frequently used as a "Works on my machine" and is just bundled up and shipped. It's even broken on Docker Hub. What's the point of an python:3.9 image, when it's never actually updated? Developers base their own Dockerfiles on these base images, but often forget that they need to add an OS update step. I don't understand why images, like the python ones, aren't continuously updated centrally.
- samvher 5y ago
  > but often forget that they need to add an OS update step. Is this standard practice? I've never done this but it makes a lot of sense. Is this basically just adding `RUN apt-get update && apt-get -y upgrade` at the top of the Dockerfile? (Assuming a Debian image.)
  - dsr_ 5y ago
    Technically, this is correct. In practice, it will eventually bite you. Someday some package that you depend on will have a security issue that causes a necessary config change that has a default you don't like. Also, note that you have just changed to a non-reproducible container, where running it on Monday is not guaranteed to be the same as running it next Friday.
    - mrweasel 5y ago
      > Also, note that you have just changed to a non-reproducible container Precisely, and that may be what stops many from just updating their images. I think ideally we'd have centralised updates, perhaps just one a week, and then use tag for that week. You could do all this in-house, but it's a lot of work for small teams. Redundant work.
jrochkind1 5y ago
i am someone who does not use containers, and every time I try to get into it (knowing how popular they are now), I just get... stuck. In feelings of wrongness, among other things. One of which is that I really don't understand the "security story" with regard to patches for vulnerabilities etc. How one is meant to know when a patch is required and how one applies it, what the "conventional" or "best practice" workflow for this is. This article is making me think maybe there's nothing I'm missing....?
- omginternets 5y ago
  I think I might have the complementary partial understanding to yours, so I'm interested in comparing notes. I use containers quite often. In my mind, it's just like managing a physical server -- I'm responsible for configuration/patching/system upgrades/etc -- but with the significant upside that I obtain a (mostly) reproducible environment in which to run the software. This is the part I really can't live without anymore. I don't have to deal with builds from a colleague that don't work on my machine. >How one is meant to know when a patch is required and how one applies it, what the "conventional" or "best practice" workflow for this is. How do you do this without containers?
  - jrochkind1 5y ago
    > How do you do this without containers? So I'm mostly in the "dev" rather than "ops" space (without containers I can stay out of it), but I believe our (non-container) machines are running nightly updates of OS and other packages via `apt-get update` etc. That is definitely a pre-immutability world, which certainly has it's own problems. Alternately, perhaps you deploy to heroku, where you hope [right or wrong] "that's heroku's responsibility, I'm not sure how they deal with it, but i assume they do" Of course, the more "modern" alternative to heroku is some kind of k8 host (or using heroku's docker-based instead of traditional process)... which I gets puts the "patch vulnerabilities in OS" back in your lane again, even if you are paying for hosted K8 of some kind? That's among what I'm trying to get out of paying for PaaS! Looking at other comments in this HN post discussion (I confess I asked versions of this question a couple times)... it seems there is some controversy over how people actually do this with container-based infrastructure, especially on small teams...
itamarst 5y ago
First, the author is being fooled by false positives due to bad scanner settings. Basically there are a huge number of CVEs that are meaningless, and closed by some distros but not others in the CVE databases. The result is a spew of "OMG LOOK AT THIS BADNESS" which security scanners vendors like because it makes them look useful, but is actually just noise. Long version: https://pythonspeed.com/articles/docker-security-scanner/ https://pythonspeed.com/articles/docker-security-scanner/ Second, the author isn't installing security updates! You need to run `apt-get upgrade`, even for official images. Long version: https://pythonspeed.com/articles/security-updates-in-docker/ https://pythonspeed.com/articles/security-updates-in-docker/ Third, Alpine has some issues in some cases, although for Go at least neither of these issues is usually relevant so Alpine is fine. 1. musl can be subtly incompatible with some applications, with annoying bugs. Personal experience: if you using minikube in a WeWork office, Alpine-based (or really, musl-based) containers would fail to resolve DNS inside Kubernetes due to a concatenation of circumstances that was mostly the fault of WeWork's ops team but which glibc handled better than musl. The problem has since been fixed by WeWork, AFAIK. 2. For Python specifically, binary precompiled packages (wheels) won't work on Alpine, which means you ahve to recompile the whole universe, which means container builds are slow. There's a PEP which might get this fixed, but for now, not worth it. https://pythonspeed.com/articles/alpine-docker-python/ https://pythonspeed.com/articles/alpine-docker-python/
- yabones 5y ago
  It's a common misunderstanding: vulnerability != exposure There was a rash of CVEs last year because Alpine or Debian or some base image had some sort of SSH exploit baked in, and a 'security researcher' got CVE numbers for pretty much every major container image. In reality, nobody exposes SSH from a container because that's stupid and not useful. Sure, there absolutely was a vulnerability in those images, but no it was never ever actually exposed. I think this blind focus on CVEs without context is doing harm to the security process, it's taking away from actual work that isn't being done. For example, installing TeamViewer on servers doesn't have a CVE number, but that almost caused a town in Florida to be poisoned.
mac-chaffee 5y ago
I'm struggling with this now too. It gets even crazier when you have dozens of microservices, each in their own container. Imagine being mandated by InfoSec to scan ~24 images or ~10 GB every release. The images are a mixture of python services and some upstream images like redis and mysql. If anyone has an idea on how to make this less painful, I'm all ears.
- itamarst 5y ago
  1. Install security updates (`apt-get upgrade`, or equivalent). 2. Only scan for security problems that _have updates available_. If there are no updates ... there's nothing you can do. Many security problems will never get updates, because it's an obscure problem that e.g. only happens if you install a NFS server in your container (which I assume you're not), or the upstream maintainer has closed it as WONTFIX. Some scanners (e.g. Trivy) have this as a command-line option, or a switch you can toggle in the UI. Long version of the latter: https://pythonspeed.com/articles/docker-security-scanner/ https://pythonspeed.com/articles/docker-security-scanner/
chociej 5y ago
I think most would be served well by using Alpine images. In my experience their quality is quite high. Unfortunately as mentioned, musl is a non-starter for a number of uses.

80 points by r4um 5y ago

Luckily most of those vulnerabilities will be dormant, impossible to reach and exploit. Hopefully. I don't think there's going to be any change in how people package containers. So perhaps what is needed is a dependency-walking Link-Time-Optimization-like tool that can trim down, perhaps by masking rather than removing, the dead code?

iximiuz 5y ago
I still have a hope that increasing awareness of the problem may eventually lead to a reduction in the number of Dockerfiles starting `FROM <something_huge_and_not_really_needed>`.
dspillett 5y ago
> Luckily most of those vulnerabilities will be dormant, impossible to reach and exploit. Hopefully. Our people who use docker for some of our infrastructure have taken to using locally rolled images (from scratch or by updating official ones) for this reason. I don't think any of the issues found when we scanned were actually exploitable in our config, but we need to be a bit more careful than just hoping (due diligence & security-in-depth and all that - an unknown flaw elsewhere could expose one of the issues and that in turn allow some form of DoS, data exfiltration, or arbitrary code execution). We were surprised how much excess stuff, much of it out of date, is in some official images. Sometimes even the required dependencies seemed slower to be updated than we would prefer.

ulzeraj 5y ago

FROM scratch CMD ['/some_golang_or_rust_program_without_dynlibs'] There you go. Never actually tried it but people here told it should work.

iximiuz 5y ago
It does work! Actually, it's my favorite type of containers so far https://iximiuz.com/en/posts/not-every-container-has-an-operating-system-inside/ https://iximiuz.com/en/posts/not-every-container-has-an-oper...
eru 5y ago
Compare https://en.wikipedia.org/wiki/Unikernel https://en.wikipedia.org/wiki/Unikernel and http://unikernel.org/ http://unikernel.org/
edflsafoiewq 5y ago
What's the difference with just using some_golang_or_rust_program_without_dynlibs directly?
- ulzeraj 5y ago
  I agree but on the other hand I can't really deny how containery stuff offers a standard way of deploying and automation.
- mrweasel 5y ago
  Maybe you want to run your application on Kubernetes? Nomad can just run an application directly, but I don't believe Kubernetes can.
- zaphar 5y ago
  In isolation there is no difference. In fact if rust or go apps are the only apps you deploy containers are probably overkill for you. But if you also deploy java/node/python/ruby then containers provide not just reproducible builds but a common deployment method that is language/framework agnostic. And the consistency is often worth it even if some of the deployments don't strictly need docker themselves. Consistency is sometimes an underrated value but it can provide teams a lot of speedup when it's there.

calpaterson 5y ago

Using alpine with Python in production is usually a mistake. Python is considerably slower with musl libc than with glibc and on top of that no one is providing built wheels for musl-based platforms so you need to drag in the entire buildchain to package most projects. Who wants to build numpy from scratch as part of their build pipeline? And having a C compiler present is not great as it allows exploits to escalate their seriousness considerably. Anyway most of these security scanners are bunkum: https://pythonspeed.com/articles/docker-security-scanner/ https://pythonspeed.com/articles/docker-security-scanner/ There are not, in fact, a huge number of known and unpatched vulnerabilities in debian stable. If your tool is finding numerous problems, it is time to look more closely at your tool.

nickjj 5y ago
> Python is considerably slower with musl libc than with glibc I've noticed this years ago too. Interacting with the pg package (PostgreSQL) which has a C dependency was noticeably slower on Alpine vs Debian when performing common database operations like selecting data. Since around 2018'ish I've been using Debian (Slim) and haven't looked back. Haven't had a single issue.
- tyingq 5y ago
  It seems like it's usually down to musl's malloc being slower than glibc's. They are working on a new malloc. https://github.com/richfelker/mallocng-draft https://github.com/richfelker/mallocng-draft
  - megous 5y ago
    New malloc is already in released version of musl.
    - tyingq 5y ago
      Ah, yep, as of 1.2.1 it seems: https://git.musl-libc.org/cgit/musl/commit/?id=ea6d7847ac128f02d1f01ffd01e68341df34ac6e https://git.musl-libc.org/cgit/musl/commit/?id=ea6d7847ac128...

nickjj 5y ago

I'm all for smaller images but his final distroless Python image to run Flask is going to fall apart as soon as his Flask app needs to connect to a database. That's because the official Python PostgreSQL DB package requires C dependencies which get built and referenced when you install the pg package. You also need certain system libraries to exist on your system in order to build them, such as libpq-dev on a Debian based system.

otabdeveloper4 5y ago
Just use nix, it solves this problem with minimal disruption for the end-user.
KptMarchewa 5y ago
You can use psycopg2-binary package. Not sure about dynamic dependencies though.
- nickjj 5y ago
  > You can use psycopg2-binary package. Yep but the author of psycopg2 recommends not using it in production. That's mentioned in the package's docs with "The binary package is a practical choice for development and testing but in production it is advised to use the package built from sources." But more generally, this is only 1 package of many that has a C dependency and requires certain system lib files to run.

raesene9 5y ago

One thing to watch out for, when using container scanning tools, is how they handle "unfixed" vulnerabilities in images based on Debian/Ubuntu. Both those distros maintain a list of CVEs that they know of but don't have a patch for. Traditional VA tools (e.g. Nessus) default to not flagging those, but a lot of container scanning tools will default to showing them, so you end up seeing wildly different results. (some more details https://raesene.github.io/blog/2020/11/22/When_Is_A_Vulnerability_Not_A_Vulnerability/ https://raesene.github.io/blog/2020/11/22/When_Is_A_Vulnerab...) Whether you consider this a problem is ofc, dependent on your threat model, but it's one to consider. (full disclosure, I work for a company that makes Trivy , but not on that project :) )

tkiolp4 5y ago

I don’t need slimmer containers, I need slimmer VMs. Honest question: I’m using Vagrant and virtualbox to locally reproduce my cloud infrastructure. I can’t do this just with containers (Some of my servers do not run containers, so I install packages via Ansible and apt, I use systemd, etc.). How do you reproduce locally infrastructure with just containers?

iximiuz 5y ago
Containers for sure aren't a replacement for VMs. And indeed, VMs still have and will have legitimate use cases. There is actually a relatively simple way to turn a container into a VM. Maybe you'll find it useful https://iximiuz.com/en/posts/from-docker-container-to-bootable-linux-disk-image/ https://iximiuz.com/en/posts/from-docker-container-to-bootab...
zozbot234 5y ago
You can't reproduce your entire configuration with containers, since kernel version might be different. But everything else is fair game given proper namespacing, and AFAICT modern Linux aims for namespacing of all possible resources.
k__ 5y ago
Have you looked into Firecracker?
raesene9 5y ago
For testing purposes, you can create container environments that are similar to VMs. The way I do it, is to use ansible playbooks, where the individual tasks can be run either on a VM or in a container, depending on my needs (one example https://github.com/raesene/container_sec_workstation https://github.com/raesene/container_sec_workstation)
ashishbijlani 5y ago
We've been working on a new hypervisor https://kwarantine.xyz https://kwarantine.xyz that can run strongly isolated containers. It’s still a WIP though.
5y ago

sonthonax 5y ago

FROM scratch COPY hello / CMD ["/hello"] This just raises the question of what the point of containerisation for applications with no dependancies.

raesene9 5y ago
There's a couple that spring to mind - Consistent packaging/execution environment. containers are delivered and executed in a standard way regardless of target env. - A level of sandboxing from the underlying host. Things like automatically applied Linux namespaces and dropped capabilities if you use Docker/runc, and then more sandboxing automatically applied if you use things like gVsior or firecracker.
- megous 5y ago
  1) Is not needed if your app has no dependencies (otherwise it would not work in an empty "container") 2) Sandboxing can be done using a systemd service configuration to a large extent.
  - raesene9 5y ago
    1) I was more thinking of the delivery mechanism via centralized registries (i.e. Docker pull works for anything) 2) Sure in theory, you can totally replicate docker's sandboxing with Linux security principles, however containerized deployment gives you an easy to use sensible starting point :)
dimes 5y ago
Genuinely curious about alternatives. I have a simple node application that all I really need to do is run npm install and node index.js. Even with such a simple setup, provisioning hosts was a huge pain. Each one needed node installed on it, and I had to deploy the app to each new host. With k8s, I just change the number of hosts I want in a config file and everything just works.
majewsky 5y ago
The main appeal of containerisation is not security, the main appeal is having a deployment process that's consistent across the entire team/org/company. Sure, if all you have are Go binaries, you can just run those directly, but if you have some Go binaries and also some Python applications and also some Java beans, you stuff them all into containers and then they can be deployed with the same tooling.

mrweasel 5y ago

Sure slimmer fixes part of the problem, because it reduces the attack surface, but that's not really the issue. Somehow everyone just assume that because something is in a container, then it won't need patching. What really happened is that we moved the patch management responsibility from operations to development. The developers just didn't notice. One issue could be that it breaks many people mental model of containers. Container images are frequently used as a "Works on my machine" and is just bundled up and shipped. It's even broken on Docker Hub. What's the point of an python:3.9 image, when it's never actually updated? Developers base their own Dockerfiles on these base images, but often forget that they need to add an OS update step. I don't understand why images, like the python ones, aren't continuously updated centrally.

samvher 5y ago
> but often forget that they need to add an OS update step. Is this standard practice? I've never done this but it makes a lot of sense. Is this basically just adding `RUN apt-get update && apt-get -y upgrade` at the top of the Dockerfile? (Assuming a Debian image.)
- dsr_ 5y ago
  Technically, this is correct. In practice, it will eventually bite you. Someday some package that you depend on will have a security issue that causes a necessary config change that has a default you don't like. Also, note that you have just changed to a non-reproducible container, where running it on Monday is not guaranteed to be the same as running it next Friday.
  - mrweasel 5y ago
    > Also, note that you have just changed to a non-reproducible container Precisely, and that may be what stops many from just updating their images. I think ideally we'd have centralised updates, perhaps just one a week, and then use tag for that week. You could do all this in-house, but it's a lot of work for small teams. Redundant work.

jrochkind1 5y ago

i am someone who does not use containers, and every time I try to get into it (knowing how popular they are now), I just get... stuck. In feelings of wrongness, among other things. One of which is that I really don't understand the "security story" with regard to patches for vulnerabilities etc. How one is meant to know when a patch is required and how one applies it, what the "conventional" or "best practice" workflow for this is. This article is making me think maybe there's nothing I'm missing....?

omginternets 5y ago
I think I might have the complementary partial understanding to yours, so I'm interested in comparing notes. I use containers quite often. In my mind, it's just like managing a physical server -- I'm responsible for configuration/patching/system upgrades/etc -- but with the significant upside that I obtain a (mostly) reproducible environment in which to run the software. This is the part I really can't live without anymore. I don't have to deal with builds from a colleague that don't work on my machine. >How one is meant to know when a patch is required and how one applies it, what the "conventional" or "best practice" workflow for this is. How do you do this without containers?
- jrochkind1 5y ago
  > How do you do this without containers? So I'm mostly in the "dev" rather than "ops" space (without containers I can stay out of it), but I believe our (non-container) machines are running nightly updates of OS and other packages via `apt-get update` etc. That is definitely a pre-immutability world, which certainly has it's own problems. Alternately, perhaps you deploy to heroku, where you hope [right or wrong] "that's heroku's responsibility, I'm not sure how they deal with it, but i assume they do" Of course, the more "modern" alternative to heroku is some kind of k8 host (or using heroku's docker-based instead of traditional process)... which I gets puts the "patch vulnerabilities in OS" back in your lane again, even if you are paying for hosted K8 of some kind? That's among what I'm trying to get out of paying for PaaS! Looking at other comments in this HN post discussion (I confess I asked versions of this question a couple times)... it seems there is some controversy over how people actually do this with container-based infrastructure, especially on small teams...

itamarst 5y ago

First, the author is being fooled by false positives due to bad scanner settings. Basically there are a huge number of CVEs that are meaningless, and closed by some distros but not others in the CVE databases. The result is a spew of "OMG LOOK AT THIS BADNESS" which security scanners vendors like because it makes them look useful, but is actually just noise. Long version: https://pythonspeed.com/articles/docker-security-scanner/ https://pythonspeed.com/articles/docker-security-scanner/ Second, the author isn't installing security updates! You need to run `apt-get upgrade`, even for official images. Long version: https://pythonspeed.com/articles/security-updates-in-docker/ https://pythonspeed.com/articles/security-updates-in-docker/ Third, Alpine has some issues in some cases, although for Go at least neither of these issues is usually relevant so Alpine is fine. 1. musl can be subtly incompatible with some applications, with annoying bugs. Personal experience: if you using minikube in a WeWork office, Alpine-based (or really, musl-based) containers would fail to resolve DNS inside Kubernetes due to a concatenation of circumstances that was mostly the fault of WeWork's ops team but which glibc handled better than musl. The problem has since been fixed by WeWork, AFAIK. 2. For Python specifically, binary precompiled packages (wheels) won't work on Alpine, which means you ahve to recompile the whole universe, which means container builds are slow. There's a PEP which might get this fixed, but for now, not worth it. https://pythonspeed.com/articles/alpine-docker-python/ https://pythonspeed.com/articles/alpine-docker-python/

yabones 5y ago
It's a common misunderstanding: vulnerability != exposure There was a rash of CVEs last year because Alpine or Debian or some base image had some sort of SSH exploit baked in, and a 'security researcher' got CVE numbers for pretty much every major container image. In reality, nobody exposes SSH from a container because that's stupid and not useful. Sure, there absolutely was a vulnerability in those images, but no it was never ever actually exposed. I think this blind focus on CVEs without context is doing harm to the security process, it's taking away from actual work that isn't being done. For example, installing TeamViewer on servers doesn't have a CVE number, but that almost caused a town in Florida to be poisoned.

mac-chaffee 5y ago

I'm struggling with this now too. It gets even crazier when you have dozens of microservices, each in their own container. Imagine being mandated by InfoSec to scan ~24 images or ~10 GB every release. The images are a mixture of python services and some upstream images like redis and mysql. If anyone has an idea on how to make this less painful, I'm all ears.

itamarst 5y ago
1. Install security updates (`apt-get upgrade`, or equivalent). 2. Only scan for security problems that _have updates available_. If there are no updates ... there's nothing you can do. Many security problems will never get updates, because it's an obscure problem that e.g. only happens if you install a NFS server in your container (which I assume you're not), or the upstream maintainer has closed it as WONTFIX. Some scanners (e.g. Trivy) have this as a command-line option, or a switch you can toggle in the UI. Long version of the latter: https://pythonspeed.com/articles/docker-security-scanner/ https://pythonspeed.com/articles/docker-security-scanner/

chociej 5y ago

I think most would be served well by using Alpine images. In my experience their quality is quite high. Unfortunately as mentioned, musl is a non-starter for a number of uses.