4 ms·

I fully agree with your last point. In my experience, the all-versions-packaged approach of containers actually leads to a worse patch state than a simple Ubunt

I fully agree with your last point. In my experience, the all-versions-packaged approach of containers actually leads to a worse patch state than a simple Ubuntu or Debian installation with unattended-upgrades.

greedo 4y ago
I've tried to explain to our Devops guys that they need to automate their patching so that a new container is fully patched before deployment. They had no idea what I was talking about. I said, "you have containers that run for months, right? And they aren't modified at all after deployment, right? So how do you address new vulnerabilities? How often are you patching your base image?" Just clueless looks.
- monkpit 4y ago
  How would you accomplish this exactly? Your use case may vary, but in my mind dependencies shouldn’t just be upgraded by some automated system without some sort of feedback mechanism (like tests). If you’re going to run tests against the new dependencies, then why not just shift that whole activity to the development process itself (not the deployment system)? When dependencies are sufficiently different from the existing prod dependencies, you can release a new version, instead of expecting ops/devops to take care of your dependencies for you.
  - greedo 4y ago
    I'm actually agnostic as to how it's achieved. I just think that the idea of running a container for three months without patching the underlying OS is nuts. Generally our guys only redeploy a container when there's been an update, so that OS is pretty stale compared to what I'm running on traditional VMs. It really was more of a reflection on how DevOps can become too focused on application development and deployment, as opposed to deploying holistic systems. We have coders writing apps that feed into our CI/CD system, we have admins who manage kubernetes/Swarm, and we have traditional sysadmins managing VMs. All with differing priorities and focuses.
    - monkpit 4y ago
      Ah, the OS itself. I thought you meant the application dependencies.
      - deleted 4y ago
        [deleted]
    - Spivak 4y ago
      I kinda stopped worrying too much about it because the "OS" (really userspace) that lives in a container isn't in a position to be exploited like the underlying VM OS is. Nothing in the container is privileged and there are no security boundaries that exist within a container. You should care exactly as much about container image updates as you do gem/npm/pip updates and I guarantee you have months old deps pinned in a lockfile somewhere. What people hear is "Oh my god you're still running Ubuntu 14.04?!?" but what is actually happening is all the defenses and security boundaries are being enforced by a fully patched RHEL 7 instance with SELinux, the completely unprivileged application code just happens to have an old version of libz bundled with it. So for sure the application is in a position to be compromised depending on how old the libs and the vulns that might exist but that's the developer's concern exactly like their NPM deps.
      - deleted 4y ago
        [deleted]
      - xyzzy123 4y ago
        Ok you got me (I laughed) I'm just so sick of the container security shills; in particular I'm tired of people selling debian advisories with zero value add on enterprise contract.
      - greedo 4y ago
        So an application runs in a container that has a vulnerable OS. This has network access, DB access, NFS or SMB access. All valuable stuff. And then the container's OS gets breached due to the vulnerability. Sure, the VM hosting the Docker containers is secure, but that doesn't really matter if the breached container had access to PII or other valuable data. This idea that people have about containers being 'the "OS (really userspace)' is fundamentally wrong. It's the equivalent of a VM, just (hopefully) stripped down to the bare essentials. There's no magic that protects it. The Docker host sure doesn't protect it, and in fact the Docker host can be vulnerable to exploitation from its containers, just as a vSphere host (or any other hypervisor) can be exploited if its VMs are insecure.
  - _joel 4y ago
    If you're using an upstream image from a distro, pull that down, update it, push it back to a registry, then use that as your base image. Do that every day in a CI/CD (or just cron) system, such that you've got logs, auditing and fires off alerts if something breaks. Add update steps to the builds of your final image containers too at the start, to catch any delta that may have happened during last base image patch. There are also container scanning tools but they can produce some false positives sometimes but useful nonetheless. You still run tests as normal against the new images before promoting, just like normal.
    - greedo 4y ago
      Exactly what I've been trying to push to the team.
  - deleted 4y ago
    [deleted]
- xyzzy123 4y ago
  If that matters why hasn't the internet exploded yet? Hardly anyone is doing that thing and it seems to not matter. People who panic only when they see $STUPIDLY_NAMED_ISSUE seem fine.
  - icedchai 4y ago
    Unless there's a vulnerability in something internet facing, you can go a very long time without updating anything. So, yes, it doesn't matter... except when it does. I've logged into production systems with multi year uptimes, with updates that haven't been applied in just as long. Many orgs have a if it's not broke, don't fix it policy. Applying needless updates are just as likely to screw something up.
    - greedo 4y ago
      Many orgs get breached (or have no way to detect if they've been breached). It depends on both your industry and your risk tolerance. We generally apply every security patch that comes along for RHEL. Application patching is different, it depends on the severity and what the vendor will support. If we can't patch, we try to apply compensating controls to mitigate the risk. And the pride we all used to have with long uptimes is really a poor badge of courage. It shows a misplaced trust in either your security, or the threat model you face. OS security patches (at least for RHEL) pose little risk of breaking applications. In the last 15 years, I've never once had to roll back a security patch from RH.
      - db48x 4y ago
        Security patches can be applied even to the kernel without restarting anything though, so having a high uptime is not mutually exclusive with staying up to date. Of course, you do have to pay extra for live patching.
        greedo 4y ago
        I'm still leery of ksplice etc. And the cost is a factor too.

greedo 4y ago

I've tried to explain to our Devops guys that they need to automate their patching so that a new container is fully patched before deployment. They had no idea what I was talking about. I said, "you have containers that run for months, right? And they aren't modified at all after deployment, right? So how do you address new vulnerabilities? How often are you patching your base image?" Just clueless looks.

monkpit 4y ago
How would you accomplish this exactly? Your use case may vary, but in my mind dependencies shouldn’t just be upgraded by some automated system without some sort of feedback mechanism (like tests). If you’re going to run tests against the new dependencies, then why not just shift that whole activity to the development process itself (not the deployment system)? When dependencies are sufficiently different from the existing prod dependencies, you can release a new version, instead of expecting ops/devops to take care of your dependencies for you.
- greedo 4y ago
  I'm actually agnostic as to how it's achieved. I just think that the idea of running a container for three months without patching the underlying OS is nuts. Generally our guys only redeploy a container when there's been an update, so that OS is pretty stale compared to what I'm running on traditional VMs. It really was more of a reflection on how DevOps can become too focused on application development and deployment, as opposed to deploying holistic systems. We have coders writing apps that feed into our CI/CD system, we have admins who manage kubernetes/Swarm, and we have traditional sysadmins managing VMs. All with differing priorities and focuses.
  - monkpit 4y ago
    Ah, the OS itself. I thought you meant the application dependencies.
    - deleted 4y ago
      [deleted]
  - Spivak 4y ago
    I kinda stopped worrying too much about it because the "OS" (really userspace) that lives in a container isn't in a position to be exploited like the underlying VM OS is. Nothing in the container is privileged and there are no security boundaries that exist within a container. You should care exactly as much about container image updates as you do gem/npm/pip updates and I guarantee you have months old deps pinned in a lockfile somewhere. What people hear is "Oh my god you're still running Ubuntu 14.04?!?" but what is actually happening is all the defenses and security boundaries are being enforced by a fully patched RHEL 7 instance with SELinux, the completely unprivileged application code just happens to have an old version of libz bundled with it. So for sure the application is in a position to be compromised depending on how old the libs and the vulns that might exist but that's the developer's concern exactly like their NPM deps.
    - deleted 4y ago
      [deleted]
    - xyzzy123 4y ago
      Ok you got me (I laughed) I'm just so sick of the container security shills; in particular I'm tired of people selling debian advisories with zero value add on enterprise contract.
    - greedo 4y ago
      So an application runs in a container that has a vulnerable OS. This has network access, DB access, NFS or SMB access. All valuable stuff. And then the container's OS gets breached due to the vulnerability. Sure, the VM hosting the Docker containers is secure, but that doesn't really matter if the breached container had access to PII or other valuable data. This idea that people have about containers being 'the "OS (really userspace)' is fundamentally wrong. It's the equivalent of a VM, just (hopefully) stripped down to the bare essentials. There's no magic that protects it. The Docker host sure doesn't protect it, and in fact the Docker host can be vulnerable to exploitation from its containers, just as a vSphere host (or any other hypervisor) can be exploited if its VMs are insecure.
- _joel 4y ago
  If you're using an upstream image from a distro, pull that down, update it, push it back to a registry, then use that as your base image. Do that every day in a CI/CD (or just cron) system, such that you've got logs, auditing and fires off alerts if something breaks. Add update steps to the builds of your final image containers too at the start, to catch any delta that may have happened during last base image patch. There are also container scanning tools but they can produce some false positives sometimes but useful nonetheless. You still run tests as normal against the new images before promoting, just like normal.
  - greedo 4y ago
    Exactly what I've been trying to push to the team.
- deleted 4y ago
  [deleted]
xyzzy123 4y ago
If that matters why hasn't the internet exploded yet? Hardly anyone is doing that thing and it seems to not matter. People who panic only when they see $STUPIDLY_NAMED_ISSUE seem fine.
- icedchai 4y ago
  Unless there's a vulnerability in something internet facing, you can go a very long time without updating anything. So, yes, it doesn't matter... except when it does. I've logged into production systems with multi year uptimes, with updates that haven't been applied in just as long. Many orgs have a if it's not broke, don't fix it policy. Applying needless updates are just as likely to screw something up.
  - greedo 4y ago
    Many orgs get breached (or have no way to detect if they've been breached). It depends on both your industry and your risk tolerance. We generally apply every security patch that comes along for RHEL. Application patching is different, it depends on the severity and what the vendor will support. If we can't patch, we try to apply compensating controls to mitigate the risk. And the pride we all used to have with long uptimes is really a poor badge of courage. It shows a misplaced trust in either your security, or the threat model you face. OS security patches (at least for RHEL) pose little risk of breaking applications. In the last 15 years, I've never once had to roll back a security patch from RH.
    - db48x 4y ago
      Security patches can be applied even to the kernel without restarting anything though, so having a high uptime is not mutually exclusive with staying up to date. Of course, you do have to pay extra for live patching.
      - greedo 4y ago
        I'm still leery of ksplice etc. And the cost is a factor too.