5 ms·

How would you accomplish this exactly? Your use case may vary, but in my mind dependencies shouldn’t just be upgraded by some automated system without some sort

How would you accomplish this exactly? Your use case may vary, but in my mind dependencies shouldn’t just be upgraded by some automated system without some sort of feedback mechanism (like tests). If you’re going to run tests against the new dependencies, then why not just shift that whole activity to the development process itself (not the deployment system)? When dependencies are sufficiently different from the existing prod dependencies, you can release a new version, instead of expecting ops/devops to take care of your dependencies for you.

greedo 4y ago
I'm actually agnostic as to how it's achieved. I just think that the idea of running a container for three months without patching the underlying OS is nuts. Generally our guys only redeploy a container when there's been an update, so that OS is pretty stale compared to what I'm running on traditional VMs. It really was more of a reflection on how DevOps can become too focused on application development and deployment, as opposed to deploying holistic systems. We have coders writing apps that feed into our CI/CD system, we have admins who manage kubernetes/Swarm, and we have traditional sysadmins managing VMs. All with differing priorities and focuses.
- monkpit 4y ago
  Ah, the OS itself. I thought you meant the application dependencies.
  - deleted 4y ago
    [deleted]
- Spivak 4y ago
  I kinda stopped worrying too much about it because the "OS" (really userspace) that lives in a container isn't in a position to be exploited like the underlying VM OS is. Nothing in the container is privileged and there are no security boundaries that exist within a container. You should care exactly as much about container image updates as you do gem/npm/pip updates and I guarantee you have months old deps pinned in a lockfile somewhere. What people hear is "Oh my god you're still running Ubuntu 14.04?!?" but what is actually happening is all the defenses and security boundaries are being enforced by a fully patched RHEL 7 instance with SELinux, the completely unprivileged application code just happens to have an old version of libz bundled with it. So for sure the application is in a position to be compromised depending on how old the libs and the vulns that might exist but that's the developer's concern exactly like their NPM deps.
  - deleted 4y ago
    [deleted]
  - xyzzy123 4y ago
    Ok you got me (I laughed) I'm just so sick of the container security shills; in particular I'm tired of people selling debian advisories with zero value add on enterprise contract.
  - greedo 4y ago
    So an application runs in a container that has a vulnerable OS. This has network access, DB access, NFS or SMB access. All valuable stuff. And then the container's OS gets breached due to the vulnerability. Sure, the VM hosting the Docker containers is secure, but that doesn't really matter if the breached container had access to PII or other valuable data. This idea that people have about containers being 'the "OS (really userspace)' is fundamentally wrong. It's the equivalent of a VM, just (hopefully) stripped down to the bare essentials. There's no magic that protects it. The Docker host sure doesn't protect it, and in fact the Docker host can be vulnerable to exploitation from its containers, just as a vSphere host (or any other hypervisor) can be exploited if its VMs are insecure.
    - Spivak 4y ago
      > And then the container's OS gets breached This can't happen. There is no "OS" actually running. It's a bunch of binaries and files sitting in a tar archive that the process running can access and link against. What are you going to breach? All the networking, firewalls, kernel, remote access like SSH, and privilege management is the hosts. The only process running in the container is the application. What you mean is "what if the application running in the container is compromised?" For example what if there's a RCE vuln in the version of libcurl that the app links against? Absolutely! Now you've got a huge problem but this is exactly the same as if there was an RCE in leftPad. In container world that's now the developer's problem instead of yours. Everything in the container is vendored, just pretend it was all statically linked. It's not like an VM where ops patches it. > It's the equivalent of a VM, just (hopefully) stripped down to the bare essentials No, no 1000 times no. It's in all ways the equivalent of a process (or tree of processes) running on the host machine. From a security perspective a container is no different than any other binary running on the host. You have it exactly right, there is exactly zero magic and no inherent security. You should lose exactly as much sleep over outdated container images as you lose over outdated rubygems in your developers' Bundle.lock. Whether that means a lot or a little depends on the company and how security critical the application is.
_joel 4y ago
If you're using an upstream image from a distro, pull that down, update it, push it back to a registry, then use that as your base image. Do that every day in a CI/CD (or just cron) system, such that you've got logs, auditing and fires off alerts if something breaks. Add update steps to the builds of your final image containers too at the start, to catch any delta that may have happened during last base image patch. There are also container scanning tools but they can produce some false positives sometimes but useful nonetheless. You still run tests as normal against the new images before promoting, just like normal.
- greedo 4y ago
  Exactly what I've been trying to push to the team.
deleted 4y ago
[deleted]

greedo 4y ago

I'm actually agnostic as to how it's achieved. I just think that the idea of running a container for three months without patching the underlying OS is nuts. Generally our guys only redeploy a container when there's been an update, so that OS is pretty stale compared to what I'm running on traditional VMs. It really was more of a reflection on how DevOps can become too focused on application development and deployment, as opposed to deploying holistic systems. We have coders writing apps that feed into our CI/CD system, we have admins who manage kubernetes/Swarm, and we have traditional sysadmins managing VMs. All with differing priorities and focuses.

monkpit 4y ago
Ah, the OS itself. I thought you meant the application dependencies.
- deleted 4y ago
  [deleted]
Spivak 4y ago
I kinda stopped worrying too much about it because the "OS" (really userspace) that lives in a container isn't in a position to be exploited like the underlying VM OS is. Nothing in the container is privileged and there are no security boundaries that exist within a container. You should care exactly as much about container image updates as you do gem/npm/pip updates and I guarantee you have months old deps pinned in a lockfile somewhere. What people hear is "Oh my god you're still running Ubuntu 14.04?!?" but what is actually happening is all the defenses and security boundaries are being enforced by a fully patched RHEL 7 instance with SELinux, the completely unprivileged application code just happens to have an old version of libz bundled with it. So for sure the application is in a position to be compromised depending on how old the libs and the vulns that might exist but that's the developer's concern exactly like their NPM deps.
- deleted 4y ago
  [deleted]
- xyzzy123 4y ago
  Ok you got me (I laughed) I'm just so sick of the container security shills; in particular I'm tired of people selling debian advisories with zero value add on enterprise contract.
- greedo 4y ago
  So an application runs in a container that has a vulnerable OS. This has network access, DB access, NFS or SMB access. All valuable stuff. And then the container's OS gets breached due to the vulnerability. Sure, the VM hosting the Docker containers is secure, but that doesn't really matter if the breached container had access to PII or other valuable data. This idea that people have about containers being 'the "OS (really userspace)' is fundamentally wrong. It's the equivalent of a VM, just (hopefully) stripped down to the bare essentials. There's no magic that protects it. The Docker host sure doesn't protect it, and in fact the Docker host can be vulnerable to exploitation from its containers, just as a vSphere host (or any other hypervisor) can be exploited if its VMs are insecure.
  - Spivak 4y ago
    > And then the container's OS gets breached This can't happen. There is no "OS" actually running. It's a bunch of binaries and files sitting in a tar archive that the process running can access and link against. What are you going to breach? All the networking, firewalls, kernel, remote access like SSH, and privilege management is the hosts. The only process running in the container is the application. What you mean is "what if the application running in the container is compromised?" For example what if there's a RCE vuln in the version of libcurl that the app links against? Absolutely! Now you've got a huge problem but this is exactly the same as if there was an RCE in leftPad. In container world that's now the developer's problem instead of yours. Everything in the container is vendored, just pretend it was all statically linked. It's not like an VM where ops patches it. > It's the equivalent of a VM, just (hopefully) stripped down to the bare essentials No, no 1000 times no. It's in all ways the equivalent of a process (or tree of processes) running on the host machine. From a security perspective a container is no different than any other binary running on the host. You have it exactly right, there is exactly zero magic and no inherent security. You should lose exactly as much sleep over outdated container images as you lose over outdated rubygems in your developers' Bundle.lock. Whether that means a lot or a little depends on the company and how security critical the application is.

_joel 4y ago

If you're using an upstream image from a distro, pull that down, update it, push it back to a registry, then use that as your base image. Do that every day in a CI/CD (or just cron) system, such that you've got logs, auditing and fires off alerts if something breaks. Add update steps to the builds of your final image containers too at the start, to catch any delta that may have happened during last base image patch. There are also container scanning tools but they can produce some false positives sometimes but useful nonetheless. You still run tests as normal against the new images before promoting, just like normal.

greedo 4y ago
Exactly what I've been trying to push to the team.

deleted 4y ago

[deleted]