4 ms·

The other replies here are great but let me give you the layman's version of what "immutable infrastructure" means: If it works for me it works for everyone.

The other replies here are great but let me give you the layman's version of what "immutable infrastructure" means: If it works for me it works for everyone. You never patch or upgrade immutable infrastructure. You just replace what you've got with a new VM or container. Containers being preferred because they can be started & stopped near instantaneously and there's nothing like a virtual BIOS that could have different configurations like with VMs. You don't stand up a VM or container then "log in to configure it". Once the VM is "up" that's it. You're done. At that point you just need to point your load balancers/DNS at the new stuff then take down the old stuff. One interesting aspect of immutable infrastructure such as this is that it is completely incompatible with loads of existing security policies and what would have been considered "best practices" just a few years ago. For example, you might have a security policy that states that everything must be scanned within 30 days for malware/out-of-date packages/whatever. Yet with immutable infrastructure your hosts or containers may only be up for a few days before being replaced! So when your security team freaks out because none of your hosts/containers are showing up in their systems you'll have a lot of explaining to do =D "We need to scan your hosts so we can ensure that you're installing security patches." "We don't do that." "You don't install security patches?!?" "Yeah, well, you see..." Trust me when I say that trying to explain how it all works and why it's more secure than old school deployments is not easy!

dastbe 11y ago
this is why your security team should be integrated into how you (securely!) build and deploy software. in this specific case, that would mean maintaining the OS layer Dockerfile and scanning the dependency tree for build artifacts.
- riskable 11y ago
  Yeah, it's actually a lot more complicated than that. Let's assume the security team creates the Dockerfile. It'll look like this: FROM ubuntu WORKDIR ${foo} # WORKDIR /bar RUN developer_script.sh So now with each new container update you'd need someone from the security team to audit/review `developer_script.sh`. It's kind of pointless if your goal is fast deployments. If you just need to make sure your developers don't make a mistake in terms of securely configuring their containers (and making sure to always use the latest software) then you simply scan them before the canary stage. The problem there is, "what are you looking for?" Also, there exists only one tool to scan Docker containers (OpenSCAP) and if your security team doesn't like it, well, you're screwed: https://github.com/OpenSCAP/container-compliance https://github.com/OpenSCAP/container-compliance The other problem is that OpenSCAP only checks the container's packages for compliance. It doesn't actually scan the container's filesystem for things like JREs and bundled libs. So if you're using Docker best practices by keeping your images as minimal as possible you may not even have a package tool inside your containers. In that case how do you check for things like out-of-date versions of Java? Another problem is you assume there's some modicum of control over what's inside the containers before they're deployed. We're in charge of creating the infrastructure for running containers with the promise that end users (who would be various application teams) can make their own containers (or at least their own Dockerfiles) and deploy them on our infrastructure. We can work with them to help develop, say, a Kubernetes pod config for their app but as far as what their app is or what gets bundled with it we'll have no knowledge after the first successful deployment.
  - dastbe 11y ago
    I never said you need or should have the package tools inside your container, but you do need to be able to track the lineage of what was put into those containers so you can easily search for ex. deployed container revisions with an out of date Java version. This is where you need a team devoted to build that creates this kind of infrastructure. Otherwise, you end up with difficult to answer questions about your infrastructure. tl;dr I assume there's some control because there should be.
cyphar 11y ago
> So when your security team freaks out because none of your hosts/containers are showing up in their systems you'll have a lot of explaining to do =D > "We need to scan your hosts so we can ensure that you're installing security patches." > "We don't do that." > "You don't install security patches?!?" > "Yeah, well, you see..." If you use a tool like zypper-docker, you can create a new image quickly that applies just the security patches. Currently it only works for SUSE-derived containers but we're planning on making it distribution agnostic. There's also some work we're doing on connecting Docker containers running on SUSE enterprises to connect with SUSE Manager (aka spacewalk), so they would show up in their systems. You can find most of this stuff in github.com/SUSE.
- riskable 11y ago
  zypper-docker is kinda pointless. Just re-build from your Dockerfile using the latest upstream image. It takes like 10 seconds (depending on how many external dependencies need to be fetched). Our Docker registry pulls down the latest images (that we use with our Dockerfiles) multiple times daily. So if the author of the Dockerfile just puts "FROM whatever:latest" they are guaranteed to have all the latest/patched software whenever they "docker build".
  - goldbrick 11y ago
    But then you risk having untold other packages move under your feet. This is what concepts like "unattended-upgrades" address.
    - riskable 11y ago
      The risk associated with breakage from an automated upgrade is an order of magnitude less than the risk associated with being hacked because you weren't keeping up to date.
  - cyphar 11y ago
    Sometimes you don't want to update all of your dependencies (isn't that the whole point of Docker), you might want to just apply security updates. Just putting "FROM whatever:latest" is a bad idea, because you don't know whether the "latest" will suddenly break your image. The point of zypper-docker is to quickly roll out security fixes with essentially no downtime. zypper-docker also allows you to quickly check the health of all of the images and containers on your servers, so you can figure out which ones need updating. Also, zypper-docker allows you to apply RPM patches (something that's quite crucial for enterprises). So no, it's not pointless. In fact it solves a problem that not many people have been working on solving: updating containers.

dastbe 11y ago

this is why your security team should be integrated into how you (securely!) build and deploy software. in this specific case, that would mean maintaining the OS layer Dockerfile and scanning the dependency tree for build artifacts.

riskable 11y ago
Yeah, it's actually a lot more complicated than that. Let's assume the security team creates the Dockerfile. It'll look like this: FROM ubuntu WORKDIR ${foo} # WORKDIR /bar RUN developer_script.sh So now with each new container update you'd need someone from the security team to audit/review `developer_script.sh`. It's kind of pointless if your goal is fast deployments. If you just need to make sure your developers don't make a mistake in terms of securely configuring their containers (and making sure to always use the latest software) then you simply scan them before the canary stage. The problem there is, "what are you looking for?" Also, there exists only one tool to scan Docker containers (OpenSCAP) and if your security team doesn't like it, well, you're screwed: https://github.com/OpenSCAP/container-compliance https://github.com/OpenSCAP/container-compliance The other problem is that OpenSCAP only checks the container's packages for compliance. It doesn't actually scan the container's filesystem for things like JREs and bundled libs. So if you're using Docker best practices by keeping your images as minimal as possible you may not even have a package tool inside your containers. In that case how do you check for things like out-of-date versions of Java? Another problem is you assume there's some modicum of control over what's inside the containers before they're deployed. We're in charge of creating the infrastructure for running containers with the promise that end users (who would be various application teams) can make their own containers (or at least their own Dockerfiles) and deploy them on our infrastructure. We can work with them to help develop, say, a Kubernetes pod config for their app but as far as what their app is or what gets bundled with it we'll have no knowledge after the first successful deployment.
- dastbe 11y ago
  I never said you need or should have the package tools inside your container, but you do need to be able to track the lineage of what was put into those containers so you can easily search for ex. deployed container revisions with an out of date Java version. This is where you need a team devoted to build that creates this kind of infrastructure. Otherwise, you end up with difficult to answer questions about your infrastructure. tl;dr I assume there's some control because there should be.

cyphar 11y ago

> So when your security team freaks out because none of your hosts/containers are showing up in their systems you'll have a lot of explaining to do =D > "We need to scan your hosts so we can ensure that you're installing security patches." > "We don't do that." > "You don't install security patches?!?" > "Yeah, well, you see..." If you use a tool like zypper-docker, you can create a new image quickly that applies just the security patches. Currently it only works for SUSE-derived containers but we're planning on making it distribution agnostic. There's also some work we're doing on connecting Docker containers running on SUSE enterprises to connect with SUSE Manager (aka spacewalk), so they would show up in their systems. You can find most of this stuff in github.com/SUSE.

riskable 11y ago
zypper-docker is kinda pointless. Just re-build from your Dockerfile using the latest upstream image. It takes like 10 seconds (depending on how many external dependencies need to be fetched). Our Docker registry pulls down the latest images (that we use with our Dockerfiles) multiple times daily. So if the author of the Dockerfile just puts "FROM whatever:latest" they are guaranteed to have all the latest/patched software whenever they "docker build".
- goldbrick 11y ago
  But then you risk having untold other packages move under your feet. This is what concepts like "unattended-upgrades" address.
  - riskable 11y ago
    The risk associated with breakage from an automated upgrade is an order of magnitude less than the risk associated with being hacked because you weren't keeping up to date.
- cyphar 11y ago
  Sometimes you don't want to update all of your dependencies (isn't that the whole point of Docker), you might want to just apply security updates. Just putting "FROM whatever:latest" is a bad idea, because you don't know whether the "latest" will suddenly break your image. The point of zypper-docker is to quickly roll out security fixes with essentially no downtime. zypper-docker also allows you to quickly check the health of all of the images and containers on your servers, so you can figure out which ones need updating. Also, zypper-docker allows you to apply RPM patches (something that's quite crucial for enterprises). So no, it's not pointless. In fact it solves a problem that not many people have been working on solving: updating containers.