6 ms·

Container Structure Tests: Unit Tests for Docker Images

84 points by mzl 9y ago

mooreds 9y ago
That's awesome. One step closer to treating infrastructure exactly as code. I'd imagine you could stop a build if the docker image generated doesn't have, say, a valid python installation because someone mistyped a command, and that doing so would be quicker than standing up the image and running an external test against it.
- dboreham 9y ago
  To me this sounds crazy. The possibilities from "mistyped commands" are essentially infinite. Instead, test that the container does what it is supposed to do, at least in some minimal way. Then if it doesn't do what it should do because there's no Python, it won't do what it should do and the test will fail. One test to catch the infinite number of typing errors.
crdoconnor 9y ago
This feels to me like the wrong approach. You shouldn't be trying to test if a file exists or a command was run as a 'unit test' (as a sanity check, maybe, but not a test) - that's testing the implementation. You should be testing for the desired behavior that having that command run or having that file there was supposed to achieve. IMHO even the use of containers should probably be considered an implementation detail.
- nitrogen 9y ago
  Agreed. What might be useful is a file list comparison across builds to alert someone if new files are present in a container that weren't in previous versions.
  - mschuster91 9y ago
    Good luck when your container contains anything from Bower or npm, it will get messy on every version upgrade in your dependencies.
  - dlor 9y ago
    Maintainer of structure-test here. We also have a tool that does this! Check out https://github.com/GoogleCloudPlatform/container-diff/ https://github.com/GoogleCloudPlatform/container-diff/
- gedrap 9y ago
  I get you, wrong wording, it's not exactly a standard unit test, etc. But it doesn't change the meaning and intention of it - treat infrastructure as you treat code (I guess that also includes debating whether something is a 'unit' test or 'other' kind of test). Also, having a file on a container (which is probably the easiest test to perform) often is _the desired behavior_ of a command or something else.
  - crdoconnor 9y ago
    >I get you, wrong wording No. Not that. >Also, having a file on a container (which is probably the easiest test to perform) often is _the desired behavior_ No it isn't. The end user doesn't give a damn whether a particular file is on the container. That's an implementation detail. The end user wants: * Pages to load quickly * To not have to face data inconsistency bugs * Pages to operate while the site is under high load * For various services your system connects to to work properly (e.g. clicking 'get two factor code' actually sends an SMS). Checking to see if a certain file is present is pointless if that doesn't lead to the desired system behavior. Moreover, if you have the means to verify the desired system behavior: * The presence of the file, if it was required, can be assumed. * If you swap out a component and stop needing that file to be present to achieve the same behavior your test will still fail even if your system works perfectly. That's an extremely undesirable property to have in a test. Checking for the presence of a file and failing hard if it is not there as part of a build is a sometimes a good way sanity checking a component, but as an outside 'test' of that component it's a bad idea.
    - yorwba 9y ago
      The "end user" of a container image may very well be another developer who wants to run his code in there and depends on the existence of said file.
      - crdoconnor 9y ago
        In which case it would make most sense to build a mock container on top which clearly exhibits the desired behavior of the underlying container and then test the behavior of that.
- puzzle 9y ago
  Sometimes the desired behaviour is actually generating a Docker image with specific contents. Think of the pause container image that every Kubernetes cluster out there uses. To put it in other way, this is for build targets producing Docker images that are needed by an arbitrary set of projects, possibly from third parties. Suppose you have an organization-wide Docker image from which tens of other images from a number of teams are derived. You could add a test to make sure that the ca.crt file is always packaged and never empty or you could let tens of downstream tests and images fail in mysterious ways when it's missing (unless you replicate the check in all projects depending on the base image). The use case is more for infrastructure that explicitly deals with or supports containers. Generic applications, as you say, shouldn't bother.
  - crdoconnor 9y ago
    >Think of the pause container image The pause container is very much an implementation detail. That's sort of my point. Why would you try to test implementation details? >You could add a test to make sure that the ca.crt file is always packaged I could, but why? If I wrote a command to put ca.crt there I would expect it to be there unless somebody deliberately took it out. It would be even better if I wrote a command to put ca.crt in the wrong place and then tested to see that it was in the wrong place. The test would be worse than useless in that case - it would fail on a working container and pass on a broken container. The ca.crt is there because it triggers desired behavior - e.g. that you can connect to services over SSL without errors. Test that instead. >The use case is more for infrastructure that explicitly deals with or supports containers. Personally I'd consider containers themselves an implementation detail most of the time - the exception being when you're building containers for other people to use. I still think it's the wrong approach for that too.
    - dlor 9y ago
      > The ca.crt is there because it triggers desired behavior - e.g. that you can connect to services over SSL without errors. Test that instead. Awesome example here! We have a test that uses container-structure-test to do exactly this over here: https://github.com/GoogleCloudPlatform/distroless/blob/1ea91a52f18cf99cc3f71f55365f6888bb414b05/java/testdata/certs.yaml https://github.com/GoogleCloudPlatform/distroless/blob/1ea91... You can mix and match between tests that check file contents and tests that actually run commands inside the container and make assertions based on their output.
willejs 9y ago
This could be quite useful to define the basic things a container should exhibit. A lot of the time you rely on certain files being present or a specific entrypoint and command combo when running containers under orchestration with sidecars or volume mapping etc. So this could help you define what is needed and why for others who will change your container build steps in future. It could also compliment your Goss or Inspec integration tests quite nicely.
ukoki 9y ago
My standard way of "smoke testing" docker images is to run a verify script at the end that just checks all the expected binaries are in the path, and tries to access any expected environment variables while `-u` is set. https://github.com/EngineerBetter/concourse-up/blob/master/ci/docker/Dockerfile#L49-L50 https://github.com/EngineerBetter/concourse-up/blob/master/c... https://github.com/EngineerBetter/concourse-up/blob/master/ci/docker/verify_image.sh https://github.com/EngineerBetter/concourse-up/blob/master/c...
- dlor 9y ago
  Maintainer of container-structure-test here. Cool example! We started with something like this, but ran into some issues when trying to test images that don't contain a shell.
- perfmode 9y ago
  a benefit of specifying this declaratively is that you can change the implementation without changing syntax.
v_lisivka 9y ago
Just package content into RPM and use rpm to perform these checks at installation time as usual. Moreover, Dockerfile will be simple "yum install package" command.
Samarth1234 9y ago
This could be very useful given the docker images are increasing day by day.

84 points by mzl 9y ago

That's awesome. One step closer to treating infrastructure exactly as code. I'd imagine you could stop a build if the docker image generated doesn't have, say, a valid python installation because someone mistyped a command, and that doing so would be quicker than standing up the image and running an external test against it.

dboreham 9y ago
To me this sounds crazy. The possibilities from "mistyped commands" are essentially infinite. Instead, test that the container does what it is supposed to do, at least in some minimal way. Then if it doesn't do what it should do because there's no Python, it won't do what it should do and the test will fail. One test to catch the infinite number of typing errors.

crdoconnor 9y ago

This feels to me like the wrong approach. You shouldn't be trying to test if a file exists or a command was run as a 'unit test' (as a sanity check, maybe, but not a test) - that's testing the implementation. You should be testing for the desired behavior that having that command run or having that file there was supposed to achieve. IMHO even the use of containers should probably be considered an implementation detail.

nitrogen 9y ago
Agreed. What might be useful is a file list comparison across builds to alert someone if new files are present in a container that weren't in previous versions.
- mschuster91 9y ago
  Good luck when your container contains anything from Bower or npm, it will get messy on every version upgrade in your dependencies.
- dlor 9y ago
  Maintainer of structure-test here. We also have a tool that does this! Check out https://github.com/GoogleCloudPlatform/container-diff/ https://github.com/GoogleCloudPlatform/container-diff/
gedrap 9y ago
I get you, wrong wording, it's not exactly a standard unit test, etc. But it doesn't change the meaning and intention of it - treat infrastructure as you treat code (I guess that also includes debating whether something is a 'unit' test or 'other' kind of test). Also, having a file on a container (which is probably the easiest test to perform) often is _the desired behavior_ of a command or something else.
- crdoconnor 9y ago
  >I get you, wrong wording No. Not that. >Also, having a file on a container (which is probably the easiest test to perform) often is _the desired behavior_ No it isn't. The end user doesn't give a damn whether a particular file is on the container. That's an implementation detail. The end user wants: * Pages to load quickly * To not have to face data inconsistency bugs * Pages to operate while the site is under high load * For various services your system connects to to work properly (e.g. clicking 'get two factor code' actually sends an SMS). Checking to see if a certain file is present is pointless if that doesn't lead to the desired system behavior. Moreover, if you have the means to verify the desired system behavior: * The presence of the file, if it was required, can be assumed. * If you swap out a component and stop needing that file to be present to achieve the same behavior your test will still fail even if your system works perfectly. That's an extremely undesirable property to have in a test. Checking for the presence of a file and failing hard if it is not there as part of a build is a sometimes a good way sanity checking a component, but as an outside 'test' of that component it's a bad idea.
  - yorwba 9y ago
    The "end user" of a container image may very well be another developer who wants to run his code in there and depends on the existence of said file.
    - crdoconnor 9y ago
      In which case it would make most sense to build a mock container on top which clearly exhibits the desired behavior of the underlying container and then test the behavior of that.
puzzle 9y ago
Sometimes the desired behaviour is actually generating a Docker image with specific contents. Think of the pause container image that every Kubernetes cluster out there uses. To put it in other way, this is for build targets producing Docker images that are needed by an arbitrary set of projects, possibly from third parties. Suppose you have an organization-wide Docker image from which tens of other images from a number of teams are derived. You could add a test to make sure that the ca.crt file is always packaged and never empty or you could let tens of downstream tests and images fail in mysterious ways when it's missing (unless you replicate the check in all projects depending on the base image). The use case is more for infrastructure that explicitly deals with or supports containers. Generic applications, as you say, shouldn't bother.
- crdoconnor 9y ago
  >Think of the pause container image The pause container is very much an implementation detail. That's sort of my point. Why would you try to test implementation details? >You could add a test to make sure that the ca.crt file is always packaged I could, but why? If I wrote a command to put ca.crt there I would expect it to be there unless somebody deliberately took it out. It would be even better if I wrote a command to put ca.crt in the wrong place and then tested to see that it was in the wrong place. The test would be worse than useless in that case - it would fail on a working container and pass on a broken container. The ca.crt is there because it triggers desired behavior - e.g. that you can connect to services over SSL without errors. Test that instead. >The use case is more for infrastructure that explicitly deals with or supports containers. Personally I'd consider containers themselves an implementation detail most of the time - the exception being when you're building containers for other people to use. I still think it's the wrong approach for that too.
  - dlor 9y ago
    > The ca.crt is there because it triggers desired behavior - e.g. that you can connect to services over SSL without errors. Test that instead. Awesome example here! We have a test that uses container-structure-test to do exactly this over here: https://github.com/GoogleCloudPlatform/distroless/blob/1ea91a52f18cf99cc3f71f55365f6888bb414b05/java/testdata/certs.yaml https://github.com/GoogleCloudPlatform/distroless/blob/1ea91... You can mix and match between tests that check file contents and tests that actually run commands inside the container and make assertions based on their output.

willejs 9y ago

This could be quite useful to define the basic things a container should exhibit. A lot of the time you rely on certain files being present or a specific entrypoint and command combo when running containers under orchestration with sidecars or volume mapping etc. So this could help you define what is needed and why for others who will change your container build steps in future. It could also compliment your Goss or Inspec integration tests quite nicely.

ukoki 9y ago

My standard way of "smoke testing" docker images is to run a verify script at the end that just checks all the expected binaries are in the path, and tries to access any expected environment variables while `-u` is set. https://github.com/EngineerBetter/concourse-up/blob/master/ci/docker/Dockerfile#L49-L50 https://github.com/EngineerBetter/concourse-up/blob/master/c... https://github.com/EngineerBetter/concourse-up/blob/master/ci/docker/verify_image.sh https://github.com/EngineerBetter/concourse-up/blob/master/c...

dlor 9y ago
Maintainer of container-structure-test here. Cool example! We started with something like this, but ran into some issues when trying to test images that don't contain a shell.
perfmode 9y ago
a benefit of specifying this declaratively is that you can change the implementation without changing syntax.

v_lisivka 9y ago

Just package content into RPM and use rpm to perform these checks at installation time as usual. Moreover, Dockerfile will be simple "yum install package" command.

Samarth1234 9y ago

This could be very useful given the docker images are increasing day by day.