11 ms·

Docker-slim: Minify your Docker container image without changing anything

379 points by LinuxBender 7y ago

gtirloni 7y ago
It seems this is basically analyzing the running application inside the container and only packaging what's needed to make it work, at a more granular level than OS-level packages. Interesting concept. I wonder how it expected to cover 100% of the app usage if certain things aren't triggered during the analysis phase.
- Twirrim 7y ago
  I guess the burden is on the app dev to ensure 100% functional coverage? That seems a little "yikes".
  - meesles 7y ago
    Is that any different than anything else? Compilers, asset pipelines, and build tools all work the same way: they make assumptions about how a system works and try to optimize on those assumptions. Test your app, run QA, etc. Most licenses make no promises that the software will work, so this tool doesn't seem any different.
    - Twirrim 7y ago
      Wow, that's an oddly defensive response. My point is merely that this is quite a significant risk. If you fail to exercise 100% of your code paths via functional testing (so you've got to have comprehensive positive and negative functional testing, which is pretty rare in my experience), you risk producing an image with docker-slim that will break. You've got to think about exercising every single possible interaction with every other component running on an OS. That's no small feat. Think about it. That's not just 100% of _your_ code paths, that's 100% of the code paths that you could possibly ever trigger in any library that you consume, and you have to think about what might influence those circumstances. There's all sorts of angles to consider, e.g. Does latency of DNS response matter? Does time of day matter? Does IPv4 vs IPv6 matter (answer is likely yes in this case, so you might need to think about running the functional tests coming from both address stacks). docker-slim is a neat idea, but it seems to come with significant risk.
    - rumanator 7y ago
      > Is that any different than anything else? Yes. In practice test coverage tends to be well below 100%. This is fine if you're just running tests but if you're deciding which part of your package should be pruned based on this sort of analysis then it's very likely that this will cause problems.
  - kylequest 7y ago
    Yes, the coverage might not always be ideal. The tool will have additional static analysis to figure out the coverage. For now it relies on you to create custom probes to ensure better coverage if you need it...
- tofflos 7y ago
  Isn't it more or less the same with or without minification? If one wants to be sure one more or less has to have a good test suite. In this case one would simply run those tests against the minified image.
- kylequest 7y ago
  Yes, it is a potential problem, but in most real world cases it's good enough and if you have a decent test coverage to begin with (which you should have :-)) you can run those tests when the container is minified and after to confirm that it's working as expected.
- samcodes 7y ago
  There is a stage during the analysis where you interact with the container. So you can do something to trigger the resource usage that you want it to pick up on.
rudolph9 7y ago
Is there somewhere with a good walk though of the constructs that docker containers default to? Not these slim one the default docker one. As someone who works with docker containers on a somewhat daily basis I only have a vague idea of what they do under the hood and don’t have much of a reference point when comparing this slim impl to the default one.
- timClicks 7y ago
  Until you're more comfortable with the container model, I recommend sticking with Ubuntu LTS images, such as ubuntu:18.04. That way if something breaks, you have a familiar environment to work with. Once you have gathered a mental model, feel free to tune your images by basing them on something lighter.
- core-questions 7y ago
  Well, you're always coming FROM something, unless you're coming FROM scratch, which is literally empty. A popular base image is Debian Buster. What's in the Dockerfile? https://github.com/debuerreotype/docker-debian-artifacts/blob/17d8dadb220460bf153bf2de493c2bc873ef52de/buster/Dockerfile https://github.com/debuerreotype/docker-debian-artifacts/blo... Wow, simple, it just adds a "rootfs" archive. What's in there? https://github.com/debuerreotype/docker-debian-artifacts/blob/17d8dadb220460bf153bf2de493c2bc873ef52de/buster/rootfs.manifest https://github.com/debuerreotype/docker-debian-artifacts/blo... Aha. So what we have in rootfs.xz is essentially the output of a Debian system with all of those packages installed in it, and the minimal configuration needed to tie them together. No init system, no kernel, just a big filesystem full of the usual stuff an absolutely minimal Debian install would have. Now, when you run FROM debian:buster, what you get is a base layer with enough to be useful. `apt-get` is there with a default repository list that works fine, and you can install things to your hearts' content.
- itamarst 7y ago
  In general, the bottom layer is a small Linux distro (Ubuntu, Debian, Alpine), just enough to get you going. Ideally you choose a base image that is getting security updates on regular basis, and that doesn't have significant changes over time for the same tag. So e.g. `fedora` is bad base image, because one day you'll jump from Fedora 30 to Fedora 31. Likewise `ubuntu` isn't great. But `ubuntu:18.04` is the Long-Term Support release and so it'll be fairly stable. In the context of Python packaging I've written a more detailed guide to choosing a base image: https://pythonspeed.com/articles/base-image-python-docker-images/ https://pythonspeed.com/articles/base-image-python-docker-im...
slimsag 7y ago
Very interesting, but I worry this will just break a lot of applications that are run through it in subtle ways. For example, removing system packages can have negative effects not noticed except in subtle edge cases like DNS resolution.
dcolkitt 7y ago
I think the importance of small Docker images is generally oversold. I regularly deploy multi-GB images on Google Cloud, and startup even on a fresh node only takes 60 seconds or so. If the node's already hot (i.e. has cached the image), starting a container takes no more than a few seconds. I think what's more important is layering the Dockerfile in the right order. You should be putting most of your large, infrequently changing assets in the lowest layers. Then put smaller, more frequently changing assets in the top layer. If you have a 4GB image, but only change the top 10MB layer, then it only requires caching 10MB of new data when you update the container. But if you change a lower layer, then it requires re-building and re-caching everything above it.
- itamarst 7y ago
  Multi-GB images still seem... big. Though to be fair in many situations those 60 seconds are meaningless, so might not be worth the time to optimize. Often images that large are the result of including build toolchains, which you can omit with multi-stage builds (e.g. for Python https://pythonspeed.com/articles/multi-stage-docker-python/ https://pythonspeed.com/articles/multi-stage-docker-python/.
- tybit 7y ago
  I agree it’s oversold for most use cases, clouds do have pretty decent networking after all. Having said that as an Australian with sub par internet at home, I do appreciate not making images gratuitously large. Just because SDK images won’t ever be deployed to prod doesn’t mean you shouldn’t strip out unnecessary junk.
  - igor47 7y ago
    Similarly, just yesterday I had to go home from the cafe I was working in because I had to push a docker image. The wifi just wasn't fast enough for that, though plenty fast for googling documentation
- cosmotic 7y ago
  Adding a minute to some deployment process can be fairly significant. The docker push and pull is probably 70% of my deployment process and I already have ~100MB or less images. I don't think the concern is 'how long does deployment take' but 'how fast can we iterate?'; Building and loading the images on a local dev machine to test a 2 second change would take much longer with larger images. Getting feedback of a PR merge from a CI build agent would take minutes longer. I don't think the importance has been oversold.
tzickel 7y ago
I've done a similar yet simpler hobby scoped project: https://github.com/tzickel/docker-trim https://github.com/tzickel/docker-trim Last time I've checked some of their open issues about cases theirs doesn't work, worked on mine. Also, mine is a few lines in python, if you want to learn how to trim a docker image.
hrdwdmrbl 7y ago
Their FAQ doesn't answer what it is that they are removing. Can someone shed light on that? As others have said, it seems to watch your application running and then remove anything it doesn't see your application using. Seems like a very high-risk + high-reward method.
- dkarras 7y ago
  Yes, this needs a "what is the catch?" section in readme.
  - soupdiver 7y ago
    exactly what I thought... there are nowhere 1-2 setences which sum up what they're doing
  - kylequest 7y ago
    The main catch is that it targets application container images and not generic base images. This is the most common gotcha many people encounter. There has to be an app/service in the container that does something specific. And, of course, it is possible that not all artifacts will be identified. There are a couple of ways to mitigate this. First, you can create custom probes for your app/service to make sure the app container can be analyzed much better. Second, you can explicitly tell docker-slim what you want to keep in your container image (you can specify files or executables)
- Havoc 7y ago
  Another commenter above seems to think it’s dynamically determined
  - kylequest 7y ago
    One of the monitors leverages FANOTIFY in Linux to determine what files your application is using. This is the same thing many AV tools use to detect malware :-) The future version will get more different runtime monitors and it will do much more with static analysis too. Right now its static analysis is limited to LDD-like dynamic library inspection for extra artifacts you want to keep in your image. There's a lot more that can be done there...
itamarst 7y ago
Alternatives if you don't want to risk missing some file that only gets loaded 10 minutes in: 1. Start with a small base image, e.g. for Python there's "python:3.7-slim". For Python I'm not a fan of Alpine, but for Go that gives you an extra small base image (see https://pythonspeed.com/articles/base-image-python-docker-images/ https://pythonspeed.com/articles/base-image-python-docker-im...). 2. Don't install unnecessary system packages (https://pythonspeed.com/articles/system-packages-docker/ https://pythonspeed.com/articles/system-packages-docker/). 3. Multi-stage builds (in Python context, https://pythonspeed.com/articles/smaller-python-docker-images/ https://pythonspeed.com/articles/smaller-python-docker-image...). You can find similar guides for non-Python as well. Basic idea being "don't install unnecessary stuff, and in final image only include the final build artifacts".
yegle 7y ago
Most people would benefit from the distroless base image: https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless It's a base image with binaries from Debian deb package, and with necessary stuff like ca-certificates, and absolutely nothing else, while still glibc-based (unlike Alpine base images). Example images I built with the base image - C binary, <10MB https://hub.docker.com/r/yegle/stubby-dns https://hub.docker.com/r/yegle/stubby-dns - Python binary, <50MB https://hub.docker.com/r/yegle/fava https://hub.docker.com/r/yegle/fava - Go binary, 5MB https://hub.docker.com/r/yegle/dns-over-https https://hub.docker.com/r/yegle/dns-over-https Another trick I use is to use https://github.com/wagoodman/dive https://github.com/wagoodman/dive to find the deltas between layers and manually remove it in my Dockerfile.
- dfsegoat 7y ago
  I just became familiar with the Alpine base images. Curious to know what the benefit conferred by it being 'glibc-based' is exactly?
  - CraftThatBlock 7y ago
    Mostly compatibility. GNU's libc and Alpine's libc implementation should match the same ABI but might not be 100% the same
    - devy 7y ago
      Yeah, because of that, it's really hard to get Alpine-based Docker images to work in some situations, for instance: this: https://github.com/grpc/grpc/issues/18150#issuecomment-479994767 https://github.com/grpc/grpc/issues/18150#issuecomment-47999... And this: https://github.com/pypa/manylinux/issues/37 https://github.com/pypa/manylinux/issues/37 I use Python. Anytime you need to compile the source to build a pip package and that the upstream package developer decided that they don't support Alpine's libc implementation (aka musl) then you will have a big problem, unless you can control your dependencies and include as few pip packages that requires compilation or find binary builds.
  - 7y ago
aktuel 7y ago
I wrote something similar but way simpler a while ago: https://github.com/ak-1/sackman https://github.com/ak-1/sackman
alpb 7y ago
It seems like this tool is built upon the assumption that the containerized program will load libraries and read files while this tool is tracing it. It seems like the biggest FAQ item is missing from the readme: What happens is my container reads a file only every so often and this tool doesn't capture it? Also, do I need to keep images running for a while for this tool to minimize the files in the rootfs? It seems impractical, especially in headless environments like CI/CD.
- Operyl 7y ago
  It seems they do try and solve this, but potentially not completely depending on what triggers your file load. It seems to be able to simulate HTTP traffic.
- kylequest 7y ago
  The temporary containers usually don't run for too long. The probes (http, for now) are there to ensure that the app/services gets to do something useful exercising different code paths. Still, it is possible that something could be missing or you might want to keep something extra. For those cases you can tell docker-slim what else you want to keep in your container image (it has a few flags for that).
kylequest 7y ago
i'll be happy to answer any questions about the tool (i'm the main author) :-)
- mbu 7y ago
  This looks pretty amazing, thanks for sharing. I've just been playing with this on a couple containers. The first went from 197MB to 71.5MB - not bad! I did some testing and it didn't work without including a few extra pieces but pretty painless. The second container went from 176MB to 4.7MB!! I've not really tested that so there's a pretty good chance things aren't going to work too well in practice (but we'll see). If it all continues as well as it's started then we'll definitely be using it.
  - kylequest 7y ago
    Do you mind sharing a bit of information about your container images? What's the app language for both? What kind of application is it? How do you init your apps in the containers? docker-slim is definitely not perfect and there's lots of room for improvement. Can you ping me offline (on github/gitter, twitter or email, my email is kcq.public@gmail.com)?
- tumetab1 7y ago
  Please add a better description of what the tools does, like "Docker-slim removes all files which you container doesn't touch when running" Considering CI/CD attacks these days I spend a while scanning if this tool was some kind of joke trying to show that developers download anything from the Internet. PS: Thanks for this :)
  - kylequest 7y ago
    Thank you for the your feedback! It's very important for a tool like this to be clear and transparent about what it's doing. The docs definitely need to be improved. It's already a lot for some info to be lost and not enough, at the same time, in other areas. By the way, if you don't feel comfortable with the minification functionality you can still use docker-slim as a Docker image inspection and profiling tool (take a look at the report files it generates). Additional package level reporting is on the todo list. Image and Dockerfile linting functionality is coming soon too.
dsprt4cultre 7y ago
Testing comments, just seeing if I've been muted again.
- reilly3000 7y ago
  I see you, but your account is flagged as new and your comment grayed out.
zzyzxd 7y ago
Instead of starting with a bulky image and guess and remove unnecessary stuff at the end, I would prefer the opposite: only include necessary packages for the executable in the first place. I am happy with my current distroless + docker multi-stage build.
alex-ant 7y ago
Would be nice to add a support for native docker flags (-t for --tag, -f for --from-dockerfile, etc.)
- kylequest 7y ago
  This is a great idea! Thank you for your feedback. Using native docker flags as-is will definitely reduce friction and simplify the use. Do you mind creating a Github issue if you have a specific list of flags you'd like to see supported first?
antpls 7y ago
Results are interesting. How are Go containers only 1.5MB, and Rust's ones 15MB? I would have expected Rust to be on par with Go (both compiled languages)

379 points by LinuxBender 7y ago

It seems this is basically analyzing the running application inside the container and only packaging what's needed to make it work, at a more granular level than OS-level packages. Interesting concept. I wonder how it expected to cover 100% of the app usage if certain things aren't triggered during the analysis phase.

Twirrim 7y ago
I guess the burden is on the app dev to ensure 100% functional coverage? That seems a little "yikes".
- meesles 7y ago
  Is that any different than anything else? Compilers, asset pipelines, and build tools all work the same way: they make assumptions about how a system works and try to optimize on those assumptions. Test your app, run QA, etc. Most licenses make no promises that the software will work, so this tool doesn't seem any different.
  - Twirrim 7y ago
    Wow, that's an oddly defensive response. My point is merely that this is quite a significant risk. If you fail to exercise 100% of your code paths via functional testing (so you've got to have comprehensive positive and negative functional testing, which is pretty rare in my experience), you risk producing an image with docker-slim that will break. You've got to think about exercising every single possible interaction with every other component running on an OS. That's no small feat. Think about it. That's not just 100% of _your_ code paths, that's 100% of the code paths that you could possibly ever trigger in any library that you consume, and you have to think about what might influence those circumstances. There's all sorts of angles to consider, e.g. Does latency of DNS response matter? Does time of day matter? Does IPv4 vs IPv6 matter (answer is likely yes in this case, so you might need to think about running the functional tests coming from both address stacks). docker-slim is a neat idea, but it seems to come with significant risk.
  - rumanator 7y ago
    > Is that any different than anything else? Yes. In practice test coverage tends to be well below 100%. This is fine if you're just running tests but if you're deciding which part of your package should be pruned based on this sort of analysis then it's very likely that this will cause problems.
- kylequest 7y ago
  Yes, the coverage might not always be ideal. The tool will have additional static analysis to figure out the coverage. For now it relies on you to create custom probes to ensure better coverage if you need it...
tofflos 7y ago
Isn't it more or less the same with or without minification? If one wants to be sure one more or less has to have a good test suite. In this case one would simply run those tests against the minified image.
kylequest 7y ago
Yes, it is a potential problem, but in most real world cases it's good enough and if you have a decent test coverage to begin with (which you should have :-)) you can run those tests when the container is minified and after to confirm that it's working as expected.
samcodes 7y ago
There is a stage during the analysis where you interact with the container. So you can do something to trigger the resource usage that you want it to pick up on.

rudolph9 7y ago

Is there somewhere with a good walk though of the constructs that docker containers default to? Not these slim one the default docker one. As someone who works with docker containers on a somewhat daily basis I only have a vague idea of what they do under the hood and don’t have much of a reference point when comparing this slim impl to the default one.

timClicks 7y ago
Until you're more comfortable with the container model, I recommend sticking with Ubuntu LTS images, such as ubuntu:18.04. That way if something breaks, you have a familiar environment to work with. Once you have gathered a mental model, feel free to tune your images by basing them on something lighter.
core-questions 7y ago
Well, you're always coming FROM something, unless you're coming FROM scratch, which is literally empty. A popular base image is Debian Buster. What's in the Dockerfile? https://github.com/debuerreotype/docker-debian-artifacts/blob/17d8dadb220460bf153bf2de493c2bc873ef52de/buster/Dockerfile https://github.com/debuerreotype/docker-debian-artifacts/blo... Wow, simple, it just adds a "rootfs" archive. What's in there? https://github.com/debuerreotype/docker-debian-artifacts/blob/17d8dadb220460bf153bf2de493c2bc873ef52de/buster/rootfs.manifest https://github.com/debuerreotype/docker-debian-artifacts/blo... Aha. So what we have in rootfs.xz is essentially the output of a Debian system with all of those packages installed in it, and the minimal configuration needed to tie them together. No init system, no kernel, just a big filesystem full of the usual stuff an absolutely minimal Debian install would have. Now, when you run FROM debian:buster, what you get is a base layer with enough to be useful. `apt-get` is there with a default repository list that works fine, and you can install things to your hearts' content.
itamarst 7y ago
In general, the bottom layer is a small Linux distro (Ubuntu, Debian, Alpine), just enough to get you going. Ideally you choose a base image that is getting security updates on regular basis, and that doesn't have significant changes over time for the same tag. So e.g. `fedora` is bad base image, because one day you'll jump from Fedora 30 to Fedora 31. Likewise `ubuntu` isn't great. But `ubuntu:18.04` is the Long-Term Support release and so it'll be fairly stable. In the context of Python packaging I've written a more detailed guide to choosing a base image: https://pythonspeed.com/articles/base-image-python-docker-images/ https://pythonspeed.com/articles/base-image-python-docker-im...

slimsag 7y ago

Very interesting, but I worry this will just break a lot of applications that are run through it in subtle ways. For example, removing system packages can have negative effects not noticed except in subtle edge cases like DNS resolution.

dcolkitt 7y ago

I think the importance of small Docker images is generally oversold. I regularly deploy multi-GB images on Google Cloud, and startup even on a fresh node only takes 60 seconds or so. If the node's already hot (i.e. has cached the image), starting a container takes no more than a few seconds. I think what's more important is layering the Dockerfile in the right order. You should be putting most of your large, infrequently changing assets in the lowest layers. Then put smaller, more frequently changing assets in the top layer. If you have a 4GB image, but only change the top 10MB layer, then it only requires caching 10MB of new data when you update the container. But if you change a lower layer, then it requires re-building and re-caching everything above it.

itamarst 7y ago
Multi-GB images still seem... big. Though to be fair in many situations those 60 seconds are meaningless, so might not be worth the time to optimize. Often images that large are the result of including build toolchains, which you can omit with multi-stage builds (e.g. for Python https://pythonspeed.com/articles/multi-stage-docker-python/ https://pythonspeed.com/articles/multi-stage-docker-python/.
tybit 7y ago
I agree it’s oversold for most use cases, clouds do have pretty decent networking after all. Having said that as an Australian with sub par internet at home, I do appreciate not making images gratuitously large. Just because SDK images won’t ever be deployed to prod doesn’t mean you shouldn’t strip out unnecessary junk.
- igor47 7y ago
  Similarly, just yesterday I had to go home from the cafe I was working in because I had to push a docker image. The wifi just wasn't fast enough for that, though plenty fast for googling documentation
cosmotic 7y ago
Adding a minute to some deployment process can be fairly significant. The docker push and pull is probably 70% of my deployment process and I already have ~100MB or less images. I don't think the concern is 'how long does deployment take' but 'how fast can we iterate?'; Building and loading the images on a local dev machine to test a 2 second change would take much longer with larger images. Getting feedback of a PR merge from a CI build agent would take minutes longer. I don't think the importance has been oversold.

tzickel 7y ago

I've done a similar yet simpler hobby scoped project: https://github.com/tzickel/docker-trim https://github.com/tzickel/docker-trim Last time I've checked some of their open issues about cases theirs doesn't work, worked on mine. Also, mine is a few lines in python, if you want to learn how to trim a docker image.

hrdwdmrbl 7y ago

Their FAQ doesn't answer what it is that they are removing. Can someone shed light on that? As others have said, it seems to watch your application running and then remove anything it doesn't see your application using. Seems like a very high-risk + high-reward method.

dkarras 7y ago
Yes, this needs a "what is the catch?" section in readme.
- soupdiver 7y ago
  exactly what I thought... there are nowhere 1-2 setences which sum up what they're doing
- kylequest 7y ago
  The main catch is that it targets application container images and not generic base images. This is the most common gotcha many people encounter. There has to be an app/service in the container that does something specific. And, of course, it is possible that not all artifacts will be identified. There are a couple of ways to mitigate this. First, you can create custom probes for your app/service to make sure the app container can be analyzed much better. Second, you can explicitly tell docker-slim what you want to keep in your container image (you can specify files or executables)
Havoc 7y ago
Another commenter above seems to think it’s dynamically determined
- kylequest 7y ago
  One of the monitors leverages FANOTIFY in Linux to determine what files your application is using. This is the same thing many AV tools use to detect malware :-) The future version will get more different runtime monitors and it will do much more with static analysis too. Right now its static analysis is limited to LDD-like dynamic library inspection for extra artifacts you want to keep in your image. There's a lot more that can be done there...

itamarst 7y ago

Alternatives if you don't want to risk missing some file that only gets loaded 10 minutes in: 1. Start with a small base image, e.g. for Python there's "python:3.7-slim". For Python I'm not a fan of Alpine, but for Go that gives you an extra small base image (see https://pythonspeed.com/articles/base-image-python-docker-images/ https://pythonspeed.com/articles/base-image-python-docker-im...). 2. Don't install unnecessary system packages (https://pythonspeed.com/articles/system-packages-docker/ https://pythonspeed.com/articles/system-packages-docker/). 3. Multi-stage builds (in Python context, https://pythonspeed.com/articles/smaller-python-docker-images/ https://pythonspeed.com/articles/smaller-python-docker-image...). You can find similar guides for non-Python as well. Basic idea being "don't install unnecessary stuff, and in final image only include the final build artifacts".

yegle 7y ago

Most people would benefit from the distroless base image: https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless It's a base image with binaries from Debian deb package, and with necessary stuff like ca-certificates, and absolutely nothing else, while still glibc-based (unlike Alpine base images). Example images I built with the base image - C binary, <10MB https://hub.docker.com/r/yegle/stubby-dns https://hub.docker.com/r/yegle/stubby-dns - Python binary, <50MB https://hub.docker.com/r/yegle/fava https://hub.docker.com/r/yegle/fava - Go binary, 5MB https://hub.docker.com/r/yegle/dns-over-https https://hub.docker.com/r/yegle/dns-over-https Another trick I use is to use https://github.com/wagoodman/dive https://github.com/wagoodman/dive to find the deltas between layers and manually remove it in my Dockerfile.

dfsegoat 7y ago
I just became familiar with the Alpine base images. Curious to know what the benefit conferred by it being 'glibc-based' is exactly?
- CraftThatBlock 7y ago
  Mostly compatibility. GNU's libc and Alpine's libc implementation should match the same ABI but might not be 100% the same
  - devy 7y ago
    Yeah, because of that, it's really hard to get Alpine-based Docker images to work in some situations, for instance: this: https://github.com/grpc/grpc/issues/18150#issuecomment-479994767 https://github.com/grpc/grpc/issues/18150#issuecomment-47999... And this: https://github.com/pypa/manylinux/issues/37 https://github.com/pypa/manylinux/issues/37 I use Python. Anytime you need to compile the source to build a pip package and that the upstream package developer decided that they don't support Alpine's libc implementation (aka musl) then you will have a big problem, unless you can control your dependencies and include as few pip packages that requires compilation or find binary builds.
- 7y ago

aktuel 7y ago

I wrote something similar but way simpler a while ago: https://github.com/ak-1/sackman https://github.com/ak-1/sackman

alpb 7y ago

It seems like this tool is built upon the assumption that the containerized program will load libraries and read files while this tool is tracing it. It seems like the biggest FAQ item is missing from the readme: What happens is my container reads a file only every so often and this tool doesn't capture it? Also, do I need to keep images running for a while for this tool to minimize the files in the rootfs? It seems impractical, especially in headless environments like CI/CD.

Operyl 7y ago
It seems they do try and solve this, but potentially not completely depending on what triggers your file load. It seems to be able to simulate HTTP traffic.
kylequest 7y ago
The temporary containers usually don't run for too long. The probes (http, for now) are there to ensure that the app/services gets to do something useful exercising different code paths. Still, it is possible that something could be missing or you might want to keep something extra. For those cases you can tell docker-slim what else you want to keep in your container image (it has a few flags for that).

kylequest 7y ago

i'll be happy to answer any questions about the tool (i'm the main author) :-)

mbu 7y ago
This looks pretty amazing, thanks for sharing. I've just been playing with this on a couple containers. The first went from 197MB to 71.5MB - not bad! I did some testing and it didn't work without including a few extra pieces but pretty painless. The second container went from 176MB to 4.7MB!! I've not really tested that so there's a pretty good chance things aren't going to work too well in practice (but we'll see). If it all continues as well as it's started then we'll definitely be using it.
- kylequest 7y ago
  Do you mind sharing a bit of information about your container images? What's the app language for both? What kind of application is it? How do you init your apps in the containers? docker-slim is definitely not perfect and there's lots of room for improvement. Can you ping me offline (on github/gitter, twitter or email, my email is kcq.public@gmail.com)?
tumetab1 7y ago
Please add a better description of what the tools does, like "Docker-slim removes all files which you container doesn't touch when running" Considering CI/CD attacks these days I spend a while scanning if this tool was some kind of joke trying to show that developers download anything from the Internet. PS: Thanks for this :)
- kylequest 7y ago
  Thank you for the your feedback! It's very important for a tool like this to be clear and transparent about what it's doing. The docs definitely need to be improved. It's already a lot for some info to be lost and not enough, at the same time, in other areas. By the way, if you don't feel comfortable with the minification functionality you can still use docker-slim as a Docker image inspection and profiling tool (take a look at the report files it generates). Additional package level reporting is on the todo list. Image and Dockerfile linting functionality is coming soon too.

dsprt4cultre 7y ago

Testing comments, just seeing if I've been muted again.

reilly3000 7y ago
I see you, but your account is flagged as new and your comment grayed out.

zzyzxd 7y ago

Instead of starting with a bulky image and guess and remove unnecessary stuff at the end, I would prefer the opposite: only include necessary packages for the executable in the first place. I am happy with my current distroless + docker multi-stage build.

alex-ant 7y ago

Would be nice to add a support for native docker flags (-t for --tag, -f for --from-dockerfile, etc.)

kylequest 7y ago
This is a great idea! Thank you for your feedback. Using native docker flags as-is will definitely reduce friction and simplify the use. Do you mind creating a Github issue if you have a specific list of flags you'd like to see supported first?

antpls 7y ago

Results are interesting. How are Go containers only 1.5MB, and Rust's ones 15MB? I would have expected Rust to be on par with Go (both compiled languages)