6 ms·
We’re currently consolidating all container images to run on Debian-slim instead of a mixture of Debian, Ubuntu and alpine. Sure, alpine is small, but with 70%
by robinhoodexe 3y ago
We’re currently consolidating all container images to run on Debian-slim instead of a mixture of Debian, Ubuntu and alpine. Sure, alpine is small, but with 70% of our 500 container images being used for Python, R or node the final image is so large (due to libraries/packages), that the difference between alpine (~30 MB) and debian-slim (~80) is negligible. We’ve been experiencing the weird DNS behaviour of alpine and other issues with musl as well. Debian is rock solid and upgrading to bookworm from bullseye and even buster in many cases didn’t cause any problems at all.
I will admit though, that Debian-slim still has some non-essential stuff that usually isn’t needed at runtime, a shell is still neat for debugging or local development. This trade off could be considered a security risk, but it’s rather simple to restrict other stuff at runtime (such as run as non-privileged, non-root user with all capabilities dropped and a read-only file system except for /tmp).
It’s a balancing act between ease-of-use and security. I don’t think I’d get popular with the developers by forcing them to use “FROM scratch” and let them figure out exactly what their application needs at runtime and what stuff to copy over from a previous build stage.
- amouat 3y agoYou might want to check out Wolfi and Chainguard Images. Wolfi is a Linux distro that we use to build minimal images that are roughly comparable to Alpine in size but, everything is compiled from source against glibc. Our images come without a shell or pacakge manager by default, but there are -dev variants that include these. https://github.com/wolfi-dev/ https://github.com/wolfi-dev/ https://github.com/chainguard-images/images https://github.com/chainguard-images/images
- kapilvt 3y agoI was using wolfi for an oss project, but the recent attempts at forcing payments when pinning version was a huge red flag, as was the messaging around it, and I’ll be migrating away from it. https://www.chainguard.dev/unchained/scaling-chainguard-images-with-a-growing-catalog-and-proactive-security-updates https://www.chainguard.dev/unchained/scaling-chainguard-imag... Pinning a language version (say python 3.11) isn’t an optional thing its a best practice, and the notion that its because of security seems intentionally misleading as the images should be refreshed in place on the tag along with signatures.
- amouat 3y agoI'm very sorry that we broke things for you. To be clear, nothing has changed with Wolfi. Wolfi is an open source community project and everything is still available there: https://github.com/wolfi-dev/ https://github.com/wolfi-dev/. We have made changes to Chainguard Images - our commercial product built on top of Wolfi - which mean you can no longer pull images by tag (other than latest). Chainguard images are rebuilt everyday and have a not inconsiderable maintenance cost (and the money we make here directly helps us support Wolfi). The easiest way to avoid this is to build the images yourself. You can rebuild identical images to ours using apko and the source files in the images repo e.g: https://github.com/chainguard-images/images/blob/main/images/nginx/configs/latest.apko.yaml https://github.com/chainguard-images/images/blob/main/images... (note you can replace package names with versioned versions). You can also just use a Dockerfile with the wolfi-base image to "apk add" packages. Full details are here: https://www.chainguard.dev/unchained/a-guide-on-how-to-use-chainguard-images-for-public-catalog-tier-users https://www.chainguard.dev/unchained/a-guide-on-how-to-use-c... I agree that pinning is a best practice. The above blog explains that you can still do it using a digest, but I accept this isn't the simplest solution. If I can help any more, please feel free to get in touch - you can find me most places including twitter https://twitter.com/adrianmouat https://twitter.com/adrianmouat
- kapilvt 3y agoI sympathize, but if wolfi is oss, please update docs and downloads to use a separate registry then your commercial one for distribution. else, frankly you claimed wolfi is oss, get customers to use your registry, and then bait and switched your early adopters. aka, major version upgrades at random, have fun!
- amouat 3y agoWolfi packages are served from https://packages.wolfi.dev/os https://packages.wolfi.dev/os which can be used with apk tools or apko. The built Chainguard images are all on the cgr.dev registry. Wolfi is completely OSS. The policies regarding Chainguard Images have changed over time, so if there are docs that don't properly reflect this, please let me know and I'll get them updated.
- adolph 3y ago> the difference between alpine (~30 MB) and debian-slim (~80) Given that it's a different layer, the your container runtime isn't going to redownload the layer anyway, right?
- robinhoodexe 3y agoExactly. Part of the appeal to consolidate all of our container images to use Debian-slim is the ability to optimise the caching of layers, both in our container registry but also on our kubernetes cluster’s nodes (which can be done in a consistent manner with kube-fledged[1]). [1] https://github.com/senthilrch/kube-fledged https://github.com/senthilrch/kube-fledged
- robertlagrant 3y agoThanks for that - that operator sounds extremely useful!
- kstrauser 3y agoAnd even if it did, in an ancient data center that only uses gigabit ethernet, that's only a .5s longer download. And even a $4 DigitalOcean server comes with 10GB of storage, so that 50MB is only 1/200th of the instance's store. (I'd also bet that nearly no one uses instances that tiny for durable production work where 50MB is going to make a difference.)
- leononame 3y agoCan you point me on where to look for more details on securing a container? I'm a developer myself, and for me, the main benefit of containers is being able to deploy the app myself easily because I can bundle all the dependencies. What would you suggest I restrict at runtime and can you point me to a tutorial or an article where I can go have a deeper read on how to do it?
- robinhoodexe 3y agoWhat you want to read is the kubernetes pod security context fields[1]. In your Dockerfile, add a non-root user with UID and GID 1000, then in the end of your Dockerfile, right before a CMD or ENTRYPOINT you change to that user. In your kubernetes yaml manifest, you can now set runAsNonRoot to true and runAsUser & runAsGroup to 1000. Then there’s the privileged and allowPrivilegeEscalation fields, these can nearly always be set to false unless you actually need the extra privileges (such as using a GPU) on a shared node. Then there’s seccomp profiles and the system capabilities. If you can run your container as the non-root user you’ve created, and you don’t need the extra privileges then these can safely also be set to the most restricted. Non-privileged non-root is the same as all capabilities are dropped. The tricky one is the readOnlyRootfilesystem field. This includes /tmp, which is considered a global writeable directory, so the workaround is to make a in-memory volume and mount it at /tmp to make it writable. Likewise, your $HOME/.cache and $HOME/.local directories (for the user you created in your Dockerfile) are usually used by third party packages, so creating mounts here can be useful as well (if for some reason you can’t point it to /tmp instead). [1] https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ https://kubernetes.io/docs/tasks/configure-pod-container/sec...
- lenova 3y ago> In your Dockerfile, add a non-root user with UID and GID 1000, then in the end of your Dockerfile, right before a CMD or ENTRYPOINT you change to that user. I'm not certain, but would it be more secure to create a non-root user in the Dockerfile, but _don't_ use UID 1000. Use volume mounts to grant host disk access instead.
- galangalalgol 3y agoI've run into the same thing for large dev images, but using pure rust often means that musl allows for a single executable and a config file in a from scratch container for deployment. In cases where a slab or bump allocator are used, musl's deficiencies seem minimized. That means duplication of musl in lots of containers, but when they are all less than 10MB its less of an issue. Linking against gnu libraries might get the same executable down to less than 2MB but you'll add that back and more for even the tiniest gnu nase images.
- synergy20 3y agowhat is pure rust,rust needs its own libraries which is a few MB as I recall
- galangalalgol 3y agoBy pure, I just meant no dependencies on c or c++ bindings other than libc. If that is the case you can do a musl build that has no dynamic dependencies, as all rust dependencies are static. So then your only dependency is the kernel, which is provided via podman/docker. A decent sized rust program with hundreds of dependencies I can get to compile down to 1.5MB. But that is depending on gnu. So if you had 4 or 5 of those on a node, it might be less data to use one gnu base image that is really small like rhel micro, and build rust for gnu. But if you have cpu hungry services like I do, then you usually have only a couple per node, so from scratch musl can be a bit smaller.
- phamilton 3y ago> cpu hungry services Have you benchmarked musl vs glibc in any way? Data I've seen is all over the place and in curious about your experience.
- galangalalgol 3y agoVery few system calls in the main loop. Not even close to being io bound, and as I said memory is all preallocated, so allocator efficiency isn't a factor. Edit: realized I didn't answer your question, I ran flamegraph both ways and it was completely dominated by tight vectorized loops and a couple sadly necessary memory copies.
- jonwest 3y agoIn the same boat here as well. Especially when you’re talking about container images using JavaScript or other interpreted languages that are bundling in a bunch of other dependencies, the developer experience is much better in my experience given that more developers are likely to have had experience working in a Debian based distro than an Alpine based one. Especially when you’re also developing within the container as well, having that be unified is absolutely worth the convenience, and honestly security and reliability as well. I realize that a container with less installed on it is inherently more secure, but if the only people who are familiar with the system are a small infrastructure/platform/ops type of team, things are more likely to get missed.
- kachnuv_ocasek 3y agoDo you have any tips regarding building R-based container images?
- robinhoodexe 3y agoR is kinda difficult and I haven’t cracked this one. Currently we’re using the rocker based ones[1] but they are based on Ubuntu and include a lot of stuff we don’t need at runtime. I’ll look into creating a more minimal R base images that’s based on Debian-slim. [1] https://github.com/rocker-org/rocker-versioned2 https://github.com/rocker-org/rocker-versioned2
- Rapzid 3y agoThis here. Honestly most orgs with uhh.. Let's say a more mature sense of ROI tradeoffs were doing this from pretty much the very beginning. Also, Ubuntu 22.04 is only 28.17MB compressed right now so it looks equiv to debian-slim. There are also these new image lines, I can't recall the funky name for them, that are even smaller.
- kstrauser 3y agoI'm pushing to go back to Debian from Ubuntu. Canonical's making decisions lately that don't appeal to me, and especially on the server I don't see a clear advantage for Ubuntu vs good ol' rock solid Debian.
- 3np 3y agoHost OS vs container image base shouldn't be the same conversation.
- Fnoord 3y agoI've done this multiple times when new Debian Stable was released (Bookworm just got released). Its great, until it isn't (ie. you need more up-to-date software). Although to be fair, you might be comparing Ubuntu LTS with Debian releases and the minor Ubuntu releases get short support. Ubuntu is akin to a polished Debian Testing snapshot. I don't like them swapping to Snap either (snappy it ain't).
- synergy20 3y agowhy is it so much smaller than debian slim?
- nickjj 3y agoI made the switch too around 4ish years ago. It has worked out nicely and I have no intention on moving away from Debian Slim. Everything "just works" and you get to re-use any previous Debian knowledge you may have picked up before using Docker.
- kelvie 3y agoMy biggest beef with the apt/deb-based distros for container images is that apt-get updates installs take frustratingly long, whereas apks always tend to be near instant. I wonder what it is in the postinst and related scripts that just take so long and can't be parallelized! Most of the reason I switched from Ubuntu -> Arch is that working with alpine allowed me to realize that installing packages and their dependencies don't have to take so long.
- p_l 3y agoHave you tried building the images with eatmydata library? Essentially, dpkg is hilariously slow due to its approach in issuing fsync calls for every file - eatmydata wrapper injects a shared library which turns that into a no-op. I haven't timed it rigorously, but I experienced up to 10x speedup in container builds by just wrapping dpkg with eatmydata.
- gorgoiler 3y ago[Edit: modern debian images have “force-unsafe-io” turned on in /etc/dpkg/ so eatmydata shouldn’t make a difference.] I’m not the OP, but in an example I just tried it took build-essential from 48s to 39s (timed on the third step, without and then with the call to eatmydata): from debian:sid run apt update && env DEBIAN_FRONTEND=noninteractive apt-get install --yes eatmydata run apt update && eatmydata env DEBIAN_FRONTEND=noninteractive apt-get install --yes build-essential Can you replicate something faster? Was I doing something wrong?
- TylerE 3y agoYes, the second invocation will have hot caches. Basically the standard is to either test in a clean environment (e.g. immediately after a reboot) or to run the command until the times stabilize, and then take, say, the middle 3 out of 5 measurements.
- gorgoiler 3y agoWhat is cached though? DNS? IP routes to the Debian mirrors? I ran it three more times. Plain install was 50s 48s 47s; eatmydata was 46s 46s 46s. Installing build-essential probably does enough non-dpkg IO to account for the difference (install scripts etc, for example when libc changes and the system scans for running daemons) but it isn’t much and the delta may well be statistically insignificant.